2024-04-25T15:36:32.919 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T15:36:32.919 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228 --verbose --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/orig.config.yaml 2024-04-25T15:36:32.975 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T15:36:33.061 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673228' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - but it is still running - objects unfound and apparently lost - MDS_SLOW_METADATA_IO - MDS_TRIM sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBGer4xfgwQKKHClW7fd5vckzVLt/MYQCove0MxZiNBSgA3lcVPlQRSIBMQvaez3OVOyYcoT2zPvqKwHethtjNQ= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGLfJ3uYds5uQosqsnxVpzAvV38Lh7qKVncNctNv4YiS/v6c8biTMkzRoMMR25sAf/G5Av40wBnQnCKube88vjc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - thrashosds: null - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T15:36:33.061 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T15:36:33.062 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T15:36:33.062 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T15:36:33.063 INFO:teuthology.task.internal:Checking packages... 2024-04-25T15:36:33.088 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T15:36:33.088 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T15:36:33.088 INFO:teuthology.packaging:ref: None 2024-04-25T15:36:33.088 INFO:teuthology.packaging:tag: None 2024-04-25T15:36:33.088 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:36:33.088 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:36:33.088 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:36:33.232 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:36:33.234 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T15:36:33.240 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T15:36:33.240 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T15:36:33.278 INFO:teuthology.task.internal:Saving configuration 2024-04-25T15:36:33.297 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T15:36:33.302 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T15:36:33.321 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:30:54.825227', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOeYaw7WaG5pkCUviIhXd251qGtP6RcpAQAAtmL4xI2e'} 2024-04-25T15:36:33.339 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:30:54.823485', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANE41wbQ9yZxHvs6NWX/lqkfQgy86K+9j5gdI+oKHVF'} 2024-04-25T15:36:33.339 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T15:36:33.346 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-25T15:36:33.346 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-25T15:36:33.346 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T15:36:33.423 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8eb842b3a0>, signals=[15]) 2024-04-25T15:36:33.423 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T15:36:33.430 INFO:teuthology.task.internal:Opening connections... 2024-04-25T15:36:33.430 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:36:33.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:36:33.513 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:36:33.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:36:33.592 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T15:36:33.598 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-25T15:36:33.604 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-25T15:36:33.604 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-25T15:36:33.651 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T15:36:33.651 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Ubuntu" 2024-04-25T15:36:33.651 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="22.04" 2024-04-25T15:36:33.651 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T15:36:33.651 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_CODENAME=jammy 2024-04-25T15:36:33.651 INFO:teuthology.orchestra.run.smithi067.stdout:ID=ubuntu 2024-04-25T15:36:33.651 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE=debian 2024-04-25T15:36:33.652 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T15:36:33.652 INFO:teuthology.orchestra.run.smithi067.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T15:36:33.652 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T15:36:33.652 INFO:teuthology.orchestra.run.smithi067.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T15:36:33.652 INFO:teuthology.orchestra.run.smithi067.stdout:UBUNTU_CODENAME=jammy 2024-04-25T15:36:33.652 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-25T15:36:33.680 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-04-25T15:36:33.685 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-04-25T15:36:33.685 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Ubuntu" 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="22.04" 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_CODENAME=jammy 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:ID=ubuntu 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE=debian 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T15:36:33.736 INFO:teuthology.orchestra.run.smithi176.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T15:36:33.737 INFO:teuthology.orchestra.run.smithi176.stdout:UBUNTU_CODENAME=jammy 2024-04-25T15:36:33.737 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-04-25T15:36:33.754 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T15:36:33.764 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T15:36:33.786 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T15:36:33.786 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:36:33.789 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:36:33.793 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T15:36:33.813 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T15:36:33.813 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:36:33.836 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:36:33.846 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T15:36:33.894 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T15:36:33.909 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T15:36:33.909 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T15:36:33.909 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:36:33.910 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:36:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:36:33.914 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-25T15:36:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:36:33.915 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-04-25T15:36:33.963 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-56-generic 2024-04-25T15:36:33.963 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.15.0-56-generic 2024-04-25T15:36:33.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-04-25T15:36:33.965 INFO:teuthology.orchestra.run.smithi176.stdout:5.15.0-56-generic 2024-04-25T15:36:33.965 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.15.0-56-generic 2024-04-25T15:36:33.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2024-04-25T15:36:34.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-04-25T15:36:34.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-04-25T15:36:34.220 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T15:36:34.221 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T15:36:34.225 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T15:36:34.252 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T15:36:34.308 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T15:36:34.356 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T15:36:34.438 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T15:36:34.463 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T15:36:34.465 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T15:36:34.571 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T15:36:34.572 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T15:36:34.592 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T15:36:34.599 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T15:36:34.605 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T15:36:34.605 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T15:36:34.606 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T15:36:34.626 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T15:36:34.628 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T15:36:34.629 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T15:36:34.639 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T15:36:34.643 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T15:36:34.644 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T15:36:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T15:36:34.674 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T15:36:34.675 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T15:36:34.675 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T15:36:34.678 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T15:36:34.680 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T15:36:34.680 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T15:36:34.685 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T15:36:34.685 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T15:36:34.686 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T15:36:34.691 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T15:36:34.702 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T15:36:34.703 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T15:36:34.705 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T15:36:34.706 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T15:36:34.718 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T15:36:34.719 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T15:36:34.719 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T15:36:34.719 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T15:36:34.731 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T15:36:34.737 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T15:36:34.737 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T15:36:34.738 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T15:36:34.775 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T15:36:34.779 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T15:36:34.779 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T15:36:34.791 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T15:36:34.793 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T15:36:34.811 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T15:36:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T15:36:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T15:36:34.817 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T15:36:34.817 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T15:36:34.837 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T15:36:34.837 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-25T15:36:34.838 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-25T15:36:34.860 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T15:36:34.861 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T15:36:34.861 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-25T15:36:34.861 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-25T15:36:34.862 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T15:36:34.877 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T15:36:34.932 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T15:36:35.138 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T15:36:35.311 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T15:36:35.326 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T15:36:35.329 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T15:36:35.422 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T15:36:35.423 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T15:36:35.432 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T15:36:35.596 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T15:36:36.316 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T15:36:36.397 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T15:36:36.424 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T15:36:36.426 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T15:36:36.427 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T15:36:36.429 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T15:36:36.431 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T15:36:36.431 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-25T15:36:36.438 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-25T15:36:36.446 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T15:36:36.610 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T15:36:36.693 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-25T15:36:36.775 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-25T15:36:36.856 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T15:36:36.886 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T15:36:40.703 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 13.2 MB in 2s (6,749 kB/s) 2024-04-25T15:36:41.337 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 13.2 MB in 3s (4,872 kB/s) 2024-04-25T15:36:42.039 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-25T15:36:42.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T15:36:42.118 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-25T15:36:42.324 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-25T15:36:42.324 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-25T15:36:42.507 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:36:42.507 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:36:42.507 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:36:42.508 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:36:42.508 INFO:teuthology.orchestra.run.smithi067.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T15:36:42.508 INFO:teuthology.orchestra.run.smithi067.stdout: python2.7-minimal 2024-04-25T15:36:42.508 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:36:42.509 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-04-25T15:36:42.509 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T15:36:42.509 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T15:36:42.509 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T15:36:42.510 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-04-25T15:36:42.511 INFO:teuthology.orchestra.run.smithi067.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T15:36:42.511 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-04-25T15:36:42.511 INFO:teuthology.orchestra.run.smithi067.stdout: thermald 2024-04-25T15:36:42.553 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-04-25T15:36:42.554 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T15:36:42.554 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T15:36:42.554 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T15:36:42.555 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-04-25T15:36:42.555 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T15:36:42.650 INFO:teuthology.orchestra.run.smithi067.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T15:36:42.650 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 113 MB of archives. 2024-04-25T15:36:42.650 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T15:36:42.650 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T15:36:42.775 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-04-25T15:36:42.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T15:36:42.855 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-04-25T15:36:43.056 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-04-25T15:36:43.057 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-04-25T15:36:43.230 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:36:43.230 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:36:43.230 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:36:43.231 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:36:43.231 INFO:teuthology.orchestra.run.smithi176.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T15:36:43.231 INFO:teuthology.orchestra.run.smithi176.stdout: python2.7-minimal 2024-04-25T15:36:43.231 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:36:43.232 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-04-25T15:36:43.232 INFO:teuthology.orchestra.run.smithi176.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T15:36:43.232 INFO:teuthology.orchestra.run.smithi176.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T15:36:43.233 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T15:36:43.234 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-04-25T15:36:43.234 INFO:teuthology.orchestra.run.smithi176.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T15:36:43.234 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2024-04-25T15:36:43.234 INFO:teuthology.orchestra.run.smithi176.stdout: thermald 2024-04-25T15:36:43.286 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-04-25T15:36:43.286 INFO:teuthology.orchestra.run.smithi176.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T15:36:43.286 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T15:36:43.286 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T15:36:43.287 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages will be upgraded: 2024-04-25T15:36:43.288 INFO:teuthology.orchestra.run.smithi176.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T15:36:43.338 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T15:36:43.397 INFO:teuthology.orchestra.run.smithi176.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T15:36:43.397 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 113 MB of archives. 2024-04-25T15:36:43.397 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T15:36:43.397 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T15:36:43.584 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T15:36:44.682 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T15:36:45.058 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T15:36:46.051 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T15:36:46.051 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T15:36:46.052 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T15:36:46.347 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T15:36:46.419 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T15:36:46.793 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T15:36:46.793 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T15:36:46.794 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T15:36:46.966 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 113 MB in 4s (29.4 MB/s) 2024-04-25T15:36:47.075 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T15:36:47.157 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T15:36:47.158 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T15:36:47.547 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 113 MB in 4s (29.3 MB/s) 2024-04-25T15:36:47.758 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T15:36:49.154 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T15:36:49.159 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:36:49.247 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:36:49.711 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T15:36:49.715 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:36:49.855 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:36:51.831 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T15:36:51.852 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:36:51.968 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:36:52.138 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T15:36:52.160 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:36:52.265 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:36:52.443 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T15:36:52.463 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:36:52.501 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:36:52.733 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T15:36:52.753 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:36:52.791 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:36:58.977 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T15:36:59.104 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T15:36:59.463 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T15:36:59.565 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T15:36:59.852 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T15:36:59.863 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T15:36:59.866 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T15:36:59.909 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T15:37:00.010 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T15:37:00.373 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T15:37:00.493 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T15:37:00.796 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T15:37:00.818 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T15:37:00.863 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T15:37:05.966 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T15:37:06.003 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:37:06.041 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:06.832 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T15:37:06.869 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T15:37:06.906 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:08.770 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T15:37:08.920 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T15:37:09.279 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T15:37:09.417 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:09.543 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T15:37:09.662 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:10.025 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T15:37:10.650 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T15:37:11.034 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T15:37:11.155 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:11.256 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T15:37:11.359 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:11.922 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T15:37:11.922 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T15:37:11.922 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T15:37:11.922 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:12.073 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:12.702 INFO:teuthology.orchestra.run.smithi176.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T15:37:12.702 INFO:teuthology.orchestra.run.smithi176.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T15:37:12.702 INFO:teuthology.orchestra.run.smithi176.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T15:37:12.703 INFO:teuthology.orchestra.run.smithi176.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:12.886 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:13.534 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:14.280 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:14.968 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T15:37:15.102 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T15:37:15.203 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:15.285 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T15:37:15.285 INFO:teuthology.orchestra.run.smithi067.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:16.448 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T15:37:16.573 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T15:37:16.725 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T15:37:16.788 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T15:37:16.788 INFO:teuthology.orchestra.run.smithi176.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:29.532 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T15:37:29.532 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub' 2024-04-25T15:37:29.564 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T15:37:29.611 INFO:teuthology.orchestra.run.smithi067.stdout:Generating grub configuration file ... 2024-04-25T15:37:29.844 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T15:37:29.863 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:30.188 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T15:37:30.194 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T15:37:30.269 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T15:37:30.275 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T15:37:30.437 INFO:teuthology.orchestra.run.smithi067.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T15:37:30.437 INFO:teuthology.orchestra.run.smithi067.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T15:37:30.437 INFO:teuthology.orchestra.run.smithi067.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T15:37:30.477 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-04-25T15:37:30.942 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T15:37:30.943 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub' 2024-04-25T15:37:30.968 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T15:37:30.985 INFO:teuthology.orchestra.run.smithi176.stdout:Generating grub configuration file ... 2024-04-25T15:37:31.025 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-04-25T15:37:31.043 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-04-25T15:37:31.043 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.105.102 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-04-25T15:37:31.044 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-04-25T15:37:31.045 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T15:37:31.045 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T15:37:31.045 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T15:37:31.045 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-25T15:37:31.045 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T15:37:31.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-04-25T15:37:31.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-04-25T15:37:31.279 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T15:37:31.300 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:31.369 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T15:37:31.465 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T15:37:31.546 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T15:37:31.629 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T15:37:31.643 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T15:37:31.648 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T15:37:31.717 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T15:37:31.723 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T15:37:31.904 INFO:teuthology.orchestra.run.smithi176.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T15:37:31.904 INFO:teuthology.orchestra.run.smithi176.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T15:37:31.904 INFO:teuthology.orchestra.run.smithi176.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T15:37:31.945 INFO:teuthology.orchestra.run.smithi176.stdout:done 2024-04-25T15:37:32.453 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic 2024-04-25T15:37:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic 2024-04-25T15:37:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2024-04-25T15:37:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2024-04-25T15:37:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2024-04-25T15:37:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 21 2024-04-25T15:37:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T15:37:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.15.0.105.102 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2024-04-25T15:37:32.472 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2024-04-25T15:37:32.472 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T15:37:32.472 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T15:37:32.472 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T15:37:32.472 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-25T15:37:32.473 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T15:37:32.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2024-04-25T15:37:32.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-04-25T15:37:32.795 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-25T15:37:32.795 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T15:37:32.807 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T15:37:32.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T15:37:32.828 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T15:37:32.865 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-25T15:37:32.913 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T15:37:33.046 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-25T15:37:33.047 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-25T15:37:33.209 INFO:teuthology.orchestra.run.smithi067.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T15:37:33.209 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:37:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:37:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:37:33.209 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:37:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T15:37:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T15:37:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:37:33.250 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T15:37:33.251 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-04-25T15:37:33.313 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.105.102 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-04-25T15:37:33.314 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-04-25T15:37:33.315 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-04-25T15:37:33.315 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-04-25T15:37:33.315 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-25T15:37:33.362 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.NfdKbq6Q3u 2024-04-25T15:37:33.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NfdKbq6Q3u 2024-04-25T15:37:33.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.NfdKbq6Q3u 2024-04-25T15:37:33.540 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.NfdKbq6Q3u is 10KB 2024-04-25T15:37:33.553 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.NfdKbq6Q3u 2024-04-25T15:37:33.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T15:37:33.614 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:37:33.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T15:37:33.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T15:37:33.682 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T15:37:33.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo update-grub 2024-04-25T15:37:34.165 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-04-25T15:37:34.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T15:37:34.234 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-04-25T15:37:34.412 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-04-25T15:37:34.412 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-04-25T15:37:34.567 INFO:teuthology.orchestra.run.smithi176.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T15:37:34.568 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:37:34.568 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:37:34.568 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:37:34.568 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:37:34.568 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T15:37:34.568 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T15:37:34.568 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:37:34.605 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T15:37:34.607 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic 2024-04-25T15:37:34.663 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic 2024-04-25T15:37:34.663 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2024-04-25T15:37:34.663 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 21 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.15.0.105.102 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2024-04-25T15:37:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2024-04-25T15:37:34.665 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-04-25T15:37:34.708 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.2W0m0QWdBK 2024-04-25T15:37:34.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub/grub.cfg /tmp/tmp.2W0m0QWdBK 2024-04-25T15:37:34.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.2W0m0QWdBK 2024-04-25T15:37:34.899 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.2W0m0QWdBK is 10KB 2024-04-25T15:37:34.913 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2024-04-25T15:37:34.914 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.2W0m0QWdBK 2024-04-25T15:37:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T15:37:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-04-25T15:37:34.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T15:37:34.976 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:37:34.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T15:37:34.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T15:37:35.060 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T15:37:35.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo update-grub 2024-04-25T15:37:35.081 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T15:37:35.087 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:35.357 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T15:37:35.362 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T15:37:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T15:37:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T15:37:35.539 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T15:37:35.539 INFO:teuthology.orchestra.run.smithi067.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T15:37:35.539 INFO:teuthology.orchestra.run.smithi067.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T15:37:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-04-25T15:37:35.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-04-25T15:37:36.365 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub' 2024-04-25T15:37:36.366 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T15:37:36.370 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-04-25T15:37:36.585 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T15:37:36.591 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T15:37:36.834 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T15:37:36.840 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T15:37:36.878 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T15:37:36.884 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T15:37:36.992 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T15:37:36.993 INFO:teuthology.orchestra.run.smithi176.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T15:37:36.993 INFO:teuthology.orchestra.run.smithi176.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T15:37:37.017 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-04-25T15:37:37.034 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-04-25T15:38:05.569 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-25T15:38:05.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:38:05.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:38:07.042 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-25T15:38:07.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:38:07.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:38:25.439 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-04-25T15:38:34.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:38:34.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:38:37.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-04-25T15:38:40.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-25T15:38:49.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:38:49.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:38:49.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:38:49.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:39:07.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-25T15:39:19.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:39:19.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:39:19.441 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-25T15:39:21.287 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-04-25T15:39:21.884 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:39:21.884 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-25T15:39:21.884 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-04-25T15:39:21.931 INFO:teuthology.orchestra.run.smithi176.stdout:5.15.0-105-generic 2024-04-25T15:39:21.932 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T15:39:21.932 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T15:39:21.932 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T15:39:22.932 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T15:39:22.933 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T15:39:22.987 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-04-25T15:39:23.005 DEBUG:teuthology.parallel:result is None 2024-04-25T15:39:34.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:39:34.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:39:34.636 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-25T15:39:35.227 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:39:35.227 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-25T15:39:35.227 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-25T15:39:35.277 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-105-generic 2024-04-25T15:39:35.277 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T15:39:35.277 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T15:39:35.277 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T15:39:36.278 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T15:39:36.279 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T15:39:36.346 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-04-25T15:39:36.377 DEBUG:teuthology.parallel:result is None 2024-04-25T15:39:36.378 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T15:39:36.387 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T15:39:36.387 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:39:36.390 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:39:36.396 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T15:39:36.428 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T15:39:36.478 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T15:39:36.478 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:39:36.481 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:39:36.526 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T15:39:36.536 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T15:39:36.536 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:39:36.545 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:39:36.563 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:39:36.571 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:39:36.584 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:39:36.591 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:39:36.593 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T15:39:36.601 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T15:39:36.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:39:36.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:39:36.647 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T15:39:36.656 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T15:39:36.656 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:39:36.673 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:39:36.696 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:39:36.724 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:39:36.772 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:39:36.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:39:36.831 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:39:36.837 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:39:36.884 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:39:36.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:39:36.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-25T15:39:36.948 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-04-25T15:39:37.019 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T15:39:37.027 INFO:teuthology.task.internal:Starting timer... 2024-04-25T15:39:37.027 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T15:39:37.075 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T15:39:37.121 DEBUG:teuthology.task.selinux:Excluding smithi067: OS 'ubuntu' does not support SELinux 2024-04-25T15:39:37.121 DEBUG:teuthology.task.selinux:Excluding smithi176: OS 'ubuntu' does not support SELinux 2024-04-25T15:39:37.121 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T15:39:37.121 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T15:39:37.122 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T15:39:37.122 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T15:39:37.172 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T15:39:37.190 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T15:39:37.191 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T15:46:38.378 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-04-25T15:46:38.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:46:38.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:46:38.465 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-25T15:46:38.546 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-25T15:46:38.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:46:38.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:46:38.630 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-04-25T15:46:38.706 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-25T15:46:38.706 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T15:46:38.717 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T15:46:38.717 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:46:38.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:46:38.720 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:46:38.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Command line: ntpd -gq 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: ---------------------------------------------------- 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: corporation. Support and training for ntp-4 are 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: available at https://www.nwtime.org/support 2024-04-25T15:46:38.748 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: ---------------------------------------------------- 2024-04-25T15:46:38.749 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: proto: precision = 0.062 usec (-24) 2024-04-25T15:46:38.749 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: basedate set to 2022-02-04 2024-04-25T15:46:38.749 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: gps base set to 2022-02-06 (week 2196) 2024-04-25T15:46:38.749 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T15:46:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T15:46:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T15:46:38.751 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T15:46:38.751 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T15:46:38.752 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T15:46:38.752 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T15:46:38.752 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Listen normally on 3 enp3s0f1 172.21.15.67:123 2024-04-25T15:46:38.752 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Listen normally on 4 lo [::1]:123 2024-04-25T15:46:38.752 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:fc0b%5]:123 2024-04-25T15:46:38.752 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:38 ntpd[17211]: Listening on routing socket on fd #22 for interface updates 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Command line: ntpd -gq 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: ---------------------------------------------------- 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: corporation. Support and training for ntp-4 are 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: available at https://www.nwtime.org/support 2024-04-25T15:46:38.774 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: ---------------------------------------------------- 2024-04-25T15:46:38.775 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: proto: precision = 0.056 usec (-24) 2024-04-25T15:46:38.775 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: basedate set to 2022-02-04 2024-04-25T15:46:38.775 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: gps base set to 2022-02-06 (week 2196) 2024-04-25T15:46:38.775 INFO:teuthology.orchestra.run.smithi176.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T15:46:38.776 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T15:46:38.776 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T15:46:38.776 INFO:teuthology.orchestra.run.smithi176.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T15:46:38.777 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T15:46:38.777 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T15:46:38.777 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T15:46:38.777 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Listen normally on 3 enp3s0f1 172.21.15.176:123 2024-04-25T15:46:38.777 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Listen normally on 4 lo [::1]:123 2024-04-25T15:46:38.777 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f61%5]:123 2024-04-25T15:46:38.777 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:38 ntpd[17238]: Listening on routing socket on fd #22 for interface updates 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:39.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:39 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:40.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:40 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:41 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:42.748 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:42 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:42.748 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:42 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:42 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:42 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:42 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:42.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:42 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:42.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:42 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:43.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:43 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:43.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:43 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:43.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:43 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:43.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:43 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:43.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:43 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:43.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:43 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:43.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:43 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:44 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:44 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:44 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:44 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:44 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:44 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:44.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:44.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:44.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:44.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:44 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:45.749 INFO:teuthology.orchestra.run.smithi067.stdout:25 Apr 15:46:45 ntpd[17211]: ntpd: time slew -0.001604 s 2024-04-25T15:46:45.750 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew -0.001604s 2024-04-25T15:46:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T15:46:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:25 Apr 15:46:45 ntpd[17211]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T15:46:45.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:45 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:45.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:45 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:45.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:45 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:45.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:45 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:45.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:45 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:45.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:45 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:45.799 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T15:46:45.799 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-04-25T15:46:45.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:45.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:45.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:45.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:46.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:46.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:46.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:46.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:46.775 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T15:46:46.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T15:46:46.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T15:46:46.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T15:46:46.776 INFO:teuthology.orchestra.run.smithi176.stdout:25 Apr 15:46:46 ntpd[17238]: ntpd: time slew +0.000990 s 2024-04-25T15:46:46.776 INFO:teuthology.orchestra.run.smithi176.stdout:ntpd: time slew +0.000990s 2024-04-25T15:46:46.776 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T15:46:46.777 INFO:teuthology.orchestra.run.smithi176.stderr:25 Apr 15:46:46 ntpd[17238]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T15:46:46.827 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T15:46:46.827 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2024-04-25T15:46:46.827 INFO:teuthology.orchestra.run.smithi176.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:46.827 INFO:teuthology.orchestra.run.smithi176.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:46.827 INFO:teuthology.orchestra.run.smithi176.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:46.827 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:46:46.828 INFO:teuthology.run_tasks:Running task install... 2024-04-25T15:46:46.836 DEBUG:teuthology.task.install:project ceph 2024-04-25T15:46:46.836 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T15:46:46.836 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T15:46:46.836 INFO:teuthology.task.install:Using flavor: default 2024-04-25T15:46:46.842 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T15:46:46.842 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T15:46:46.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-key list | grep Ceph 2024-04-25T15:46:46.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-key list | grep Ceph 2024-04-25T15:46:46.912 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T15:46:46.927 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T15:46:46.945 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T15:46:46.946 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T15:46:46.946 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-25T15:46:46.946 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-25T15:46:46.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:46:46.956 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T15:46:46.956 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T15:46:46.956 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-25T15:46:46.956 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-25T15:46:46.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:46:47.095 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default/ 2024-04-25T15:46:47.095 INFO:teuthology.task.install.deb:Package version is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:46:47.175 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default/ 2024-04-25T15:46:47.175 INFO:teuthology.task.install.deb:Package version is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:46:47.266 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:46:47.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T15:46:47.279 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:46:47.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T15:46:47.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-04-25T15:46:47.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-04-25T15:46:47.462 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T15:46:47.475 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T15:46:47.476 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T15:46:47.495 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T15:46:47.509 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T15:46:47.530 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T15:46:47.553 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T15:46:47.575 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,406 B] 2024-04-25T15:46:47.581 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T15:46:47.591 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T15:46:47.598 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T15:46:47.603 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-25T15:46:47.628 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T15:46:47.665 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T15:46:47.665 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T15:46:47.702 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,406 B] 2024-04-25T15:46:47.744 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-25T15:46:47.746 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T15:46:47.842 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 29.2 kB in 0s (78.3 kB/s) 2024-04-25T15:46:47.950 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 29.2 kB in 0s (60.7 kB/s) 2024-04-25T15:46:48.827 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-25T15:46:48.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3291-g994e2f20-1jammy cephadm=19.0.0-3291-g994e2f20-1jammy ceph-mds=19.0.0-3291-g994e2f20-1jammy ceph-mgr=19.0.0-3291-g994e2f20-1jammy ceph-common=19.0.0-3291-g994e2f20-1jammy ceph-fuse=19.0.0-3291-g994e2f20-1jammy ceph-test=19.0.0-3291-g994e2f20-1jammy ceph-volume=19.0.0-3291-g994e2f20-1jammy radosgw=19.0.0-3291-g994e2f20-1jammy python3-rados=19.0.0-3291-g994e2f20-1jammy python3-rgw=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy python3-rbd=19.0.0-3291-g994e2f20-1jammy libcephfs2=19.0.0-3291-g994e2f20-1jammy libcephfs-dev=19.0.0-3291-g994e2f20-1jammy librados2=19.0.0-3291-g994e2f20-1jammy librbd1=19.0.0-3291-g994e2f20-1jammy rbd-fuse=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy cephfs-shell=19.0.0-3291-g994e2f20-1jammy cephfs-top=19.0.0-3291-g994e2f20-1jammy cephfs-mirror=19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:46:48.879 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-04-25T15:46:48.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3291-g994e2f20-1jammy cephadm=19.0.0-3291-g994e2f20-1jammy ceph-mds=19.0.0-3291-g994e2f20-1jammy ceph-mgr=19.0.0-3291-g994e2f20-1jammy ceph-common=19.0.0-3291-g994e2f20-1jammy ceph-fuse=19.0.0-3291-g994e2f20-1jammy ceph-test=19.0.0-3291-g994e2f20-1jammy ceph-volume=19.0.0-3291-g994e2f20-1jammy radosgw=19.0.0-3291-g994e2f20-1jammy python3-rados=19.0.0-3291-g994e2f20-1jammy python3-rgw=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy python3-rbd=19.0.0-3291-g994e2f20-1jammy libcephfs2=19.0.0-3291-g994e2f20-1jammy libcephfs-dev=19.0.0-3291-g994e2f20-1jammy librados2=19.0.0-3291-g994e2f20-1jammy librbd1=19.0.0-3291-g994e2f20-1jammy rbd-fuse=19.0.0-3291-g994e2f20-1jammy python3-cephfs=19.0.0-3291-g994e2f20-1jammy cephfs-shell=19.0.0-3291-g994e2f20-1jammy cephfs-top=19.0.0-3291-g994e2f20-1jammy cephfs-mirror=19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:46:48.896 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-25T15:46:48.958 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-04-25T15:46:49.081 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-25T15:46:49.081 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-25T15:46:49.157 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-04-25T15:46:49.158 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-04-25T15:46:49.262 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:46:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:46:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:46:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:46:49.262 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T15:46:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T15:46:49.263 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:46:49.263 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-04-25T15:46:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T15:46:49.263 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T15:46:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T15:46:49.264 INFO:teuthology.orchestra.run.smithi067.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-25T15:46:49.265 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-25T15:46:49.266 INFO:teuthology.orchestra.run.smithi067.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-04-25T15:46:49.267 INFO:teuthology.orchestra.run.smithi067.stdout: btrfs-tools 2024-04-25T15:46:49.358 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:46:49.358 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:46:49.358 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:46:49.358 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T15:46:49.361 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-25T15:46:49.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2024-04-25T15:46:49.363 INFO:teuthology.orchestra.run.smithi176.stdout: btrfs-tools 2024-04-25T15:46:49.373 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-04-25T15:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T15:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T15:46:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-25T15:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-25T15:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T15:46:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-25T15:46:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-25T15:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-25T15:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T15:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T15:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T15:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T15:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-25T15:46:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: socat unzip xmlstarlet zip 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-25T15:46:49.468 INFO:teuthology.orchestra.run.smithi176.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-25T15:46:49.469 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T15:46:49.470 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T15:46:49.470 INFO:teuthology.orchestra.run.smithi176.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T15:46:49.470 INFO:teuthology.orchestra.run.smithi176.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T15:46:49.470 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-25T15:46:49.470 INFO:teuthology.orchestra.run.smithi176.stdout: socat unzip xmlstarlet zip 2024-04-25T15:46:49.518 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T15:46:49.518 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 175 MB of archives. 2024-04-25T15:46:49.518 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 887 MB of additional disk space will be used. 2024-04-25T15:46:49.518 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3291-g994e2f20-1jammy [3,525 kB] 2024-04-25T15:46:49.564 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-25T15:46:49.611 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T15:46:49.611 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 175 MB of archives. 2024-04-25T15:46:49.611 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 887 MB of additional disk space will be used. 2024-04-25T15:46:49.611 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3291-g994e2f20-1jammy [3,525 kB] 2024-04-25T15:46:49.651 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T15:46:49.657 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-25T15:46:49.702 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-25T15:46:49.740 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T15:46:49.785 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-25T15:46:49.903 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T15:46:49.934 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3291-g994e2f20-1jammy [3,287 kB] 2024-04-25T15:46:49.987 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T15:46:50.013 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T15:46:50.026 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T15:46:50.054 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3291-g994e2f20-1jammy [3,287 kB] 2024-04-25T15:46:50.079 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T15:46:50.082 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T15:46:50.095 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T15:46:50.132 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3291-g994e2f20-1jammy [726 kB] 2024-04-25T15:46:50.147 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T15:46:50.167 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3291-g994e2f20-1jammy [356 kB] 2024-04-25T15:46:50.191 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3291-g994e2f20-1jammy [32.8 kB] 2024-04-25T15:46:50.192 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3291-g994e2f20-1jammy [180 kB] 2024-04-25T15:46:50.211 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3291-g994e2f20-1jammy [67.1 kB] 2024-04-25T15:46:50.213 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3291-g994e2f20-1jammy [330 kB] 2024-04-25T15:46:50.223 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3291-g994e2f20-1jammy [7,189 kB] 2024-04-25T15:46:50.291 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3291-g994e2f20-1jammy [726 kB] 2024-04-25T15:46:50.303 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T15:46:50.314 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T15:46:50.321 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T15:46:50.330 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T15:46:50.333 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3291-g994e2f20-1jammy [356 kB] 2024-04-25T15:46:50.343 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T15:46:50.355 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T15:46:50.357 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T15:46:50.364 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3291-g994e2f20-1jammy [32.8 kB] 2024-04-25T15:46:50.365 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T15:46:50.365 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3291-g994e2f20-1jammy [180 kB] 2024-04-25T15:46:50.372 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3291-g994e2f20-1jammy [67.1 kB] 2024-04-25T15:46:50.374 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3291-g994e2f20-1jammy [330 kB] 2024-04-25T15:46:50.405 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3291-g994e2f20-1jammy [7,189 kB] 2024-04-25T15:46:50.523 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T15:46:50.531 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T15:46:50.656 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3291-g994e2f20-1jammy [112 kB] 2024-04-25T15:46:50.658 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3291-g994e2f20-1jammy [471 kB] 2024-04-25T15:46:50.696 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3291-g994e2f20-1jammy [25.6 MB] 2024-04-25T15:46:50.932 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3291-g994e2f20-1jammy [112 kB] 2024-04-25T15:46:50.961 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3291-g994e2f20-1jammy [471 kB] 2024-04-25T15:46:50.977 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3291-g994e2f20-1jammy [25.6 MB] 2024-04-25T15:46:51.097 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T15:46:51.112 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T15:46:51.127 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T15:46:51.135 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T15:46:51.136 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T15:46:51.137 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T15:46:51.154 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T15:46:51.154 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T15:46:51.178 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T15:46:51.191 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T15:46:51.192 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T15:46:51.192 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T15:46:51.204 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T15:46:51.207 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T15:46:51.208 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T15:46:51.222 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T15:46:51.247 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T15:46:51.263 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T15:46:51.268 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T15:46:51.268 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T15:46:51.378 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T15:46:51.442 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T15:46:51.461 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T15:46:51.483 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T15:46:51.525 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T15:46:51.532 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T15:46:51.543 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T15:46:51.582 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T15:46:51.604 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T15:46:51.606 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T15:46:51.631 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T15:46:51.632 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T15:46:51.648 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T15:46:51.665 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T15:46:51.814 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T15:46:51.830 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T15:46:51.837 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T15:46:51.842 INFO:teuthology.orchestra.run.smithi176.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T15:46:51.863 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T15:46:51.884 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T15:46:51.893 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T15:46:51.898 INFO:teuthology.orchestra.run.smithi067.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T15:46:52.027 INFO:teuthology.orchestra.run.smithi176.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T15:46:52.092 INFO:teuthology.orchestra.run.smithi067.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T15:46:52.111 INFO:teuthology.orchestra.run.smithi176.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T15:46:52.124 INFO:teuthology.orchestra.run.smithi176.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T15:46:52.180 INFO:teuthology.orchestra.run.smithi067.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T15:46:52.197 INFO:teuthology.orchestra.run.smithi067.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T15:46:52.294 INFO:teuthology.orchestra.run.smithi067.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3291-g994e2f20-1jammy [4,917 kB] 2024-04-25T15:46:52.303 INFO:teuthology.orchestra.run.smithi176.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T15:46:52.350 INFO:teuthology.orchestra.run.smithi176.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T15:46:52.362 INFO:teuthology.orchestra.run.smithi176.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T15:46:52.384 INFO:teuthology.orchestra.run.smithi067.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T15:46:52.438 INFO:teuthology.orchestra.run.smithi067.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T15:46:52.451 INFO:teuthology.orchestra.run.smithi067.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T15:46:52.505 INFO:teuthology.orchestra.run.smithi067.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3291-g994e2f20-1jammy [244 kB] 2024-04-25T15:46:52.524 INFO:teuthology.orchestra.run.smithi176.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T15:46:52.526 INFO:teuthology.orchestra.run.smithi176.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T15:46:52.526 INFO:teuthology.orchestra.run.smithi067.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3291-g994e2f20-1jammy [125 kB] 2024-04-25T15:46:52.530 INFO:teuthology.orchestra.run.smithi176.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T15:46:52.531 INFO:teuthology.orchestra.run.smithi067.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3291-g994e2f20-1jammy [1,542 kB] 2024-04-25T15:46:52.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T15:46:52.549 INFO:teuthology.orchestra.run.smithi176.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T15:46:52.609 INFO:teuthology.orchestra.run.smithi067.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T15:46:52.610 INFO:teuthology.orchestra.run.smithi067.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T15:46:52.615 INFO:teuthology.orchestra.run.smithi067.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T15:46:52.620 INFO:teuthology.orchestra.run.smithi067.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T15:46:52.627 INFO:teuthology.orchestra.run.smithi067.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3291-g994e2f20-1jammy [6,148 kB] 2024-04-25T15:46:52.634 INFO:teuthology.orchestra.run.smithi067.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T15:46:52.734 INFO:teuthology.orchestra.run.smithi176.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T15:46:52.821 INFO:teuthology.orchestra.run.smithi067.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T15:46:52.981 INFO:teuthology.orchestra.run.smithi176.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3291-g994e2f20-1jammy [4,917 kB] 2024-04-25T15:46:52.985 INFO:teuthology.orchestra.run.smithi067.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3291-g994e2f20-1jammy [22.5 MB] 2024-04-25T15:46:53.063 INFO:teuthology.orchestra.run.smithi176.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T15:46:53.087 INFO:teuthology.orchestra.run.smithi067.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T15:46:53.178 INFO:teuthology.orchestra.run.smithi176.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T15:46:53.205 INFO:teuthology.orchestra.run.smithi067.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T15:46:53.220 INFO:teuthology.orchestra.run.smithi176.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3291-g994e2f20-1jammy [244 kB] 2024-04-25T15:46:53.220 INFO:teuthology.orchestra.run.smithi176.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T15:46:53.246 INFO:teuthology.orchestra.run.smithi176.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3291-g994e2f20-1jammy [125 kB] 2024-04-25T15:46:53.249 INFO:teuthology.orchestra.run.smithi067.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T15:46:53.250 INFO:teuthology.orchestra.run.smithi176.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3291-g994e2f20-1jammy [1,542 kB] 2024-04-25T15:46:53.304 INFO:teuthology.orchestra.run.smithi176.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T15:46:53.316 INFO:teuthology.orchestra.run.smithi176.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T15:46:53.329 INFO:teuthology.orchestra.run.smithi176.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T15:46:53.332 INFO:teuthology.orchestra.run.smithi067.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T15:46:53.345 INFO:teuthology.orchestra.run.smithi067.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T15:46:53.357 INFO:teuthology.orchestra.run.smithi067.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T15:46:53.360 INFO:teuthology.orchestra.run.smithi176.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3291-g994e2f20-1jammy [6,148 kB] 2024-04-25T15:46:53.371 INFO:teuthology.orchestra.run.smithi176.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T15:46:53.412 INFO:teuthology.orchestra.run.smithi067.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T15:46:53.634 INFO:teuthology.orchestra.run.smithi176.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T15:46:53.636 INFO:teuthology.orchestra.run.smithi176.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T15:46:53.638 INFO:teuthology.orchestra.run.smithi176.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T15:46:53.706 INFO:teuthology.orchestra.run.smithi067.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T15:46:53.723 INFO:teuthology.orchestra.run.smithi067.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T15:46:53.726 INFO:teuthology.orchestra.run.smithi067.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T15:46:53.803 INFO:teuthology.orchestra.run.smithi176.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T15:46:53.803 INFO:teuthology.orchestra.run.smithi176.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3291-g994e2f20-1jammy [22.5 MB] 2024-04-25T15:46:53.893 INFO:teuthology.orchestra.run.smithi067.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T15:46:54.140 INFO:teuthology.orchestra.run.smithi176.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T15:46:54.221 INFO:teuthology.orchestra.run.smithi067.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T15:46:54.358 INFO:teuthology.orchestra.run.smithi067.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3291-g994e2f20-1jammy [14.1 kB] 2024-04-25T15:46:54.358 INFO:teuthology.orchestra.run.smithi067.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3291-g994e2f20-1jammy [856 kB] 2024-04-25T15:46:54.375 INFO:teuthology.orchestra.run.smithi067.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3291-g994e2f20-1jammy [2,367 kB] 2024-04-25T15:46:54.507 INFO:teuthology.orchestra.run.smithi067.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3291-g994e2f20-1jammy [791 kB] 2024-04-25T15:46:54.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T15:46:54.544 INFO:teuthology.orchestra.run.smithi176.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T15:46:54.546 INFO:teuthology.orchestra.run.smithi176.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T15:46:54.562 INFO:teuthology.orchestra.run.smithi067.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3291-g994e2f20-1jammy [148 kB] 2024-04-25T15:46:54.568 INFO:teuthology.orchestra.run.smithi067.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3291-g994e2f20-1jammy [2,297 kB] 2024-04-25T15:46:54.614 INFO:teuthology.orchestra.run.smithi176.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T15:46:54.614 INFO:teuthology.orchestra.run.smithi176.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T15:46:54.616 INFO:teuthology.orchestra.run.smithi176.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T15:46:54.617 INFO:teuthology.orchestra.run.smithi176.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T15:46:54.619 INFO:teuthology.orchestra.run.smithi176.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T15:46:54.620 INFO:teuthology.orchestra.run.smithi176.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T15:46:54.712 INFO:teuthology.orchestra.run.smithi067.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3291-g994e2f20-1jammy [8,625 kB] 2024-04-25T15:46:54.718 INFO:teuthology.orchestra.run.smithi067.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T15:46:54.730 INFO:teuthology.orchestra.run.smithi067.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T15:46:54.731 INFO:teuthology.orchestra.run.smithi067.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T15:46:54.784 INFO:teuthology.orchestra.run.smithi176.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T15:46:54.810 INFO:teuthology.orchestra.run.smithi067.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T15:46:54.811 INFO:teuthology.orchestra.run.smithi067.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T15:46:54.813 INFO:teuthology.orchestra.run.smithi067.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T15:46:54.814 INFO:teuthology.orchestra.run.smithi067.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T15:46:54.816 INFO:teuthology.orchestra.run.smithi067.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T15:46:54.817 INFO:teuthology.orchestra.run.smithi067.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T15:46:54.984 INFO:teuthology.orchestra.run.smithi067.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T15:46:54.985 INFO:teuthology.orchestra.run.smithi176.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T15:46:55.200 INFO:teuthology.orchestra.run.smithi067.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T15:46:55.213 INFO:teuthology.orchestra.run.smithi067.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3291-g994e2f20-1jammy [14.2 kB] 2024-04-25T15:46:55.214 INFO:teuthology.orchestra.run.smithi067.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3291-g994e2f20-1jammy [44.9 MB] 2024-04-25T15:46:55.286 INFO:teuthology.orchestra.run.smithi176.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T15:46:55.329 INFO:teuthology.orchestra.run.smithi176.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T15:46:55.355 INFO:teuthology.orchestra.run.smithi176.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T15:46:55.364 INFO:teuthology.orchestra.run.smithi176.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T15:46:55.411 INFO:teuthology.orchestra.run.smithi176.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T15:46:55.436 INFO:teuthology.orchestra.run.smithi176.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-25T15:46:55.437 INFO:teuthology.orchestra.run.smithi176.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T15:46:55.444 INFO:teuthology.orchestra.run.smithi176.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T15:46:55.446 INFO:teuthology.orchestra.run.smithi176.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T15:46:55.563 INFO:teuthology.orchestra.run.smithi067.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T15:46:55.589 INFO:teuthology.orchestra.run.smithi176.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3291-g994e2f20-1jammy [14.1 kB] 2024-04-25T15:46:55.589 INFO:teuthology.orchestra.run.smithi176.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3291-g994e2f20-1jammy [856 kB] 2024-04-25T15:46:55.606 INFO:teuthology.orchestra.run.smithi176.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3291-g994e2f20-1jammy [2,367 kB] 2024-04-25T15:46:55.611 INFO:teuthology.orchestra.run.smithi067.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T15:46:55.659 INFO:teuthology.orchestra.run.smithi176.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3291-g994e2f20-1jammy [791 kB] 2024-04-25T15:46:55.667 INFO:teuthology.orchestra.run.smithi067.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T15:46:55.678 INFO:teuthology.orchestra.run.smithi067.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T15:46:55.706 INFO:teuthology.orchestra.run.smithi176.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3291-g994e2f20-1jammy [148 kB] 2024-04-25T15:46:55.708 INFO:teuthology.orchestra.run.smithi176.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3291-g994e2f20-1jammy [2,297 kB] 2024-04-25T15:46:55.728 INFO:teuthology.orchestra.run.smithi067.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T15:46:55.752 INFO:teuthology.orchestra.run.smithi176.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T15:46:55.759 INFO:teuthology.orchestra.run.smithi067.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-25T15:46:55.762 INFO:teuthology.orchestra.run.smithi067.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T15:46:55.770 INFO:teuthology.orchestra.run.smithi067.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T15:46:55.774 INFO:teuthology.orchestra.run.smithi067.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T15:46:55.857 INFO:teuthology.orchestra.run.smithi176.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3291-g994e2f20-1jammy [8,625 kB] 2024-04-25T15:46:55.983 INFO:teuthology.orchestra.run.smithi067.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T15:46:56.006 INFO:teuthology.orchestra.run.smithi176.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T15:46:56.237 INFO:teuthology.orchestra.run.smithi067.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T15:46:56.241 INFO:teuthology.orchestra.run.smithi176.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T15:46:56.244 INFO:teuthology.orchestra.run.smithi176.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T15:46:56.263 INFO:teuthology.orchestra.run.smithi176.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T15:46:56.268 INFO:teuthology.orchestra.run.smithi176.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T15:46:56.304 INFO:teuthology.orchestra.run.smithi176.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T15:46:56.317 INFO:teuthology.orchestra.run.smithi176.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T15:46:56.389 INFO:teuthology.orchestra.run.smithi176.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3291-g994e2f20-1jammy [14.2 kB] 2024-04-25T15:46:56.389 INFO:teuthology.orchestra.run.smithi176.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3291-g994e2f20-1jammy [44.9 MB] 2024-04-25T15:46:56.408 INFO:teuthology.orchestra.run.smithi176.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T15:46:56.410 INFO:teuthology.orchestra.run.smithi176.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T15:46:56.411 INFO:teuthology.orchestra.run.smithi176.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T15:46:56.485 INFO:teuthology.orchestra.run.smithi067.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T15:46:56.488 INFO:teuthology.orchestra.run.smithi067.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T15:46:56.505 INFO:teuthology.orchestra.run.smithi067.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T15:46:56.515 INFO:teuthology.orchestra.run.smithi067.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T15:46:56.560 INFO:teuthology.orchestra.run.smithi067.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T15:46:56.575 INFO:teuthology.orchestra.run.smithi067.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T15:46:56.590 INFO:teuthology.orchestra.run.smithi176.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T15:46:56.685 INFO:teuthology.orchestra.run.smithi067.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T15:46:56.687 INFO:teuthology.orchestra.run.smithi067.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T15:46:56.689 INFO:teuthology.orchestra.run.smithi067.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T15:46:56.863 INFO:teuthology.orchestra.run.smithi176.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T15:46:56.867 INFO:teuthology.orchestra.run.smithi067.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T15:46:57.142 INFO:teuthology.orchestra.run.smithi067.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T15:46:58.443 INFO:teuthology.orchestra.run.smithi067.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3291-g994e2f20-1jammy [124 kB] 2024-04-25T15:46:58.445 INFO:teuthology.orchestra.run.smithi067.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3291-g994e2f20-1jammy [236 kB] 2024-04-25T15:46:58.450 INFO:teuthology.orchestra.run.smithi067.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3291-g994e2f20-1jammy [33.9 kB] 2024-04-25T15:46:58.452 INFO:teuthology.orchestra.run.smithi067.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3291-g994e2f20-1jammy [23.4 kB] 2024-04-25T15:46:58.504 INFO:teuthology.orchestra.run.smithi067.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3291-g994e2f20-1jammy [39.0 kB] 2024-04-25T15:46:58.505 INFO:teuthology.orchestra.run.smithi067.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3291-g994e2f20-1jammy [14.2 MB] 2024-04-25T15:46:59.331 INFO:teuthology.orchestra.run.smithi067.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3291-g994e2f20-1jammy [92.6 kB] 2024-04-25T15:46:59.381 INFO:teuthology.orchestra.run.smithi176.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3291-g994e2f20-1jammy [124 kB] 2024-04-25T15:46:59.384 INFO:teuthology.orchestra.run.smithi176.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3291-g994e2f20-1jammy [236 kB] 2024-04-25T15:46:59.389 INFO:teuthology.orchestra.run.smithi176.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3291-g994e2f20-1jammy [33.9 kB] 2024-04-25T15:46:59.390 INFO:teuthology.orchestra.run.smithi176.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3291-g994e2f20-1jammy [23.4 kB] 2024-04-25T15:46:59.394 INFO:teuthology.orchestra.run.smithi176.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3291-g994e2f20-1jammy [39.0 kB] 2024-04-25T15:46:59.433 INFO:teuthology.orchestra.run.smithi176.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3291-g994e2f20-1jammy [14.2 MB] 2024-04-25T15:46:59.784 INFO:teuthology.orchestra.run.smithi176.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3291-g994e2f20-1jammy [92.6 kB] 2024-04-25T15:46:59.977 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 175 MB in 10s (17.6 MB/s) 2024-04-25T15:47:00.113 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-25T15:47:00.206 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T15:47:00.214 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-25T15:47:00.252 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-25T15:47:00.472 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 175 MB in 10s (16.8 MB/s) 2024-04-25T15:47:00.555 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T15:47:00.582 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T15:47:00.608 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-25T15:47:00.622 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T15:47:00.707 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T15:47:00.713 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-25T15:47:00.756 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-25T15:47:00.947 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cmd2. 2024-04-25T15:47:00.974 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-25T15:47:01.001 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T15:47:01.016 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-25T15:47:01.030 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T15:47:01.059 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T15:47:01.319 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T15:47:01.346 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T15:47:01.353 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cmd2. 2024-04-25T15:47:01.380 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-25T15:47:01.420 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-25T15:47:01.444 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T15:47:01.782 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T15:47:01.806 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T15:47:01.807 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T15:47:01.833 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T15:47:01.873 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T15:47:01.890 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T15:47:02.209 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T15:47:02.236 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T15:47:02.243 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T15:47:02.269 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T15:47:02.275 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T15:47:02.310 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T15:47:02.621 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T15:47:02.645 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T15:47:02.647 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T15:47:02.672 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T15:47:02.687 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T15:47:02.720 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:47:03.116 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T15:47:03.140 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T15:47:03.142 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T15:47:03.167 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T15:47:03.191 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:47:03.206 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:47:03.560 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T15:47:03.587 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T15:47:03.611 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T15:47:03.626 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:47:03.637 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T15:47:03.677 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:47:03.937 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T15:47:03.964 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T15:47:03.995 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T15:47:04.014 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T15:47:04.040 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T15:47:04.072 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:47:04.315 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librados2. 2024-04-25T15:47:04.342 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../010-librados2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:04.382 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:04.425 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T15:47:04.451 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T15:47:04.492 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T15:47:04.828 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librados2. 2024-04-25T15:47:04.854 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../010-librados2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:04.859 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librbd1. 2024-04-25T15:47:04.886 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../011-librbd1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:04.894 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:04.926 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:05.372 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librbd1. 2024-04-25T15:47:05.395 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T15:47:05.398 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../011-librbd1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:05.422 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:05.439 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:05.462 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rados. 2024-04-25T15:47:05.755 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:05.798 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:05.841 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T15:47:05.868 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:05.908 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:06.125 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T15:47:06.154 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:06.192 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:06.202 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rados. 2024-04-25T15:47:06.228 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:06.269 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:06.444 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T15:47:06.471 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:06.511 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:06.596 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T15:47:06.623 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:06.663 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:06.907 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T15:47:06.922 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:06.955 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T15:47:06.957 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:06.982 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:07.088 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:07.366 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T15:47:07.393 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T15:47:07.424 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T15:47:07.643 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T15:47:07.670 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:07.677 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T15:47:07.704 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:07.710 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:07.743 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:08.030 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T15:47:08.056 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T15:47:08.096 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T15:47:08.113 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T15:47:08.140 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T15:47:08.179 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T15:47:08.366 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T15:47:08.381 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:08.424 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:08.457 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T15:47:08.473 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T15:47:08.515 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T15:47:08.777 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T15:47:08.806 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T15:47:08.843 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T15:47:08.851 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T15:47:08.867 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T15:47:08.909 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T15:47:09.087 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T15:47:09.103 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T15:47:09.137 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T15:47:09.271 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua5.1. 2024-04-25T15:47:09.298 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T15:47:09.338 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T15:47:09.432 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T15:47:09.447 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T15:47:09.490 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T15:47:09.675 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-any. 2024-04-25T15:47:09.701 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-25T15:47:09.741 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T15:47:09.843 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua5.1. 2024-04-25T15:47:09.870 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T15:47:09.910 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T15:47:09.960 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package zip. 2024-04-25T15:47:09.976 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-25T15:47:10.010 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T15:47:10.213 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-any. 2024-04-25T15:47:10.239 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-25T15:47:10.271 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T15:47:10.296 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package unzip. 2024-04-25T15:47:10.323 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T15:47:10.362 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T15:47:10.499 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package zip. 2024-04-25T15:47:10.514 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-25T15:47:10.557 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T15:47:10.750 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package luarocks. 2024-04-25T15:47:10.777 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T15:47:10.816 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T15:47:10.868 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package unzip. 2024-04-25T15:47:10.895 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T15:47:10.935 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T15:47:11.199 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librgw2. 2024-04-25T15:47:11.226 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../027-librgw2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:11.265 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:11.347 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package luarocks. 2024-04-25T15:47:11.374 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T15:47:11.414 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T15:47:11.755 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librgw2. 2024-04-25T15:47:11.782 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../027-librgw2_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:11.813 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:11.867 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T15:47:11.885 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:11.917 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:12.395 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T15:47:12.399 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T15:47:12.423 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T15:47:12.425 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:12.462 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T15:47:12.465 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:12.767 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T15:47:12.794 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:12.833 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:13.002 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T15:47:13.028 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T15:47:13.068 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T15:47:13.195 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-common. 2024-04-25T15:47:13.222 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:13.278 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:13.354 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T15:47:13.381 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:13.421 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:13.766 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-common. 2024-04-25T15:47:13.792 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:13.832 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:14.809 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-base. 2024-04-25T15:47:14.836 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:14.893 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:15.346 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-base. 2024-04-25T15:47:15.372 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:15.421 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:15.530 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T15:47:15.557 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T15:47:15.604 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T15:47:15.940 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T15:47:15.968 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T15:47:16.007 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T15:47:16.025 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T15:47:16.052 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T15:47:16.091 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T15:47:16.335 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T15:47:16.352 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T15:47:16.362 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T15:47:16.368 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T15:47:16.402 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T15:47:16.402 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T15:47:16.663 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T15:47:16.688 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T15:47:16.690 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T15:47:16.715 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T15:47:16.721 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T15:47:16.754 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T15:47:17.025 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T15:47:17.052 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T15:47:17.057 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T15:47:17.084 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T15:47:17.091 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T15:47:17.124 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T15:47:17.394 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T15:47:17.421 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T15:47:17.461 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T15:47:17.518 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T15:47:17.546 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T15:47:17.593 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T15:47:17.722 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T15:47:17.748 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T15:47:17.780 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T15:47:17.921 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-portend. 2024-04-25T15:47:17.948 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-25T15:47:17.988 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T15:47:18.050 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-portend. 2024-04-25T15:47:18.076 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-25T15:47:18.116 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T15:47:18.274 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T15:47:18.301 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T15:47:18.332 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T15:47:18.452 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T15:47:18.479 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T15:47:18.519 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T15:47:18.585 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T15:47:18.601 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T15:47:18.637 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T15:47:18.788 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T15:47:18.815 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T15:47:18.855 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T15:47:18.997 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T15:47:19.024 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T15:47:19.063 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T15:47:19.200 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T15:47:19.227 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T15:47:19.266 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T15:47:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T15:47:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T15:47:19.416 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T15:47:19.578 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T15:47:19.605 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T15:47:19.644 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T15:47:19.686 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T15:47:19.713 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T15:47:19.752 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T15:47:19.922 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T15:47:19.949 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T15:47:19.989 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T15:47:20.064 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-mako. 2024-04-25T15:47:20.091 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T15:47:20.130 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T15:47:20.300 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-mako. 2024-04-25T15:47:20.327 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T15:47:20.366 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T15:47:20.475 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T15:47:20.502 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T15:47:20.542 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T15:47:20.703 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T15:47:20.730 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T15:47:20.770 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T15:47:20.786 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T15:47:20.813 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T15:47:20.845 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T15:47:21.064 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T15:47:21.091 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T15:47:21.130 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T15:47:21.139 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webob. 2024-04-25T15:47:21.166 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T15:47:21.205 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T15:47:21.392 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webob. 2024-04-25T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T15:47:21.450 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T15:47:21.583 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T15:47:21.610 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T15:47:21.662 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T15:47:21.803 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T15:47:21.830 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T15:47:21.872 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T15:47:22.147 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T15:47:22.174 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T15:47:22.214 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T15:47:22.510 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-paste. 2024-04-25T15:47:22.537 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T15:47:22.577 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T15:47:22.703 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T15:47:22.730 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T15:47:22.770 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T15:47:23.031 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-paste. 2024-04-25T15:47:23.058 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T15:47:23.089 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T15:47:23.418 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T15:47:23.433 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T15:47:23.475 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T15:47:23.589 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T15:47:23.615 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T15:47:23.655 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T15:47:23.737 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T15:47:23.764 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T15:47:23.795 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T15:47:23.958 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T15:47:23.984 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T15:47:24.024 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T15:47:24.073 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T15:47:24.100 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T15:47:24.139 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T15:47:24.302 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T15:47:24.329 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T15:47:24.386 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T15:47:24.451 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T15:47:24.478 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T15:47:24.517 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T15:47:24.647 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T15:47:24.673 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T15:47:24.713 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T15:47:24.864 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T15:47:24.891 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T15:47:24.930 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T15:47:25.085 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T15:47:25.111 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T15:47:25.151 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T15:47:25.258 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-idna. 2024-04-25T15:47:25.285 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-25T15:47:25.324 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T15:47:25.471 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-idna. 2024-04-25T15:47:25.497 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-25T15:47:25.529 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T15:47:25.603 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T15:47:25.630 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T15:47:25.661 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T15:47:25.823 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T15:47:25.850 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T15:47:25.890 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T15:47:25.931 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests. 2024-04-25T15:47:25.957 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T15:47:25.988 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T15:47:26.201 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests. 2024-04-25T15:47:26.228 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T15:47:26.268 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T15:47:26.284 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T15:47:26.310 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T15:47:26.349 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T15:47:26.554 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T15:47:26.580 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T15:47:26.611 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T15:47:26.612 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T15:47:26.627 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:26.661 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:26.915 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T15:47:26.942 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:26.981 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:26.991 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T15:47:27.018 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:27.049 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:27.320 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T15:47:27.346 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:27.377 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T15:47:27.378 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:27.404 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T15:47:27.446 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T15:47:27.697 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T15:47:27.724 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T15:47:27.763 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T15:47:27.771 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T15:47:27.799 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T15:47:28.066 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T15:47:28.088 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T15:47:28.093 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T15:47:28.133 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T15:47:28.399 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T15:47:28.416 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:28.457 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:28.694 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T15:47:28.722 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:28.761 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:28.836 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T15:47:28.863 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:28.902 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:29.197 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T15:47:29.224 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:29.264 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:29.514 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T15:47:29.541 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:29.580 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:29.892 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T15:47:29.918 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:29.958 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:30.642 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph. 2024-04-25T15:47:30.658 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../068-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:30.691 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:30.986 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T15:47:31.012 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph. 2024-04-25T15:47:31.013 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:31.027 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../068-ceph_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:31.052 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:31.061 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:31.339 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T15:47:31.366 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:31.405 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:31.422 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T15:47:31.450 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:31.489 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:31.775 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T15:47:31.802 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:31.841 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:31.850 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephadm. 2024-04-25T15:47:31.866 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../071-cephadm_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:31.908 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:32.211 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T15:47:32.236 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephadm. 2024-04-25T15:47:32.238 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T15:47:32.263 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../071-cephadm_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:32.277 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T15:47:32.303 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:32.589 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T15:47:32.614 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T15:47:32.615 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T15:47:32.630 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T15:47:32.655 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T15:47:32.698 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T15:47:32.966 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T15:47:32.981 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T15:47:33.032 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T15:47:33.761 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-babel. 2024-04-25T15:47:33.788 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T15:47:33.827 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T15:47:34.038 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-babel. 2024-04-25T15:47:34.065 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T15:47:34.104 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T15:47:34.121 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T15:47:34.149 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T15:47:34.179 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T15:47:34.390 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T15:47:34.417 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T15:47:34.456 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T15:47:34.499 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T15:47:34.529 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:34.566 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:34.810 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T15:47:34.836 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:34.876 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:34.877 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T15:47:34.894 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T15:47:34.927 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T15:47:35.213 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T15:47:35.213 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-routes. 2024-04-25T15:47:35.239 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T15:47:35.240 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T15:47:35.279 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T15:47:35.280 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T15:47:35.582 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-routes. 2024-04-25T15:47:35.600 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T15:47:35.609 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T15:47:35.627 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:35.648 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T15:47:35.666 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:35.985 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T15:47:36.012 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:36.051 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:36.880 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T15:47:36.907 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T15:47:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T15:47:37.281 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T15:47:37.297 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T15:47:37.339 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T15:47:37.458 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T15:47:37.486 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T15:47:37.524 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T15:47:37.835 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T15:47:37.862 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T15:47:37.901 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T15:47:37.911 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T15:47:37.939 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T15:47:38.064 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T15:47:38.230 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T15:47:38.256 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T15:47:38.381 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T15:47:38.550 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T15:47:38.577 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T15:47:38.608 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T15:47:38.684 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T15:47:38.711 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T15:47:38.759 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T15:47:39.342 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T15:47:39.370 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:39.425 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:39.451 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T15:47:39.479 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:39.518 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:40.420 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T15:47:40.421 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T15:47:40.448 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T15:47:40.448 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T15:47:40.479 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T15:47:40.486 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T15:47:40.757 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T15:47:40.784 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T15:47:40.823 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T15:47:40.823 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T15:47:40.850 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T15:47:40.897 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T15:47:41.168 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T15:47:41.196 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T15:47:41.235 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T15:47:41.284 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T15:47:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T15:47:41.359 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T15:47:41.546 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T15:47:41.573 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-25T15:47:41.612 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T15:47:41.737 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T15:47:41.765 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-25T15:47:41.804 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T15:47:41.907 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T15:47:41.935 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T15:47:41.966 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T15:47:42.141 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T15:47:42.168 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T15:47:42.207 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T15:47:42.269 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T15:47:42.296 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T15:47:42.335 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T15:47:42.536 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T15:47:42.563 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T15:47:42.602 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T15:47:42.663 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T15:47:42.691 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T15:47:42.730 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T15:47:42.914 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T15:47:42.941 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T15:47:42.980 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T15:47:43.050 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T15:47:43.077 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T15:47:43.116 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T15:47:43.417 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T15:47:43.444 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T15:47:43.483 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T15:47:44.019 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T15:47:44.047 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:44.130 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T15:47:44.159 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:44.197 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:44.255 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:44.525 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T15:47:44.553 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T15:47:44.582 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T15:47:44.591 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T15:47:44.609 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T15:47:44.640 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T15:47:44.903 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T15:47:44.931 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T15:47:44.969 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T15:47:44.994 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T15:47:45.021 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T15:47:45.060 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T15:47:45.247 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package jq. 2024-04-25T15:47:45.275 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T15:47:45.313 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T15:47:45.329 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package jq. 2024-04-25T15:47:45.345 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T15:47:45.388 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T15:47:45.566 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package socat. 2024-04-25T15:47:45.594 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T15:47:45.624 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package socat. 2024-04-25T15:47:45.634 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T15:47:45.651 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T15:47:45.690 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T15:47:45.951 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T15:47:45.967 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T15:47:45.969 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T15:47:45.997 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T15:47:46.001 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T15:47:46.044 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T15:47:46.347 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-test. 2024-04-25T15:47:46.375 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:46.406 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-test. 2024-04-25T15:47:46.413 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:46.423 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:46.465 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:48.676 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T15:47:48.703 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:48.742 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:48.803 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T15:47:48.831 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:48.869 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:49.130 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-25T15:47:49.157 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:49.196 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:49.240 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-25T15:47:49.268 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:49.306 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:49.483 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-colorama. 2024-04-25T15:47:49.510 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-25T15:47:49.540 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-25T15:47:49.617 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-colorama. 2024-04-25T15:47:49.646 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-25T15:47:49.684 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-25T15:47:49.768 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephfs-shell. 2024-04-25T15:47:49.796 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:49.835 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:49.970 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephfs-shell. 2024-04-25T15:47:49.998 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:50.036 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:50.104 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephfs-top. 2024-04-25T15:47:50.131 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:50.170 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:50.297 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephfs-top. 2024-04-25T15:47:50.314 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3291-g994e2f20-1jammy_all.deb ... 2024-04-25T15:47:50.347 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:50.439 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T15:47:50.467 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:50.497 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:50.549 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T15:47:50.566 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:50.608 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:50.792 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T15:47:50.819 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T15:47:50.858 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T15:47:50.894 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T15:47:50.922 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T15:47:50.969 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T15:47:51.196 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T15:47:51.224 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T15:47:51.262 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T15:47:51.307 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T15:47:51.335 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T15:47:51.373 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T15:47:51.633 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T15:47:51.661 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T15:47:51.668 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T15:47:51.697 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T15:47:51.699 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T15:47:51.726 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T15:47:52.130 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package pkg-config. 2024-04-25T15:47:52.140 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package pkg-config. 2024-04-25T15:47:52.158 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T15:47:52.168 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T15:47:52.196 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T15:47:52.206 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T15:47:52.457 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T15:47:52.459 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T15:47:52.476 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T15:47:52.485 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T15:47:52.509 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T15:47:52.523 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T15:47:52.946 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T15:47:52.975 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T15:47:52.994 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T15:47:53.012 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T15:47:53.021 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T15:47:53.060 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T15:47:53.290 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T15:47:53.315 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T15:47:53.356 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T15:47:53.363 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T15:47:53.390 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T15:47:53.429 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T15:47:53.634 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T15:47:53.663 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T15:47:53.693 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T15:47:53.749 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T15:47:53.776 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T15:47:53.815 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T15:47:54.151 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T15:47:54.179 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T15:47:54.218 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T15:47:54.265 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T15:47:54.293 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T15:47:54.331 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T15:47:54.571 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-py. 2024-04-25T15:47:54.598 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-25T15:47:54.645 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T15:47:54.651 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-py. 2024-04-25T15:47:54.679 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-25T15:47:54.717 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T15:47:54.949 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T15:47:54.976 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T15:47:54.987 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T15:47:55.004 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T15:47:55.015 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T15:47:55.037 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T15:47:55.461 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T15:47:55.482 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T15:47:55.489 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T15:47:55.511 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T15:47:55.527 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T15:47:55.549 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T15:47:55.814 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-toml. 2024-04-25T15:47:55.827 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-toml. 2024-04-25T15:47:55.841 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-25T15:47:55.857 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-25T15:47:55.880 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T15:47:55.885 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T15:47:56.154 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T15:47:56.183 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T15:47:56.183 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T15:47:56.211 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T15:47:56.220 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T15:47:56.249 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T15:47:56.541 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T15:47:56.570 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T15:47:56.578 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T15:47:56.606 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T15:47:56.607 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T15:47:56.644 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T15:47:56.852 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T15:47:56.881 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T15:47:56.910 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T15:47:56.922 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T15:47:56.950 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T15:47:56.988 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T15:47:57.694 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package radosgw. 2024-04-25T15:47:57.706 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package radosgw. 2024-04-25T15:47:57.720 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../122-radosgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:57.734 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../122-radosgw_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:57.750 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:57.772 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:58.645 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T15:47:58.661 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:58.703 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:58.775 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T15:47:58.792 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3291-g994e2f20-1jammy_amd64.deb ... 2024-04-25T15:47:58.833 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:47:59.016 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T15:47:59.138 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T15:47:59.249 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T15:47:59.489 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T15:47:59.519 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T15:47:59.614 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T15:47:59.875 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T15:47:59.902 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T15:48:00.202 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T15:48:00.266 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T15:48:00.629 INFO:teuthology.orchestra.run.smithi067.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T15:48:00.677 INFO:teuthology.orchestra.run.smithi176.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T15:48:00.691 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:00.736 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:01.240 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user cephadm....done 2024-04-25T15:48:01.269 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user cephadm....done 2024-04-25T15:48:01.355 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T15:48:01.408 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T15:48:01.632 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T15:48:01.684 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T15:48:01.891 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T15:48:01.910 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T15:48:02.034 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T15:48:02.036 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T15:48:02.296 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T15:48:02.302 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T15:48:02.547 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T15:48:02.578 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T15:48:02.673 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T15:48:02.729 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T15:48:02.966 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T15:48:03.055 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T15:48:03.327 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T15:48:03.424 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T15:48:03.561 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-25T15:48:03.684 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-25T15:48:03.821 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T15:48:03.952 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T15:48:04.097 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T15:48:04.236 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T15:48:04.365 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T15:48:04.496 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T15:48:04.641 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T15:48:04.764 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T15:48:04.925 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T15:48:05.190 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T15:48:05.228 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:05.458 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-argparse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:05.488 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T15:48:05.614 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T15:48:05.725 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T15:48:05.739 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-25T15:48:05.876 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T15:48:05.990 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T15:48:06.027 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-25T15:48:06.116 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T15:48:06.312 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T15:48:06.367 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T15:48:06.438 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T15:48:06.722 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T15:48:06.759 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T15:48:06.789 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T15:48:06.854 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T15:48:06.980 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T15:48:07.128 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T15:48:07.162 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T15:48:07.242 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T15:48:07.331 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T15:48:07.393 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T15:48:07.432 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T15:48:07.691 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T15:48:07.786 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T15:48:07.920 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T15:48:07.967 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T15:48:08.188 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T15:48:08.260 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T15:48:08.394 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T15:48:08.456 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T15:48:08.799 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T15:48:08.925 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T15:48:09.046 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T15:48:09.281 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T15:48:09.406 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T15:48:09.532 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T15:48:09.627 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T15:48:09.817 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T15:48:09.886 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T15:48:10.037 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T15:48:10.163 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T15:48:10.201 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T15:48:10.435 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T15:48:10.465 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T15:48:10.805 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T15:48:10.849 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T15:48:11.088 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-25T15:48:11.133 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T15:48:11.381 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T15:48:11.535 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T15:48:11.716 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T15:48:11.828 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-25T15:48:11.966 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T15:48:12.084 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:12.162 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T15:48:12.360 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T15:48:12.514 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T15:48:12.670 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T15:48:12.765 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T15:48:12.891 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:13.192 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T15:48:13.510 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T15:48:13.867 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T15:48:14.118 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:48:14.253 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T15:48:14.520 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T15:48:14.655 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T15:48:14.733 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T15:48:14.756 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T15:48:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:48:15.032 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T15:48:15.061 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T15:48:15.308 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T15:48:15.336 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T15:48:15.479 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T15:48:15.559 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:48:15.614 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T15:48:15.660 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T15:48:15.923 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T15:48:15.970 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T15:48:16.208 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T15:48:16.220 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T15:48:16.463 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T15:48:16.467 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:48:16.618 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T15:48:16.714 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T15:48:16.953 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T15:48:16.957 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T15:48:17.216 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T15:48:17.220 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T15:48:17.342 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T15:48:17.489 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T15:48:17.627 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T15:48:17.748 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T15:48:17.752 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T15:48:18.024 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T15:48:18.045 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T15:48:18.283 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T15:48:18.347 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T15:48:18.384 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T15:48:18.598 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T15:48:18.644 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T15:48:18.723 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T15:48:18.795 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T15:48:18.849 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T15:48:18.938 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T15:48:19.096 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T15:48:19.167 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T15:48:19.268 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T15:48:19.405 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T15:48:19.665 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T15:48:19.686 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T15:48:19.808 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T15:48:19.942 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T15:48:19.946 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T15:48:20.047 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T15:48:20.072 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T15:48:20.277 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T15:48:20.323 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:48:20.428 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T15:48:20.449 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T15:48:20.766 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T15:48:20.846 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T15:48:21.068 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T15:48:21.131 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T15:48:21.257 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T15:48:21.335 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T15:48:21.516 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T15:48:21.547 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T15:48:21.634 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T15:48:21.806 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T15:48:21.943 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T15:48:22.082 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T15:48:22.269 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T15:48:22.342 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T15:48:22.562 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T15:48:22.727 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T15:48:22.799 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T15:48:23.020 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T15:48:23.092 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T15:48:23.347 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T15:48:23.359 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T15:48:23.473 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:23.583 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:23.627 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T15:48:23.709 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T15:48:24.029 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T15:48:24.053 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:24.171 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:24.272 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T15:48:24.331 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T15:48:24.531 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:24.649 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:24.700 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T15:48:24.758 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librados2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:24.927 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:24.956 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T15:48:24.956 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T15:48:25.070 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T15:48:25.426 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:25.489 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:25.552 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:25.624 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libradosstriper1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:25.687 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:25.750 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T15:48:25.804 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:25.905 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T15:48:26.034 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librbd1 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-modules-core (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:26.311 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:26.556 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T15:48:26.556 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T15:48:27.046 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs-dev (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.100 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.172 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rados (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.209 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.298 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librgw2 (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.352 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.424 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephfs-top (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.478 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.575 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T15:48:27.604 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:27.730 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:28.264 INFO:teuthology.orchestra.run.smithi176.stdout:Adding group ceph....done 2024-04-25T15:48:28.649 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user ceph....done 2024-04-25T15:48:28.783 INFO:teuthology.orchestra.run.smithi176.stdout:Setting system user ceph properties....done 2024-04-25T15:48:28.795 INFO:teuthology.orchestra.run.smithi176.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T15:48:28.862 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rbd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:28.932 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T15:48:29.013 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-fuse (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:29.156 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:29.233 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T15:48:29.283 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cephfs (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:29.408 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephfs-shell (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:29.534 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-common (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:29.689 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:29.898 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:48:29.899 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:48:30.323 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:30.461 INFO:teuthology.orchestra.run.smithi067.stdout:Adding group ceph....done 2024-04-25T15:48:30.741 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:30.836 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user ceph....done 2024-04-25T15:48:30.981 INFO:teuthology.orchestra.run.smithi067.stdout:Setting system user ceph properties....done 2024-04-25T15:48:30.997 INFO:teuthology.orchestra.run.smithi067.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T15:48:31.141 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T15:48:31.197 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:48:31.197 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:48:31.512 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T15:48:31.634 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:31.980 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T15:48:31.994 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephfs-mirror (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:32.227 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:48:32.227 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:48:32.444 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:32.674 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T15:48:32.674 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T15:48:32.687 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-test (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:32.813 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up radosgw (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:33.121 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:33.278 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:48:33.279 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:48:33.321 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T15:48:33.321 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T15:48:33.740 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-base (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:33.789 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:34.075 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T15:48:34.075 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T15:48:34.108 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T15:48:34.549 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:34.600 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mds (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:34.674 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:34.825 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:34.835 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T15:48:34.835 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T15:48:35.037 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T15:48:35.037 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T15:48:35.285 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:35.493 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:35.509 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T15:48:35.510 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T15:48:35.653 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:35.804 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:35.955 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:35.978 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-osd (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:36.267 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T15:48:36.267 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T15:48:36.300 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T15:48:36.409 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T15:48:36.545 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T15:48:36.738 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:36.864 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mon (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:37.236 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T15:48:37.236 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T15:48:37.603 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T15:48:37.708 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-cephadm (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:37.851 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:37.960 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-dashboard (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:38.111 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-volume (19.0.0-3291-g994e2f20-1jammy) ... 2024-04-25T15:48:38.267 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T15:48:38.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-25T15:48:38.331 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-04-25T15:48:38.497 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T15:48:38.541 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-04-25T15:48:38.542 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-04-25T15:48:38.615 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T15:48:38.737 INFO:teuthology.orchestra.run.smithi176.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-25T15:48:38.738 INFO:teuthology.orchestra.run.smithi176.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-25T15:48:38.738 INFO:teuthology.orchestra.run.smithi176.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-25T15:48:38.738 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:48:38.738 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:48:38.738 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:48:38.738 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:48:38.739 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T15:48:38.739 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T15:48:38.739 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:48:38.741 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-04-25T15:48:38.741 INFO:teuthology.orchestra.run.smithi176.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-25T15:48:38.741 INFO:teuthology.orchestra.run.smithi176.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-25T15:48:38.741 INFO:teuthology.orchestra.run.smithi176.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-25T15:48:38.742 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-25T15:48:38.742 INFO:teuthology.orchestra.run.smithi176.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T15:48:38.742 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-04-25T15:48:38.742 INFO:teuthology.orchestra.run.smithi176.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-25T15:48:38.742 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-25T15:48:38.742 INFO:teuthology.orchestra.run.smithi176.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-25T15:48:38.779 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-04-25T15:48:38.780 INFO:teuthology.orchestra.run.smithi176.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-25T15:48:38.780 INFO:teuthology.orchestra.run.smithi176.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-25T15:48:38.781 INFO:teuthology.orchestra.run.smithi176.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-25T15:48:38.781 INFO:teuthology.orchestra.run.smithi176.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-25T15:48:38.781 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-25T15:48:38.781 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-25T15:48:38.781 INFO:teuthology.orchestra.run.smithi176.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T15:48:38.782 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages will be upgraded: 2024-04-25T15:48:38.783 INFO:teuthology.orchestra.run.smithi176.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-25T15:48:38.784 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T15:48:38.867 INFO:teuthology.orchestra.run.smithi176.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-25T15:48:38.867 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 55.7 MB of archives. 2024-04-25T15:48:38.867 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-25T15:48:38.867 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-25T15:48:39.049 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-25T15:48:39.058 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-25T15:48:39.102 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-25T15:48:39.103 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-25T15:48:39.104 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-25T15:48:39.115 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-25T15:48:39.116 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-25T15:48:39.118 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-25T15:48:39.119 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-25T15:48:39.120 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-25T15:48:39.121 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-25T15:48:39.141 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-25T15:48:39.144 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-25T15:48:39.144 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-25T15:48:39.145 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-25T15:48:39.569 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-25T15:48:39.571 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-25T15:48:39.571 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-25T15:48:39.575 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-25T15:48:39.575 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-25T15:48:39.582 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-25T15:48:39.584 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-25T15:48:39.594 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-25T15:48:39.594 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-25T15:48:39.623 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-25T15:48:39.623 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-25T15:48:39.635 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-25T15:48:39.655 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-25T15:48:39.658 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-25T15:48:39.658 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-25T15:48:39.659 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-25T15:48:39.660 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-25T15:48:39.671 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-25T15:48:39.673 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-25T15:48:39.939 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-25T15:48:39.939 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-25T15:48:39.939 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-25T15:48:39.939 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-25T15:48:39.939 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-25T15:48:39.939 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-25T15:48:39.951 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-25T15:48:39.951 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-25T15:48:40.002 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T15:48:40.355 INFO:teuthology.orchestra.run.smithi176.stdout:Preconfiguring packages ... 2024-04-25T15:48:40.553 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 55.7 MB in 1s (48.0 MB/s) 2024-04-25T15:48:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T15:48:40.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-25T15:48:40.783 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-25T15:48:40.882 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T15:48:40.888 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T15:48:40.994 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-25T15:48:40.995 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-25T15:48:41.033 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T15:48:41.194 INFO:teuthology.orchestra.run.smithi067.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-25T15:48:41.194 INFO:teuthology.orchestra.run.smithi067.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-25T15:48:41.194 INFO:teuthology.orchestra.run.smithi067.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-25T15:48:41.194 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T15:48:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T15:48:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T15:48:41.194 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T15:48:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T15:48:41.195 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T15:48:41.195 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T15:48:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-04-25T15:48:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-25T15:48:41.196 INFO:teuthology.orchestra.run.smithi067.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-25T15:48:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-25T15:48:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-25T15:48:41.197 INFO:teuthology.orchestra.run.smithi067.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T15:48:41.198 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-04-25T15:48:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-25T15:48:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-25T15:48:41.198 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-25T15:48:41.242 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-04-25T15:48:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-25T15:48:41.243 INFO:teuthology.orchestra.run.smithi067.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-25T15:48:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-25T15:48:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-25T15:48:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-25T15:48:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-25T15:48:41.244 INFO:teuthology.orchestra.run.smithi067.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-25T15:48:41.245 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-04-25T15:48:41.246 INFO:teuthology.orchestra.run.smithi067.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-25T15:48:41.306 INFO:teuthology.orchestra.run.smithi067.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-25T15:48:41.307 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 55.7 MB of archives. 2024-04-25T15:48:41.307 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-25T15:48:41.307 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-25T15:48:41.470 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-25T15:48:41.478 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-25T15:48:41.520 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-25T15:48:41.520 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-25T15:48:41.522 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-25T15:48:41.533 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-25T15:48:41.534 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-25T15:48:41.536 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-25T15:48:41.537 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-25T15:48:41.538 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-25T15:48:41.539 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-25T15:48:41.555 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-25T15:48:41.581 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-25T15:48:41.581 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-25T15:48:41.583 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-25T15:48:41.647 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-25T15:48:42.097 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-25T15:48:42.100 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-25T15:48:42.101 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-25T15:48:42.105 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-25T15:48:42.106 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-25T15:48:42.114 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-25T15:48:42.116 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-25T15:48:42.129 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-25T15:48:42.130 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-25T15:48:42.164 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-25T15:48:42.164 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-25T15:48:42.179 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-25T15:48:42.212 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-25T15:48:42.223 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-25T15:48:42.224 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-25T15:48:42.225 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-25T15:48:42.226 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-25T15:48:42.243 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-25T15:48:42.245 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-25T15:48:42.494 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-25T15:48:42.494 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-25T15:48:42.494 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-25T15:48:42.494 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-25T15:48:42.494 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-25T15:48:42.495 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-25T15:48:42.507 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-25T15:48:42.507 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-25T15:48:42.546 INFO:teuthology.orchestra.run.smithi176.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-25T15:48:42.805 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package flex. 2024-04-25T15:48:42.907 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T15:48:42.913 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-25T15:48:42.929 INFO:teuthology.orchestra.run.smithi067.stdout:Preconfiguring packages ... 2024-04-25T15:48:42.961 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-25T15:48:43.090 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 55.7 MB in 1s (44.4 MB/s) 2024-04-25T15:48:43.417 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T15:48:43.423 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T15:48:43.436 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T15:48:43.584 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T15:48:43.592 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-25T15:48:43.994 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T15:48:44.229 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-25T15:48:44.514 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T15:48:44.519 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T15:48:44.641 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T15:48:45.092 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T15:48:45.158 INFO:teuthology.orchestra.run.smithi067.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-25T15:48:45.247 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T15:48:45.431 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package flex. 2024-04-25T15:48:45.529 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-25T15:48:45.535 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-25T15:48:45.579 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-25T15:48:45.599 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T15:48:46.091 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package bison. 2024-04-25T15:48:46.197 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T15:48:46.202 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-25T15:48:46.252 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T15:48:46.256 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T15:48:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-25T15:48:46.660 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-25T15:48:46.689 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-25T15:48:46.727 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T15:48:46.857 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T15:48:47.072 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package dump. 2024-04-25T15:48:47.100 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-25T15:48:47.138 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking dump (0.4b47-3) ... 2024-04-25T15:48:47.401 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T15:48:47.407 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T15:48:47.500 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package indent. 2024-04-25T15:48:47.516 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:47.550 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T15:48:47.569 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T15:48:47.819 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-25T15:48:47.835 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-25T15:48:47.877 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T15:48:48.006 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-25T15:48:48.134 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-25T15:48:48.215 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-25T15:48:48.243 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-25T15:48:48.281 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T15:48:48.487 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T15:48:48.534 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-25T15:48:48.562 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-25T15:48:48.601 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T15:48:48.844 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package bison. 2024-04-25T15:48:48.912 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-25T15:48:48.940 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-25T15:48:48.950 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-25T15:48:48.956 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-25T15:48:48.979 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T15:48:49.013 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T15:48:49.231 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-25T15:48:49.247 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-25T15:48:49.290 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T15:48:49.429 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-25T15:48:49.458 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-25T15:48:49.496 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T15:48:49.542 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjson-perl. 2024-04-25T15:48:49.558 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-25T15:48:49.600 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-25T15:48:49.920 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-25T15:48:49.941 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package dump. 2024-04-25T15:48:49.948 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-25T15:48:49.969 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-25T15:48:49.987 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T15:48:50.007 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking dump (0.4b47-3) ... 2024-04-25T15:48:50.336 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package indent. 2024-04-25T15:48:50.364 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:50.419 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T15:48:50.714 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-25T15:48:50.730 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-25T15:48:50.772 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T15:48:51.110 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-25T15:48:51.138 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-25T15:48:51.176 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T15:48:51.479 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-25T15:48:51.507 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-25T15:48:51.570 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T15:48:51.740 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-25T15:48:51.768 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T15:48:51.806 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:48:51.907 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-25T15:48:51.935 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-25T15:48:51.974 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T15:48:52.042 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-25T15:48:52.070 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T15:48:52.100 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:48:52.235 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-25T15:48:52.251 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-25T15:48:52.294 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T15:48:52.405 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-25T15:48:52.433 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T15:48:52.472 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:48:52.529 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjson-perl. 2024-04-25T15:48:52.545 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-25T15:48:52.587 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-25T15:48:52.840 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-25T15:48:52.849 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-25T15:48:52.869 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-25T15:48:52.878 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-25T15:48:52.907 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T15:48:52.916 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T15:48:53.211 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-25T15:48:53.239 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T15:48:53.269 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T15:48:53.630 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-25T15:48:53.658 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:53.697 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:48:54.050 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-25T15:48:54.078 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:54.116 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:48:54.402 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-25T15:48:54.418 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:54.452 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:48:54.618 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-25T15:48:54.646 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T15:48:54.679 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-25T15:48:54.685 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:48:54.707 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T15:48:54.746 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T15:48:54.946 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-25T15:48:54.974 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-25T15:48:55.012 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:48:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-25T15:48:55.253 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-25T15:48:55.273 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-25T15:48:55.291 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T15:48:55.299 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T15:48:55.332 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:48:55.644 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package modemmanager. 2024-04-25T15:48:55.672 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T15:48:55.727 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:48:55.778 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-25T15:48:55.807 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-25T15:48:55.844 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T15:48:56.172 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-25T15:48:56.201 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T15:48:56.239 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T15:48:56.427 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package network-manager. 2024-04-25T15:48:56.455 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T15:48:56.510 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-25T15:48:56.516 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-25T15:48:56.545 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:56.575 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:48:56.928 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-25T15:48:56.956 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:56.995 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:48:57.225 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ppp. 2024-04-25T15:48:57.253 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-25T15:48:57.297 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-25T15:48:57.325 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:57.364 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:48:57.523 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T15:48:57.583 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-25T15:48:57.599 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-25T15:48:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T15:48:57.814 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package pptp-linux. 2024-04-25T15:48:57.840 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-25T15:48:57.871 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-25T15:48:58.116 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-25T15:48:58.128 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-25T15:48:58.143 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-25T15:48:58.157 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-25T15:48:58.182 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-25T15:48:58.195 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T15:48:58.498 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package modemmanager. 2024-04-25T15:48:58.508 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-25T15:48:58.526 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-25T15:48:58.535 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-25T15:48:58.556 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:48:58.576 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-client-common (238) ... 2024-04-25T15:48:58.880 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-25T15:48:58.908 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:48:58.947 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:48:59.196 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package network-manager. 2024-04-25T15:48:59.225 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-25T15:48:59.271 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-25T15:48:59.426 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-common. 2024-04-25T15:48:59.454 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-25T15:48:59.503 INFO:teuthology.orchestra.run.smithi176.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-25T15:48:59.533 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-common (238) ... 2024-04-25T15:48:59.983 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-14. 2024-04-25T15:49:00.011 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:49:00.037 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ppp. 2024-04-25T15:49:00.052 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:00.066 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-25T15:49:00.347 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T15:49:00.658 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package pptp-linux. 2024-04-25T15:49:00.687 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-25T15:49:00.725 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-25T15:49:01.629 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-25T15:49:01.646 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-25T15:49:01.696 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-25T15:49:02.003 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql. 2024-04-25T15:49:02.020 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-25T15:49:02.057 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-25T15:49:02.061 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql (14+238) ... 2024-04-25T15:49:02.085 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-25T15:49:02.123 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-client-common (238) ... 2024-04-25T15:49:02.347 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-client. 2024-04-25T15:49:02.363 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-25T15:49:02.414 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-client (14+238) ... 2024-04-25T15:49:02.436 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-25T15:49:02.464 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:49:02.502 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:02.658 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-25T15:49:02.686 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-25T15:49:02.724 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-25T15:49:03.027 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package tcl8.6. 2024-04-25T15:49:03.055 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-25T15:49:03.093 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T15:49:03.124 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-common. 2024-04-25T15:49:03.153 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-25T15:49:03.209 INFO:teuthology.orchestra.run.smithi067.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-25T15:49:03.239 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-common (238) ... 2024-04-25T15:49:03.337 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package tcl. 2024-04-25T15:49:03.354 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-25T15:49:03.392 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-25T15:49:03.673 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-14. 2024-04-25T15:49:03.702 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-25T15:49:03.732 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package wpasupplicant. 2024-04-25T15:49:03.741 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:03.760 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-25T15:49:03.798 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T15:49:04.186 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-25T15:49:04.213 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-25T15:49:04.252 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-25T15:49:04.630 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-25T15:49:04.658 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-25T15:49:04.705 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T15:49:04.984 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-client-common (238) ... 2024-04-25T15:49:05.133 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql. 2024-04-25T15:49:05.151 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-25T15:49:05.183 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql (14+238) ... 2024-04-25T15:49:05.194 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-25T15:49:05.320 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T15:49:05.421 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:05.427 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-client. 2024-04-25T15:49:05.456 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-25T15:49:05.494 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-client (14+238) ... 2024-04-25T15:49:05.522 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T15:49:05.671 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-25T15:49:05.677 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T15:49:05.688 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-25T15:49:05.721 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-25T15:49:05.807 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:06.040 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package tcl8.6. 2024-04-25T15:49:06.069 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-25T15:49:06.106 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T15:49:06.304 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-25T15:49:06.387 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T15:49:06.392 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package tcl. 2024-04-25T15:49:06.421 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-25T15:49:06.464 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-25T15:49:06.742 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T15:49:06.742 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T15:49:06.762 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package wpasupplicant. 2024-04-25T15:49:06.791 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-25T15:49:06.820 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T15:49:07.166 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-25T15:49:07.195 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-25T15:49:07.224 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T15:49:07.233 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-25T15:49:07.350 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T15:49:07.476 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T15:49:07.602 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-25T15:49:07.619 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-25T15:49:07.648 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-25T15:49:07.695 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T15:49:07.728 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T15:49:07.837 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T15:49:07.938 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T15:49:08.025 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-client-common (238) ... 2024-04-25T15:49:08.038 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T15:49:08.208 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-25T15:49:08.334 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-25T15:49:08.461 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:08.586 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-25T15:49:08.694 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T15:49:08.712 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-25T15:49:08.820 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up dump (0.4b47-3) ... 2024-04-25T15:49:08.838 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:08.913 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-25T15:49:08.979 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:49:09.214 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T15:49:09.342 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-25T15:49:09.365 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-25T15:49:09.436 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-25T15:49:09.491 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-25T15:49:09.617 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T15:49:09.706 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-25T15:49:09.777 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T15:49:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T15:49:09.823 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-25T15:49:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T15:49:10.078 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T15:49:10.188 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:49:10.289 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-25T15:49:10.322 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-25T15:49:10.431 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-25T15:49:10.465 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-client (14+238) ... 2024-04-25T15:49:10.532 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-25T15:49:10.591 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-25T15:49:10.633 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-25T15:49:10.759 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-25T15:49:10.894 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-25T15:49:11.020 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-25T15:49:11.054 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:11.054 INFO:teuthology.orchestra.run.smithi176.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-25T15:49:11.054 INFO:teuthology.orchestra.run.smithi176.stdout:which means they are currently configured by ifupdown: 2024-04-25T15:49:11.054 INFO:teuthology.orchestra.run.smithi176.stdout:- enp3s0f1 2024-04-25T15:49:11.054 INFO:teuthology.orchestra.run.smithi176.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-25T15:49:11.054 INFO:teuthology.orchestra.run.smithi176.stdout:remove their configuration from /etc/network/interfaces. 2024-04-25T15:49:11.054 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:11.146 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-25T15:49:11.159 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-25T15:49:11.458 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-25T15:49:11.757 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-25T15:49:12.293 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-25T15:49:12.410 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:49:12.427 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up dump (0.4b47-3) ... 2024-04-25T15:49:12.521 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-25T15:49:12.553 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T15:49:12.587 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:49:12.671 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-25T15:49:12.713 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-25T15:49:12.772 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-common (238) ... 2024-04-25T15:49:12.839 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-25T15:49:12.957 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-25T15:49:13.058 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-25T15:49:13.131 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-25T15:49:13.192 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-25T15:49:13.318 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-25T15:49:13.495 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-25T15:49:13.621 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:49:13.676 INFO:teuthology.orchestra.run.smithi176.stdout:Adding user postgres to group ssl-cert 2024-04-25T15:49:13.747 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-25T15:49:13.852 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:13.853 INFO:teuthology.orchestra.run.smithi176.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-25T15:49:13.915 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-client (14+238) ... 2024-04-25T15:49:14.024 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-25T15:49:14.039 INFO:teuthology.orchestra.run.smithi176.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-25T15:49:14.039 INFO:teuthology.orchestra.run.smithi176.stdout:Removing obsolete dictionary files: 2024-04-25T15:49:14.450 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:14.450 INFO:teuthology.orchestra.run.smithi067.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-25T15:49:14.450 INFO:teuthology.orchestra.run.smithi067.stdout:which means they are currently configured by ifupdown: 2024-04-25T15:49:14.450 INFO:teuthology.orchestra.run.smithi067.stdout:- enp3s0f1 2024-04-25T15:49:14.450 INFO:teuthology.orchestra.run.smithi067.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-25T15:49:14.450 INFO:teuthology.orchestra.run.smithi067.stdout:remove their configuration from /etc/network/interfaces. 2024-04-25T15:49:14.451 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:14.562 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-25T15:49:14.580 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-25T15:49:14.901 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-25T15:49:14.956 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:15.231 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-25T15:49:15.282 INFO:teuthology.orchestra.run.smithi176.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-25T15:49:15.295 INFO:teuthology.orchestra.run.smithi176.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-25T15:49:15.331 INFO:teuthology.orchestra.run.smithi176.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-25T15:49:15.331 INFO:teuthology.orchestra.run.smithi176.stdout:This user must also own the server process. 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout:The database cluster will be initialized with locale "en_US". 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout:The default text search configuration will be set to "english". 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout:Data page checksums are disabled. 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-25T15:49:15.332 INFO:teuthology.orchestra.run.smithi176.stdout:creating subdirectories ... ok 2024-04-25T15:49:15.333 INFO:teuthology.orchestra.run.smithi176.stdout:selecting dynamic shared memory implementation ... posix 2024-04-25T15:49:15.348 INFO:teuthology.orchestra.run.smithi176.stdout:selecting default max_connections ... 100 2024-04-25T15:49:15.366 INFO:teuthology.orchestra.run.smithi176.stdout:selecting default shared_buffers ... 128MB 2024-04-25T15:49:15.374 INFO:teuthology.orchestra.run.smithi176.stdout:selecting default time zone ... Etc/UTC 2024-04-25T15:49:15.375 INFO:teuthology.orchestra.run.smithi176.stdout:creating configuration files ... ok 2024-04-25T15:49:15.488 INFO:teuthology.orchestra.run.smithi176.stdout:running bootstrap script ... ok 2024-04-25T15:49:15.735 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-25T15:49:15.886 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-25T15:49:16.012 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-25T15:49:16.138 INFO:teuthology.orchestra.run.smithi176.stdout:performing post-bootstrap initialization ... ok 2024-04-25T15:49:16.155 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-common (238) ... 2024-04-25T15:49:16.743 INFO:teuthology.orchestra.run.smithi176.stdout:syncing data to disk ... ok 2024-04-25T15:49:17.027 INFO:teuthology.orchestra.run.smithi067.stdout:Adding user postgres to group ssl-cert 2024-04-25T15:49:17.116 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-25T15:49:17.261 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:17.261 INFO:teuthology.orchestra.run.smithi067.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-25T15:49:17.439 INFO:teuthology.orchestra.run.smithi067.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-25T15:49:17.439 INFO:teuthology.orchestra.run.smithi067.stdout:Removing obsolete dictionary files: 2024-04-25T15:49:17.981 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-25T15:49:18.373 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-25T15:49:18.721 INFO:teuthology.orchestra.run.smithi067.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-25T15:49:18.732 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-25T15:49:18.772 INFO:teuthology.orchestra.run.smithi067.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-25T15:49:18.772 INFO:teuthology.orchestra.run.smithi067.stdout:This user must also own the server process. 2024-04-25T15:49:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:The database cluster will be initialized with locale "en_US". 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:The default text search configuration will be set to "english". 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:Data page checksums are disabled. 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:creating subdirectories ... ok 2024-04-25T15:49:18.773 INFO:teuthology.orchestra.run.smithi067.stdout:selecting dynamic shared memory implementation ... posix 2024-04-25T15:49:18.788 INFO:teuthology.orchestra.run.smithi067.stdout:selecting default max_connections ... 100 2024-04-25T15:49:18.806 INFO:teuthology.orchestra.run.smithi067.stdout:selecting default shared_buffers ... 128MB 2024-04-25T15:49:18.817 INFO:teuthology.orchestra.run.smithi067.stdout:selecting default time zone ... Etc/UTC 2024-04-25T15:49:18.818 INFO:teuthology.orchestra.run.smithi067.stdout:creating configuration files ... ok 2024-04-25T15:49:18.930 INFO:teuthology.orchestra.run.smithi067.stdout:running bootstrap script ... ok 2024-04-25T15:49:19.613 INFO:teuthology.orchestra.run.smithi067.stdout:performing post-bootstrap initialization ... ok 2024-04-25T15:49:19.870 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:49:20.022 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-25T15:49:20.156 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql (14+238) ... 2024-04-25T15:49:20.167 INFO:teuthology.orchestra.run.smithi067.stdout:syncing data to disk ... ok 2024-04-25T15:49:20.273 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:49:20.383 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:49:20.554 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-25T15:49:20.620 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T15:49:20.621 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T15:49:21.137 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T15:49:21.227 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-25T15:49:21.529 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T15:49:21.690 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T15:49:23.042 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-25T15:49:23.134 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T15:49:23.304 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:49:23.430 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-25T15:49:23.531 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql (14+238) ... 2024-04-25T15:49:23.595 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T15:49:23.601 DEBUG:teuthology.parallel:result is None 2024-04-25T15:49:23.632 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-25T15:49:23.741 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-25T15:49:24.006 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T15:49:24.006 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T15:49:24.544 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T15:49:24.637 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-25T15:49:24.939 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T15:49:25.099 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T15:49:26.421 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-25T15:49:26.520 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T15:49:26.988 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T15:49:26.992 DEBUG:teuthology.parallel:result is None 2024-04-25T15:49:26.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:27.137 DEBUG:teuthology.orchestra.run.smithi067:> dpkg-query -W -f '${Version}' ceph 2024-04-25T15:49:27.156 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:49:27.157 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:49:27.157 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291-g994e2f20-1jammy is installed. 2024-04-25T15:49:27.158 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:27.304 DEBUG:teuthology.orchestra.run.smithi176:> dpkg-query -W -f '${Version}' ceph 2024-04-25T15:49:27.323 INFO:teuthology.orchestra.run.smithi176.stdout:19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:49:27.323 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291-g994e2f20-1jammy 2024-04-25T15:49:27.323 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291-g994e2f20-1jammy is installed. 2024-04-25T15:49:27.325 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T15:49:27.325 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:27.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:49:27.340 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:27.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:49:27.379 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T15:49:27.379 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:27.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:49:27.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:49:27.452 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:27.452 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:49:27.465 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:49:27.518 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T15:49:27.518 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:27.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:49:27.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:49:27.588 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:27.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:49:27.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:49:27.654 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T15:49:27.654 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:27.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:49:27.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:49:27.723 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:27.752 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:49:27.765 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:49:27.818 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T15:49:27.904 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T15:49:27.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-04-25T15:49:27.907 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-04-25T15:49:27.920 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T15:49:27.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T15:49:27.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T15:49:27.974 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T15:49:27.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T15:49:28.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T15:49:28.038 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T15:49:28.039 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T15:49:28.039 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T15:49:28.039 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:28.039 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T15:49:28.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T15:49:28.152 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:28.152 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T15:49:28.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T15:49:28.241 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T15:49:28.241 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T15:49:28.242 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228', 'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7673228', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1880, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'sleep_before_teardown': 0, 'subset': '178/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240425.015853-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBGer4xfgwQKKHClW7fd5vckzVLt/MYQCove0MxZiNBSgA3lcVPlQRSIBMQvaez3OVOyYcoT2zPvqKwHethtjNQ=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGLfJ3uYds5uQosqsnxVpzAvV38Lh7qKVncNctNv4YiS/v6c8biTMkzRoMMR25sAf/G5Av40wBnQnCKube88vjc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'thrashosds': None}, {'workunit': {'clients': {'all': ['suites/iozone.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_10:00:23', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T15:49:28.243 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T15:49:28.245 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T15:49:28.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T15:49:28.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T15:49:28.339 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:28.339 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:49:28.352 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:49:28.352 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:43:22.420111129 +0000 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:43:22.152113523 +0000 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:43:22.152113523 +0000 2024-04-25T15:49:28.401 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-25T15:49:28.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:49:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:49:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:49:28.455 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000365206 s, 1.4 MB/s 2024-04-25T15:49:28.457 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:49:28.505 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:43:22.952106377 +0000 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:43:22.684108771 +0000 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:43:22.684108771 +0000 2024-04-25T15:49:28.552 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-25T15:49:28.552 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:49:28.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:49:28.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:49:28.606 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000436042 s, 1.2 MB/s 2024-04-25T15:49:28.608 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:49:28.657 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-04-25T15:49:28.704 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:49:28.705 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-04-25T15:49:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:43:23.464101803 +0000 2024-04-25T15:49:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:43:23.196104198 +0000 2024-04-25T15:49:28.705 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:43:23.196104198 +0000 2024-04-25T15:49:28.705 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-25T15:49:28.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:49:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:49:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:49:28.759 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00038922 s, 1.3 MB/s 2024-04-25T15:49:28.760 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:49:28.809 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-04-25T15:49:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:49:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:28.856 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 936 Links: 1 2024-04-25T15:49:28.856 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:28.856 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-25 15:43:23.948097480 +0000 2024-04-25T15:49:28.856 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:43:23.684099838 +0000 2024-04-25T15:49:28.857 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:43:23.684099838 +0000 2024-04-25T15:49:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-25T15:49:28.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:49:28.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-25T15:49:28.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-25T15:49:28.911 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000373584 s, 1.4 MB/s 2024-04-25T15:49:28.912 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:49:28.961 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:28.962 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:28.962 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:49:28.967 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:49:28.968 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:43:19.268919464 +0000 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:43:19.000921857 +0000 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:43:19.000921857 +0000 2024-04-25T15:49:29.016 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-04-25T15:49:29.016 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:49:29.070 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:49:29.070 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:49:29.071 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000422845 s, 1.2 MB/s 2024-04-25T15:49:29.072 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:49:29.120 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:43:19.484917534 +0000 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:43:19.480917570 +0000 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:43:19.480917570 +0000 2024-04-25T15:49:29.168 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-04-25T15:49:29.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:49:29.221 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:49:29.221 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:49:29.221 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000310626 s, 1.6 MB/s 2024-04-25T15:49:29.223 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:49:29.272 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:43:19.984913069 +0000 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:43:19.980913104 +0000 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:43:19.980913104 +0000 2024-04-25T15:49:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-04-25T15:49:29.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:49:29.373 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:49:29.374 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:49:29.374 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000421227 s, 1.2 MB/s 2024-04-25T15:49:29.375 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:49:29.425 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-04-25T15:49:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:49:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:49:29.472 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-04-25T15:49:29.472 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:49:29.472 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-25 15:43:20.676906888 +0000 2024-04-25T15:49:29.472 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-25 15:43:20.440908996 +0000 2024-04-25T15:49:29.472 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-25 15:43:20.440908996 +0000 2024-04-25T15:49:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-04-25T15:49:29.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:49:29.526 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-25T15:49:29.526 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-25T15:49:29.526 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000383944 s, 1.3 MB/s 2024-04-25T15:49:29.527 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:49:29.576 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:29.576 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T15:49:29.577 INFO:tasks.ceph:Generating config... 2024-04-25T15:49:29.578 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T15:49:29.578 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-04-25T15:49:29.578 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T15:49:29.578 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T15:49:29.578 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T15:49:29.578 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T15:49:29.578 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T15:49:29.578 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T15:49:29.578 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T15:49:29.578 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-04-25T15:49:29.578 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T15:49:29.579 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T15:49:29.580 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T15:49:29.580 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-25T15:49:29.580 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-25T15:49:29.580 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-25T15:49:29.580 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T15:49:29.580 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T15:49:29.580 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-25T15:49:29.581 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T15:49:29.582 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T15:49:29.582 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T15:49:29.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T15:49:29.649 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T15:49:29.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T15:49:29.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:49:29.747 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.b', '172.21.15.176'), ('mon.c', '[v2:172.21.15.176:3301,v1:172.21.15.176:6790]')] 2024-04-25T15:49:29.747 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T15:49:29.748 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:29.748 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T15:49:29.792 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --add b 172.21.15.176 --addv c '[v2:172.21.15.176:3301,v1:172.21.15.176:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid fd5b4894-828e-4cb5-af55-b54a791f04c8 2024-04-25T15:49:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-04-25T15:49:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-04-25T15:49:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:fsid fd5b4894-828e-4cb5-af55-b54a791f04c8 2024-04-25T15:49:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-04-25T15:49:29.911989+0000 2024-04-25T15:49:29.909 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-25T15:49:29.911989+0000 2024-04-25T15:49:29.910 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-04-25T15:49:29.910 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-04-25T15:49:29.910 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-04-25T15:49:29.910 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-04-25T15:49:29.910 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] mon.c 2024-04-25T15:49:29.910 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T15:49:29.914 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T15:49:29.919 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fd5b4894-828e-4cb5-af55-b54a791f04c8... 2024-04-25T15:49:29.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T15:49:29.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.986 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-04-25T15:49:29.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.987 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-04-25T15:49:29.987 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-04-25T15:49:29.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.987 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-04-25T15:49:29.987 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-04-25T15:49:29.988 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-04-25T15:49:29.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-04-25T15:49:29.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-04-25T15:49:29.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-04-25T15:49:29.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-04-25T15:49:29.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-04-25T15:49:29.989 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T15:49:29.990 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T15:49:29.990 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-04-25T15:49:29.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.990 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T15:49:29.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.990 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-04-25T15:49:29.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.991 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-04-25T15:49:29.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.991 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-04-25T15:49:29.991 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-04-25T15:49:29.991 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-04-25T15:49:29.992 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]" 2024-04-25T15:49:29.992 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = fd5b4894-828e-4cb5-af55-b54a791f04c8 2024-04-25T15:49:29.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.992 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-04-25T15:49:29.992 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-04-25T15:49:29.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.992 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-04-25T15:49:29.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-04-25T15:49:29.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-04-25T15:49:29.993 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-04-25T15:49:29.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-04-25T15:49:29.993 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.993 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-04-25T15:49:29.994 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2024-04-25T15:49:29.995 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay max = 1 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay probability = 0.005 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay type = client mds 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 2500 2024-04-25T15:49:29.996 INFO:teuthology.orchestra.run.smithi067.stdout: osd blocked scrub grace period = 3600 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: osd max markdown count = 1000 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.997 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-04-25T15:49:29.998 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T15:49:29.999 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: mon down mkfs grace = 300 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-04-25T15:49:30.000 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: client_shutdown_timeout = 120 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.e] 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-25T15:49:30.001 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.f] 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal fragment size max = 10000 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal merge size = 5 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal split bits = 3 2024-04-25T15:49:30.002 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal split size = 100 2024-04-25T15:49:30.003 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-04-25T15:49:30.003 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-04-25T15:49:30.003 INFO:teuthology.orchestra.run.smithi067.stdout: mds oft prefetch dirfrags = True 2024-04-25T15:49:30.003 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-04-25T15:49:30.003 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-04-25T15:49:30.003 INFO:teuthology.orchestra.run.smithi067.stdout: mds_dir_prefetch = True 2024-04-25T15:49:30.004 INFO:teuthology.orchestra.run.smithi067.stdout: mds_sleep_rank_change = 5000000.0 2024-04-25T15:49:30.004 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-04-25T15:49:30.004 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay max = 1 2024-04-25T15:49:30.004 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay probability = 0.005 2024-04-25T15:49:30.004 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay type = client mds osd 2024-04-25T15:49:30.004 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 2500 2024-04-25T15:49:30.004 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-25T15:49:30.005 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-25T15:49:30.005 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-25T15:49:30.006 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-04-25T15:49:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-04-25T15:49:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T15:49:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-04-25T15:49:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-04-25T15:49:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "172.21.15.67,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]" 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = fd5b4894-828e-4cb5-af55-b54a791f04c8 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-04-25T15:49:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = bitmap 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2024-04-25T15:49:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1/20 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1/20 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 4/10 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2024-04-25T15:49:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay max = 1 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay probability = 0.005 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay type = client mds 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 2500 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: osd blocked scrub grace period = 3600 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: osd max markdown count = 1000 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2024-04-25T15:49:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.015 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-04-25T15:49:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:49:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-04-25T15:49:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-04-25T15:49:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-04-25T15:49:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: debug client = 20 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-04-25T15:49:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-04-25T15:49:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-04-25T15:49:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-04-25T15:49:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-04-25T15:49:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-04-25T15:49:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-04-25T15:49:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-04-25T15:49:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: mon down mkfs grace = 300 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: mon op complaint time = 120 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T15:49:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T15:49:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: client mount timeout = 600 2024-04-25T15:49:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: client_shutdown_timeout = 120 2024-04-25T15:49:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: debug client = 20 2024-04-25T15:49:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:49:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 900 2024-04-25T15:49:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 900 2024-04-25T15:49:30.020 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.e] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.d] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.f] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:[mds] 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: debug mds = 20 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: debug mds balancer = 20 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal fragment size max = 10000 2024-04-25T15:49:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal merge size = 5 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split bits = 3 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds bal split size = 100 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug frag = True 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds debug scatterstat = True 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds oft prefetch dirfrags = True 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds op complaint time = 180 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds verify scatter = True 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds_dir_prefetch = True 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mds_sleep_rank_change = 5000000.0 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay max = 1 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay probability = 0.005 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay type = client mds osd 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 2500 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: osd op complaint time = 180 2024-04-25T15:49:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: rados mon op timeout = 900 2024-04-25T15:49:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: rados osd op timeout = 900 2024-04-25T15:49:30.023 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T15:49:30.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T15:49:30.135 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T15:49:30.135 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:30.135 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T15:49:30.141 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:30.141 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T15:49:30.188 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:49:30.188 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:30.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T15:49:30.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:49:30.252 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:49:30.252 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:30.300 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:49:30.300 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:30.300 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T15:49:30.300 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T15:49:30.323 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:49:30.323 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:30.371 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T15:49:30.371 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T15:49:30.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T15:49:30.449 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T15:49:30.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T15:49:30.530 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T15:49:30.541 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T15:49:30.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T15:49:30.621 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T15:49:30.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T15:49:30.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T15:49:30.742 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T15:49:30.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T15:49:30.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-04-25T15:49:30.862 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-04-25T15:49:30.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-04-25T15:49:30.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T15:49:30.954 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T15:49:30.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T15:49:30.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T15:49:31.083 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T15:49:31.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T15:49:31.102 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-04-25T15:49:31.209 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-04-25T15:49:31.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-04-25T15:49:31.226 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T15:49:31.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T15:49:31.294 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T15:49:31.309 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T15:49:31.309 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T15:49:31.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T15:49:31.321 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:31.322 INFO:tasks.ceph:role: osd.0 2024-04-25T15:49:31.322 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:49:31.322 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T15:49:31.382 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:31.383 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:31.873 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:49:31.875 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:49:31.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T15:49:32.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T15:49:32.034 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:32.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T15:49:32.092 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:32.092 INFO:tasks.ceph:role: osd.1 2024-04-25T15:49:32.092 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:49:32.092 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:32.603 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:49:32.605 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:49:32.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T15:49:32.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T15:49:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:32.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T15:49:32.763 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:32.763 INFO:tasks.ceph:role: osd.2 2024-04-25T15:49:32.763 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:49:32.763 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T15:49:32.822 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:32.822 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:32.822 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:32.823 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:32.823 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:32.823 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:32.823 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:32.823 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:32.823 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:32.823 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:33.280 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:49:33.282 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:49:33.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T15:49:33.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T15:49:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:33.407 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:33.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T15:49:33.464 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:33.464 INFO:tasks.ceph:role: osd.3 2024-04-25T15:49:33.464 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:49:33.464 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:33.528 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:33.529 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:34.020 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-25T15:49:34.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-25T15:49:34.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T15:49:34.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T15:49:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:34.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:34.328+0000 7f36ad1b57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T15:49:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:34.328+0000 7f36ad1b57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T15:49:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:34.328+0000 7f36ad1b57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:49:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:34.328+0000 7f36ad1b57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:49:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:34.328+0000 7f36ad1b57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T15:49:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:34.328+0000 7f36ad1b57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T15:49:37.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T15:49:37.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:37.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:37.456+0000 7fab42c647c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T15:49:37.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:37.456+0000 7fab42c647c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T15:49:37.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:37.456+0000 7fab42c647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:49:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:37.456+0000 7fab42c647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:49:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:37.456+0000 7fab42c647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T15:49:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:37.456+0000 7fab42c647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T15:49:40.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T15:49:40.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:40.576+0000 7f18c1e687c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T15:49:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:40.576+0000 7f18c1e687c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T15:49:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:40.576+0000 7f18c1e687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:49:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:40.576+0000 7f18c1e687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:49:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:40.576+0000 7f18c1e687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T15:49:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:40.576+0000 7f18c1e687c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T15:49:43.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T15:49:43.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:43.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:43.708+0000 7f1a9c6c27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T15:49:43.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:43.708+0000 7f1a9c6c27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T15:49:43.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:43.708+0000 7f1a9c6c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:49:43.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:43.708+0000 7f1a9c6c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:49:43.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:43.708+0000 7f1a9c6c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T15:49:43.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:49:43.708+0000 7f1a9c6c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T15:49:46.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T15:49:46.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T15:49:46.761 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:46.761 INFO:tasks.ceph:role: osd.4 2024-04-25T15:49:46.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:49:46.761 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T15:49:46.823 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:46.824 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:47.291 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:49:47.293 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:49:47.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T15:49:47.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T15:49:47.429 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:47.430 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T15:49:47.483 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:47.484 INFO:tasks.ceph:role: osd.5 2024-04-25T15:49:47.484 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:49:47.484 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:47.547 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:47.548 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:47.548 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:47.980 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:49:47.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:49:47.983 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T15:49:48.049 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T15:49:48.100 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:48.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T15:49:48.153 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:48.153 INFO:tasks.ceph:role: osd.6 2024-04-25T15:49:48.153 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:49:48.153 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T15:49:48.215 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:48.215 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:48.215 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:48.215 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:48.215 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:48.215 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:48.216 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:48.216 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:48.216 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:48.216 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:48.683 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:49:48.685 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:49:48.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T15:49:48.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T15:49:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:48.797 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:48.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T15:49:48.849 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T15:49:48.849 INFO:tasks.ceph:role: osd.7 2024-04-25T15:49:48.849 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:49:48.849 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T15:49:48.912 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T15:49:48.912 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T15:49:48.912 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T15:49:48.913 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T15:49:48.913 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T15:49:48.913 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-04-25T15:49:48.913 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T15:49:48.913 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T15:49:48.913 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T15:49:48.913 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T15:49:49.375 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-04-25T15:49:49.377 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-04-25T15:49:49.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T15:49:49.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T15:49:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:49:49.550 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T15:49:49.550 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:49.692 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:49.689+0000 7f3152b827c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T15:49:49.696 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:49.693+0000 7f3152b827c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T15:49:49.696 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:49.693+0000 7f3152b827c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:49:49.696 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:49.693+0000 7f3152b827c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:49:49.696 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:49.693+0000 7f3152b827c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T15:49:49.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:49.697+0000 7f3152b827c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T15:49:52.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T15:49:52.730 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:52.848 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:52.845+0000 7f1d16f027c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T15:49:52.849 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:52.845+0000 7f1d16f027c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T15:49:52.849 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:52.845+0000 7f1d16f027c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:49:52.849 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:52.845+0000 7f1d16f027c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:49:52.849 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:52.845+0000 7f1d16f027c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T15:49:52.850 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:52.845+0000 7f1d16f027c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T15:49:55.839 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T15:49:55.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:55.970 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:55.965+0000 7fba633b37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T15:49:55.970 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:55.965+0000 7fba633b37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T15:49:55.970 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:55.965+0000 7fba633b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:49:55.970 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:55.965+0000 7fba633b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:49:55.970 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:55.965+0000 7fba633b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T15:49:55.971 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:55.969+0000 7fba633b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T15:49:58.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:49:58.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:49:58.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T15:49:58.983 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:49:59.097 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:59.093+0000 7f38da9fe7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T15:49:59.097 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:59.093+0000 7f38da9fe7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T15:49:59.097 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:59.093+0000 7f38da9fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:49:59.098 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:59.093+0000 7f38da9fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:49:59.098 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:59.093+0000 7f38da9fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T15:49:59.098 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:49:59.093+0000 7f38da9fe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T15:50:02.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T15:50:02.109 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T15:50:02.109 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T15:50:02.124 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T15:50:02.180 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T15:50:02.239 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-04-25T15:50:02.296 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T15:50:02.351 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T15:50:02.408 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T15:50:02.464 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T15:50:02.519 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T15:50:02.533 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T15:50:02.586 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T15:50:02.643 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-04-25T15:50:02.698 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.698 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T15:50:02.751 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T15:50:02.807 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.807 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T15:50:02.863 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:02.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T15:50:02.918 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:02.918 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T15:50:02.924 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T15:50:02.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T15:50:02.969 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T15:50:02.978 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-04-25T15:50:02.978 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCKeypmt8r7GhAAsqfkLrPbqThyVrxmmDpN9A== 2024-04-25T15:50:02.978 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCKeypmNl80JRAACyWbWysCnKil/ROum+2dmg== 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCKeypmys5mLBAAzLqc4kKWN89/QWKdXJHLYQ== 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.e] 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCKeypmorGOMxAAuCu4UOK8mkygNeiwDzd3WA== 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCOeypmw5u4ExAAWci1MUlp1v2rPMrkNNlgow== 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCReypmNCVsGxAAd5TgUouhJN/t6v/Xd0NiCQ== 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCUeypm2BeBIhAA+WkbE8dA079QasKcHsGRHA== 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCXeypmALh8KhAAUrALWbsl1F5z3aI1y5QcAw== 2024-04-25T15:50:02.979 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCKeypmPT6sHxAAeNTGnNPPy7Q2uP/Rc2xT4Q== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCKeypmUhLvOBAAm1uKPKSXEplP6DEDHB+U6A== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCLeypmU0MLBRAARMXPxrtWKKaQKYJ1Uoa2sA== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.f] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCLeypmRcyJDBAAFKXDzwe7dZCfCvWXvr/UoQ== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCdeypmJMlPKRAALni1CE4s50svQrpatg3mrQ== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCgeypm+8ehMhAAERGJ3lW+MIqGeD6E9QIBoA== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCjeypmeGfdORAA69huMNAFw2dyJhACGRmMvg== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCneypmG2rYBRAA85PFtW0mA5Xo1luqpM5z4g== 2024-04-25T15:50:02.980 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-25T15:50:02.981 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCLeypm0ca9ERAAwvQkWHzhCWKnNqzY8sNvww== 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.y] 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCKeypmt8r7GhAAsqfkLrPbqThyVrxmmDpN9A== 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCKeypmNl80JRAACyWbWysCnKil/ROum+2dmg== 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCKeypmys5mLBAAzLqc4kKWN89/QWKdXJHLYQ== 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.e] 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCKeypmorGOMxAAuCu4UOK8mkygNeiwDzd3WA== 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-04-25T15:50:02.982 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCOeypmw5u4ExAAWci1MUlp1v2rPMrkNNlgow== 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCReypmNCVsGxAAd5TgUouhJN/t6v/Xd0NiCQ== 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCUeypm2BeBIhAA+WkbE8dA079QasKcHsGRHA== 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.3] 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCXeypmALh8KhAAUrALWbsl1F5z3aI1y5QcAw== 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCKeypmPT6sHxAAeNTGnNPPy7Q2uP/Rc2xT4Q== 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCKeypmUhLvOBAAm1uKPKSXEplP6DEDHB+U6A== 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.d] 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCLeypmU0MLBRAARMXPxrtWKKaQKYJ1Uoa2sA== 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.f] 2024-04-25T15:50:02.983 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCLeypmRcyJDBAAFKXDzwe7dZCfCvWXvr/UoQ== 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.4] 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCdeypmJMlPKRAALni1CE4s50svQrpatg3mrQ== 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.5] 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCgeypm+8ehMhAAERGJ3lW+MIqGeD6E9QIBoA== 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.6] 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCjeypmeGfdORAA69huMNAFw2dyJhACGRmMvg== 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.7] 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCneypmG2rYBRAA85PFtW0mA5Xo1luqpM5z4g== 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-04-25T15:50:02.984 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQCLeypm0ca9ERAAwvQkWHzhCWKnNqzY8sNvww== 2024-04-25T15:50:02.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:50:03.026 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:50:03.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.334 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.405 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.620 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:50:03.622 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T15:50:03.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.698 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.848 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T15:50:03.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.921 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:03.991 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:04.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:04.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:04.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:04.132 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T15:50:04.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:50:04.203 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T15:50:04.277 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T15:50:04.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T15:50:04.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:50:04.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T15:50:04.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T15:50:04.424 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:50:04.549 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T15:50:04.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T15:50:04.612 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T15:50:04.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T15:50:04.744 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:50:04.746 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T15:50:04.790 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T15:50:04.791 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T15:50:04.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T15:50:04.793 INFO:tasks.ceph.mon.a:Started 2024-04-25T15:50:04.793 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T15:50:04.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T15:50:04.837 INFO:tasks.ceph.mon.b:Started 2024-04-25T15:50:04.837 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T15:50:04.837 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T15:50:04.839 INFO:tasks.ceph.mon.c:Started 2024-04-25T15:50:04.839 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T15:50:04.839 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T15:50:04.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T15:50:04.842 INFO:tasks.ceph.mgr.y:Started 2024-04-25T15:50:04.842 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T15:50:04.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T15:50:04.845 INFO:tasks.ceph.mgr.x:Started 2024-04-25T15:50:04.845 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T15:50:04.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T15:50:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30dd31e640 1 Processor -- start 2024-04-25T15:50:04.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30dd31e640 1 -- start start 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30dd31e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30dd31e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30dd31e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30d81651d0 con 0x7f30d80c4e30 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30dd31e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f30d80c54e0 con 0x7f30d8071af0 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30dd31e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f30d80c5970 con 0x7f30d8169620 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30cffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30cffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35776/0 (socket says 172.21.15.67:35776) 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30cffff640 1 -- 172.21.15.67:0/806126041 learned_addr learned my addr 172.21.15.67:0/806126041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:04.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30cffff640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 msgr2=0x7f30d8071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30cffff640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 msgr2=0x7f30d8071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30cffff640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30cffff640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 msgr2=0x7f30d816ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 msgr2=0x7f30d816ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:50:04.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:04.984+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30cffff640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30cffff640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 msgr2=0x7f30d8071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30cffff640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 msgr2=0x7f30d8071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30cffff640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 msgr2=0x7f30d816ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 msgr2=0x7f30d816ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:50:05.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:50:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30cffff640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T15:50:05.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.188+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T15:50:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.208+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3497568953 0 0) 0x7f30d81651d0 con 0x7f30d80c4e30 2024-04-25T15:50:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30b8003530 con 0x7f30d80c4e30 2024-04-25T15:50:05.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4159949480 0 0) 0x7f30b8003530 con 0x7f30d80c4e30 2024-04-25T15:50:05.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 msgr2=0x7f30d816ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:50:05.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.176:3300/0 conn(0x7f30d8169620 0x7f30d816ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 msgr2=0x7f30d8071f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 --2- 172.21.15.67:0/806126041 >> v2:172.21.15.67:3300/0 conn(0x7f30d8071af0 0x7f30d8071f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d8164300 con 0x7f30d80c4e30 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f30c8001680 con 0x7f30d80c4e30 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 >> v1:172.21.15.176:6789/0 conn(0x7f30d80c4e30 legacy=0x7f30d8164ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30b80039c0 0x7f30b8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8004810 0x7f30b8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b8005280 0x7f30b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30d80c54e0 con 0x7f30b8005280 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f30d80c5970 con 0x7f30b80039c0 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f30b800c220 con 0x7f30b8004810 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d6ffd640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30b80039c0 0x7f30b8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b8005280 0x7f30b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30cffff640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8004810 0x7f30b8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8004810 msgr2=0x7f30b8004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:05.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8004810 0x7f30b8004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30b80039c0 msgr2=0x7f30b8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30b80039c0 0x7f30b8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d8164300 con 0x7f30b8005280 2024-04-25T15:50:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30cffff640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8004810 0x7f30b8004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.212+0000 7f30d6ffd640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30b80039c0 0x7f30b8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.220+0000 7f30d77fe640 1 -- 172.21.15.67:0/806126041 reap_dead start 2024-04-25T15:50:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.220+0000 7f30d77fe640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b8005280 0x7f30b800b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009a40 tx=0x7f30d001d010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614d01cb882f5360 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c000f040 con 0x7f30b8005280 2024-04-25T15:50:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d00ac640 con 0x7f30b8005280 2024-04-25T15:50:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30d67fc640 1 -- 172.21.15.67:0/806126041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0009640 con 0x7f30b8005280 2024-04-25T15:50:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30dd31e640 1 -- 172.21.15.67:0/806126041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b8005280 msgr2=0x7f30b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30dd31e640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b8005280 0x7f30b800b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009a40 tx=0x7f30d001d010 comp rx=0 tx=0).stop 2024-04-25T15:50:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30dd31e640 1 -- 172.21.15.67:0/806126041 shutdown_connections 2024-04-25T15:50:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30dd31e640 1 --2- 172.21.15.67:0/806126041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30b8005280 0x7f30b800b9d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30dd31e640 1 -- 172.21.15.67:0/806126041 >> 172.21.15.67:0/806126041 conn(0x7f30d80700b0 msgr2=0x7f30d80704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30dd31e640 1 -- 172.21.15.67:0/806126041 shutdown_connections 2024-04-25T15:50:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.224+0000 7f30dd31e640 1 -- 172.21.15.67:0/806126041 wait complete. 2024-04-25T15:50:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.228+0000 7f30dd31e640 1 Processor -- start 2024-04-25T15:50:05.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30dd31e640 1 -- start start 2024-04-25T15:50:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30dd31e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8005280 0x7f30d815f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30dd31e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30d8160f90 0x7f30d81540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30dd31e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d8154630 0x7f30d814eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30dd31e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30d80c5ef0 con 0x7f30d8154630 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30dd31e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f30d80c57d0 con 0x7f30d8160f90 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30dd31e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f30d81645a0 con 0x7f30b8005280 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30d6ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8005280 0x7f30d815f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30cffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30d8160f90 0x7f30d81540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30cffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30d8160f90 0x7f30d81540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:43358/0 (socket says 172.21.15.67:43358) 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30d6ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8005280 0x7f30d815f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:45736/0 (socket says 172.21.15.67:45736) 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30cffff640 1 -- 172.21.15.67:0/3067467626 learned_addr learned my addr 172.21.15.67:0/3067467626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.232+0000 7f30d77fe640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d8154630 0x7f30d814eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30d6ffd640 1 -- 172.21.15.67:0/3067467626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30d8160f90 msgr2=0x7f30d81540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30d6ffd640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30d8160f90 0x7f30d81540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30d6ffd640 1 -- 172.21.15.67:0/3067467626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d8154630 msgr2=0x7f30d814eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30d6ffd640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d8154630 0x7f30d814eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30d6ffd640 1 -- 172.21.15.67:0/3067467626 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d815f6c0 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30d6ffd640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8005280 0x7f30d815f180 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f30c8010850 tx=0x7f30c8010c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30cf7fe640 1 -- 172.21.15.67:0/3067467626 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8018020 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30cf7fe640 1 -- 172.21.15.67:0/3067467626 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c8005290 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30cf7fe640 1 -- 172.21.15.67:0/3067467626 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c800e740 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d8160140 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d8154a40 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30cf7fe640 1 -- 172.21.15.67:0/3067467626 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f30c801b020 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30cf7fe640 1 -- 172.21.15.67:0/3067467626 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f30c8003ba0 con 0x7f30b8005280 2024-04-25T15:50:05.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.236+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d806a480 con 0x7f30b8005280 2024-04-25T15:50:05.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.240+0000 7f30cf7fe640 1 -- 172.21.15.67:0/3067467626 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f30c8022030 con 0x7f30b8005280 2024-04-25T15:50:05.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.304+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f30d815eda0 con 0x7f30b8005280 2024-04-25T15:50:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30cf7fe640 1 -- 172.21.15.67:0/3067467626 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f30c8003660 con 0x7f30b8005280 2024-04-25T15:50:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8005280 msgr2=0x7f30d815f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8005280 0x7f30d815f180 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f30c8010850 tx=0x7f30c8010c50 comp rx=0 tx=0).stop 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 shutdown_connections 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30d8154630 0x7f30d814eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f30d8160f90 0x7f30d81540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 --2- 172.21.15.67:0/3067467626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30b8005280 0x7f30d815f180 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 >> 172.21.15.67:0/3067467626 conn(0x7f30d80700b0 msgr2=0x7f30d80cbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 shutdown_connections 2024-04-25T15:50:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:05.316+0000 7f30dd31e640 1 -- 172.21.15.67:0/3067467626 wait complete. 2024-04-25T15:50:05.337 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T15:50:05.337 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T15:50:05.337 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:05.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T15:50:05.348 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:05.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T15:50:05.402 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:05.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T15:50:05.415 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:05.413+0000 7f69edc6d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:50:05.455 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:50:05.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T15:50:05.511 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:05.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T15:50:05.520 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:05.520+0000 7fd161081180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:50:05.524 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:05.524 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T15:50:05.578 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:05.578 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T15:50:05.630 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-25T15:50:05.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T15:50:05.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 2ae04396-30a5-4c1d-86e5-63bc4f26f4c0 0 2024-04-25T15:50:05.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f2163e53640 1 Processor -- start 2024-04-25T15:50:05.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f2163e53640 1 -- start start 2024-04-25T15:50:05.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f2163e53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c0c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f2163e53640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f215c0c9680 0x7f215c163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f2163e53640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f215c0c5770 con 0x7f215c06db50 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f2163e53640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f215c0c8620 con 0x7f215c0c9680 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f2163e53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f215c0ca0d0 con 0x7f215c153980 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21623f0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f215c0c9680 0x7f215c163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21623f0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f215c0c9680 0x7f215c163e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40820/0 (socket says 172.21.15.176:40820) 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21623f0640 1 -- 172.21.15.176:0/569314734 learned_addr learned my addr 172.21.15.176:0/569314734 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21613ee640 1 --2- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c0c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21623f0640 1 -- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 msgr2=0x7f215c0c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21623f0640 1 --2- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c0c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21623f0640 1 -- 172.21.15.176:0/569314734 >> v1:172.21.15.176:6789/0 conn(0x7f215c06db50 legacy=0x7f215c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21623f0640 1 -- 172.21.15.176:0/569314734 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215c0ca3b0 con 0x7f215c0c9680 2024-04-25T15:50:05.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.865+0000 7f21613ee640 1 --2- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c0c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:05.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f21623f0640 1 --2- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c0c9680 0x7f215c163e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2154009e50 tx=0x7f215401ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62b8c3a59587ebd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:05.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2160bed640 1 -- 172.21.15.176:0/569314734 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21540a5810 con 0x7f215c0c9680 2024-04-25T15:50:05.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2160bed640 1 -- 172.21.15.176:0/569314734 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21540a59b0 con 0x7f215c0c9680 2024-04-25T15:50:05.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2160bed640 1 -- 172.21.15.176:0/569314734 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21540a5d80 con 0x7f215c0c9680 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 -- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c0c9680 msgr2=0x7f215c163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 --2- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c0c9680 0x7f215c163e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2154009e50 tx=0x7f215401ca30 comp rx=0 tx=0).stop 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 -- 172.21.15.176:0/569314734 shutdown_connections 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 --2- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c0c9680 0x7f215c163e60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 --2- 172.21.15.176:0/569314734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c0c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 -- 172.21.15.176:0/569314734 >> 172.21.15.176:0/569314734 conn(0x7f215c0700c0 msgr2=0x7f215c0704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 -- 172.21.15.176:0/569314734 shutdown_connections 2024-04-25T15:50:05.874 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.869+0000 7f2163e53640 1 -- 172.21.15.176:0/569314734 wait complete. 2024-04-25T15:50:05.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 Processor -- start 2024-04-25T15:50:05.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 -- start start 2024-04-25T15:50:05.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c06db50 0x7f215c1f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f215c0c9680 0x7f215c151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c162d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:05.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f215c0c8d30 con 0x7f215c0c9680 2024-04-25T15:50:05.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f215c0c8bb0 con 0x7f215c06db50 2024-04-25T15:50:05.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f215c166bc0 con 0x7f215c153980 2024-04-25T15:50:05.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2161bef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c06db50 0x7f215c1f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c162d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2161bef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c06db50 0x7f215c1f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40832/0 (socket says 172.21.15.176:40832) 2024-04-25T15:50:05.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c162d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52072/0 (socket says 172.21.15.176:52072) 2024-04-25T15:50:05.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2161bef640 1 -- 172.21.15.176:0/3228181170 learned_addr learned my addr 172.21.15.176:0/3228181170 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:05.878 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f69edc6d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21613ee640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f215c0c9680 0x7f215c151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 -- 172.21.15.176:0/3228181170 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c06db50 msgr2=0x7f215c1f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c06db50 0x7f215c1f5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 -- 172.21.15.176:0/3228181170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f215c0c9680 msgr2=0x7f215c151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f215c0c9680 0x7f215c151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 -- 172.21.15.176:0/3228181170 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215c14a710 con 0x7f215c153980 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21623f0640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c162d00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f215c0cbba0 tx=0x7f21540197d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21527fc640 1 -- 172.21.15.176:0/3228181170 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154009510 con 0x7f215c153980 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21527fc640 1 -- 172.21.15.176:0/3228181170 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21540be070 con 0x7f215c153980 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215c14a9f0 con 0x7f215c153980 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21527fc640 1 -- 172.21.15.176:0/3228181170 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21540b88d0 con 0x7f215c153980 2024-04-25T15:50:05.880 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f215c1f5db0 con 0x7f215c153980 2024-04-25T15:50:05.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21527fc640 1 -- 172.21.15.176:0/3228181170 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f21540b8a70 con 0x7f215c153980 2024-04-25T15:50:05.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.873+0000 7f21527fc640 1 -- 172.21.15.176:0/3228181170 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f21540c3aa0 con 0x7f215c153980 2024-04-25T15:50:05.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.877+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f215c06a480 con 0x7f215c153980 2024-04-25T15:50:05.881 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.877+0000 7f21527fc640 1 -- 172.21.15.176:0/3228181170 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2154016620 con 0x7f215c153980 2024-04-25T15:50:05.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.945+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", "id": 0} v 0) -- 0x7f215c0c7480 con 0x7f215c153980 2024-04-25T15:50:05.955 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f21527fc640 1 -- 172.21.15.176:0/3228181170 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f21540c3460 con 0x7f215c153980 2024-04-25T15:50:05.955 INFO:teuthology.orchestra.run.smithi176.stdout:0 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 msgr2=0x7f215c162d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c162d00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f215c0cbba0 tx=0x7f21540197d0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 shutdown_connections 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f215c153980 0x7f215c162d00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f215c0c9680 0x7f215c151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 --2- 172.21.15.176:0/3228181170 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f215c06db50 0x7f215c1f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 >> 172.21.15.176:0/3228181170 conn(0x7f215c0700c0 msgr2=0x7f215c0cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 shutdown_connections 2024-04-25T15:50:05.957 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:05.953+0000 7f2163e53640 1 -- 172.21.15.176:0/3228181170 wait complete. 2024-04-25T15:50:05.974 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new efacae28-481c-42e9-9d26-5a019cb078b1 1 2024-04-25T15:50:06.017 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:06.016+0000 7fd161081180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:50:06.072 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:06.069+0000 7f69edc6d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:50:06.114 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.109+0000 7f64d76b7640 1 Processor -- start 2024-04-25T15:50:06.114 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.109+0000 7f64d76b7640 1 -- start start 2024-04-25T15:50:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d76b7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f64d0071bc0 0x7f64d00cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d76b7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64d00cc100 0x7f64d015a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d76b7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d00ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d76b7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f64d015ebb0 con 0x7f64d00cc100 2024-04-25T15:50:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d76b7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f64d015e720 con 0x7f64d0071bc0 2024-04-25T15:50:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d76b7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64d015de10 con 0x7f64d015ac60 2024-04-25T15:50:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f64d0071bc0 0x7f64d00cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5c54640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d00ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5c54640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d00ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52088/0 (socket says 172.21.15.176:52088) 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f64d0071bc0 0x7f64d00cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40840/0 (socket says 172.21.15.176:40840) 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5c54640 1 -- 172.21.15.176:0/576835623 learned_addr learned my addr 172.21.15.176:0/576835623 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d4c52640 1 --2- 172.21.15.176:0/576835623 >> v2:172.21.15.67:3300/0 conn(0x7f64d00cc100 0x7f64d015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 -- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 msgr2=0x7f64d00ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 --2- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d00ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 -- 172.21.15.176:0/576835623 >> v2:172.21.15.67:3300/0 conn(0x7f64d00cc100 msgr2=0x7f64d015a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 --2- 172.21.15.176:0/576835623 >> v2:172.21.15.67:3300/0 conn(0x7f64d00cc100 0x7f64d015a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 -- 172.21.15.176:0/576835623 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d0072390 con 0x7f64d0071bc0 2024-04-25T15:50:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5c54640 1 --2- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d00ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:06.118 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64d5453640 1 --2- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d0071bc0 0x7f64d00cbbc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f64d0074880 tx=0x7f64cc01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b874f2baaa06a574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.118 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64c7fff640 1 -- 172.21.15.176:0/576835623 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0b0070 con 0x7f64d0071bc0 2024-04-25T15:50:06.118 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64c7fff640 1 -- 172.21.15.176:0/576835623 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64cc004230 con 0x7f64d0071bc0 2024-04-25T15:50:06.118 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.113+0000 7f64c7fff640 1 -- 172.21.15.176:0/576835623 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc003620 con 0x7f64d0071bc0 2024-04-25T15:50:06.119 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 -- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d0071bc0 msgr2=0x7f64d00cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.119 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 --2- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d0071bc0 0x7f64d00cbbc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f64d0074880 tx=0x7f64cc01c480 comp rx=0 tx=0).stop 2024-04-25T15:50:06.119 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 -- 172.21.15.176:0/576835623 shutdown_connections 2024-04-25T15:50:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 --2- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d00ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 --2- 172.21.15.176:0/576835623 >> v2:172.21.15.67:3300/0 conn(0x7f64d00cc100 0x7f64d015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 --2- 172.21.15.176:0/576835623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d0071bc0 0x7f64d00cbbc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 -- 172.21.15.176:0/576835623 >> 172.21.15.176:0/576835623 conn(0x7f64d00700b0 msgr2=0x7f64d00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 -- 172.21.15.176:0/576835623 shutdown_connections 2024-04-25T15:50:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 -- 172.21.15.176:0/576835623 wait complete. 2024-04-25T15:50:06.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 Processor -- start 2024-04-25T15:50:06.122 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.117+0000 7f64d76b7640 1 -- start start 2024-04-25T15:50:06.125 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0071bc0 0x7f64d01534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d00cc100 0x7f64d01f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d01517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64d0072150 con 0x7f64d0071bc0 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f64d0071fd0 con 0x7f64d00cc100 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64d00d02e0 con 0x7f64d015ac60 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d00cc100 0x7f64d01f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d5c54640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d01517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d00cc100 0x7f64d01f4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40844/0 (socket says 172.21.15.176:40844) 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 -- 172.21.15.176:0/508643936 learned_addr learned my addr 172.21.15.176:0/508643936 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d5453640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0071bc0 0x7f64d01534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 -- 172.21.15.176:0/508643936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 msgr2=0x7f64d01517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d01517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 -- 172.21.15.176:0/508643936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0071bc0 msgr2=0x7f64d01534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0071bc0 0x7f64d01534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 -- 172.21.15.176:0/508643936 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d01f5050 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d4c52640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d00cc100 0x7f64d01f4b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f64c000d860 tx=0x7f64c000dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64c5ffb640 1 -- 172.21.15.176:0/508643936 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c0018030 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64c5ffb640 1 -- 172.21.15.176:0/508643936 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c00096a0 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64c5ffb640 1 -- 172.21.15.176:0/508643936 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c0009890 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d014d630 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d015da80 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64c5ffb640 1 -- 172.21.15.176:0/508643936 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f64c001b070 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64c5ffb640 1 -- 172.21.15.176:0/508643936 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f64c00147c0 con 0x7f64d00cc100 2024-04-25T15:50:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.121+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d006a480 con 0x7f64d00cc100 2024-04-25T15:50:06.129 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.125+0000 7f64c5ffb640 1 -- 172.21.15.176:0/508643936 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f64c001dc60 con 0x7f64d00cc100 2024-04-25T15:50:06.196 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.193+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", "id": 1} v 0) -- 0x7f64d015aa40 con 0x7f64d00cc100 2024-04-25T15:50:06.203 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64c5ffb640 1 -- 172.21.15.176:0/508643936 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f64c0009ad0 con 0x7f64d00cc100 2024-04-25T15:50:06.203 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-04-25T15:50:06.204 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d00cc100 msgr2=0x7f64d01f4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.204 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d00cc100 0x7f64d01f4b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f64c000d860 tx=0x7f64c000dd20 comp rx=0 tx=0).stop 2024-04-25T15:50:06.204 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 shutdown_connections 2024-04-25T15:50:06.204 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64d015ac60 0x7f64d01517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.205 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64d00cc100 0x7f64d01f4b10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.205 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 --2- 172.21.15.176:0/508643936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64d0071bc0 0x7f64d01534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.205 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 >> 172.21.15.176:0/508643936 conn(0x7f64d00700b0 msgr2=0x7f64d00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.205 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 shutdown_connections 2024-04-25T15:50:06.205 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.201+0000 7f64d76b7640 1 -- 172.21.15.176:0/508643936 wait complete. 2024-04-25T15:50:06.215 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:06.216+0000 7fd161081180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:50:06.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 60e87127-223e-4fd7-aefb-f00326f9deca 2 2024-04-25T15:50:06.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe88299f640 1 Processor -- start 2024-04-25T15:50:06.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe88299f640 1 -- start start 2024-04-25T15:50:06.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe88299f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe87c0c9230 0x7fe87c154850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe88299f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87c0ce2c0 con 0x7fe87c0ccac0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe88299f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87c0c8be0 con 0x7fe87c0cc6b0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe88299f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe87c0cae10 con 0x7fe87c0c9230 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87b7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe87c0cc6b0 0x7fe87c06d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:34158/0 (socket says 172.21.15.176:34158) 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87b7fe640 1 -- 172.21.15.176:0/906572466 learned_addr learned my addr 172.21.15.176:0/906572466 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87bfff640 1 --2- 172.21.15.176:0/906572466 >> v2:172.21.15.67:3300/0 conn(0x7fe87c0c9230 0x7fe87c154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679827226 0 0) 0x7fe87c0c8be0 con 0x7fe87c0cc6b0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe864003530 con 0x7fe87c0cc6b0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 752357308 0 0) 0x7fe864003530 con 0x7fe87c0cc6b0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 >> v2:172.21.15.67:3300/0 conn(0x7fe87c0c9230 msgr2=0x7fe87c154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 --2- 172.21.15.176:0/906572466 >> v2:172.21.15.67:3300/0 conn(0x7fe87c0c9230 0x7fe87c154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 >> v1:172.21.15.67:6789/0 conn(0x7fe87c0ccac0 legacy=0x7fe87c164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c0ce580 con 0x7fe87c0cc6b0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fe870001680 con 0x7fe87c0cc6b0 2024-04-25T15:50:06.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 >> v1:172.21.15.176:6789/0 conn(0x7fe87c0cc6b0 legacy=0x7fe87c06d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8640039c0 0x7fe864003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe864004810 0x7fe864004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe864005280 0x7fe86400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c0cae10 con 0x7fe864004810 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe86400c200 con 0x7fe864005280 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe86400c400 con 0x7fe8640039c0 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe880f3c640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe864005280 0x7fe86400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87bfff640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8640039c0 0x7fe864003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87b7fe640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe864004810 0x7fe864004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe880f3c640 1 -- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8640039c0 msgr2=0x7fe864003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe880f3c640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8640039c0 0x7fe864003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe880f3c640 1 -- 172.21.15.176:0/906572466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe864004810 msgr2=0x7fe864004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe880f3c640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe864004810 0x7fe864004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe880f3c640 1 -- 172.21.15.176:0/906572466 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe864003530 con 0x7fe864005280 2024-04-25T15:50:06.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87b7fe640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe864004810 0x7fe864004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:06.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.361+0000 7fe87bfff640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8640039c0 0x7fe864003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:06.370 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe880f3c640 1 -- 172.21.15.176:0/906572466 reap_dead start 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe880f3c640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe864005280 0x7fe86400b9d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe874001e70 tx=0x7fe87401c780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34514665368502ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00b600 con 0x7fe864005280 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe874003ba0 con 0x7fe864005280 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe87affd640 1 -- 172.21.15.176:0/906572466 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874001a90 con 0x7fe864005280 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe88299f640 1 -- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe864005280 msgr2=0x7fe86400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe88299f640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe864005280 0x7fe86400b9d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe874001e70 tx=0x7fe87401c780 comp rx=0 tx=0).stop 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe88299f640 1 -- 172.21.15.176:0/906572466 shutdown_connections 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe88299f640 1 --2- 172.21.15.176:0/906572466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe864005280 0x7fe86400b9d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe88299f640 1 -- 172.21.15.176:0/906572466 >> 172.21.15.176:0/906572466 conn(0x7fe87c0705c0 msgr2=0x7fe87c0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe88299f640 1 -- 172.21.15.176:0/906572466 shutdown_connections 2024-04-25T15:50:06.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.365+0000 7fe88299f640 1 -- 172.21.15.176:0/906572466 wait complete. 2024-04-25T15:50:06.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 Processor -- start 2024-04-25T15:50:06.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 -- start start 2024-04-25T15:50:06.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe864005280 0x7fe87c1e0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe87c14ede0 0x7fe87c1e3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c14f1f0 0x7fe87c14df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c15a540 con 0x7fe87c14f1f0 2024-04-25T15:50:06.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c0caa80 con 0x7fe87c14ede0 2024-04-25T15:50:06.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe87c15a6c0 con 0x7fe864005280 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe864005280 0x7fe87c1e0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe864005280 0x7fe87c1e0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52112/0 (socket says 172.21.15.176:52112) 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 -- 172.21.15.176:0/3718247413 learned_addr learned my addr 172.21.15.176:0/3718247413 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87b7fe640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe87c14ede0 0x7fe87c1e3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe880f3c640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c14f1f0 0x7fe87c14df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 -- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe87c14ede0 msgr2=0x7fe87c1e3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe87c14ede0 0x7fe87c1e3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 -- 172.21.15.176:0/3718247413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c14f1f0 msgr2=0x7fe87c14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c14f1f0 0x7fe87c14df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 -- 172.21.15.176:0/3718247413 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c1e0b40 con 0x7fe864005280 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87b7fe640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe87c14ede0 0x7fe87c1e3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe87bfff640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe864005280 0x7fe87c1e0600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00b570 tx=0x7fe86c00ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe878ff9640 1 -- 172.21.15.176:0/3718247413 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c018070 con 0x7fe864005280 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe878ff9640 1 -- 172.21.15.176:0/3718247413 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c0106e0 con 0x7fe864005280 2024-04-25T15:50:06.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c1f2300 con 0x7fe864005280 2024-04-25T15:50:06.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe878ff9640 1 -- 172.21.15.176:0/3718247413 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c010a90 con 0x7fe864005280 2024-04-25T15:50:06.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c1f2b60 con 0x7fe864005280 2024-04-25T15:50:06.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe878ff9640 1 -- 172.21.15.176:0/3718247413 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe86c01b070 con 0x7fe864005280 2024-04-25T15:50:06.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe878ff9640 1 -- 172.21.15.176:0/3718247413 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fe86c01dc70 con 0x7fe864005280 2024-04-25T15:50:06.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.369+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c050750 con 0x7fe864005280 2024-04-25T15:50:06.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.373+0000 7fe878ff9640 1 -- 172.21.15.176:0/3718247413 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe86c010c30 con 0x7fe864005280 2024-04-25T15:50:06.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.441+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", "id": 2} v 0) -- 0x7fe87c0c87b0 con 0x7fe864005280 2024-04-25T15:50:06.452 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe878ff9640 1 -- 172.21.15.176:0/3718247413 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fe86c01d450 con 0x7fe864005280 2024-04-25T15:50:06.452 INFO:teuthology.orchestra.run.smithi176.stdout:2 2024-04-25T15:50:06.453 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe864005280 msgr2=0x7fe87c1e0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.453 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe864005280 0x7fe87c1e0600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00b570 tx=0x7fe86c00ba30 comp rx=0 tx=0).stop 2024-04-25T15:50:06.453 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 shutdown_connections 2024-04-25T15:50:06.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c14f1f0 0x7fe87c14df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe87c14ede0 0x7fe87c1e3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 --2- 172.21.15.176:0/3718247413 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe864005280 0x7fe87c1e0600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 >> 172.21.15.176:0/3718247413 conn(0x7fe87c0705c0 msgr2=0x7fe87c150ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 shutdown_connections 2024-04-25T15:50:06.454 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.449+0000 7fe88299f640 1 -- 172.21.15.176:0/3718247413 wait complete. 2024-04-25T15:50:06.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 85ccf60b-0609-40a0-aae5-1e6a37eebdf2 3 2024-04-25T15:50:06.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.613+0000 7fa470dc5640 1 Processor -- start 2024-04-25T15:50:06.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.613+0000 7fa470dc5640 1 -- start start 2024-04-25T15:50:06.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa470dc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c0736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa470dc5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa46c15aaf0 0x7fa46c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa470dc5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa46c0caa60 con 0x7fa46c1547e0 2024-04-25T15:50:06.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa470dc5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa46c068590 con 0x7fa46c15aaf0 2024-04-25T15:50:06.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa470dc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa46c15a5f0 con 0x7fa46c06d460 2024-04-25T15:50:06.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c0736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa469d9b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa46c1547e0 0x7fa46c06c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:34170/0 (socket says 172.21.15.176:34170) 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa469d9b640 1 -- 172.21.15.176:0/1937871472 learned_addr learned my addr 172.21.15.176:0/1937871472 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46ad9d640 1 --2- 172.21.15.176:0/1937871472 >> v2:172.21.15.67:3300/0 conn(0x7fa46c15aaf0 0x7fa46c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46959a640 1 -- 172.21.15.176:0/1937871472 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269776040 0 0) 0x7fa46c0caa60 con 0x7fa46c1547e0 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46959a640 1 -- 172.21.15.176:0/1937871472 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa454003530 con 0x7fa46c1547e0 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46a59c640 1 -- 172.21.15.176:0/1937871472 >> v2:172.21.15.67:3300/0 conn(0x7fa46c15aaf0 msgr2=0x7fa46c15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46a59c640 1 --2- 172.21.15.176:0/1937871472 >> v2:172.21.15.67:3300/0 conn(0x7fa46c15aaf0 0x7fa46c15af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46a59c640 1 -- 172.21.15.176:0/1937871472 >> v1:172.21.15.176:6789/0 conn(0x7fa46c1547e0 legacy=0x7fa46c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46a59c640 1 -- 172.21.15.176:0/1937871472 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c0ca100 con 0x7fa46c06d460 2024-04-25T15:50:06.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46a59c640 1 --2- 172.21.15.176:0/1937871472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c0736d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa464009d90 tx=0x7fa46401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c32f856bbd1b352 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46959a640 1 -- 172.21.15.176:0/1937871472 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4640b2070 con 0x7fa46c06d460 2024-04-25T15:50:06.622 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46959a640 1 -- 172.21.15.176:0/1937871472 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4640a5a00 con 0x7fa46c06d460 2024-04-25T15:50:06.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.617+0000 7fa46959a640 1 -- 172.21.15.176:0/1937871472 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4640a5dd0 con 0x7fa46c06d460 2024-04-25T15:50:06.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- 172.21.15.176:0/1937871472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 msgr2=0x7fa46c0736d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 --2- 172.21.15.176:0/1937871472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c0736d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa464009d90 tx=0x7fa46401c940 comp rx=0 tx=0).stop 2024-04-25T15:50:06.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- 172.21.15.176:0/1937871472 shutdown_connections 2024-04-25T15:50:06.624 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 --2- 172.21.15.176:0/1937871472 >> v2:172.21.15.67:3300/0 conn(0x7fa46c15aaf0 0x7fa46c15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.624 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 --2- 172.21.15.176:0/1937871472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c0736d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.624 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- 172.21.15.176:0/1937871472 >> 172.21.15.176:0/1937871472 conn(0x7fa46c0ce1c0 msgr2=0x7fa46c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.624 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- 172.21.15.176:0/1937871472 shutdown_connections 2024-04-25T15:50:06.624 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- 172.21.15.176:0/1937871472 wait complete. 2024-04-25T15:50:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 Processor -- start 2024-04-25T15:50:06.625 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- start start 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa46c1547e0 0x7fa46c0c3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15aaf0 0x7fa46c14f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c154bf0 con 0x7fa46c15aaf0 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c15b440 con 0x7fa46c1547e0 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa470dc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa46c154d70 con 0x7fa46c06d460 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa46a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.628 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa469d9b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa46c1547e0 0x7fa46c0c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa469d9b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa46c1547e0 0x7fa46c0c3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40876/0 (socket says 172.21.15.176:40876) 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa46a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c14fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52142/0 (socket says 172.21.15.176:52142) 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.621+0000 7fa469d9b640 1 -- 172.21.15.176:0/895651202 learned_addr learned my addr 172.21.15.176:0/895651202 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa46ad9d640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15aaf0 0x7fa46c14f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa469d9b640 1 -- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 msgr2=0x7fa46c14fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa469d9b640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c14fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa469d9b640 1 -- 172.21.15.176:0/895651202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15aaf0 msgr2=0x7fa46c14f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa469d9b640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15aaf0 0x7fa46c14f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa469d9b640 1 -- 172.21.15.176:0/895651202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c1501f0 con 0x7fa46c1547e0 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa46a59c640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa469d9b640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa46c1547e0 0x7fa46c0c3390 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa46000c370 tx=0x7fa46000c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa45affd640 1 -- 172.21.15.176:0/895651202 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460017070 con 0x7fa46c1547e0 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa45affd640 1 -- 172.21.15.176:0/895651202 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4600074e0 con 0x7fa46c1547e0 2024-04-25T15:50:06.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46c0c38d0 con 0x7fa46c1547e0 2024-04-25T15:50:06.630 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa45affd640 1 -- 172.21.15.176:0/895651202 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4600059d0 con 0x7fa46c1547e0 2024-04-25T15:50:06.630 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46c1f6100 con 0x7fa46c1547e0 2024-04-25T15:50:06.630 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa45affd640 1 -- 172.21.15.176:0/895651202 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa46001a070 con 0x7fa46c1547e0 2024-04-25T15:50:06.630 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa45affd640 1 -- 172.21.15.176:0/895651202 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fa460012890 con 0x7fa46c1547e0 2024-04-25T15:50:06.630 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.625+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46c06a480 con 0x7fa46c1547e0 2024-04-25T15:50:06.631 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.629+0000 7fa45affd640 1 -- 172.21.15.176:0/895651202 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa460021070 con 0x7fa46c1547e0 2024-04-25T15:50:06.647 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:06.641+0000 7f69edc6d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:50:06.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.697+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", "id": 3} v 0) -- 0x7fa46c159510 con 0x7fa46c1547e0 2024-04-25T15:50:06.701 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:06.697+0000 7f69edc6d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:50:06.706 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.701+0000 7fa45affd640 1 -- 172.21.15.176:0/895651202 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fa460012400 con 0x7fa46c1547e0 2024-04-25T15:50:06.706 INFO:teuthology.orchestra.run.smithi176.stdout:3 2024-04-25T15:50:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa46c1547e0 msgr2=0x7fa46c0c3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa46c1547e0 0x7fa46c0c3390 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa46000c370 tx=0x7fa46000c830 comp rx=0 tx=0).stop 2024-04-25T15:50:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 shutdown_connections 2024-04-25T15:50:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15aaf0 0x7fa46c14f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa46c1547e0 0x7fa46c0c3390 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.707 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 --2- 172.21.15.176:0/895651202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa46c06d460 0x7fa46c14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 >> 172.21.15.176:0/895651202 conn(0x7fa46c0ce1c0 msgr2=0x7fa46c0cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 shutdown_connections 2024-04-25T15:50:06.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.705+0000 7fa470dc5640 1 -- 172.21.15.176:0/895651202 wait complete. 2024-04-25T15:50:06.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new ce4919c0-d2fb-4210-b281-e843e84c94c1 4 2024-04-25T15:50:06.757 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:06.753+0000 7f69edc6d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:50:06.793 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:06.792+0000 7fd161081180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:50:06.838 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:06.833+0000 7f69edc6d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:50:06.849 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:06.848+0000 7fd161081180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:50:06.865 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.861+0000 7feab9ec1640 1 Processor -- start 2024-04-25T15:50:06.866 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.861+0000 7feab9ec1640 1 -- start start 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab9ec1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7feab40cbea0 0x7feab4071af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab9ec1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab4167230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab9ec1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab40c57b0 con 0x7feab40caff0 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab9ec1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7feab40c5a80 con 0x7feab40cbea0 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab9ec1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feab40c5d60 con 0x7feab4072030 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feaabfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7feab40cbea0 0x7feab4071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab4167230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab4167230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52156/0 (socket says 172.21.15.176:52156) 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feaabfff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7feab40cbea0 0x7feab4071af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40890/0 (socket says 172.21.15.176:40890) 2024-04-25T15:50:06.868 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 -- 172.21.15.176:0/401197757 learned_addr learned my addr 172.21.15.176:0/401197757 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 -- 172.21.15.176:0/401197757 >> v2:172.21.15.176:3300/0 conn(0x7feab40cbea0 msgr2=0x7feab4071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 --2- 172.21.15.176:0/401197757 >> v2:172.21.15.176:3300/0 conn(0x7feab40cbea0 0x7feab4071af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 -- 172.21.15.176:0/401197757 >> v1:172.21.15.67:6789/0 conn(0x7feab40caff0 legacy=0x7feab41608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 -- 172.21.15.176:0/401197757 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab415cb10 con 0x7feab4072030 2024-04-25T15:50:06.869 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feaabfff640 1 --2- 172.21.15.176:0/401197757 >> v2:172.21.15.176:3300/0 conn(0x7feab40cbea0 0x7feab4071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab3fff640 1 --2- 172.21.15.176:0/401197757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab4167230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feaac009e50 tx=0x7feaac01c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ec1e4dfbc2c40f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab2ffd640 1 -- 172.21.15.176:0/401197757 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0a5670 con 0x7feab4072030 2024-04-25T15:50:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab2ffd640 1 -- 172.21.15.176:0/401197757 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaac0a5810 con 0x7feab4072030 2024-04-25T15:50:06.870 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.865+0000 7feab2ffd640 1 -- 172.21.15.176:0/401197757 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac0a5be0 con 0x7feab4072030 2024-04-25T15:50:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- 172.21.15.176:0/401197757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 msgr2=0x7feab4167230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 --2- 172.21.15.176:0/401197757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab4167230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feaac009e50 tx=0x7feaac01c9f0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- 172.21.15.176:0/401197757 shutdown_connections 2024-04-25T15:50:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 --2- 172.21.15.176:0/401197757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab4167230 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 --2- 172.21.15.176:0/401197757 >> v2:172.21.15.176:3300/0 conn(0x7feab40cbea0 0x7feab4071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- 172.21.15.176:0/401197757 >> 172.21.15.176:0/401197757 conn(0x7feab40700b0 msgr2=0x7feab40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- 172.21.15.176:0/401197757 shutdown_connections 2024-04-25T15:50:06.872 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- 172.21.15.176:0/401197757 wait complete. 2024-04-25T15:50:06.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 Processor -- start 2024-04-25T15:50:06.873 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- start start 2024-04-25T15:50:06.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab414ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab40caff0 0x7feab414b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.875 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feab40cbea0 0x7feab414bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feab40cc2b0 con 0x7feab40caff0 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feab40c6040 con 0x7feab40cbea0 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab9ec1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feab40caac0 con 0x7feab4072030 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feab40cbea0 0x7feab414bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab414ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feab40cbea0 0x7feab414bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40900/0 (socket says 172.21.15.176:40900) 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab414ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52162/0 (socket says 172.21.15.176:52162) 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab3fff640 1 -- 172.21.15.176:0/1950511540 learned_addr learned my addr 172.21.15.176:0/1950511540 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feaabfff640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab40caff0 0x7feab414b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab37fe640 1 -- 172.21.15.176:0/1950511540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feab40cbea0 msgr2=0x7feab414bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab37fe640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feab40cbea0 0x7feab414bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab37fe640 1 -- 172.21.15.176:0/1950511540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab40caff0 msgr2=0x7feab414b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab37fe640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab40caff0 0x7feab414b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.869+0000 7feab37fe640 1 -- 172.21.15.176:0/1950511540 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab414cfb0 con 0x7feab4072030 2024-04-25T15:50:06.876 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab37fe640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab414ca70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00b410 tx=0x7fea9c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab0ff9640 1 -- 172.21.15.176:0/1950511540 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c015030 con 0x7feab4072030 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab0ff9640 1 -- 172.21.15.176:0/1950511540 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea9c004020 con 0x7feab4072030 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab41f7280 con 0x7feab4072030 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab0ff9640 1 -- 172.21.15.176:0/1950511540 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c004410 con 0x7feab4072030 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feab40cde70 con 0x7feab4072030 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab0ff9640 1 -- 172.21.15.176:0/1950511540 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea9c018070 con 0x7feab4072030 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab0ff9640 1 -- 172.21.15.176:0/1950511540 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fea9c011b80 con 0x7feab4072030 2024-04-25T15:50:06.877 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab406f770 con 0x7feab4072030 2024-04-25T15:50:06.878 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.873+0000 7feab0ff9640 1 -- 172.21.15.176:0/1950511540 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fea9c011d80 con 0x7feab4072030 2024-04-25T15:50:06.892 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:06.889+0000 7f69edc6d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:50:06.904 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:06.904+0000 7fd161081180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:50:06.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.941+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", "id": 4} v 0) -- 0x7feab406a480 con 0x7feab4072030 2024-04-25T15:50:06.954 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.949+0000 7feab0ff9640 1 -- 172.21.15.176:0/1950511540 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fea9c011770 con 0x7feab4072030 2024-04-25T15:50:06.954 INFO:teuthology.orchestra.run.smithi176.stdout:4 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 msgr2=0x7feab414ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab414ca70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00b410 tx=0x7fea9c00b8d0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 shutdown_connections 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feab40cbea0 0x7feab414bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab40caff0 0x7feab414b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 --2- 172.21.15.176:0/1950511540 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feab4072030 0x7feab414ca70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 >> 172.21.15.176:0/1950511540 conn(0x7feab40700b0 msgr2=0x7feab415f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 shutdown_connections 2024-04-25T15:50:06.956 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:06.953+0000 7feab9ec1640 1 -- 172.21.15.176:0/1950511540 wait complete. 2024-04-25T15:50:06.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 47dcf3c9-5c23-4584-922c-b774345f2175 5 2024-04-25T15:50:06.973 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:06.969+0000 7f69edc6d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:50:06.985 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:06.984+0000 7fd161081180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:50:07.026 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:07.021+0000 7f69edc6d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:50:07.040 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:07.040+0000 7fd161081180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:50:07.122 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:07.120+0000 7fd161081180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:50:07.163 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.157+0000 7fdf72f4d640 1 Processor -- start 2024-04-25T15:50:07.163 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.157+0000 7fdf72f4d640 1 -- start start 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf72f4d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf72f4d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdf6c15a0b0 0x7fdf6c0746a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf72f4d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6c0c3bc0 con 0x7fdf6c153a00 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf72f4d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdf6c153e10 con 0x7fdf6c15a0b0 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf72f4d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6c068c40 con 0x7fdf6c0c4010 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf714ea640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdf6c15a0b0 0x7fdf6c0746a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.165 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf714ea640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdf6c15a0b0 0x7fdf6c0746a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40902/0 (socket says 172.21.15.176:40902) 2024-04-25T15:50:07.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c0725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52174/0 (socket says 172.21.15.176:52174) 2024-04-25T15:50:07.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf714ea640 1 -- 172.21.15.176:0/1191256551 learned_addr learned my addr 172.21.15.176:0/1191256551 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:07.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf714ea640 1 -- 172.21.15.176:0/1191256551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 msgr2=0x7fdf6c0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:07.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf714ea640 1 --2- 172.21.15.176:0/1191256551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c0725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf714ea640 1 -- 172.21.15.176:0/1191256551 >> v1:172.21.15.67:6789/0 conn(0x7fdf6c153a00 legacy=0x7fdf6c073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:07.166 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.161+0000 7fdf714ea640 1 -- 172.21.15.176:0/1191256551 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c0c53a0 con 0x7fdf6c15a0b0 2024-04-25T15:50:07.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf714ea640 1 --2- 172.21.15.176:0/1191256551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c0746a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009e50 tx=0x7fdf6401c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4531f3809046a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:07.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf6b7fe640 1 -- 172.21.15.176:0/1191256551 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf640a5670 con 0x7fdf6c15a0b0 2024-04-25T15:50:07.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf6b7fe640 1 -- 172.21.15.176:0/1191256551 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf640a5810 con 0x7fdf6c15a0b0 2024-04-25T15:50:07.167 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf6b7fe640 1 -- 172.21.15.176:0/1191256551 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf640a5be0 con 0x7fdf6c15a0b0 2024-04-25T15:50:07.168 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 -- 172.21.15.176:0/1191256551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c15a0b0 msgr2=0x7fdf6c0746a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.168 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 --2- 172.21.15.176:0/1191256551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c0746a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009e50 tx=0x7fdf6401c9f0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.168 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 -- 172.21.15.176:0/1191256551 shutdown_connections 2024-04-25T15:50:07.168 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 --2- 172.21.15.176:0/1191256551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c0746a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 --2- 172.21.15.176:0/1191256551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c0725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 -- 172.21.15.176:0/1191256551 >> 172.21.15.176:0/1191256551 conn(0x7fdf6c0ce1c0 msgr2=0x7fdf6c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:07.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 -- 172.21.15.176:0/1191256551 shutdown_connections 2024-04-25T15:50:07.169 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 -- 172.21.15.176:0/1191256551 wait complete. 2024-04-25T15:50:07.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 Processor -- start 2024-04-25T15:50:07.170 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.165+0000 7fdf72f4d640 1 -- start start 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c153a00 0x7fdf6c1635b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c0686c0 con 0x7fdf6c15a0b0 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c0c4420 con 0x7fdf6c153a00 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6c074be0 con 0x7fdf6c0c4010 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c153a00 0x7fdf6c1635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf70ce9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c153a00 0x7fdf6c1635b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40916/0 (socket says 172.21.15.176:40916) 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 -- 172.21.15.176:0/981023202 learned_addr learned my addr 172.21.15.176:0/981023202 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf714ea640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 -- 172.21.15.176:0/981023202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 msgr2=0x7fdf6c149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c149bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 -- 172.21.15.176:0/981023202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf6c15a0b0 msgr2=0x7fdf6c1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c1620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 -- 172.21.15.176:0/981023202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c14a100 con 0x7fdf6c153a00 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf714ea640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c1620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf6bfff640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c153a00 0x7fdf6c1635b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800e8a0 tx=0x7fdf5800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf697fa640 1 -- 172.21.15.176:0/981023202 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58017070 con 0x7fdf6c153a00 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf697fa640 1 -- 172.21.15.176:0/981023202 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5800c940 con 0x7fdf6c153a00 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c162d30 con 0x7fdf6c153a00 2024-04-25T15:50:07.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf697fa640 1 -- 172.21.15.176:0/981023202 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5800ccf0 con 0x7fdf6c153a00 2024-04-25T15:50:07.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c1490e0 con 0x7fdf6c153a00 2024-04-25T15:50:07.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf697fa640 1 -- 172.21.15.176:0/981023202 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf5801a070 con 0x7fdf6c153a00 2024-04-25T15:50:07.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf697fa640 1 -- 172.21.15.176:0/981023202 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fdf5801c9d0 con 0x7fdf6c153a00 2024-04-25T15:50:07.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.169+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c162aa0 con 0x7fdf6c153a00 2024-04-25T15:50:07.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.173+0000 7fdf697fa640 1 -- 172.21.15.176:0/981023202 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdf5801c450 con 0x7fdf6c153a00 2024-04-25T15:50:07.176 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:07.176+0000 7fd161081180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:50:07.184 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:07.181+0000 7f69edc6d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:50:07.246 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.241+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", "id": 5} v 0) -- 0x7fdf6c06a480 con 0x7fdf6c153a00 2024-04-25T15:50:07.247 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:07.245+0000 7f69edc6d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:50:07.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.249+0000 7fdf697fa640 1 -- 172.21.15.176:0/981023202 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fdf5801c630 con 0x7fdf6c153a00 2024-04-25T15:50:07.254 INFO:teuthology.orchestra.run.smithi176.stdout:5 2024-04-25T15:50:07.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c153a00 msgr2=0x7fdf6c1635b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c153a00 0x7fdf6c1635b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800e8a0 tx=0x7fdf5800ed60 comp rx=0 tx=0).stop 2024-04-25T15:50:07.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 shutdown_connections 2024-04-25T15:50:07.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf6c15a0b0 0x7fdf6c1620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf6c153a00 0x7fdf6c1635b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 --2- 172.21.15.176:0/981023202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf6c0c4010 0x7fdf6c149bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 >> 172.21.15.176:0/981023202 conn(0x7fdf6c0ce1c0 msgr2=0x7fdf6c073ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:07.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 shutdown_connections 2024-04-25T15:50:07.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.253+0000 7fdf72f4d640 1 -- 172.21.15.176:0/981023202 wait complete. 2024-04-25T15:50:07.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 312d2166-f5ad-4cf3-8339-16a49bf48b19 6 2024-04-25T15:50:07.334 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:07.332+0000 7fd161081180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:50:07.398 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:07.396+0000 7fd161081180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:50:07.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 Processor -- start 2024-04-25T15:50:07.455 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 -- start start 2024-04-25T15:50:07.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf64071bc0 0x7fbf640cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.457 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf640cc100 0x7fbf6415a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbf6415ac60 0x7fbf640ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbf6415ebb0 con 0x7fbf64071bc0 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbf6415e720 con 0x7fbf6415ac60 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6c6bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbf6415de10 con 0x7fbf640cc100 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf640cc100 0x7fbf6415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6ac5a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbf6415ac60 0x7fbf640ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf640cc100 0x7fbf6415a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:52194/0 (socket says 172.21.15.176:52194) 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 -- 172.21.15.176:0/2132164556 learned_addr learned my addr 172.21.15.176:0/2132164556 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6a459640 1 --2- 172.21.15.176:0/2132164556 >> v2:172.21.15.67:3300/0 conn(0x7fbf64071bc0 0x7fbf640cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 -- 172.21.15.176:0/2132164556 >> v2:172.21.15.176:3300/0 conn(0x7fbf6415ac60 msgr2=0x7fbf640ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 --2- 172.21.15.176:0/2132164556 >> v2:172.21.15.176:3300/0 conn(0x7fbf6415ac60 0x7fbf640ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 -- 172.21.15.176:0/2132164556 >> v2:172.21.15.67:3300/0 conn(0x7fbf64071bc0 msgr2=0x7fbf640cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:07.458 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 --2- 172.21.15.176:0/2132164556 >> v2:172.21.15.67:3300/0 conn(0x7fbf64071bc0 0x7fbf640cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf69c58640 1 -- 172.21.15.176:0/2132164556 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf64072390 con 0x7fbf640cc100 2024-04-25T15:50:07.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.453+0000 7fbf6a459640 1 --2- 172.21.15.176:0/2132164556 >> v2:172.21.15.67:3300/0 conn(0x7fbf64071bc0 0x7fbf640cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:07.459 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf69c58640 1 --2- 172.21.15.176:0/2132164556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf640cc100 0x7fbf6415a720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbf6415e8a0 tx=0x7fbf5401c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43a7a3425703505 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:07.460 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf69457640 1 -- 172.21.15.176:0/2132164556 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540ab5f0 con 0x7fbf640cc100 2024-04-25T15:50:07.460 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf69457640 1 -- 172.21.15.176:0/2132164556 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf540b0070 con 0x7fbf640cc100 2024-04-25T15:50:07.460 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf69457640 1 -- 172.21.15.176:0/2132164556 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540ab9b0 con 0x7fbf640cc100 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2132164556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf640cc100 msgr2=0x7fbf6415a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2132164556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf640cc100 0x7fbf6415a720 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbf6415e8a0 tx=0x7fbf5401c520 comp rx=0 tx=0).stop 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2132164556 shutdown_connections 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2132164556 >> v2:172.21.15.176:3300/0 conn(0x7fbf6415ac60 0x7fbf640ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2132164556 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf640cc100 0x7fbf6415a720 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2132164556 >> v2:172.21.15.67:3300/0 conn(0x7fbf64071bc0 0x7fbf640cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2132164556 >> 172.21.15.176:0/2132164556 conn(0x7fbf640700b0 msgr2=0x7fbf640704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2132164556 shutdown_connections 2024-04-25T15:50:07.461 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2132164556 wait complete. 2024-04-25T15:50:07.462 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 Processor -- start 2024-04-25T15:50:07.463 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.457+0000 7fbf6c6bd640 1 -- start start 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf64071bc0 0x7fbf64153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf640cc100 0x7fbf64150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6415ac60 0x7fbf64152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64072150 con 0x7fbf640cc100 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64071fd0 con 0x7fbf64071bc0 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbf640d02e0 con 0x7fbf6415ac60 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6a459640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf64071bc0 0x7fbf64153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6a459640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf64071bc0 0x7fbf64153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40924/0 (socket says 172.21.15.176:40924) 2024-04-25T15:50:07.466 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6a459640 1 -- 172.21.15.176:0/2190374723 learned_addr learned my addr 172.21.15.176:0/2190374723 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6ac5a640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6415ac60 0x7fbf64152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf69c58640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf640cc100 0x7fbf64150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6ac5a640 1 -- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf64071bc0 msgr2=0x7fbf64153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6ac5a640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf64071bc0 0x7fbf64153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6ac5a640 1 -- 172.21.15.176:0/2190374723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf640cc100 msgr2=0x7fbf64150880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6ac5a640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf640cc100 0x7fbf64150880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6ac5a640 1 -- 172.21.15.176:0/2190374723 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf64154410 con 0x7fbf6415ac60 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6a459640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf64071bc0 0x7fbf64153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf69c58640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf640cc100 0x7fbf64150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6ac5a640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6415ac60 0x7fbf64152f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbf640767d0 tx=0x7fbf5c00c0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf5affd640 1 -- 172.21.15.176:0/2190374723 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c018020 con 0x7fbf6415ac60 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf5affd640 1 -- 172.21.15.176:0/2190374723 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf5c00cc20 con 0x7fbf6415ac60 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf5affd640 1 -- 172.21.15.176:0/2190374723 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c014480 con 0x7fbf6415ac60 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf64157090 con 0x7fbf6415ac60 2024-04-25T15:50:07.467 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf64157650 con 0x7fbf6415ac60 2024-04-25T15:50:07.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf5affd640 1 -- 172.21.15.176:0/2190374723 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbf5c01b020 con 0x7fbf6415ac60 2024-04-25T15:50:07.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf5affd640 1 -- 172.21.15.176:0/2190374723 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fbf5c01db80 con 0x7fbf6415ac60 2024-04-25T15:50:07.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.461+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf6406a480 con 0x7fbf6415ac60 2024-04-25T15:50:07.468 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.465+0000 7fbf5affd640 1 -- 172.21.15.176:0/2190374723 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fbf5c022030 con 0x7fbf6415ac60 2024-04-25T15:50:07.533 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.529+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", "id": 6} v 0) -- 0x7fbf640c7730 con 0x7fbf6415ac60 2024-04-25T15:50:07.540 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf5affd640 1 -- 172.21.15.176:0/2190374723 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fbf5c010040 con 0x7fbf6415ac60 2024-04-25T15:50:07.540 INFO:teuthology.orchestra.run.smithi176.stdout:6 2024-04-25T15:50:07.541 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6415ac60 msgr2=0x7fbf64152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.541 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6415ac60 0x7fbf64152f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbf640767d0 tx=0x7fbf5c00c0c0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 shutdown_connections 2024-04-25T15:50:07.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6415ac60 0x7fbf64152f20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf640cc100 0x7fbf64150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 --2- 172.21.15.176:0/2190374723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf64071bc0 0x7fbf64153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 >> 172.21.15.176:0/2190374723 conn(0x7fbf640700b0 msgr2=0x7fbf640704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:07.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 shutdown_connections 2024-04-25T15:50:07.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.537+0000 7fbf6c6bd640 1 -- 172.21.15.176:0/2190374723 wait complete. 2024-04-25T15:50:07.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 922d3949-fd7b-4d1e-8b59-7311998bbc0b 7 2024-04-25T15:50:07.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.741+0000 7f3ee38cf640 1 Processor -- start 2024-04-25T15:50:07.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.741+0000 7f3ee38cf640 1 -- start start 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee38cf640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3edc0caf60 0x7f3edc15d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee38cf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3edc073bf0 0x7f3edc163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee38cf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3edc0c77a0 con 0x7f3edc15d790 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee38cf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3edc0cab60 con 0x7f3edc073bf0 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee38cf640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3edc159620 con 0x7f3edc0caf60 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee166b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3edc0caf60 0x7f3edc15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee166b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3edc0caf60 0x7f3edc15d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40932/0 (socket says 172.21.15.176:40932) 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee166b640 1 -- 172.21.15.176:0/51352465 learned_addr learned my addr 172.21.15.176:0/51352465 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee1e6c640 1 --2- 172.21.15.176:0/51352465 >> v2:172.21.15.67:3300/0 conn(0x7f3edc073bf0 0x7f3edc163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee166b640 1 -- 172.21.15.176:0/51352465 >> v2:172.21.15.67:3300/0 conn(0x7f3edc073bf0 msgr2=0x7f3edc163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:07.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee166b640 1 --2- 172.21.15.176:0/51352465 >> v2:172.21.15.67:3300/0 conn(0x7f3edc073bf0 0x7f3edc163ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee166b640 1 -- 172.21.15.176:0/51352465 >> v1:172.21.15.67:6789/0 conn(0x7f3edc15d790 legacy=0x7f3edc06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:07.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.745+0000 7f3ee166b640 1 -- 172.21.15.176:0/51352465 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc0c7a60 con 0x7f3edc0caf60 2024-04-25T15:50:07.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee166b640 1 --2- 172.21.15.176:0/51352465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc0caf60 0x7f3edc15d250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0720b0 tx=0x7f3ed801c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1ba7fc2af4c28ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:07.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ed3fff640 1 -- 172.21.15.176:0/51352465 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80b0070 con 0x7f3edc0caf60 2024-04-25T15:50:07.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ed3fff640 1 -- 172.21.15.176:0/51352465 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed80ab600 con 0x7f3edc0caf60 2024-04-25T15:50:07.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ed3fff640 1 -- 172.21.15.176:0/51352465 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80ab9d0 con 0x7f3edc0caf60 2024-04-25T15:50:07.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 -- 172.21.15.176:0/51352465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc0caf60 msgr2=0x7f3edc15d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 --2- 172.21.15.176:0/51352465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc0caf60 0x7f3edc15d250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0720b0 tx=0x7f3ed801c4e0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 -- 172.21.15.176:0/51352465 shutdown_connections 2024-04-25T15:50:07.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 --2- 172.21.15.176:0/51352465 >> v2:172.21.15.67:3300/0 conn(0x7f3edc073bf0 0x7f3edc163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 --2- 172.21.15.176:0/51352465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc0caf60 0x7f3edc15d250 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 -- 172.21.15.176:0/51352465 >> 172.21.15.176:0/51352465 conn(0x7f3edc070db0 msgr2=0x7f3edc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:07.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 -- 172.21.15.176:0/51352465 shutdown_connections 2024-04-25T15:50:07.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 -- 172.21.15.176:0/51352465 wait complete. 2024-04-25T15:50:07.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 Processor -- start 2024-04-25T15:50:07.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.749+0000 7f3ee38cf640 1 -- start start 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc073bf0 0x7f3edc160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3edc0caf60 0x7f3edc15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3edc15d790 0x7f3edc14f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc15a570 con 0x7f3edc0caf60 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3edc15a3f0 con 0x7f3edc073bf0 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3edc166c20 con 0x7f3edc15d790 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc073bf0 0x7f3edc160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc073bf0 0x7f3edc160370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40944/0 (socket says 172.21.15.176:40944) 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee1e6c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3edc15d790 0x7f3edc14f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 -- 172.21.15.176:0/179975357 learned_addr learned my addr 172.21.15.176:0/179975357 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-04-25T15:50:07.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee0e6a640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3edc0caf60 0x7f3edc15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 -- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3edc15d790 msgr2=0x7f3edc14f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3edc15d790 0x7f3edc14f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 -- 172.21.15.176:0/179975357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3edc0caf60 msgr2=0x7f3edc15bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3edc0caf60 0x7f3edc15bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 -- 172.21.15.176:0/179975357 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3edc0c8680 con 0x7f3edc073bf0 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee1e6c640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3edc15d790 0x7f3edc14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee0e6a640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3edc0caf60 0x7f3edc15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee166b640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc073bf0 0x7f3edc160370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80ae040 tx=0x7f3ed801ad20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ed1ffb640 1 -- 172.21.15.176:0/179975357 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80b0040 con 0x7f3edc073bf0 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ed1ffb640 1 -- 172.21.15.176:0/179975357 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ed80bc070 con 0x7f3edc073bf0 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3edc1608b0 con 0x7f3edc073bf0 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ed1ffb640 1 -- 172.21.15.176:0/179975357 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80b6880 con 0x7f3edc073bf0 2024-04-25T15:50:07.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3edc1f7c50 con 0x7f3edc073bf0 2024-04-25T15:50:07.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ed1ffb640 1 -- 172.21.15.176:0/179975357 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3ed80b6a20 con 0x7f3edc073bf0 2024-04-25T15:50:07.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ed1ffb640 1 -- 172.21.15.176:0/179975357 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f3ed80c03c0 con 0x7f3edc073bf0 2024-04-25T15:50:07.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.753+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3edc06a480 con 0x7f3edc073bf0 2024-04-25T15:50:07.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.757+0000 7f3ed1ffb640 1 -- 172.21.15.176:0/179975357 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3edc0c8680 con 0x7f3edc073bf0 2024-04-25T15:50:07.829 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.825+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", "id": 7} v 0) -- 0x7f3edc0c8a40 con 0x7f3edc073bf0 2024-04-25T15:50:07.841 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ed1ffb640 1 -- 172.21.15.176:0/179975357 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f3ed80160d0 con 0x7f3edc073bf0 2024-04-25T15:50:07.841 INFO:teuthology.orchestra.run.smithi176.stdout:7 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc073bf0 msgr2=0x7f3edc160370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc073bf0 0x7f3edc160370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80ae040 tx=0x7f3ed801ad20 comp rx=0 tx=0).stop 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 shutdown_connections 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3edc15d790 0x7f3edc14f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3edc0caf60 0x7f3edc15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 --2- 172.21.15.176:0/179975357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3edc073bf0 0x7f3edc160370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 >> 172.21.15.176:0/179975357 conn(0x7f3edc070db0 msgr2=0x7f3edc06d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 shutdown_connections 2024-04-25T15:50:07.843 INFO:teuthology.orchestra.run.smithi176.stderr:2024-04-25T15:50:07.837+0000 7f3ee38cf640 1 -- 172.21.15.176:0/179975357 wait complete. 2024-04-25T15:50:07.860 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T15:50:07.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T15:50:07.863 INFO:tasks.ceph.osd.0:Started 2024-04-25T15:50:07.863 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T15:50:07.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T15:50:07.866 INFO:tasks.ceph.osd.1:Started 2024-04-25T15:50:07.866 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T15:50:07.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T15:50:07.868 INFO:tasks.ceph.osd.2:Started 2024-04-25T15:50:07.868 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T15:50:07.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T15:50:07.870 INFO:tasks.ceph.osd.3:Started 2024-04-25T15:50:07.871 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T15:50:07.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T15:50:07.873 INFO:tasks.ceph.osd.4:Started 2024-04-25T15:50:07.873 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T15:50:07.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T15:50:07.875 INFO:tasks.ceph.osd.5:Started 2024-04-25T15:50:07.875 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T15:50:07.875 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T15:50:07.877 INFO:tasks.ceph.osd.6:Started 2024-04-25T15:50:07.877 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T15:50:07.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T15:50:07.879 INFO:tasks.ceph.osd.7:Started 2024-04-25T15:50:07.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:07.891 INFO:tasks.ceph.mgr.x.smithi176.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-25T15:50:07.891 INFO:tasks.ceph.mgr.x.smithi176.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-25T15:50:07.891 INFO:tasks.ceph.mgr.x.smithi176.stderr: from numpy import show_config as show_numpy_config 2024-04-25T15:50:07.955 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:07.953+0000 7f69edc6d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:50:08.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2ac22b640 1 Processor -- start 2024-04-25T15:50:08.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2ac22b640 1 -- start start 2024-04-25T15:50:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2ac22b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c8910 0x7fc2a4156c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2ac22b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2a415cc60 con 0x7fc2a40c6c10 2024-04-25T15:50:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2ac22b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2a415cf40 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2ac22b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a415d220 con 0x7fc2a40c8910 2024-04-25T15:50:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2a9fc7640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc2a40c6c10 0x7fc2a40d2000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52280/0 (socket says 172.21.15.67:52280) 2024-04-25T15:50:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2a9fc7640 1 -- 172.21.15.67:0/4174398629 learned_addr learned my addr 172.21.15.67:0/4174398629 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2aa7c8640 1 --2- 172.21.15.67:0/4174398629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c8910 0x7fc2a4156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901244051 0 0) 0x7fc2a415cc60 con 0x7fc2a40c6c10 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc28c003530 con 0x7fc2a40c6c10 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1259468773 0 0) 0x7fc2a415cf40 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2a415cc60 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 465098731 0 0) 0x7fc28c003530 con 0x7fc2a40c6c10 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2aa7c8640 1 -- 172.21.15.67:0/4174398629 >> v1:172.21.15.176:6789/0 conn(0x7fc2a40c9bc0 legacy=0x7fc2a40c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2aa7c8640 1 -- 172.21.15.67:0/4174398629 >> v1:172.21.15.67:6789/0 conn(0x7fc2a40c6c10 legacy=0x7fc2a40d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:08.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.032+0000 7fc2aa7c8640 1 -- 172.21.15.67:0/4174398629 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a415d500 con 0x7fc2a40c8910 2024-04-25T15:50:08.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2aa7c8640 1 --2- 172.21.15.67:0/4174398629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c8910 0x7fc2a4156c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc294009dc0 tx=0x7fc29401f0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99a47bf394fc5b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:08.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2940b5070 con 0x7fc2a40c8910 2024-04-25T15:50:08.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2940a8c30 con 0x7fc2a40c8910 2024-04-25T15:50:08.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2a8fc5640 1 -- 172.21.15.67:0/4174398629 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c001680 con 0x7fc2a40c8910 2024-04-25T15:50:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4174398629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c8910 msgr2=0x7fc2a4156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2ac22b640 1 --2- 172.21.15.67:0/4174398629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c8910 0x7fc2a4156c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc294009dc0 tx=0x7fc29401f0a0 comp rx=0 tx=0).stop 2024-04-25T15:50:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4174398629 shutdown_connections 2024-04-25T15:50:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2ac22b640 1 --2- 172.21.15.67:0/4174398629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c8910 0x7fc2a4156c60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4174398629 >> 172.21.15.67:0/4174398629 conn(0x7fc2a4068760 msgr2=0x7fc2a4068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4174398629 shutdown_connections 2024-04-25T15:50:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.036+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4174398629 wait complete. 2024-04-25T15:50:08.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 Processor -- start 2024-04-25T15:50:08.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 -- start start 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c6c10 0x7fc2a414ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2a40c8910 0x7fc2a414d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a40c9bc0 0x7fc2a4155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a415c8b0 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a40c7ce0 con 0x7fc2a40c8910 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a414dbe0 con 0x7fc2a40c6c10 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a40c9bc0 0x7fc2a4155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a40c9bc0 0x7fc2a4155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35860/0 (socket says 172.21.15.67:35860) 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 -- 172.21.15.67:0/4179786632 learned_addr learned my addr 172.21.15.67:0/4179786632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2a97c6640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2a40c8910 0x7fc2a414d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 -- 172.21.15.67:0/4179786632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c6c10 msgr2=0x7fc2a414ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c6c10 0x7fc2a414ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 -- 172.21.15.67:0/4179786632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2a40c8910 msgr2=0x7fc2a414d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2a40c8910 0x7fc2a414d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 -- 172.21.15.67:0/4179786632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a414d710 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2aa7c8640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a40c9bc0 0x7fc2a4155700 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc29401d750 tx=0x7fc29401be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2a27fc640 1 -- 172.21.15.67:0/4179786632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2940b5040 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2a27fc640 1 -- 172.21.15.67:0/4179786632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2940c2070 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a41f66a0 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2a27fc640 1 -- 172.21.15.67:0/4179786632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc294009450 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a41f6c60 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2a27fc640 1 -- 172.21.15.67:0/4179786632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc2940bb470 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2a27fc640 1 -- 172.21.15.67:0/4179786632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fc2940c7b30 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.040+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a406a480 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.044+0000 7fc2a27fc640 1 -- 172.21.15.67:0/4179786632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc294018c00 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.049 INFO:tasks.ceph.mgr.y.smithi067.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-25T15:50:08.049 INFO:tasks.ceph.mgr.y.smithi067.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-25T15:50:08.049 INFO:tasks.ceph.mgr.y.smithi067.stderr: from numpy import show_config as show_numpy_config 2024-04-25T15:50:08.096 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:08.093+0000 7f69edc6d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:50:08.103 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:08.100+0000 7fd161081180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:50:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.100+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2a40c8da0 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2a27fc640 1 -- 172.21.15.67:0/4179786632 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fc2940c7460 con 0x7fc2a40c9bc0 2024-04-25T15:50:08.104 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:08.104 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:07.834016+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a40c9bc0 msgr2=0x7fc2a4155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a40c9bc0 0x7fc2a4155700 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc29401d750 tx=0x7fc29401be40 comp rx=0 tx=0).stop 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 shutdown_connections 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2a40c9bc0 0x7fc2a4155700 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2a40c8910 0x7fc2a414d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 --2- 172.21.15.67:0/4179786632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2a40c6c10 0x7fc2a414ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 >> 172.21.15.67:0/4179786632 conn(0x7fc2a4068760 msgr2=0x7fc2a40ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 shutdown_connections 2024-04-25T15:50:08.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:08.104+0000 7fc2ac22b640 1 -- 172.21.15.67:0/4179786632 wait complete. 2024-04-25T15:50:08.124 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T15:50:08.125 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T15:50:08.244 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:08.241+0000 7f69edc6d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:50:08.254 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:08.252+0000 7fd161081180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:50:08.308 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:08.305+0000 7f69edc6d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:50:08.434 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:08.432+0000 7fd161081180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:50:08.441 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:08.437+0000 7f69edc6d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:50:08.503 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:08.500+0000 7fd161081180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:50:08.505 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:50:08.504+0000 7fee7bb7d7c0 -1 Falling back to public interface 2024-04-25T15:50:08.529 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:50:08.528+0000 7f3aff1247c0 -1 Falling back to public interface 2024-04-25T15:50:08.529 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:50:08.528+0000 7f7be38d37c0 -1 Falling back to public interface 2024-04-25T15:50:08.529 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:50:08.528+0000 7f4994b1a7c0 -1 Falling back to public interface 2024-04-25T15:50:08.532 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:50:08.529+0000 7fd30293d7c0 -1 Falling back to public interface 2024-04-25T15:50:08.540 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:50:08.537+0000 7f7b7f52f7c0 -1 Falling back to public interface 2024-04-25T15:50:08.540 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:50:08.537+0000 7f077dd437c0 -1 Falling back to public interface 2024-04-25T15:50:08.556 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:50:08.553+0000 7fb8fe2dd7c0 -1 Falling back to public interface 2024-04-25T15:50:08.648 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:08.648+0000 7fd161081180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:50:08.930 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:09.073 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce5043c640 1 Processor -- start 2024-04-25T15:50:09.073 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce5043c640 1 -- start start 2024-04-25T15:50:09.075 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce5043c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce480c5e80 0x7fce48156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:09.075 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce5043c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce4806d780 0x7fce4806db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:09.075 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce5043c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce480c7900 con 0x7fce48159a80 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce5043c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fce480ca950 con 0x7fce480c5e80 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce5043c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fce481570f0 con 0x7fce4806d780 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e1d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce480c5e80 0x7fce48156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e9d9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fce48159a80 0x7fce48159e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52296/0 (socket says 172.21.15.67:52296) 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e9d9640 1 -- 172.21.15.67:0/3116828233 learned_addr learned my addr 172.21.15.67:0/3116828233 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e1d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce480c5e80 0x7fce48156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35862/0 (socket says 172.21.15.67:35862) 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4d9d7640 1 --2- 172.21.15.67:0/3116828233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce4806d780 0x7fce4806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e1d8640 1 -- 172.21.15.67:0/3116828233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce4806d780 msgr2=0x7fce4806db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e1d8640 1 --2- 172.21.15.67:0/3116828233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce4806d780 0x7fce4806db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e1d8640 1 -- 172.21.15.67:0/3116828233 >> v1:172.21.15.67:6789/0 conn(0x7fce48159a80 legacy=0x7fce48159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4e1d8640 1 -- 172.21.15.67:0/3116828233 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce480c7be0 con 0x7fce480c5e80 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4d1d6640 1 -- 172.21.15.67:0/3116828233 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1505246127 0 0) 0x7fce480c7900 con 0x7fce48159a80 2024-04-25T15:50:09.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.072+0000 7fce4d9d7640 1 --2- 172.21.15.67:0/3116828233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce4806d780 0x7fce4806db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:09.077 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:09.073+0000 7f69edc6d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:50:09.077 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce4e1d8640 1 --2- 172.21.15.67:0/3116828233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce480c5e80 0x7fce48156bb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fce380097c0 tx=0x7fce3801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f4204de51bb651 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:09.077 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce4d1d6640 1 -- 172.21.15.67:0/3116828233 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce380b5070 con 0x7fce480c5e80 2024-04-25T15:50:09.077 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce4d1d6640 1 -- 172.21.15.67:0/3116828233 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce380a8990 con 0x7fce480c5e80 2024-04-25T15:50:09.078 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce4d1d6640 1 -- 172.21.15.67:0/3116828233 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce380a8d60 con 0x7fce480c5e80 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 -- 172.21.15.67:0/3116828233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce480c5e80 msgr2=0x7fce48156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 --2- 172.21.15.67:0/3116828233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce480c5e80 0x7fce48156bb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fce380097c0 tx=0x7fce3801ec90 comp rx=0 tx=0).stop 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 -- 172.21.15.67:0/3116828233 shutdown_connections 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 --2- 172.21.15.67:0/3116828233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce4806d780 0x7fce4806db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 --2- 172.21.15.67:0/3116828233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce480c5e80 0x7fce48156bb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 -- 172.21.15.67:0/3116828233 >> 172.21.15.67:0/3116828233 conn(0x7fce48070f20 msgr2=0x7fce480cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 -- 172.21.15.67:0/3116828233 shutdown_connections 2024-04-25T15:50:09.079 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.076+0000 7fce5043c640 1 -- 172.21.15.67:0/3116828233 wait complete. 2024-04-25T15:50:09.080 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 Processor -- start 2024-04-25T15:50:09.080 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 -- start start 2024-04-25T15:50:09.083 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce4806d780 0x7fce481f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:09.083 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce480c5e80 0x7fce48151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:09.083 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce48159a80 0x7fce481496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:09.083 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fce480ca5a0 con 0x7fce4806d780 2024-04-25T15:50:09.083 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fce480c81a0 con 0x7fce480c5e80 2024-04-25T15:50:09.083 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fce481e70e0 con 0x7fce48159a80 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce4806d780 0x7fce481f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce4806d780 0x7fce481f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35868/0 (socket says 172.21.15.67:35868) 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 -- 172.21.15.67:0/429754013 learned_addr learned my addr 172.21.15.67:0/429754013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e9d9640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce48159a80 0x7fce481496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 -- 172.21.15.67:0/429754013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce48159a80 msgr2=0x7fce481496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce48159a80 0x7fce481496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 -- 172.21.15.67:0/429754013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce480c5e80 msgr2=0x7fce48151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce480c5e80 0x7fce48151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 -- 172.21.15.67:0/429754013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce481e6c10 con 0x7fce4806d780 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce4e1d8640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce4806d780 0x7fce481f4d10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fce3801c8c0 tx=0x7fce3801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce3effd640 1 -- 172.21.15.67:0/429754013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce380bf070 con 0x7fce4806d780 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce3effd640 1 -- 172.21.15.67:0/429754013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce380b0820 con 0x7fce4806d780 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4814eb50 con 0x7fce4806d780 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce3effd640 1 -- 172.21.15.67:0/429754013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce380b5040 con 0x7fce4806d780 2024-04-25T15:50:09.084 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce481f4510 con 0x7fce4806d780 2024-04-25T15:50:09.085 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce3effd640 1 -- 172.21.15.67:0/429754013 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fce380c2070 con 0x7fce4806d780 2024-04-25T15:50:09.085 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.080+0000 7fce3effd640 1 -- 172.21.15.67:0/429754013 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fce380b9ac0 con 0x7fce4806d780 2024-04-25T15:50:09.085 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.084+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce48077030 con 0x7fce4806d780 2024-04-25T15:50:09.085 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.084+0000 7fce3effd640 1 -- 172.21.15.67:0/429754013 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fce380187f0 con 0x7fce4806d780 2024-04-25T15:50:09.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.140+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce4814e1f0 con 0x7fce4806d780 2024-04-25T15:50:09.144 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce3effd640 1 -- 172.21.15.67:0/429754013 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fce380b9cc0 con 0x7fce4806d780 2024-04-25T15:50:09.144 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-25T15:50:09.144 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:07.834016+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:09.145 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce4806d780 msgr2=0x7fce481f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:09.145 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce4806d780 0x7fce481f4d10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fce3801c8c0 tx=0x7fce3801ba30 comp rx=0 tx=0).stop 2024-04-25T15:50:09.145 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 shutdown_connections 2024-04-25T15:50:09.146 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fce48159a80 0x7fce481496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.146 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fce480c5e80 0x7fce48151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.146 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 --2- 172.21.15.67:0/429754013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce4806d780 0x7fce481f4d10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:09.146 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 >> 172.21.15.67:0/429754013 conn(0x7fce48070f20 msgr2=0x7fce480c97c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:09.146 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 shutdown_connections 2024-04-25T15:50:09.146 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:09.144+0000 7fce5043c640 1 -- 172.21.15.67:0/429754013 wait complete. 2024-04-25T15:50:09.152 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:09.149+0000 7f69edc6d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:50:09.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T15:50:09.302 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:09.300+0000 7fd161081180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:50:09.334 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:09.329+0000 7f69edc6d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:50:09.381 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:09.380+0000 7fd161081180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:50:09.407 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:09.405+0000 7f69edc6d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:50:09.571 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:09.568+0000 7fd161081180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:50:09.648 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:09.648+0000 7fd161081180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:50:09.699 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:09.697+0000 7f69edc6d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:50:09.761 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:09.757+0000 7f69edc6d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:50:09.959 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:09.960+0000 7fd161081180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:50:10.024 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-25T15:50:10.024+0000 7fd161081180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:50:10.813 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:10.809+0000 7f6994720640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T15:50:10.813 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-04-25T15:50:10.809+0000 7f6994720640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T15:50:12.317 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:50:12.315+0000 7f7be38d37c0 -1 osd.2 0 log_to_monitors true 2024-04-25T15:50:12.399 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:50:12.393+0000 7fd30293d7c0 -1 osd.7 0 log_to_monitors true 2024-04-25T15:50:12.400 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:50:12.399+0000 7f3aff1247c0 -1 osd.1 0 log_to_monitors true 2024-04-25T15:50:12.419 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:50:12.417+0000 7f7b7f52f7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T15:50:12.432 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:50:12.429+0000 7f077dd437c0 -1 osd.6 0 log_to_monitors true 2024-04-25T15:50:12.437 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:50:12.433+0000 7fb8fe2dd7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T15:50:12.446 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:50:12.443+0000 7fee7bb7d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T15:50:12.533 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:50:12.531+0000 7f4994b1a7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T15:50:15.969 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:16.010 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:50:16.007+0000 7f3afb0cb640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T15:50:16.025 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:50:16.023+0000 7f7bdf87a640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T15:50:16.031 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:50:16.021+0000 7f077a4fe640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T15:50:16.037 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:50:16.033+0000 7fb8fa284640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T15:50:16.050 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:50:16.047+0000 7f4990ac1640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T15:50:16.055 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:50:16.053+0000 7f0771aeb640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.057 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:50:16.053+0000 7fd2ff0f8640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T15:50:16.060 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:50:16.057+0000 7f7b7b4d6640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T15:50:16.060 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:50:16.059+0000 7f3af2ecc640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.069 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:50:16.067+0000 7f7bd767b640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.072 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:50:16.071+0000 7fee77b24640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T15:50:16.076 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:50:16.073+0000 7fd2f66e5640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.076 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:50:16.073+0000 7fb8f2085640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.076 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:50:16.075+0000 7f49888c2640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.099 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:50:16.097+0000 7f7b732d7640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.105 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:50:16.103+0000 7fee6f925640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:50:16.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.127+0000 7f4b29c91640 1 Processor -- start 2024-04-25T15:50:16.131 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.127+0000 7f4b29c91640 1 -- start start 2024-04-25T15:50:16.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b29c91640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240ca150 0x7f4b240c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:16.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b29c91640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4b241534a0 0x7f4b241538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:16.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b29c91640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b240c8d10 con 0x7f4b240c9230 2024-04-25T15:50:16.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b29c91640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4b24070730 con 0x7f4b241534a0 2024-04-25T15:50:16.133 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b29c91640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b240c8750 con 0x7f4b240ca150 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240ca150 0x7f4b240c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240ca150 0x7f4b240c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:34618/0 (socket says 172.21.15.67:34618) 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b22ffd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4b241534a0 0x7f4b241538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 -- 172.21.15.67:0/2193390842 learned_addr learned my addr 172.21.15.67:0/2193390842 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b23fff640 1 --1- 172.21.15.67:0/2193390842 >> v1:172.21.15.176:6789/0 conn(0x7f4b240c9230 0x7f4b2415cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:40696/0 (socket says 172.21.15.67:40696) 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 -- 172.21.15.67:0/2193390842 >> v2:172.21.15.176:3300/0 conn(0x7f4b241534a0 msgr2=0x7f4b241538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 --2- 172.21.15.67:0/2193390842 >> v2:172.21.15.176:3300/0 conn(0x7f4b241534a0 0x7f4b241538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b227fc640 1 -- 172.21.15.67:0/2193390842 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421936977 0 0) 0x7f4b240c8d10 con 0x7f4b240c9230 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 -- 172.21.15.67:0/2193390842 >> v1:172.21.15.176:6789/0 conn(0x7f4b240c9230 legacy=0x7f4b2415cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:16.134 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 -- 172.21.15.67:0/2193390842 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b240c8a30 con 0x7f4b240ca150 2024-04-25T15:50:16.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b237fe640 1 --2- 172.21.15.67:0/2193390842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240ca150 0x7f4b240c76a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009870 tx=0x7f4b1801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ebc762ff2f9044 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:16.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.131+0000 7f4b227fc640 1 -- 172.21.15.67:0/2193390842 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b180ab650 con 0x7f4b240ca150 2024-04-25T15:50:16.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b227fc640 1 -- 172.21.15.67:0/2193390842 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b180b0070 con 0x7f4b240ca150 2024-04-25T15:50:16.136 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b227fc640 1 -- 172.21.15.67:0/2193390842 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b180aba10 con 0x7f4b240ca150 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 -- 172.21.15.67:0/2193390842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240ca150 msgr2=0x7f4b240c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 --2- 172.21.15.67:0/2193390842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240ca150 0x7f4b240c76a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009870 tx=0x7f4b1801c560 comp rx=0 tx=0).stop 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 -- 172.21.15.67:0/2193390842 shutdown_connections 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 --2- 172.21.15.67:0/2193390842 >> v2:172.21.15.176:3300/0 conn(0x7f4b241534a0 0x7f4b241538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 --2- 172.21.15.67:0/2193390842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240ca150 0x7f4b240c76a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 -- 172.21.15.67:0/2193390842 >> 172.21.15.67:0/2193390842 conn(0x7f4b240cb330 msgr2=0x7f4b240cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 -- 172.21.15.67:0/2193390842 shutdown_connections 2024-04-25T15:50:16.137 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 -- 172.21.15.67:0/2193390842 wait complete. 2024-04-25T15:50:16.138 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 Processor -- start 2024-04-25T15:50:16.139 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.135+0000 7f4b29c91640 1 -- start start 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240c9230 0x7f4b241f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b240ca150 0x7f4b24141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b241534a0 0x7f4b24140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b240c7d60 con 0x7f4b241534a0 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4b240c7be0 con 0x7f4b240ca150 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b240c5d80 con 0x7f4b240c9230 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b241534a0 0x7f4b24140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b241534a0 0x7f4b24140280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60336/0 (socket says 172.21.15.67:60336) 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 -- 172.21.15.67:0/3344462421 learned_addr learned my addr 172.21.15.67:0/3344462421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b22ffd640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b240ca150 0x7f4b24141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b237fe640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240c9230 0x7f4b241f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:16.141 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 -- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240c9230 msgr2=0x7f4b241f7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:16.142 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240c9230 0x7f4b241f7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.142 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 -- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b240ca150 msgr2=0x7f4b24141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:16.142 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b240ca150 0x7f4b24141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.142 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 -- 172.21.15.67:0/3344462421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2415b180 con 0x7f4b241534a0 2024-04-25T15:50:16.142 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b23fff640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b241534a0 0x7f4b24140280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00c700 tx=0x7f4b1c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:16.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b0bfff640 1 -- 172.21.15.67:0/3344462421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c018070 con 0x7f4b241534a0 2024-04-25T15:50:16.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b0bfff640 1 -- 172.21.15.67:0/3344462421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1c00ce10 con 0x7f4b241534a0 2024-04-25T15:50:16.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b24152620 con 0x7f4b241534a0 2024-04-25T15:50:16.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b0bfff640 1 -- 172.21.15.67:0/3344462421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c00a880 con 0x7f4b241534a0 2024-04-25T15:50:16.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b2415ac80 con 0x7f4b241534a0 2024-04-25T15:50:16.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b2406a480 con 0x7f4b241534a0 2024-04-25T15:50:16.143 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b0bfff640 1 -- 172.21.15.67:0/3344462421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 93151+0+0 (secure 0 0 0) 0x7f4b1c01b070 con 0x7f4b241534a0 2024-04-25T15:50:16.144 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b0bfff640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b1006f3a0 0x7f4b100718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:16.144 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b0bfff640 1 -- 172.21.15.67:0/3344462421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f4b1c08d880 con 0x7f4b241534a0 2024-04-25T15:50:16.144 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.139+0000 7f4b237fe640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b1006f3a0 0x7f4b100718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:16.144 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.143+0000 7f4b237fe640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b1006f3a0 0x7f4b100718b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4b180af000 tx=0x7f4b180a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:16.148 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.147+0000 7f4b0bfff640 1 -- 172.21.15.67:0/3344462421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4b1c05b020 con 0x7f4b241534a0 2024-04-25T15:50:16.305 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.303+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b241530e0 con 0x7f4b241534a0 2024-04-25T15:50:16.306 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.303+0000 7f4b0bfff640 1 -- 172.21.15.67:0/3344462421 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f4b1c05f320 con 0x7f4b241534a0 2024-04-25T15:50:16.307 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-25T15:50:16.307 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":12,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:13.810006+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:16.308 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b1006f3a0 msgr2=0x7f4b100718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:16.308 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b1006f3a0 0x7f4b100718b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4b180af000 tx=0x7f4b180a7000 comp rx=0 tx=0).stop 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b241534a0 msgr2=0x7f4b24140280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b241534a0 0x7f4b24140280 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00c700 tx=0x7f4b1c00cbc0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 shutdown_connections 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b1006f3a0 0x7f4b100718b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b241534a0 0x7f4b24140280 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b240ca150 0x7f4b24141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 --2- 172.21.15.67:0/3344462421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b240c9230 0x7f4b241f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 >> 172.21.15.67:0/3344462421 conn(0x7f4b240cb330 msgr2=0x7f4b24150ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 shutdown_connections 2024-04-25T15:50:16.309 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:16.307+0000 7f4b29c91640 1 -- 172.21.15.67:0/3344462421 wait complete. 2024-04-25T15:50:16.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T15:50:23.132 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:23.285 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe06b705640 1 Processor -- start 2024-04-25T15:50:23.286 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe06b705640 1 -- start start 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe06b705640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe06415ff60 0x7fe0640d20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe06b705640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0640d2620 0x7fe0640c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe06b705640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe064160580 con 0x7fe0640c6be0 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe06b705640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe0640b5be0 con 0x7fe0640d2620 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe06b705640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe0640c7730 con 0x7fe06415ff60 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe069ca2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0640d2620 0x7fe0640c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe069ca2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0640d2620 0x7fe0640c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52322/0 (socket says 172.21.15.67:52322) 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe069ca2640 1 -- 172.21.15.67:0/210573260 learned_addr learned my addr 172.21.15.67:0/210573260 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe068ca0640 1 --2- 172.21.15.67:0/210573260 >> v2:172.21.15.176:3300/0 conn(0x7fe06415ff60 0x7fe0640d20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe069ca2640 1 -- 172.21.15.67:0/210573260 >> v2:172.21.15.176:3300/0 conn(0x7fe06415ff60 msgr2=0x7fe0640d20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:23.288 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe069ca2640 1 --2- 172.21.15.67:0/210573260 >> v2:172.21.15.176:3300/0 conn(0x7fe06415ff60 0x7fe0640d20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.289 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe069ca2640 1 -- 172.21.15.67:0/210573260 >> v1:172.21.15.176:6789/0 conn(0x7fe0640c6be0 legacy=0x7fe0641568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:23.289 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.282+0000 7fe069ca2640 1 -- 172.21.15.67:0/210573260 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe064160e00 con 0x7fe0640d2620 2024-04-25T15:50:23.290 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe069ca2640 1 --2- 172.21.15.67:0/210573260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0640d2620 0x7fe0640c7bc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe0640b7e30 tx=0x7fe05c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea809bab98d984c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:23.290 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe057fff640 1 -- 172.21.15.67:0/210573260 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0b2070 con 0x7fe0640d2620 2024-04-25T15:50:23.290 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe057fff640 1 -- 172.21.15.67:0/210573260 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe05c0a9070 con 0x7fe0640d2620 2024-04-25T15:50:23.290 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe057fff640 1 -- 172.21.15.67:0/210573260 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0ad880 con 0x7fe0640d2620 2024-04-25T15:50:23.291 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 -- 172.21.15.67:0/210573260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0640d2620 msgr2=0x7fe0640c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.291 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 --2- 172.21.15.67:0/210573260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0640d2620 0x7fe0640c7bc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe0640b7e30 tx=0x7fe05c01c450 comp rx=0 tx=0).stop 2024-04-25T15:50:23.291 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 -- 172.21.15.67:0/210573260 shutdown_connections 2024-04-25T15:50:23.291 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 --2- 172.21.15.67:0/210573260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0640d2620 0x7fe0640c7bc0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.291 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 --2- 172.21.15.67:0/210573260 >> v2:172.21.15.176:3300/0 conn(0x7fe06415ff60 0x7fe0640d20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.291 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 -- 172.21.15.67:0/210573260 >> 172.21.15.67:0/210573260 conn(0x7fe064068880 msgr2=0x7fe064068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:23.291 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 -- 172.21.15.67:0/210573260 shutdown_connections 2024-04-25T15:50:23.292 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.286+0000 7fe06b705640 1 -- 172.21.15.67:0/210573260 wait complete. 2024-04-25T15:50:23.293 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 Processor -- start 2024-04-25T15:50:23.293 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 -- start start 2024-04-25T15:50:23.295 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0640c6be0 0x7fe0641f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.295 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0640d2620 0x7fe0641fa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.295 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06415ff60 0x7fe0640d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe064169b10 con 0x7fe06415ff60 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe0640c8510 con 0x7fe0640c6be0 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe064169c90 con 0x7fe0640d2620 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06415ff60 0x7fe0640d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06415ff60 0x7fe0640d05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52326/0 (socket says 172.21.15.67:52326) 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 -- 172.21.15.67:0/3157460669 learned_addr learned my addr 172.21.15.67:0/3157460669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe0694a1640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0640c6be0 0x7fe0641f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe068ca0640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0640d2620 0x7fe0641fa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 -- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0640d2620 msgr2=0x7fe0641fa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0640d2620 0x7fe0641fa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 -- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0640c6be0 msgr2=0x7fe0641f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0640c6be0 0x7fe0641f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 -- 172.21.15.67:0/3157460669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0640d0b20 con 0x7fe06415ff60 2024-04-25T15:50:23.296 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe069ca2640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06415ff60 0x7fe0640d05e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe05c01a080 tx=0x7fe05c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:23.297 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe055ffb640 1 -- 172.21.15.67:0/3157460669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0bb070 con 0x7fe06415ff60 2024-04-25T15:50:23.297 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe055ffb640 1 -- 172.21.15.67:0/3157460669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe05c0add70 con 0x7fe06415ff60 2024-04-25T15:50:23.297 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0641f6c70 con 0x7fe06415ff60 2024-04-25T15:50:23.297 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe055ffb640 1 -- 172.21.15.67:0/3157460669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0b2040 con 0x7fe06415ff60 2024-04-25T15:50:23.297 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0640cfde0 con 0x7fe06415ff60 2024-04-25T15:50:23.298 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.290+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe06406a480 con 0x7fe06415ff60 2024-04-25T15:50:23.298 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.294+0000 7fe055ffb640 1 -- 172.21.15.67:0/3157460669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe05c0be070 con 0x7fe06415ff60 2024-04-25T15:50:23.298 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.294+0000 7fe055ffb640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe04806f430 0x7fe048071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.298 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.294+0000 7fe055ffb640 1 -- 172.21.15.67:0/3157460669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe05c128090 con 0x7fe06415ff60 2024-04-25T15:50:23.298 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.294+0000 7fe0694a1640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe04806f430 0x7fe048071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.298 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.294+0000 7fe0694a1640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe04806f430 0x7fe048071940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe058009f20 tx=0x7fe0580094d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:23.312 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.310+0000 7fe055ffb640 1 -- 172.21.15.67:0/3157460669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe05c0160d0 con 0x7fe06415ff60 2024-04-25T15:50:23.479 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.474+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0641f69c0 con 0x7fe06415ff60 2024-04-25T15:50:23.481 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.478+0000 7fe055ffb640 1 -- 172.21.15.67:0/3157460669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7fe05c0f5480 con 0x7fe06415ff60 2024-04-25T15:50:23.482 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-25T15:50:23.482 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:20.083015+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:23.484 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.478+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe04806f430 msgr2=0x7fe048071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.478+0000 7fe06b705640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe04806f430 0x7fe048071940 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe058009f20 tx=0x7fe0580094d0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.478+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06415ff60 msgr2=0x7fe0640d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.478+0000 7fe06b705640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06415ff60 0x7fe0640d05e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe05c01a080 tx=0x7fe05c0191f0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.478+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 shutdown_connections 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.482+0000 7fe06b705640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe04806f430 0x7fe048071940 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.482+0000 7fe06b705640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe06415ff60 0x7fe0640d05e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.482+0000 7fe06b705640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0640d2620 0x7fe0641fa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.482+0000 7fe06b705640 1 --2- 172.21.15.67:0/3157460669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0640c6be0 0x7fe0641f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.482+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 >> 172.21.15.67:0/3157460669 conn(0x7fe064068880 msgr2=0x7fe0641568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.482+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 shutdown_connections 2024-04-25T15:50:23.485 INFO:teuthology.misc.health.smithi067.stderr:2024-04-25T15:50:23.482+0000 7fe06b705640 1 -- 172.21.15.67:0/3157460669 wait complete. 2024-04-25T15:50:23.503 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T15:50:23.503 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T15:50:23.503 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T15:50:23.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T15:50:23.505 INFO:tasks.ceph.mds.a:Started 2024-04-25T15:50:23.505 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T15:50:23.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T15:50:23.507 INFO:tasks.ceph.mds.c:Started 2024-04-25T15:50:23.507 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-25T15:50:23.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-25T15:50:23.509 INFO:tasks.ceph.mds.e:Started 2024-04-25T15:50:23.509 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T15:50:23.509 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T15:50:23.512 INFO:tasks.ceph.mds.b:Started 2024-04-25T15:50:23.512 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T15:50:23.512 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T15:50:23.514 INFO:tasks.ceph.mds.d:Started 2024-04-25T15:50:23.514 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-25T15:50:23.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-25T15:50:23.517 INFO:tasks.ceph.mds.f:Started 2024-04-25T15:50:23.517 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T15:50:23.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:23.605 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:50:23.605 INFO:tasks.ceph.mds.e.smithi067.stdout:starting mds.e at 2024-04-25T15:50:23.610 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:50:23.623 INFO:tasks.ceph.mds.f.smithi176.stdout:starting mds.f at 2024-04-25T15:50:23.626 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:50:23.627 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:50:23.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a14b5d640 1 Processor -- start 2024-04-25T15:50:23.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a14b5d640 1 -- start start 2024-04-25T15:50:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a14b5d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8a1006ca60 0x7f8a1006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a14b5d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 0x7f8a10160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a14b5d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a1006d780 con 0x7f8a100b5660 2024-04-25T15:50:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a14b5d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8a1015d510 con 0x7f8a1006ca60 2024-04-25T15:50:23.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a14b5d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a10160be0 con 0x7f8a1015fe00 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a0dd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a100b5660 0x7f8a100b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49276/0 (socket says 172.21.15.67:49276) 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.662+0000 7f8a0dd9b640 1 -- 172.21.15.67:0/2585840501 learned_addr learned my addr 172.21.15.67:0/2585840501 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2585840501 >> v2:172.21.15.176:3300/0 conn(0x7f8a1006ca60 0x7f8a1006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0ed9d640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 0x7f8a10160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814790202 0 0) 0x7f8a1006d780 con 0x7f8a100b5660 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89fc003530 con 0x7f8a100b5660 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2811190844 0 0) 0x7f89fc003530 con 0x7f8a100b5660 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 msgr2=0x7f8a10160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 0x7f8a10160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 >> v2:172.21.15.176:3300/0 conn(0x7f8a1006ca60 msgr2=0x7f8a1006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 --2- 172.21.15.67:0/2585840501 >> v2:172.21.15.176:3300/0 conn(0x7f8a1006ca60 0x7f8a1006ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1015cbe0 con 0x7f8a100b5660 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f89f80016a0 con 0x7f8a100b5660 2024-04-25T15:50:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 >> v1:172.21.15.67:6789/0 conn(0x7f8a100b5660 legacy=0x7f8a100b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 0x7f89fc0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89fc003f40 0x7f89fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89fc004d50 0x7f89fc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1015d510 con 0x7f89fc004d50 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a10160be0 con 0x7f89fc003f40 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89fc00bab0 con 0x7f8a1015fe00 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 0x7f89fc0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0dd9b640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89fc003f40 0x7f89fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0ed9d640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89fc004d50 0x7f89fc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 -- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89fc003f40 msgr2=0x7f89fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89fc003f40 0x7f89fc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 -- 172.21.15.67:0/2585840501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89fc004d50 msgr2=0x7f89fc005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89fc004d50 0x7f89fc005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 -- 172.21.15.67:0/2585840501 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc003530 con 0x7f8a1015fe00 2024-04-25T15:50:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0ed9d640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89fc004d50 0x7f89fc005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 0x7f89fc0039c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a100cd350 tx=0x7f8a0001ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6b4411e6d83623 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a000b2070 con 0x7f8a1015fe00 2024-04-25T15:50:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a000a5640 con 0x7f8a1015fe00 2024-04-25T15:50:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.666+0000 7f8a0d59a640 1 -- 172.21.15.67:0/2585840501 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a000096b0 con 0x7f8a1015fe00 2024-04-25T15:50:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 msgr2=0x7f89fc0039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 --2- 172.21.15.67:0/2585840501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1015fe00 0x7f89fc0039c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a100cd350 tx=0x7f8a0001ca50 comp rx=0 tx=0).stop 2024-04-25T15:50:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a0ed9d640 1 -- 172.21.15.67:0/2585840501 reap_dead start 2024-04-25T15:50:23.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2585840501 shutdown_connections 2024-04-25T15:50:23.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2585840501 >> 172.21.15.67:0/2585840501 conn(0x7f8a1006e420 msgr2=0x7f8a1006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:23.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2585840501 shutdown_connections 2024-04-25T15:50:23.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2585840501 wait complete. 2024-04-25T15:50:23.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 Processor -- start 2024-04-25T15:50:23.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.670+0000 7f8a14b5d640 1 -- start start 2024-04-25T15:50:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1015fe00 0x7f8a10158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a10158510 0x7f8a1015b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a10150570 0x7f8a10150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1015c890 con 0x7f8a1015fe00 2024-04-25T15:50:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1015d1a0 con 0x7f8a10158510 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a100b6e90 con 0x7f8a10150570 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1015fe00 0x7f8a10158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1015fe00 0x7f8a10158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52394/0 (socket says 172.21.15.67:52394) 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 -- 172.21.15.67:0/2892698967 learned_addr learned my addr 172.21.15.67:0/2892698967 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0ed9d640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a10150570 0x7f8a10150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0dd9b640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a10158510 0x7f8a1015b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 -- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a10150570 msgr2=0x7f8a10150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a10150570 0x7f8a10150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 -- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a10158510 msgr2=0x7f8a1015b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a10158510 0x7f8a1015b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 -- 172.21.15.67:0/2892698967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a101593d0 con 0x7f8a1015fe00 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a0e59c640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1015fe00 0x7f8a10158e90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8a000b0040 tx=0x7f89fc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a06ffd640 1 -- 172.21.15.67:0/2892698967 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a000be070 con 0x7f8a1015fe00 2024-04-25T15:50:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a06ffd640 1 -- 172.21.15.67:0/2892698967 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a000ad440 con 0x7f8a1015fe00 2024-04-25T15:50:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a101f6520 con 0x7f8a1015fe00 2024-04-25T15:50:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a06ffd640 1 -- 172.21.15.67:0/2892698967 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a000b2040 con 0x7f8a1015fe00 2024-04-25T15:50:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a101f6800 con 0x7f8a1015fe00 2024-04-25T15:50:23.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a1006a480 con 0x7f8a1015fe00 2024-04-25T15:50:23.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a06ffd640 1 -- 172.21.15.67:0/2892698967 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8a000ad830 con 0x7f8a1015fe00 2024-04-25T15:50:23.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.674+0000 7f8a06ffd640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89f006f430 0x7f89f0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:23.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.678+0000 7f8a06ffd640 1 -- 172.21.15.67:0/2892698967 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f8a00128be0 con 0x7f8a1015fe00 2024-04-25T15:50:23.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.678+0000 7f8a0dd9b640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89f006f430 0x7f89f0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:23.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.678+0000 7f8a0dd9b640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89f006f430 0x7f89f0071940 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89f8000c10 tx=0x7f89f8007900 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.682+0000 7f8a06ffd640 1 -- 172.21.15.67:0/2892698967 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8a00016760 con 0x7f8a1015fe00 2024-04-25T15:50:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.838+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a100cd380 con 0x7f8a1015fe00 2024-04-25T15:50:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a06ffd640 1 -- 172.21.15.67:0/2892698967 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7f8a000f57e0 con 0x7f8a1015fe00 2024-04-25T15:50:23.846 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:23.846 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:20.083015+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89f006f430 msgr2=0x7f89f0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89f006f430 0x7f89f0071940 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f89f8000c10 tx=0x7f89f8007900 comp rx=0 tx=0).stop 2024-04-25T15:50:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1015fe00 msgr2=0x7f8a10158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1015fe00 0x7f8a10158e90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8a000b0040 tx=0x7f89fc005840 comp rx=0 tx=0).stop 2024-04-25T15:50:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 shutdown_connections 2024-04-25T15:50:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89f006f430 0x7f89f0071940 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a10150570 0x7f8a10150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a10158510 0x7f8a1015b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 --2- 172.21.15.67:0/2892698967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1015fe00 0x7f8a10158e90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:23.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 >> 172.21.15.67:0/2892698967 conn(0x7f8a1006e420 msgr2=0x7f8a101557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:23.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 shutdown_connections 2024-04-25T15:50:23.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:23.842+0000 7f8a14b5d640 1 -- 172.21.15.67:0/2892698967 wait complete. 2024-04-25T15:50:23.865 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:50:23.866 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:50:24.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.018+0000 7fbcae5ac640 1 Processor -- start 2024-04-25T15:50:24.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.018+0000 7fbcae5ac640 1 -- start start 2024-04-25T15:50:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcae5ac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbca8071b20 0x7fbca815d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcae5ac640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbca815d710 0x7fbca80c7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcae5ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca80c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcae5ac640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbca815cd40 con 0x7fbca8071b20 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcae5ac640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbca80c6890 con 0x7fbca815d710 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcae5ac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbca80c6b70 con 0x7fbca80c7b50 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbca8071b20 0x7fbca815d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbca8071b20 0x7fbca815d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52396/0 (socket says 172.21.15.67:52396) 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 -- 172.21.15.67:0/3149524892 learned_addr learned my addr 172.21.15.67:0/3149524892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca77fe640 1 --2- 172.21.15.67:0/3149524892 >> v2:172.21.15.176:3300/0 conn(0x7fbca815d710 0x7fbca80c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcacb49640 1 --2- 172.21.15.67:0/3149524892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca80c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 -- 172.21.15.67:0/3149524892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 msgr2=0x7fbca80c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 --2- 172.21.15.67:0/3149524892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca80c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 -- 172.21.15.67:0/3149524892 >> v2:172.21.15.176:3300/0 conn(0x7fbca815d710 msgr2=0x7fbca80c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 --2- 172.21.15.67:0/3149524892 >> v2:172.21.15.176:3300/0 conn(0x7fbca815d710 0x7fbca80c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 -- 172.21.15.67:0/3149524892 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca80c6e50 con 0x7fbca8071b20 2024-04-25T15:50:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbcacb49640 1 --2- 172.21.15.67:0/3149524892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca80c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca7fff640 1 --2- 172.21.15.67:0/3149524892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca815d1d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbca806d910 tx=0x7fbc9801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f45ac6e51621dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:24.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca6ffd640 1 -- 172.21.15.67:0/3149524892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc980ab5f0 con 0x7fbca8071b20 2024-04-25T15:50:24.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca6ffd640 1 -- 172.21.15.67:0/3149524892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc980b0070 con 0x7fbca8071b20 2024-04-25T15:50:24.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.022+0000 7fbca6ffd640 1 -- 172.21.15.67:0/3149524892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc980ab9b0 con 0x7fbca8071b20 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3149524892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 msgr2=0x7fbca815d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3149524892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca815d1d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbca806d910 tx=0x7fbc9801c520 comp rx=0 tx=0).stop 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3149524892 shutdown_connections 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3149524892 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca80c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3149524892 >> v2:172.21.15.176:3300/0 conn(0x7fbca815d710 0x7fbca80c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3149524892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca815d1d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3149524892 >> 172.21.15.67:0/3149524892 conn(0x7fbca806fe70 msgr2=0x7fbca80cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:24.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3149524892 shutdown_connections 2024-04-25T15:50:24.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3149524892 wait complete. 2024-04-25T15:50:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 Processor -- start 2024-04-25T15:50:24.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.026+0000 7fbcae5ac640 1 -- start start 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca81f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca8151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbca815d710 0x7fbca814be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8159db0 con 0x7fbca8071b20 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8159c30 con 0x7fbca815d710 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbca8159f30 con 0x7fbca80c7b50 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca81f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca81f7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52410/0 (socket says 172.21.15.67:52410) 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 -- 172.21.15.67:0/3155251962 learned_addr learned my addr 172.21.15.67:0/3155251962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:24.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca77fe640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca8151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcacb49640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbca815d710 0x7fbca814be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 -- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 msgr2=0x7fbca8151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca8151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 -- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbca815d710 msgr2=0x7fbca814be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbca815d710 0x7fbca814be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 -- 172.21.15.67:0/3155251962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca81f8130 con 0x7fbca8071b20 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca7fff640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca81f7bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbc9801c350 tx=0x7fbc980192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca4ff9640 1 -- 172.21.15.67:0/3155251962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc980be3f0 con 0x7fbca8071b20 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca4ff9640 1 -- 172.21.15.67:0/3155251962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc980be590 con 0x7fbca8071b20 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca81f7430 con 0x7fbca8071b20 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca4ff9640 1 -- 172.21.15.67:0/3155251962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc980be940 con 0x7fbca8071b20 2024-04-25T15:50:24.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca8163720 con 0x7fbca8071b20 2024-04-25T15:50:24.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca806a480 con 0x7fbca8071b20 2024-04-25T15:50:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca4ff9640 1 -- 172.21.15.67:0/3155251962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbc980b43f0 con 0x7fbca8071b20 2024-04-25T15:50:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca4ff9640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9406f400 0x7fbc94071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.030+0000 7fbca4ff9640 1 -- 172.21.15.67:0/3155251962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbc98128c10 con 0x7fbca8071b20 2024-04-25T15:50:24.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.034+0000 7fbca77fe640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9406f400 0x7fbc94071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.034+0000 7fbca77fe640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9406f400 0x7fbc94071910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbca814d520 tx=0x7fbc9c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.038+0000 7fbca4ff9640 1 -- 172.21.15.67:0/3155251962 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbc98016110 con 0x7fbca8071b20 2024-04-25T15:50:24.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbca80c4700 con 0x7fbca8071b20 2024-04-25T15:50:24.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbca4ff9640 1 -- 172.21.15.67:0/3155251962 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fbc980f5920 con 0x7fbca8071b20 2024-04-25T15:50:24.198 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:50:24.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9406f400 msgr2=0x7fbc94071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9406f400 0x7fbc94071910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbca814d520 tx=0x7fbc9c009190 comp rx=0 tx=0).stop 2024-04-25T15:50:24.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 msgr2=0x7fbca81f7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca81f7bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbc9801c350 tx=0x7fbc980192c0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 shutdown_connections 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9406f400 0x7fbc94071910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbca815d710 0x7fbca814be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbca80c7b50 0x7fbca8151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.194+0000 7fbcae5ac640 1 --2- 172.21.15.67:0/3155251962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbca8071b20 0x7fbca81f7bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.198+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 >> 172.21.15.67:0/3155251962 conn(0x7fbca806fe70 msgr2=0x7fbca80caaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.198+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 shutdown_connections 2024-04-25T15:50:24.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.198+0000 7fbcae5ac640 1 -- 172.21.15.67:0/3155251962 wait complete. 2024-04-25T15:50:24.218 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:50:24.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d9dbfd640 1 Processor -- start 2024-04-25T15:50:24.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d9dbfd640 1 -- start start 2024-04-25T15:50:24.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d9dbfd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d980caf60 0x7f0d9815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d9dbfd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d98073bf0 0x7f0d98163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d9dbfd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d980c77a0 con 0x7f0d9815d790 2024-04-25T15:50:24.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d9dbfd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0d980cab60 con 0x7f0d98073bf0 2024-04-25T15:50:24.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d9dbfd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0d98159620 con 0x7f0d980caf60 2024-04-25T15:50:24.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d97fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d98073bf0 0x7f0d98163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d97fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d98073bf0 0x7f0d98163ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52418/0 (socket says 172.21.15.67:52418) 2024-04-25T15:50:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d97fff640 1 -- 172.21.15.67:0/1404683348 learned_addr learned my addr 172.21.15.67:0/1404683348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d977fe640 1 --2- 172.21.15.67:0/1404683348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d980caf60 0x7f0d9815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d97fff640 1 -- 172.21.15.67:0/1404683348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d980caf60 msgr2=0x7f0d9815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d97fff640 1 --2- 172.21.15.67:0/1404683348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d980caf60 0x7f0d9815d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d97fff640 1 -- 172.21.15.67:0/1404683348 >> v1:172.21.15.176:6789/0 conn(0x7f0d9815d790 legacy=0x7f0d9806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.362+0000 7f0d97fff640 1 -- 172.21.15.67:0/1404683348 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d980c7a60 con 0x7f0d98073bf0 2024-04-25T15:50:24.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d97fff640 1 --2- 172.21.15.67:0/1404683348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d98073bf0 0x7f0d98163ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000d150 tx=0x7f0d9001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e058b293bd08d165 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:24.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d967fc640 1 -- 172.21.15.67:0/1404683348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9000b2d0 con 0x7f0d98073bf0 2024-04-25T15:50:24.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d967fc640 1 -- 172.21.15.67:0/1404683348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d9000b470 con 0x7f0d98073bf0 2024-04-25T15:50:24.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d967fc640 1 -- 172.21.15.67:0/1404683348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900a7a20 con 0x7f0d98073bf0 2024-04-25T15:50:24.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/1404683348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d98073bf0 msgr2=0x7f0d98163ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/1404683348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d98073bf0 0x7f0d98163ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000d150 tx=0x7f0d9001ec30 comp rx=0 tx=0).stop 2024-04-25T15:50:24.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/1404683348 shutdown_connections 2024-04-25T15:50:24.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/1404683348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d98073bf0 0x7f0d98163ec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/1404683348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d980caf60 0x7f0d9815d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/1404683348 >> 172.21.15.67:0/1404683348 conn(0x7f0d98070db0 msgr2=0x7f0d980ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:24.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/1404683348 shutdown_connections 2024-04-25T15:50:24.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.366+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/1404683348 wait complete. 2024-04-25T15:50:24.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 Processor -- start 2024-04-25T15:50:24.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 -- start start 2024-04-25T15:50:24.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d98073bf0 0x7f0d981567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980caf60 0x7f0d981f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d9815d790 0x7f0d981f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9815a570 con 0x7f0d9815d790 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9815a3f0 con 0x7f0d980caf60 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9806ce50 con 0x7f0d98073bf0 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d9815d790 0x7f0d981f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d9815d790 0x7f0d981f54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52430/0 (socket says 172.21.15.67:52430) 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 -- 172.21.15.67:0/2558694046 learned_addr learned my addr 172.21.15.67:0/2558694046 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d977fe640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d98073bf0 0x7f0d981567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d96ffd640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980caf60 0x7f0d981f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 -- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d98073bf0 msgr2=0x7f0d981567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d98073bf0 0x7f0d981567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 -- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980caf60 msgr2=0x7f0d981f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980caf60 0x7f0d981f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 -- 172.21.15.67:0/2558694046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d98156cf0 con 0x7f0d9815d790 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d97fff640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d9815d790 0x7f0d981f54a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0d900b1040 tx=0x7f0d9001b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d7ffff640 1 -- 172.21.15.67:0/2558694046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90009600 con 0x7f0d9815d790 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d7ffff640 1 -- 172.21.15.67:0/2558694046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d900097c0 con 0x7f0d9815d790 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d7ffff640 1 -- 172.21.15.67:0/2558694046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90009bf0 con 0x7f0d9815d790 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d98153800 con 0x7f0d9815d790 2024-04-25T15:50:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.370+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d980ce1a0 con 0x7f0d9815d790 2024-04-25T15:50:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.374+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d9806a480 con 0x7f0d9815d790 2024-04-25T15:50:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.374+0000 7f0d7ffff640 1 -- 172.21.15.67:0/2558694046 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0d900c2020 con 0x7f0d9815d790 2024-04-25T15:50:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.374+0000 7f0d7ffff640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0d8406f400 0x7f0d84071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.374+0000 7f0d7ffff640 1 -- 172.21.15.67:0/2558694046 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f0d9012b8d0 con 0x7f0d9815d790 2024-04-25T15:50:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.374+0000 7f0d977fe640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0d8406f400 0x7f0d84071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.374+0000 7f0d977fe640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0d8406f400 0x7f0d84071910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800b930 tx=0x7f0d88008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:24.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.378+0000 7f0d7ffff640 1 -- 172.21.15.67:0/2558694046 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0d900c9020 con 0x7f0d9815d790 2024-04-25T15:50:24.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.578+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d9814c0d0 con 0x7f0d9815d790 2024-04-25T15:50:24.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.578+0000 7f0d7ffff640 1 -- 172.21.15.67:0/2558694046 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f0d98156cf0 con 0x7f0d9815d790 2024-04-25T15:50:24.584 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:24.584 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:50:24.584 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-04-25T15:50:24.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0d8406f400 msgr2=0x7f0d84071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0d8406f400 0x7f0d84071910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800b930 tx=0x7f0d88008040 comp rx=0 tx=0).stop 2024-04-25T15:50:24.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d9815d790 msgr2=0x7f0d981f54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:24.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d9815d790 0x7f0d981f54a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0d900b1040 tx=0x7f0d9001b9d0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 shutdown_connections 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0d8406f400 0x7f0d84071910 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d9815d790 0x7f0d981f54a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980caf60 0x7f0d981f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 --2- 172.21.15.67:0/2558694046 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d98073bf0 0x7f0d981567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 >> 172.21.15.67:0/2558694046 conn(0x7f0d98070db0 msgr2=0x7f0d980c8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 shutdown_connections 2024-04-25T15:50:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:24.582+0000 7f0d9dbfd640 1 -- 172.21.15.67:0/2558694046 wait complete. 2024-04-25T15:50:26.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:50:26.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed28d9b640 1 Processor -- start 2024-04-25T15:50:26.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed28d9b640 1 -- start start 2024-04-25T15:50:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed28d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed2415a510 0x7fed2415a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed28d9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fed24162e80 0x7fed241652a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed28d9b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed240ce970 con 0x7fed24068840 2024-04-25T15:50:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed28d9b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fed240ca1e0 con 0x7fed2415a510 2024-04-25T15:50:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed28d9b640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fed2415ae60 con 0x7fed24162e80 2024-04-25T15:50:26.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed21d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed2415a510 0x7fed2415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed21d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed2415a510 0x7fed2415a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52432/0 (socket says 172.21.15.67:52432) 2024-04-25T15:50:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed21d9b640 1 -- 172.21.15.67:0/1540110249 learned_addr learned my addr 172.21.15.67:0/1540110249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed22d9d640 1 --2- 172.21.15.67:0/1540110249 >> v2:172.21.15.176:3300/0 conn(0x7fed24162e80 0x7fed241652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed21d9b640 1 -- 172.21.15.67:0/1540110249 >> v2:172.21.15.176:3300/0 conn(0x7fed24162e80 msgr2=0x7fed241652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed21d9b640 1 --2- 172.21.15.67:0/1540110249 >> v2:172.21.15.176:3300/0 conn(0x7fed24162e80 0x7fed241652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed21d9b640 1 -- 172.21.15.67:0/1540110249 >> v1:172.21.15.176:6789/0 conn(0x7fed24068840 legacy=0x7fed240c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.770+0000 7fed21d9b640 1 -- 172.21.15.67:0/1540110249 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed240cb630 con 0x7fed2415a510 2024-04-25T15:50:26.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed21d9b640 1 --2- 172.21.15.67:0/1540110249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed2415a510 0x7fed2415a920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fed2406fa40 tx=0x7fed1801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63ac06f5d77c8e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed2159a640 1 -- 172.21.15.67:0/1540110249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed180b5070 con 0x7fed2415a510 2024-04-25T15:50:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed2159a640 1 -- 172.21.15.67:0/1540110249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed180a89d0 con 0x7fed2415a510 2024-04-25T15:50:26.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed2159a640 1 -- 172.21.15.67:0/1540110249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed180a8da0 con 0x7fed2415a510 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 -- 172.21.15.67:0/1540110249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed2415a510 msgr2=0x7fed2415a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1540110249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed2415a510 0x7fed2415a920 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fed2406fa40 tx=0x7fed1801ecd0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 -- 172.21.15.67:0/1540110249 shutdown_connections 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1540110249 >> v2:172.21.15.176:3300/0 conn(0x7fed24162e80 0x7fed241652a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1540110249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed2415a510 0x7fed2415a920 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 -- 172.21.15.67:0/1540110249 >> 172.21.15.67:0/1540110249 conn(0x7fed24069220 msgr2=0x7fed240cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 -- 172.21.15.67:0/1540110249 shutdown_connections 2024-04-25T15:50:26.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.774+0000 7fed28d9b640 1 -- 172.21.15.67:0/1540110249 wait complete. 2024-04-25T15:50:26.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 Processor -- start 2024-04-25T15:50:26.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 -- start start 2024-04-25T15:50:26.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed24068840 0x7fed24155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:26.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed2415a510 0x7fed24146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:26.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed24162e80 0x7fed24155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:26.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fed240caa40 con 0x7fed24068840 2024-04-25T15:50:26.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fed240ca760 con 0x7fed2415a510 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fed240725d0 con 0x7fed24162e80 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed24068840 0x7fed24155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed24068840 0x7fed24155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52438/0 (socket says 172.21.15.67:52438) 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 -- 172.21.15.67:0/1288188065 learned_addr learned my addr 172.21.15.67:0/1288188065 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed22d9d640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed24162e80 0x7fed24155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed21d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed2415a510 0x7fed24146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 -- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed24162e80 msgr2=0x7fed24155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed24162e80 0x7fed24155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 -- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed2415a510 msgr2=0x7fed24146b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed2415a510 0x7fed24146b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 -- 172.21.15.67:0/1288188065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed24150610 con 0x7fed24068840 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed21d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed2415a510 0x7fed24146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed2259c640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed24068840 0x7fed24155b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fed1400b410 tx=0x7fed1400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed12ffd640 1 -- 172.21.15.67:0/1288188065 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14015030 con 0x7fed24068840 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed12ffd640 1 -- 172.21.15.67:0/1288188065 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed14004020 con 0x7fed24068840 2024-04-25T15:50:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed12ffd640 1 -- 172.21.15.67:0/1288188065 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14004450 con 0x7fed24068840 2024-04-25T15:50:26.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed241f2a70 con 0x7fed24068840 2024-04-25T15:50:26.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed241f2ff0 con 0x7fed24068840 2024-04-25T15:50:26.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.778+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed2406a480 con 0x7fed24068840 2024-04-25T15:50:26.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.782+0000 7fed12ffd640 1 -- 172.21.15.67:0/1288188065 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fed14018070 con 0x7fed24068840 2024-04-25T15:50:26.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.782+0000 7fed12ffd640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fed0406f400 0x7fed04071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:26.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.782+0000 7fed12ffd640 1 -- 172.21.15.67:0/1288188065 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fed1408f310 con 0x7fed24068840 2024-04-25T15:50:26.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.782+0000 7fed21d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fed0406f400 0x7fed04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:26.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.782+0000 7fed21d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fed0406f400 0x7fed04071910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fed180b2000 tx=0x7fed180aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:26.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.786+0000 7fed12ffd640 1 -- 172.21.15.67:0/1288188065 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fed24150610 con 0x7fed24068840 2024-04-25T15:50:26.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.982+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed240723b0 con 0x7fed24068840 2024-04-25T15:50:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.982+0000 7fed12ffd640 1 -- 172.21.15.67:0/1288188065 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5515 (secure 0 0 0) 0x7fed1405c5c0 con 0x7fed24068840 2024-04-25T15:50:26.984 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:26.984 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/920422475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":920422475},{"type":"v1","addr":"172.21.15.67:6835","nonce":920422475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/144842397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":144842397},{"type":"v1","addr":"172.21.15.67:6834","nonce":144842397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6838/2023672746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2023672746},{"type":"v1","addr":"172.21.15.176:6838","nonce":2023672746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/2385091301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2385091301},{"type":"v1","addr":"172.21.15.176:6835","nonce":2385091301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6839/1886671623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6837","nonce":1886671623},{"type":"v1","addr":"172.21.15.176:6839","nonce":1886671623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T15:50:26.984 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-04-25T15:50:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fed0406f400 msgr2=0x7fed04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:26.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fed0406f400 0x7fed04071910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fed180b2000 tx=0x7fed180aa070 comp rx=0 tx=0).stop 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed24068840 msgr2=0x7fed24155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed24068840 0x7fed24155b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fed1400b410 tx=0x7fed1400b8d0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 shutdown_connections 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fed0406f400 0x7fed04071910 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed24162e80 0x7fed24155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed2415a510 0x7fed24146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 --2- 172.21.15.67:0/1288188065 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed24068840 0x7fed24155b10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 >> 172.21.15.67:0/1288188065 conn(0x7fed24069220 msgr2=0x7fed24070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 shutdown_connections 2024-04-25T15:50:26.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:26.986+0000 7fed28d9b640 1 -- 172.21.15.67:0/1288188065 wait complete. 2024-04-25T15:50:27.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc0732f3640 1 Processor -- start 2024-04-25T15:50:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc0732f3640 1 -- start start 2024-04-25T15:50:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc0732f3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc06c15a020 0x7fc06c153730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc0732f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c0c9800 0x7fc06c0c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc0732f3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06c072060 con 0x7fc06c15d090 2024-04-25T15:50:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc0732f3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc06c072300 con 0x7fc06c15a020 2024-04-25T15:50:27.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc0732f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc06c153c70 con 0x7fc06c0c9800 2024-04-25T15:50:27.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc07108f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc06c15a020 0x7fc06c153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc07088e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c0c9800 0x7fc06c0c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc071890640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc06c15d090 0x7fc06c15d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:44850/0 (socket says 172.21.15.67:44850) 2024-04-25T15:50:27.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.154+0000 7fc071890640 1 -- 172.21.15.67:0/1597480548 learned_addr learned my addr 172.21.15.67:0/1597480548 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:27.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc07108f640 1 -- 172.21.15.67:0/1597480548 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c0c9800 msgr2=0x7fc06c0c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:27.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc07108f640 1 --2- 172.21.15.67:0/1597480548 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c0c9800 0x7fc06c0c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc07108f640 1 -- 172.21.15.67:0/1597480548 >> v1:172.21.15.176:6789/0 conn(0x7fc06c15d090 legacy=0x7fc06c15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc07108f640 1 -- 172.21.15.67:0/1597480548 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c06c6e0 con 0x7fc06c15a020 2024-04-25T15:50:27.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc07108f640 1 --2- 172.21.15.67:0/1597480548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 0x7fc06c153730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc0680097c0 tx=0x7fc06801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4967dfabdf7cfd9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc063fff640 1 -- 172.21.15.67:0/1597480548 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0680b0070 con 0x7fc06c15a020 2024-04-25T15:50:27.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc063fff640 1 -- 172.21.15.67:0/1597480548 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0680ab600 con 0x7fc06c15a020 2024-04-25T15:50:27.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc063fff640 1 -- 172.21.15.67:0/1597480548 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0680ab9d0 con 0x7fc06c15a020 2024-04-25T15:50:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 -- 172.21.15.67:0/1597480548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 msgr2=0x7fc06c153730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 --2- 172.21.15.67:0/1597480548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 0x7fc06c153730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc0680097c0 tx=0x7fc06801c4a0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 -- 172.21.15.67:0/1597480548 shutdown_connections 2024-04-25T15:50:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 --2- 172.21.15.67:0/1597480548 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c0c9800 0x7fc06c0c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 --2- 172.21.15.67:0/1597480548 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 0x7fc06c153730 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 -- 172.21.15.67:0/1597480548 >> 172.21.15.67:0/1597480548 conn(0x7fc06c06f860 msgr2=0x7fc06c06fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 -- 172.21.15.67:0/1597480548 shutdown_connections 2024-04-25T15:50:27.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.158+0000 7fc0732f3640 1 -- 172.21.15.67:0/1597480548 wait complete. 2024-04-25T15:50:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.162+0000 7fc0732f3640 1 Processor -- start 2024-04-25T15:50:27.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.162+0000 7fc0732f3640 1 -- start start 2024-04-25T15:50:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.162+0000 7fc0732f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc06c0c9800 0x7fc06c1f9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 0x7fc06c162970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c15d090 0x7fc06c1544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c0725a0 con 0x7fc06c0c9800 2024-04-25T15:50:27.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc06c15a430 con 0x7fc06c15a020 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc06c166720 con 0x7fc06c15d090 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc06c0c9800 0x7fc06c1f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc06c0c9800 0x7fc06c1f9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52444/0 (socket says 172.21.15.67:52444) 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 -- 172.21.15.67:0/3421722570 learned_addr learned my addr 172.21.15.67:0/3421722570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07088e640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 0x7fc06c162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc071890640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c15d090 0x7fc06c1544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 -- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c15d090 msgr2=0x7fc06c1544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c15d090 0x7fc06c1544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 -- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 msgr2=0x7fc06c162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 0x7fc06c162970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 -- 172.21.15.67:0/3421722570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c162eb0 con 0x7fc06c0c9800 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc071890640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c15d090 0x7fc06c1544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:27.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07108f640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc06c0c9800 0x7fc06c1f9230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc06801a0d0 tx=0x7fc068019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc061ffb640 1 -- 172.21.15.67:0/3421722570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0680ba070 con 0x7fc06c0c9800 2024-04-25T15:50:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc061ffb640 1 -- 172.21.15.67:0/3421722570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0680b43f0 con 0x7fc06c0c9800 2024-04-25T15:50:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc06c1f9bc0 con 0x7fc06c0c9800 2024-04-25T15:50:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc061ffb640 1 -- 172.21.15.67:0/3421722570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0680b0040 con 0x7fc06c0c9800 2024-04-25T15:50:27.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc06c162170 con 0x7fc06c0c9800 2024-04-25T15:50:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc06c06a480 con 0x7fc06c0c9800 2024-04-25T15:50:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc061ffb640 1 -- 172.21.15.67:0/3421722570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc0680b47c0 con 0x7fc06c0c9800 2024-04-25T15:50:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc061ffb640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc05006f400 0x7fc050071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc061ffb640 1 -- 172.21.15.67:0/3421722570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fc068127f50 con 0x7fc06c0c9800 2024-04-25T15:50:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.166+0000 7fc07088e640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc05006f400 0x7fc050071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.170+0000 7fc07088e640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc05006f400 0x7fc050071910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc06c153870 tx=0x7fc05c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.174+0000 7fc061ffb640 1 -- 172.21.15.67:0/3421722570 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc068016090 con 0x7fc06c0c9800 2024-04-25T15:50:27.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.330+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc06c0c4670 con 0x7fc06c0c9800 2024-04-25T15:50:27.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.330+0000 7fc061ffb640 1 -- 172.21.15.67:0/3421722570 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12848 (secure 0 0 0) 0x7fc0680f51a0 con 0x7fc06c0c9800 2024-04-25T15:50:27.333 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:27.333 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:20.083015+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc05006f400 msgr2=0x7fc050071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc05006f400 0x7fc050071910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc06c153870 tx=0x7fc05c008040 comp rx=0 tx=0).stop 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc06c0c9800 msgr2=0x7fc06c1f9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc06c0c9800 0x7fc06c1f9230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc06801a0d0 tx=0x7fc068019240 comp rx=0 tx=0).stop 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 shutdown_connections 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc05006f400 0x7fc050071910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc06c15d090 0x7fc06c1544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc06c15a020 0x7fc06c162970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 --2- 172.21.15.67:0/3421722570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc06c0c9800 0x7fc06c1f9230 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 >> 172.21.15.67:0/3421722570 conn(0x7fc06c06f860 msgr2=0x7fc06c15d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 shutdown_connections 2024-04-25T15:50:27.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.334+0000 7fc0732f3640 1 -- 172.21.15.67:0/3421722570 wait complete. 2024-04-25T15:50:27.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:50:27.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:50:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 Processor -- start 2024-04-25T15:50:27.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 -- start start 2024-04-25T15:50:27.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff20071b20 0x7eff2015d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff2015d710 0x7eff200c7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7eff200c7b50 0x7eff200c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7eff2015cd40 con 0x7eff20071b20 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7eff200c6890 con 0x7eff200c7b50 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff287d0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff200c6b70 con 0x7eff2015d710 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff20071b20 0x7eff2015d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff20071b20 0x7eff2015d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52452/0 (socket says 172.21.15.67:52452) 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 -- 172.21.15.67:0/2651887285 learned_addr learned my addr 172.21.15.67:0/2651887285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff26d6d640 1 --2- 172.21.15.67:0/2651887285 >> v2:172.21.15.176:3300/0 conn(0x7eff200c7b50 0x7eff200c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff25d6b640 1 --2- 172.21.15.67:0/2651887285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff2015d710 0x7eff200c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 -- 172.21.15.67:0/2651887285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff2015d710 msgr2=0x7eff200c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 --2- 172.21.15.67:0/2651887285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff2015d710 0x7eff200c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 -- 172.21.15.67:0/2651887285 >> v2:172.21.15.176:3300/0 conn(0x7eff200c7b50 msgr2=0x7eff200c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 --2- 172.21.15.67:0/2651887285 >> v2:172.21.15.176:3300/0 conn(0x7eff200c7b50 0x7eff200c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff2656c640 1 -- 172.21.15.67:0/2651887285 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff200c6e50 con 0x7eff20071b20 2024-04-25T15:50:27.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.510+0000 7eff25d6b640 1 --2- 172.21.15.67:0/2651887285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff2015d710 0x7eff200c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff2656c640 1 --2- 172.21.15.67:0/2651887285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff2015d1d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff2015a530 tx=0x7eff1c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1e14f971702640 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff2556a640 1 -- 172.21.15.67:0/2651887285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0ab5f0 con 0x7eff20071b20 2024-04-25T15:50:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff2556a640 1 -- 172.21.15.67:0/2651887285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff1c0b0070 con 0x7eff20071b20 2024-04-25T15:50:27.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff2556a640 1 -- 172.21.15.67:0/2651887285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0ab9b0 con 0x7eff20071b20 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 -- 172.21.15.67:0/2651887285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 msgr2=0x7eff2015d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 --2- 172.21.15.67:0/2651887285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff2015d1d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff2015a530 tx=0x7eff1c01c520 comp rx=0 tx=0).stop 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 -- 172.21.15.67:0/2651887285 shutdown_connections 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 --2- 172.21.15.67:0/2651887285 >> v2:172.21.15.176:3300/0 conn(0x7eff200c7b50 0x7eff200c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 --2- 172.21.15.67:0/2651887285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff2015d710 0x7eff200c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 --2- 172.21.15.67:0/2651887285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff2015d1d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 -- 172.21.15.67:0/2651887285 >> 172.21.15.67:0/2651887285 conn(0x7eff2006fe70 msgr2=0x7eff200cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 -- 172.21.15.67:0/2651887285 shutdown_connections 2024-04-25T15:50:27.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 -- 172.21.15.67:0/2651887285 wait complete. 2024-04-25T15:50:27.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.514+0000 7eff287d0640 1 Processor -- start 2024-04-25T15:50:27.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 -- start start 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff20156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff200c7b50 0x7eff20158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff2015d710 0x7eff20143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7eff20159db0 con 0x7eff20071b20 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff20159c30 con 0x7eff2015d710 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff20159f30 con 0x7eff200c7b50 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff20156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff20156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52466/0 (socket says 172.21.15.67:52466) 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 -- 172.21.15.67:0/923923676 learned_addr learned my addr 172.21.15.67:0/923923676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:27.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff26d6d640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff2015d710 0x7eff20143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff25d6b640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff200c7b50 0x7eff20158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 -- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff200c7b50 msgr2=0x7eff20158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff200c7b50 0x7eff20158a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 -- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff2015d710 msgr2=0x7eff20143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff2015d710 0x7eff20143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 -- 172.21.15.67:0/923923676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2014c430 con 0x7eff20071b20 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff26d6d640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff2015d710 0x7eff20143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff2656c640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff20156190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0ad040 tx=0x7eff1c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff16ffd640 1 -- 172.21.15.67:0/923923676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0be3f0 con 0x7eff20071b20 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff16ffd640 1 -- 172.21.15.67:0/923923676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff1c0be590 con 0x7eff20071b20 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff20149f20 con 0x7eff20071b20 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff16ffd640 1 -- 172.21.15.67:0/923923676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0be980 con 0x7eff20071b20 2024-04-25T15:50:27.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2014bc30 con 0x7eff20071b20 2024-04-25T15:50:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff2006a480 con 0x7eff20071b20 2024-04-25T15:50:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.518+0000 7eff16ffd640 1 -- 172.21.15.67:0/923923676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7eff1c0c7440 con 0x7eff20071b20 2024-04-25T15:50:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.522+0000 7eff16ffd640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff0406f430 0x7eff04071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.522+0000 7eff16ffd640 1 -- 172.21.15.67:0/923923676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7eff1c129320 con 0x7eff20071b20 2024-04-25T15:50:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.522+0000 7eff25d6b640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff0406f430 0x7eff04071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.522+0000 7eff25d6b640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff0406f430 0x7eff04071940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7eff20072d80 tx=0x7eff10008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.526+0000 7eff16ffd640 1 -- 172.21.15.67:0/923923676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7eff1c016110 con 0x7eff20071b20 2024-04-25T15:50:27.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.682+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff2014cc60 con 0x7eff20071b20 2024-04-25T15:50:27.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.682+0000 7eff16ffd640 1 -- 172.21.15.67:0/923923676 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7eff1c0f67a0 con 0x7eff20071b20 2024-04-25T15:50:27.684 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff0406f430 msgr2=0x7eff04071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff0406f430 0x7eff04071940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7eff20072d80 tx=0x7eff10008040 comp rx=0 tx=0).stop 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 msgr2=0x7eff20156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff20156190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0ad040 tx=0x7eff1c0192c0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 shutdown_connections 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff0406f430 0x7eff04071940 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff2015d710 0x7eff20143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff200c7b50 0x7eff20158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 --2- 172.21.15.67:0/923923676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff20071b20 0x7eff20156190 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 >> 172.21.15.67:0/923923676 conn(0x7eff2006fe70 msgr2=0x7eff2006c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 shutdown_connections 2024-04-25T15:50:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.686+0000 7eff287d0640 1 -- 172.21.15.67:0/923923676 wait complete. 2024-04-25T15:50:27.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:50:27.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd768e11640 1 Processor -- start 2024-04-25T15:50:27.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd768e11640 1 -- start start 2024-04-25T15:50:27.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd768e11640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd764074370 0x7fd764074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd768e11640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7640c92d0 con 0x7fd7640c7fc0 2024-04-25T15:50:27.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd768e11640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd76406e210 con 0x7fd76406c640 2024-04-25T15:50:27.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd768e11640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd76406e4d0 con 0x7fd764074370 2024-04-25T15:50:27.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd762d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd7640c7fc0 0x7fd764164f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49286/0 (socket says 172.21.15.67:49286) 2024-04-25T15:50:27.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd762d9d640 1 -- 172.21.15.67:0/533259038 learned_addr learned my addr 172.21.15.67:0/533259038 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:27.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd761d9b640 1 --2- 172.21.15.67:0/533259038 >> v2:172.21.15.67:3300/0 conn(0x7fd764074370 0x7fd764074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd76159a640 1 -- 172.21.15.67:0/533259038 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367657599 0 0) 0x7fd7640c92d0 con 0x7fd7640c7fc0 2024-04-25T15:50:27.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd76159a640 1 -- 172.21.15.67:0/533259038 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd74c003530 con 0x7fd7640c7fc0 2024-04-25T15:50:27.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd761d9b640 1 -- 172.21.15.67:0/533259038 >> v1:172.21.15.176:6789/0 conn(0x7fd76406c640 legacy=0x7fd76415b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd761d9b640 1 -- 172.21.15.67:0/533259038 >> v1:172.21.15.67:6789/0 conn(0x7fd7640c7fc0 legacy=0x7fd764164f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.850+0000 7fd761d9b640 1 -- 172.21.15.67:0/533259038 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76406e7b0 con 0x7fd764074370 2024-04-25T15:50:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd761d9b640 1 --2- 172.21.15.67:0/533259038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd764074780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd758009d90 tx=0x7fd75801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca9661278356516 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd76159a640 1 -- 172.21.15.67:0/533259038 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7580b4070 con 0x7fd764074370 2024-04-25T15:50:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd76159a640 1 -- 172.21.15.67:0/533259038 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7580af600 con 0x7fd764074370 2024-04-25T15:50:27.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd76159a640 1 -- 172.21.15.67:0/533259038 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7580af9d0 con 0x7fd764074370 2024-04-25T15:50:27.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd768e11640 1 -- 172.21.15.67:0/533259038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 msgr2=0x7fd764074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd768e11640 1 --2- 172.21.15.67:0/533259038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd764074780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd758009d90 tx=0x7fd75801f150 comp rx=0 tx=0).stop 2024-04-25T15:50:27.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd768e11640 1 -- 172.21.15.67:0/533259038 shutdown_connections 2024-04-25T15:50:27.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd768e11640 1 --2- 172.21.15.67:0/533259038 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd764074780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd768e11640 1 -- 172.21.15.67:0/533259038 >> 172.21.15.67:0/533259038 conn(0x7fd764070db0 msgr2=0x7fd7640ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd768e11640 1 -- 172.21.15.67:0/533259038 shutdown_connections 2024-04-25T15:50:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.854+0000 7fd768e11640 1 -- 172.21.15.67:0/533259038 wait complete. 2024-04-25T15:50:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 Processor -- start 2024-04-25T15:50:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 -- start start 2024-04-25T15:50:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd76406c640 0x7fd764156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd7641f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7640c7fc0 0x7fd7641526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd764073e50 con 0x7fd764074370 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd764073b70 con 0x7fd7640c7fc0 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd764073420 con 0x7fd76406c640 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd7641f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd7641f1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52496/0 (socket says 172.21.15.67:52496) 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 -- 172.21.15.67:0/4052996657 learned_addr learned my addr 172.21.15.67:0/4052996657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 -- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd76406c640 msgr2=0x7fd764156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd762d9d640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7640c7fc0 0x7fd7641526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd76259c640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd76406c640 0x7fd764156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd76406c640 0x7fd764156cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 -- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7640c7fc0 msgr2=0x7fd7641526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7640c7fc0 0x7fd7641526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 -- 172.21.15.67:0/4052996657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7641f19c0 con 0x7fd764074370 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd762d9d640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7640c7fc0 0x7fd7641526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd761d9b640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd7641f1480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd7580b2040 tx=0x7fd75801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd752ffd640 1 -- 172.21.15.67:0/4052996657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7580b4040 con 0x7fd764074370 2024-04-25T15:50:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd752ffd640 1 -- 172.21.15.67:0/4052996657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7580c1070 con 0x7fd764074370 2024-04-25T15:50:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76415bed0 con 0x7fd764074370 2024-04-25T15:50:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd752ffd640 1 -- 172.21.15.67:0/4052996657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758009450 con 0x7fd764074370 2024-04-25T15:50:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7641564c0 con 0x7fd764074370 2024-04-25T15:50:27.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.858+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd76406a480 con 0x7fd764074370 2024-04-25T15:50:27.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.862+0000 7fd752ffd640 1 -- 172.21.15.67:0/4052996657 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd7580ba6b0 con 0x7fd764074370 2024-04-25T15:50:27.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.862+0000 7fd752ffd640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd74406f400 0x7fd744071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:27.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.862+0000 7fd752ffd640 1 -- 172.21.15.67:0/4052996657 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd75812e250 con 0x7fd764074370 2024-04-25T15:50:27.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.862+0000 7fd76259c640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd74406f400 0x7fd744071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:27.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.862+0000 7fd76259c640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd74406f400 0x7fd744071910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd75c000f70 tx=0x7fd75c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:27.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:27.866+0000 7fd752ffd640 1 -- 172.21.15.67:0/4052996657 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd758018cb0 con 0x7fd764074370 2024-04-25T15:50:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.030+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd76406d490 con 0x7fd764074370 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.030+0000 7fd752ffd640 1 -- 172.21.15.67:0/4052996657 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7fd7580fb570 con 0x7fd764074370 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:50:28.032 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-25T15:50:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd74406f400 msgr2=0x7fd744071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd74406f400 0x7fd744071910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd75c000f70 tx=0x7fd75c009040 comp rx=0 tx=0).stop 2024-04-25T15:50:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 msgr2=0x7fd7641f1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd7641f1480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd7580b2040 tx=0x7fd75801bef0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 shutdown_connections 2024-04-25T15:50:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd74406f400 0x7fd744071910 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7640c7fc0 0x7fd7641526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd764074370 0x7fd7641f1480 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 --2- 172.21.15.67:0/4052996657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd76406c640 0x7fd764156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 >> 172.21.15.67:0/4052996657 conn(0x7fd764070db0 msgr2=0x7fd7640cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 shutdown_connections 2024-04-25T15:50:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.034+0000 7fd768e11640 1 -- 172.21.15.67:0/4052996657 wait complete. 2024-04-25T15:50:28.053 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:50:28.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:50:28.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.214+0000 7fd23e97f640 1 Processor -- start 2024-04-25T15:50:28.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.214+0000 7fd23e97f640 1 -- start start 2024-04-25T15:50:28.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd23e97f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2380ca700 0x7fd2380cee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd23e97f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd23815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd23e97f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2380cab10 con 0x7fd238157db0 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd23e97f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd23815b0b0 con 0x7fd2380ca700 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd23e97f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2381581c0 con 0x7fd23815a760 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd2377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2380ca700 0x7fd2380cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd2377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd2380ca700 0x7fd2380cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52504/0 (socket says 172.21.15.67:52504) 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd2377fe640 1 -- 172.21.15.67:0/594657191 learned_addr learned my addr 172.21.15.67:0/594657191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd23cf1c640 1 --2- 172.21.15.67:0/594657191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd23815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd2377fe640 1 -- 172.21.15.67:0/594657191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 msgr2=0x7fd23815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd2377fe640 1 --2- 172.21.15.67:0/594657191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd23815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd2377fe640 1 -- 172.21.15.67:0/594657191 >> v1:172.21.15.176:6789/0 conn(0x7fd238157db0 legacy=0x7fd2380cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd2377fe640 1 -- 172.21.15.67:0/594657191 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23815b2f0 con 0x7fd2380ca700 2024-04-25T15:50:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.218+0000 7fd23cf1c640 1 --2- 172.21.15.67:0/594657191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd23815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd2377fe640 1 --2- 172.21.15.67:0/594657191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ca700 0x7fd2380cee40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0098a0 tx=0x7fd22c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442227732c108cba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd236ffd640 1 -- 172.21.15.67:0/594657191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0ab650 con 0x7fd2380ca700 2024-04-25T15:50:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd236ffd640 1 -- 172.21.15.67:0/594657191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd22c0b0070 con 0x7fd2380ca700 2024-04-25T15:50:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd236ffd640 1 -- 172.21.15.67:0/594657191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0aba10 con 0x7fd2380ca700 2024-04-25T15:50:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 -- 172.21.15.67:0/594657191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ca700 msgr2=0x7fd2380cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 --2- 172.21.15.67:0/594657191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ca700 0x7fd2380cee40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0098a0 tx=0x7fd22c01c540 comp rx=0 tx=0).stop 2024-04-25T15:50:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 -- 172.21.15.67:0/594657191 shutdown_connections 2024-04-25T15:50:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 --2- 172.21.15.67:0/594657191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd23815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 --2- 172.21.15.67:0/594657191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2380ca700 0x7fd2380cee40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 -- 172.21.15.67:0/594657191 >> 172.21.15.67:0/594657191 conn(0x7fd238071dd0 msgr2=0x7fd2380721e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 -- 172.21.15.67:0/594657191 shutdown_connections 2024-04-25T15:50:28.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 -- 172.21.15.67:0/594657191 wait complete. 2024-04-25T15:50:28.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 Processor -- start 2024-04-25T15:50:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.222+0000 7fd23e97f640 1 -- start start 2024-04-25T15:50:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2380ca700 0x7fd2381f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd238157db0 0x7fd2380c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd2381569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2380cc210 con 0x7fd238157db0 2024-04-25T15:50:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2380ccb50 con 0x7fd2380ca700 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2380cdff0 con 0x7fd23815a760 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd238157db0 0x7fd2380c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd238157db0 0x7fd2380c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52512/0 (socket says 172.21.15.67:52512) 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 -- 172.21.15.67:0/1386293241 learned_addr learned my addr 172.21.15.67:0/1386293241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd237fff640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2380ca700 0x7fd2381f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23cf1c640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd2381569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 -- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 msgr2=0x7fd2381569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd2381569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 -- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2380ca700 msgr2=0x7fd2381f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2380ca700 0x7fd2381f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 -- 172.21.15.67:0/1386293241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2380cba40 con 0x7fd238157db0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd2377fe640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd238157db0 0x7fd2380c6080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0ad040 tx=0x7fd22c01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd234ff9640 1 -- 172.21.15.67:0/1386293241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0abe60 con 0x7fd238157db0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd234ff9640 1 -- 172.21.15.67:0/1386293241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd22c0b4820 con 0x7fd238157db0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2381f3d40 con 0x7fd238157db0 2024-04-25T15:50:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd234ff9640 1 -- 172.21.15.67:0/1386293241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0b4c10 con 0x7fd238157db0 2024-04-25T15:50:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2381f7350 con 0x7fd238157db0 2024-04-25T15:50:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd23806a480 con 0x7fd238157db0 2024-04-25T15:50:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd234ff9640 1 -- 172.21.15.67:0/1386293241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd22c0c7390 con 0x7fd238157db0 2024-04-25T15:50:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.226+0000 7fd234ff9640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd22406f400 0x7fd224071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.230+0000 7fd234ff9640 1 -- 172.21.15.67:0/1386293241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd22c129700 con 0x7fd238157db0 2024-04-25T15:50:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.230+0000 7fd237fff640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd22406f400 0x7fd224071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.230+0000 7fd237fff640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd22406f400 0x7fd224071910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2280015b0 tx=0x7fd228009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.234+0000 7fd234ff9640 1 -- 172.21.15.67:0/1386293241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd22c0c4060 con 0x7fd238157db0 2024-04-25T15:50:28.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:50:28.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:50:28.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:28.398+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd2380ce380 con 0x7fd238157db0 2024-04-25T15:50:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.134+0000 7fd234ff9640 1 -- 172.21.15.67:0/1386293241 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fd22c016130 con 0x7fd238157db0 2024-04-25T15:50:29.136 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-25T15:50:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd22406f400 msgr2=0x7fd224071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd22406f400 0x7fd224071910 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2280015b0 tx=0x7fd228009040 comp rx=0 tx=0).stop 2024-04-25T15:50:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd238157db0 msgr2=0x7fd2380c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd238157db0 0x7fd2380c6080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0ad040 tx=0x7fd22c01ad80 comp rx=0 tx=0).stop 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 shutdown_connections 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd22406f400 0x7fd224071910 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd23815a760 0x7fd2381569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd238157db0 0x7fd2380c6080 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 --2- 172.21.15.67:0/1386293241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2380ca700 0x7fd2381f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 >> 172.21.15.67:0/1386293241 conn(0x7fd238071dd0 msgr2=0x7fd2380cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 shutdown_connections 2024-04-25T15:50:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.138+0000 7fd23e97f640 1 -- 172.21.15.67:0/1386293241 wait complete. 2024-04-25T15:50:29.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:50:29.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f331080a640 1 Processor -- start 2024-04-25T15:50:29.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f331080a640 1 -- start start 2024-04-25T15:50:29.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f331080a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 0x7f330c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f331080a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f330c0caa60 con 0x7f330c06d460 2024-04-25T15:50:29.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f331080a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f330c15a5f0 con 0x7f330c06cb80 2024-04-25T15:50:29.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f331080a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f330c068590 con 0x7f330c0736d0 2024-04-25T15:50:29.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f330c06d460 0x7f330c1547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49296/0 (socket says 172.21.15.67:49296) 2024-04-25T15:50:29.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330a59c640 1 -- 172.21.15.67:0/900229678 learned_addr learned my addr 172.21.15.67:0/900229678 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:29.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f3309d9b640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 0x7f330c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1801975966 0 0) 0x7f330c0caa60 con 0x7f330c06d460 2024-04-25T15:50:29.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32f4003530 con 0x7f330c06d460 2024-04-25T15:50:29.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1139820759 0 0) 0x7f32f4003530 con 0x7f330c06d460 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 msgr2=0x7f330c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 0x7f330c06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 >> v1:172.21.15.176:6789/0 conn(0x7f330c06cb80 legacy=0x7f330c15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c0ca100 con 0x7f330c06d460 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f32fc001680 con 0x7f330c06d460 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 >> v1:172.21.15.67:6789/0 conn(0x7f330c06d460 legacy=0x7f330c1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f40039f0 0x7f32f4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 0x7f32f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4004d50 0x7f32f4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f330c068590 con 0x7f32f4004d50 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f32f400bab0 con 0x7f32f40039f0 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f32f400bcb0 con 0x7f330c0736d0 2024-04-25T15:50:29.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330ad9d640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4004d50 0x7f32f4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f330a59c640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f40039f0 0x7f32f4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.378+0000 7f3309d9b640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 0x7f32f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330ad9d640 1 -- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 msgr2=0x7f32f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330ad9d640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c0736d0 0x7f32f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330ad9d640 1 -- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f40039f0 msgr2=0x7f32f4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330ad9d640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32f40039f0 0x7f32f4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330ad9d640 1 -- 172.21.15.67:0/900229678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f4003530 con 0x7f32f4004d50 2024-04-25T15:50:29.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330ad9d640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4004d50 0x7f32f4005160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f330c154f10 tx=0x7f330401c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc380cef56d5be1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:29.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33040ac070 con 0x7f32f4004d50 2024-04-25T15:50:29.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33040a6630 con 0x7f32f4004d50 2024-04-25T15:50:29.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330959a640 1 -- 172.21.15.67:0/900229678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304001b60 con 0x7f32f4004d50 2024-04-25T15:50:29.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f331080a640 1 -- 172.21.15.67:0/900229678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4004d50 msgr2=0x7f32f4005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:29.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f331080a640 1 --2- 172.21.15.67:0/900229678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32f4004d50 0x7f32f4005160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f330c154f10 tx=0x7f330401c840 comp rx=0 tx=0).stop 2024-04-25T15:50:29.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f330ad9d640 1 -- 172.21.15.67:0/900229678 reap_dead start 2024-04-25T15:50:29.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f331080a640 1 -- 172.21.15.67:0/900229678 shutdown_connections 2024-04-25T15:50:29.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f331080a640 1 -- 172.21.15.67:0/900229678 >> 172.21.15.67:0/900229678 conn(0x7f330c0ce1c0 msgr2=0x7f330c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:29.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f331080a640 1 -- 172.21.15.67:0/900229678 shutdown_connections 2024-04-25T15:50:29.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.382+0000 7f331080a640 1 -- 172.21.15.67:0/900229678 wait complete. 2024-04-25T15:50:29.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 Processor -- start 2024-04-25T15:50:29.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 -- start start 2024-04-25T15:50:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1f0ec0 0x7f330c1f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c1f4250 0x7f330c0b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0c5bb0 0x7f330c0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f330c068ac0 con 0x7f330c1f0ec0 2024-04-25T15:50:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f330c0687e0 con 0x7f330c0c5bb0 2024-04-25T15:50:29.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f330c15b200 con 0x7f330c1f4250 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1f0ec0 0x7f330c1f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1f0ec0 0x7f330c1f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52542/0 (socket says 172.21.15.67:52542) 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 -- 172.21.15.67:0/2398082979 learned_addr learned my addr 172.21.15.67:0/2398082979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f3309d9b640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c1f4250 0x7f330c0b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330ad9d640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0c5bb0 0x7f330c0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 -- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c1f4250 msgr2=0x7f330c0b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c1f4250 0x7f330c0b2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 -- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0c5bb0 msgr2=0x7f330c0c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0c5bb0 0x7f330c0c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 -- 172.21.15.67:0/2398082979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c0c82b0 con 0x7f330c1f0ec0 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f330a59c640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1f0ec0 0x7f330c1f1740 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32fc004850 tx=0x7f32fc011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f32faffd640 1 -- 172.21.15.67:0/2398082979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc010040 con 0x7f330c1f0ec0 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f32faffd640 1 -- 172.21.15.67:0/2398082979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32fc01a070 con 0x7f330c1f0ec0 2024-04-25T15:50:29.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330c0c8590 con 0x7f330c1f0ec0 2024-04-25T15:50:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f32faffd640 1 -- 172.21.15.67:0/2398082979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc00fda0 con 0x7f330c1f0ec0 2024-04-25T15:50:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.386+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330c1f06c0 con 0x7f330c1f0ec0 2024-04-25T15:50:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.390+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f330c06a480 con 0x7f330c1f0ec0 2024-04-25T15:50:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.390+0000 7f32faffd640 1 -- 172.21.15.67:0/2398082979 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f32fc01f460 con 0x7f330c1f0ec0 2024-04-25T15:50:29.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.390+0000 7f32faffd640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32ec06f400 0x7f32ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.390+0000 7f32faffd640 1 -- 172.21.15.67:0/2398082979 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f32fc092cf0 con 0x7f330c1f0ec0 2024-04-25T15:50:29.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.390+0000 7f3309d9b640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32ec06f400 0x7f32ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:29.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.390+0000 7f3309d9b640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32ec06f400 0x7f32ec071910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3300009a10 tx=0x7f3300008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:29.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.394+0000 7f32faffd640 1 -- 172.21.15.67:0/2398082979 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f32fc05ff30 con 0x7f330c1f0ec0 2024-04-25T15:50:29.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:29.558+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f330c0cce40 con 0x7f330c1f0ec0 2024-04-25T15:50:30.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.146+0000 7f32faffd640 1 -- 172.21.15.67:0/2398082979 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f32fc05f830 con 0x7f330c1f0ec0 2024-04-25T15:50:30.148 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-25T15:50:30.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32ec06f400 msgr2=0x7f32ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:30.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32ec06f400 0x7f32ec071910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3300009a10 tx=0x7f3300008040 comp rx=0 tx=0).stop 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1f0ec0 msgr2=0x7f330c1f1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1f0ec0 0x7f330c1f1740 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32fc004850 tx=0x7f32fc011ca0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 shutdown_connections 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32ec06f400 0x7f32ec071910 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0c5bb0 0x7f330c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f330c1f4250 0x7f330c0b2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 --2- 172.21.15.67:0/2398082979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f330c1f0ec0 0x7f330c1f1740 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 >> 172.21.15.67:0/2398082979 conn(0x7f330c0ce1c0 msgr2=0x7f330c0cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 shutdown_connections 2024-04-25T15:50:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.150+0000 7f331080a640 1 -- 172.21.15.67:0/2398082979 wait complete. 2024-04-25T15:50:30.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:50:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 Processor -- start 2024-04-25T15:50:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 -- start start 2024-04-25T15:50:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f497c071b20 0x7f497c15d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f497c15d710 0x7f497c0c7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c0c7b50 0x7f497c0c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f497c15cd40 con 0x7f497c15d710 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f497c0c6890 con 0x7f497c071b20 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4981638640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f497c0c6b70 con 0x7f497c0c7b50 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f497c15d710 0x7f497c0c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f497c15d710 0x7f497c0c7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52550/0 (socket says 172.21.15.67:52550) 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 -- 172.21.15.67:0/557896184 learned_addr learned my addr 172.21.15.67:0/557896184 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f497b7fe640 1 --2- 172.21.15.67:0/557896184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c0c7b50 0x7f497c0c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f497affd640 1 --2- 172.21.15.67:0/557896184 >> v2:172.21.15.176:3300/0 conn(0x7f497c071b20 0x7f497c15d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 -- 172.21.15.67:0/557896184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c0c7b50 msgr2=0x7f497c0c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 --2- 172.21.15.67:0/557896184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c0c7b50 0x7f497c0c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 -- 172.21.15.67:0/557896184 >> v2:172.21.15.176:3300/0 conn(0x7f497c071b20 msgr2=0x7f497c15d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 --2- 172.21.15.67:0/557896184 >> v2:172.21.15.176:3300/0 conn(0x7f497c071b20 0x7f497c15d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.354+0000 7f4973fff640 1 -- 172.21.15.67:0/557896184 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c0c6e50 con 0x7f497c15d710 2024-04-25T15:50:30.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4973fff640 1 --2- 172.21.15.67:0/557896184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c0c7610 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f497c070a50 tx=0x7f496401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdadc9df42c7840 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:30.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f497a7fc640 1 -- 172.21.15.67:0/557896184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49640ab5f0 con 0x7f497c15d710 2024-04-25T15:50:30.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f497a7fc640 1 -- 172.21.15.67:0/557896184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49640b0070 con 0x7f497c15d710 2024-04-25T15:50:30.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f497a7fc640 1 -- 172.21.15.67:0/557896184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49640ab9b0 con 0x7f497c15d710 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 -- 172.21.15.67:0/557896184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 msgr2=0x7f497c0c7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 --2- 172.21.15.67:0/557896184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c0c7610 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f497c070a50 tx=0x7f496401c520 comp rx=0 tx=0).stop 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 -- 172.21.15.67:0/557896184 shutdown_connections 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 --2- 172.21.15.67:0/557896184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c0c7b50 0x7f497c0c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 --2- 172.21.15.67:0/557896184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c0c7610 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 --2- 172.21.15.67:0/557896184 >> v2:172.21.15.176:3300/0 conn(0x7f497c071b20 0x7f497c15d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 -- 172.21.15.67:0/557896184 >> 172.21.15.67:0/557896184 conn(0x7f497c06fe70 msgr2=0x7f497c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 -- 172.21.15.67:0/557896184 shutdown_connections 2024-04-25T15:50:30.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 -- 172.21.15.67:0/557896184 wait complete. 2024-04-25T15:50:30.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.358+0000 7f4981638640 1 Processor -- start 2024-04-25T15:50:30.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 -- start start 2024-04-25T15:50:30.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c071b20 0x7f497c1f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:30.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f497c0c7b50 0x7f497c14a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:30.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c14c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:30.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f497c159db0 con 0x7f497c15d710 2024-04-25T15:50:30.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f497c159c30 con 0x7f497c0c7b50 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f497c159f30 con 0x7f497c071b20 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c14c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c14c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52554/0 (socket says 172.21.15.67:52554) 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 -- 172.21.15.67:0/658517967 learned_addr learned my addr 172.21.15.67:0/658517967 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497affd640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c071b20 0x7f497c1f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4973fff640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f497c0c7b50 0x7f497c14a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 -- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c071b20 msgr2=0x7f497c1f6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c071b20 0x7f497c1f6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 -- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f497c0c7b50 msgr2=0x7f497c14a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f497c0c7b50 0x7f497c14a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 -- 172.21.15.67:0/658517967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c1f70e0 con 0x7f497c15d710 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f497b7fe640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c14c010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f497400d500 tx=0x7f497400d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f49737fe640 1 -- 172.21.15.67:0/658517967 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49740125f0 con 0x7f497c15d710 2024-04-25T15:50:30.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f49737fe640 1 -- 172.21.15.67:0/658517967 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4974019070 con 0x7f497c15d710 2024-04-25T15:50:30.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c1fa100 con 0x7f497c15d710 2024-04-25T15:50:30.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f49737fe640 1 -- 172.21.15.67:0/658517967 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49740129d0 con 0x7f497c15d710 2024-04-25T15:50:30.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c1fa6c0 con 0x7f497c15d710 2024-04-25T15:50:30.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.362+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c06a480 con 0x7f497c15d710 2024-04-25T15:50:30.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.366+0000 7f49737fe640 1 -- 172.21.15.67:0/658517967 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4974012b70 con 0x7f497c15d710 2024-04-25T15:50:30.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.366+0000 7f49737fe640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f495c06f400 0x7f495c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:30.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.366+0000 7f49737fe640 1 -- 172.21.15.67:0/658517967 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f4974090070 con 0x7f497c15d710 2024-04-25T15:50:30.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.366+0000 7f497affd640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f495c06f400 0x7f495c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:30.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.366+0000 7f497affd640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f495c06f400 0x7f495c071910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f497c0c5960 tx=0x7f496c003a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:30.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.370+0000 7f49737fe640 1 -- 172.21.15.67:0/658517967 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f497405d0e0 con 0x7f497c15d710 2024-04-25T15:50:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:30.566+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f497c1f9e90 con 0x7f497c15d710 2024-04-25T15:50:31.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.194+0000 7f49737fe640 1 -- 172.21.15.67:0/658517967 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f497405c9e0 con 0x7f497c15d710 2024-04-25T15:50:31.196 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T15:50:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f495c06f400 msgr2=0x7f495c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f495c06f400 0x7f495c071910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f497c0c5960 tx=0x7f496c003a40 comp rx=0 tx=0).stop 2024-04-25T15:50:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 msgr2=0x7f497c14c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c14c010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f497400d500 tx=0x7f497400d9c0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 shutdown_connections 2024-04-25T15:50:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f495c06f400 0x7f495c071910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f497c15d710 0x7f497c14c010 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f497c0c7b50 0x7f497c14a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 --2- 172.21.15.67:0/658517967 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f497c071b20 0x7f497c1f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.198+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 >> 172.21.15.67:0/658517967 conn(0x7f497c06fe70 msgr2=0x7f497c0ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.202+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 shutdown_connections 2024-04-25T15:50:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.202+0000 7f4981638640 1 -- 172.21.15.67:0/658517967 wait complete. 2024-04-25T15:50:31.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:31.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13e5db0640 1 Processor -- start 2024-04-25T15:50:31.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13e5db0640 1 -- start start 2024-04-25T15:50:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13e5db0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13e0068840 0x7f13e00c7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13e5db0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e00cb320 con 0x7f13e0162e80 2024-04-25T15:50:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13e5db0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e00732d0 con 0x7f13e00c7bd0 2024-04-25T15:50:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13e5db0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f13e00cb5c0 con 0x7f13e0068840 2024-04-25T15:50:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13df7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13e0068840 0x7f13e00c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13df7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13e0068840 0x7f13e00c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52568/0 (socket says 172.21.15.67:52568) 2024-04-25T15:50:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13df7fe640 1 -- 172.21.15.67:0/1149384301 learned_addr learned my addr 172.21.15.67:0/1149384301 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13dffff640 1 --1- 172.21.15.67:0/1149384301 >> v1:172.21.15.67:6789/0 conn(0x7f13e0162e80 0x7f13e01652a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49304/0 (socket says 172.21.15.67:49304) 2024-04-25T15:50:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13df7fe640 1 -- 172.21.15.67:0/1149384301 >> v1:172.21.15.176:6789/0 conn(0x7f13e00c7bd0 legacy=0x7f13e015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13df7fe640 1 -- 172.21.15.67:0/1149384301 >> v1:172.21.15.67:6789/0 conn(0x7f13e0162e80 legacy=0x7f13e01652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13deffd640 1 -- 172.21.15.67:0/1149384301 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3858783840 0 0) 0x7f13e00cb320 con 0x7f13e0162e80 2024-04-25T15:50:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.378+0000 7f13df7fe640 1 -- 172.21.15.67:0/1149384301 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e00cb8a0 con 0x7f13e0068840 2024-04-25T15:50:31.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13df7fe640 1 --2- 172.21.15.67:0/1149384301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c7690 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18714b210a31f543 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13deffd640 1 -- 172.21.15.67:0/1149384301 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00b0070 con 0x7f13e0068840 2024-04-25T15:50:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13deffd640 1 -- 172.21.15.67:0/1149384301 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d00ab600 con 0x7f13e0068840 2024-04-25T15:50:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13deffd640 1 -- 172.21.15.67:0/1149384301 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00ab9d0 con 0x7f13e0068840 2024-04-25T15:50:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 -- 172.21.15.67:0/1149384301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 msgr2=0x7f13e00c7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 --2- 172.21.15.67:0/1149384301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c7690 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d001c560 comp rx=0 tx=0).stop 2024-04-25T15:50:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 -- 172.21.15.67:0/1149384301 shutdown_connections 2024-04-25T15:50:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 --2- 172.21.15.67:0/1149384301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c7690 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 -- 172.21.15.67:0/1149384301 >> 172.21.15.67:0/1149384301 conn(0x7f13e0069220 msgr2=0x7f13e00cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 -- 172.21.15.67:0/1149384301 shutdown_connections 2024-04-25T15:50:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 -- 172.21.15.67:0/1149384301 wait complete. 2024-04-25T15:50:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 Processor -- start 2024-04-25T15:50:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.382+0000 7f13e5db0640 1 -- start start 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13e00c7bd0 0x7f13e01f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13e0162e80 0x7f13e01406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13e00725d0 con 0x7f13e0068840 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f13e00ca120 con 0x7f13e00c7bd0 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f13e00c8980 con 0x7f13e0162e80 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52572/0 (socket says 172.21.15.67:52572) 2024-04-25T15:50:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 -- 172.21.15.67:0/3375430719 learned_addr learned my addr 172.21.15.67:0/3375430719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13dffff640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13e0162e80 0x7f13e01406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13d7fff640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13e00c7bd0 0x7f13e01f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 -- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13e0162e80 msgr2=0x7f13e01406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13e0162e80 0x7f13e01406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 -- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13e00c7bd0 msgr2=0x7f13e01f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13e00c7bd0 0x7f13e01f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 -- 172.21.15.67:0/3375430719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e00c60e0 con 0x7f13e0068840 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13df7fe640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c6410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13dcff9640 1 -- 172.21.15.67:0/3375430719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00bc070 con 0x7f13e0068840 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13dcff9640 1 -- 172.21.15.67:0/3375430719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d0004030 con 0x7f13e0068840 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e01f70f0 con 0x7f13e0068840 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13dcff9640 1 -- 172.21.15.67:0/3375430719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00b0040 con 0x7f13e0068840 2024-04-25T15:50:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e0144ec0 con 0x7f13e0068840 2024-04-25T15:50:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e006a480 con 0x7f13e0068840 2024-04-25T15:50:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.386+0000 7f13dcff9640 1 -- 172.21.15.67:0/3375430719 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f13d00bf070 con 0x7f13e0068840 2024-04-25T15:50:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.390+0000 7f13dcff9640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f13c006f430 0x7f13c0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.390+0000 7f13dcff9640 1 -- 172.21.15.67:0/3375430719 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f13d0128d20 con 0x7f13e0068840 2024-04-25T15:50:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.390+0000 7f13d7fff640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f13c006f430 0x7f13c0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.390+0000 7f13d7fff640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f13c006f430 0x7f13c0071940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13e0150750 tx=0x7f13c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:31.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.394+0000 7f13dcff9640 1 -- 172.21.15.67:0/3375430719 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f13d00c6080 con 0x7f13e0068840 2024-04-25T15:50:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.558+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13e0072750 con 0x7f13e0068840 2024-04-25T15:50:31.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13dcff9640 1 -- 172.21.15.67:0/3375430719 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16582 (secure 0 0 0) 0x7f13d0016150 con 0x7f13e0068840 2024-04-25T15:50:31.562 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:31.562 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:30.570747+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f13c006f430 msgr2=0x7f13c0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f13c006f430 0x7f13c0071940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13e0150750 tx=0x7f13c8009040 comp rx=0 tx=0).stop 2024-04-25T15:50:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 msgr2=0x7f13e00c6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c6410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009870 tx=0x7f13d001ada0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 shutdown_connections 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f13c006f430 0x7f13c0071940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13e0162e80 0x7f13e01406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13e00c7bd0 0x7f13e01f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 --2- 172.21.15.67:0/3375430719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13e0068840 0x7f13e00c6410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 >> 172.21.15.67:0/3375430719 conn(0x7f13e0069220 msgr2=0x7f13e00ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 shutdown_connections 2024-04-25T15:50:31.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.562+0000 7f13e5db0640 1 -- 172.21.15.67:0/3375430719 wait complete. 2024-04-25T15:50:31.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb134308640 1 Processor -- start 2024-04-25T15:50:31.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb134308640 1 -- start start 2024-04-25T15:50:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb134308640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb12c06c640 0x7fb12c06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb134308640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12c158d20 con 0x7fb12c0cdfb0 2024-04-25T15:50:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb134308640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12c158fe0 con 0x7fb12c152b00 2024-04-25T15:50:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb134308640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb12c0722c0 con 0x7fb12c06c640 2024-04-25T15:50:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1328a5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb12c0cdfb0 0x7fb12c0c8680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49314/0 (socket says 172.21.15.67:49314) 2024-04-25T15:50:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1328a5640 1 -- 172.21.15.67:0/3535087702 learned_addr learned my addr 172.21.15.67:0/3535087702 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:31.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1318a3640 1 --2- 172.21.15.67:0/3535087702 >> v2:172.21.15.176:3300/0 conn(0x7fb12c06c640 0x7fb12c06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886868105 0 0) 0x7fb12c158d20 con 0x7fb12c0cdfb0 2024-04-25T15:50:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb118003530 con 0x7fb12c0cdfb0 2024-04-25T15:50:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 963605850 0 0) 0x7fb118003530 con 0x7fb12c0cdfb0 2024-04-25T15:50:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 >> v2:172.21.15.176:3300/0 conn(0x7fb12c06c640 msgr2=0x7fb12c06ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 --2- 172.21.15.67:0/3535087702 >> v2:172.21.15.176:3300/0 conn(0x7fb12c06c640 0x7fb12c06ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 >> v1:172.21.15.176:6789/0 conn(0x7fb12c152b00 legacy=0x7fb12c0c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c1592c0 con 0x7fb12c0cdfb0 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fb124001a00 con 0x7fb12c0cdfb0 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 >> v1:172.21.15.67:6789/0 conn(0x7fb12c0cdfb0 legacy=0x7fb12c0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1180039c0 0x7fb118003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb118004810 0x7fb118004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180051d0 0x7fb11800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c0722c0 con 0x7fb1180039c0 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb11800c170 con 0x7fb1180051d0 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb11800c370 con 0x7fb118004810 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1180039c0 0x7fb118003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1318a3640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb118004810 0x7fb118004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1328a5640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180051d0 0x7fb11800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1320a4640 1 -- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb118004810 msgr2=0x7fb118004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb118004810 0x7fb118004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1320a4640 1 -- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180051d0 msgr2=0x7fb11800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:31.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180051d0 0x7fb11800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1320a4640 1 -- 172.21.15.67:0/3535087702 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb118003530 con 0x7fb1180039c0 2024-04-25T15:50:31.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.734+0000 7fb1328a5640 1 -- 172.21.15.67:0/3535087702 reap_dead start 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1180039c0 0x7fb118003e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb128001780 tx=0x7fb12801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68df7c752d4b8ca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1280013a0 con 0x7fb1180039c0 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1280052f0 con 0x7fb1180039c0 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb1310a2640 1 -- 172.21.15.67:0/3535087702 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128003840 con 0x7fb1180039c0 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb134308640 1 -- 172.21.15.67:0/3535087702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1180039c0 msgr2=0x7fb118003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb134308640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1180039c0 0x7fb118003e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb128001780 tx=0x7fb12801c340 comp rx=0 tx=0).stop 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb134308640 1 -- 172.21.15.67:0/3535087702 shutdown_connections 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb134308640 1 --2- 172.21.15.67:0/3535087702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1180039c0 0x7fb118003e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb134308640 1 -- 172.21.15.67:0/3535087702 >> 172.21.15.67:0/3535087702 conn(0x7fb12c06fe70 msgr2=0x7fb12c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb134308640 1 -- 172.21.15.67:0/3535087702 shutdown_connections 2024-04-25T15:50:31.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.738+0000 7fb134308640 1 -- 172.21.15.67:0/3535087702 wait complete. 2024-04-25T15:50:31.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 Processor -- start 2024-04-25T15:50:31.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 -- start start 2024-04-25T15:50:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180039c0 0x7fb12c158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb12c073930 0x7fb12c148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb12c151a60 0x7fb12c151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c152f10 con 0x7fb12c151a60 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c0ca100 con 0x7fb1180039c0 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb12c0c8680 con 0x7fb12c073930 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb12c151a60 0x7fb12c151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb12c151a60 0x7fb12c151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52600/0 (socket says 172.21.15.67:52600) 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 -- 172.21.15.67:0/3272287647 learned_addr learned my addr 172.21.15.67:0/3272287647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1318a3640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb12c073930 0x7fb12c148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180039c0 0x7fb12c158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 -- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb12c073930 msgr2=0x7fb12c148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb12c073930 0x7fb12c148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 -- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180039c0 msgr2=0x7fb12c158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180039c0 0x7fb12c158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 -- 172.21.15.67:0/3272287647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c1f3ee0 con 0x7fb12c151a60 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180039c0 0x7fb12c158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1318a3640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb12c073930 0x7fb12c148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:31.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb1328a5640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb12c151a60 0x7fb12c151e70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb12c074e50 tx=0x7fb124007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb122ffd640 1 -- 172.21.15.67:0/3272287647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12400ea80 con 0x7fb12c151a60 2024-04-25T15:50:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb122ffd640 1 -- 172.21.15.67:0/3272287647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12400ec40 con 0x7fb12c151a60 2024-04-25T15:50:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12c1f41c0 con 0x7fb12c151a60 2024-04-25T15:50:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb122ffd640 1 -- 172.21.15.67:0/3272287647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124014430 con 0x7fb12c151a60 2024-04-25T15:50:31.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.742+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12c1f3bb0 con 0x7fb12c151a60 2024-04-25T15:50:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.746+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12c076730 con 0x7fb12c151a60 2024-04-25T15:50:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.746+0000 7fb122ffd640 1 -- 172.21.15.67:0/3272287647 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb1240145f0 con 0x7fb12c151a60 2024-04-25T15:50:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.746+0000 7fb122ffd640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb11006f400 0x7fb110071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.746+0000 7fb122ffd640 1 -- 172.21.15.67:0/3272287647 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fb12408efb0 con 0x7fb12c151a60 2024-04-25T15:50:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.746+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb11006f400 0x7fb110071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.746+0000 7fb1320a4640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb11006f400 0x7fb110071910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb128001750 tx=0x7fb1280011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:31.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.750+0000 7fb122ffd640 1 -- 172.21.15.67:0/3272287647 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb12405bfc0 con 0x7fb12c151a60 2024-04-25T15:50:31.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.914+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb12c06a480 con 0x7fb12c151a60 2024-04-25T15:50:31.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.914+0000 7fb122ffd640 1 -- 172.21.15.67:0/3272287647 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16582 (secure 0 0 0) 0x7fb12405b8c0 con 0x7fb12c151a60 2024-04-25T15:50:31.916 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:31.916 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:30.570747+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:31.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb11006f400 msgr2=0x7fb110071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb11006f400 0x7fb110071910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb128001750 tx=0x7fb1280011a0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb12c151a60 msgr2=0x7fb12c151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb12c151a60 0x7fb12c151e70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb12c074e50 tx=0x7fb124007bc0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 shutdown_connections 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb11006f400 0x7fb110071910 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb12c151a60 0x7fb12c151e70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb12c073930 0x7fb12c148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 --2- 172.21.15.67:0/3272287647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1180039c0 0x7fb12c158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 >> 172.21.15.67:0/3272287647 conn(0x7fb12c06fe70 msgr2=0x7fb12c14e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 shutdown_connections 2024-04-25T15:50:31.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:31.918+0000 7fb134308640 1 -- 172.21.15.67:0/3272287647 wait complete. 2024-04-25T15:50:31.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:50:32.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec444e7640 1 Processor -- start 2024-04-25T15:50:32.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec444e7640 1 -- start start 2024-04-25T15:50:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec444e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c15d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec444e7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec3c0c5bb0 0x7fec3c0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec444e7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec3c0c6890 con 0x7fec3c15d710 2024-04-25T15:50:32.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec444e7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fec3c15cd40 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec444e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fec3c0c6b70 con 0x7fec3c071b20 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42a84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec3c0c5bb0 0x7fec3c0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42a84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec3c0c5bb0 0x7fec3c0c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52604/0 (socket says 172.21.15.67:52604) 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42a84640 1 -- 172.21.15.67:0/437899000 learned_addr learned my addr 172.21.15.67:0/437899000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42283640 1 --2- 172.21.15.67:0/437899000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c15d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42a84640 1 -- 172.21.15.67:0/437899000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 msgr2=0x7fec3c15d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42a84640 1 --2- 172.21.15.67:0/437899000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c15d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42a84640 1 -- 172.21.15.67:0/437899000 >> v1:172.21.15.176:6789/0 conn(0x7fec3c15d710 legacy=0x7fec3c0c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42a84640 1 -- 172.21.15.67:0/437899000 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c0c6e50 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.090+0000 7fec42283640 1 --2- 172.21.15.67:0/437899000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c15d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:32.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec42a84640 1 --2- 172.21.15.67:0/437899000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c0c5fc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec3400d150 tx=0x7fec3401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6456c4b967ea811 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec41281640 1 -- 172.21.15.67:0/437899000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3400b2d0 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec41281640 1 -- 172.21.15.67:0/437899000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec3400b470 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec41281640 1 -- 172.21.15.67:0/437899000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec340a7b30 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 -- 172.21.15.67:0/437899000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 msgr2=0x7fec3c0c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 --2- 172.21.15.67:0/437899000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c0c5fc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec3400d150 tx=0x7fec3401ec30 comp rx=0 tx=0).stop 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 -- 172.21.15.67:0/437899000 shutdown_connections 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 --2- 172.21.15.67:0/437899000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c0c5fc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 --2- 172.21.15.67:0/437899000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c15d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 -- 172.21.15.67:0/437899000 >> 172.21.15.67:0/437899000 conn(0x7fec3c06fe70 msgr2=0x7fec3c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 -- 172.21.15.67:0/437899000 shutdown_connections 2024-04-25T15:50:32.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.094+0000 7fec444e7640 1 -- 172.21.15.67:0/437899000 wait complete. 2024-04-25T15:50:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 Processor -- start 2024-04-25T15:50:32.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 -- start start 2024-04-25T15:50:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c14abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec3c15d710 0x7fec3c148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c06ce50 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fec3c0c7d20 con 0x7fec3c15d710 2024-04-25T15:50:32.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fec3c06cfd0 con 0x7fec3c071b20 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c15c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52612/0 (socket says 172.21.15.67:52612) 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 -- 172.21.15.67:0/2492120800 learned_addr learned my addr 172.21.15.67:0/2492120800 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec42a84640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec3c15d710 0x7fec3c148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec42283640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c14abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 -- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 msgr2=0x7fec3c14abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c14abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 -- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec3c15d710 msgr2=0x7fec3c148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec3c15d710 0x7fec3c148a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 -- 172.21.15.67:0/2492120800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3c14a6f0 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec42283640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c14abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec41a82640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c15c3a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec3800c420 tx=0x7fec3800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec32ffd640 1 -- 172.21.15.67:0/2492120800 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38017070 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec32ffd640 1 -- 172.21.15.67:0/2492120800 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec380052d0 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec3c14b780 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec32ffd640 1 -- 172.21.15.67:0/2492120800 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38003840 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.098+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3c14ba20 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.102+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec3c06a480 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.102+0000 7fec32ffd640 1 -- 172.21.15.67:0/2492120800 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fec3801a070 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.102+0000 7fec32ffd640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fec2006f430 0x7fec20071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.102+0000 7fec32ffd640 1 -- 172.21.15.67:0/2492120800 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fec3808e400 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.102+0000 7fec42283640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fec2006f430 0x7fec20071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:32.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.102+0000 7fec42283640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fec2006f430 0x7fec20071940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0c5960 tx=0x7fec2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:32.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.106+0000 7fec32ffd640 1 -- 172.21.15.67:0/2492120800 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fec38005480 con 0x7fec3c0c5bb0 2024-04-25T15:50:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:32.310+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fec3c15b050 con 0x7fec3c0c5bb0 2024-04-25T15:50:33.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.198+0000 7fec32ffd640 1 -- 172.21.15.67:0/2492120800 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7fec38063370 con 0x7fec3c0c5bb0 2024-04-25T15:50:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fec2006f430 msgr2=0x7fec20071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fec2006f430 0x7fec20071940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec3c0c5960 tx=0x7fec2c008040 comp rx=0 tx=0).stop 2024-04-25T15:50:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 msgr2=0x7fec3c15c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c15c3a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec3800c420 tx=0x7fec3800c8e0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 shutdown_connections 2024-04-25T15:50:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fec2006f430 0x7fec20071940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec3c15d710 0x7fec3c148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec3c0c5bb0 0x7fec3c15c3a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 --2- 172.21.15.67:0/2492120800 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec3c071b20 0x7fec3c14abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 >> 172.21.15.67:0/2492120800 conn(0x7fec3c06fe70 msgr2=0x7fec3c072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:33.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 shutdown_connections 2024-04-25T15:50:33.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.202+0000 7fec444e7640 1 -- 172.21.15.67:0/2492120800 wait complete. 2024-04-25T15:50:33.230 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2024-04-25T15:50:33.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T15:50:33.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b62429640 1 Processor -- start 2024-04-25T15:50:33.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b62429640 1 -- start start 2024-04-25T15:50:33.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b62429640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b5c0ca6d0 0x7f4b5c06e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b62429640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b5c0ccef0 con 0x7f4b5c0ca2c0 2024-04-25T15:50:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b62429640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b5c0cafa0 con 0x7f4b5c06e610 2024-04-25T15:50:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b62429640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4b5c15ca20 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b53fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b5c0ca6d0 0x7f4b5c06e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b5bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4b5c0ca2c0 0x7f4b5c15ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35946/0 (socket says 172.21.15.67:35946) 2024-04-25T15:50:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b5bfff640 1 -- 172.21.15.67:0/2042240568 learned_addr learned my addr 172.21.15.67:0/2042240568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:33.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b53fff640 1 -- 172.21.15.67:0/2042240568 >> v1:172.21.15.176:6789/0 conn(0x7f4b5c06e610 legacy=0x7f4b5c167040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:33.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b53fff640 1 -- 172.21.15.67:0/2042240568 >> v1:172.21.15.67:6789/0 conn(0x7f4b5c0ca2c0 legacy=0x7f4b5c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:33.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.394+0000 7f4b53fff640 1 -- 172.21.15.67:0/2042240568 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b5c0cb280 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b53fff640 1 --2- 172.21.15.67:0/2042240568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c06e0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b440097c0 tx=0x7f4b4401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a474e39fda47207c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:33.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b5b7fe640 1 -- 172.21.15.67:0/2042240568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440b5070 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b5b7fe640 1 -- 172.21.15.67:0/2042240568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b440a8950 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b5b7fe640 1 -- 172.21.15.67:0/2042240568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440a8d20 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b62429640 1 -- 172.21.15.67:0/2042240568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 msgr2=0x7f4b5c06e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:33.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b62429640 1 --2- 172.21.15.67:0/2042240568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c06e0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b440097c0 tx=0x7f4b4401ec50 comp rx=0 tx=0).stop 2024-04-25T15:50:33.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b62429640 1 -- 172.21.15.67:0/2042240568 shutdown_connections 2024-04-25T15:50:33.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b62429640 1 --2- 172.21.15.67:0/2042240568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c06e0d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b62429640 1 -- 172.21.15.67:0/2042240568 >> 172.21.15.67:0/2042240568 conn(0x7f4b5c073c50 msgr2=0x7f4b5c074060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:33.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b62429640 1 -- 172.21.15.67:0/2042240568 shutdown_connections 2024-04-25T15:50:33.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.398+0000 7f4b62429640 1 -- 172.21.15.67:0/2042240568 wait complete. 2024-04-25T15:50:33.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 Processor -- start 2024-04-25T15:50:33.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 -- start start 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c06e610 0x7f4b5c157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b5c0ca2c0 0x7f4b5c14db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c15de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c15d5c0 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5c0cb840 con 0x7f4b5c06e610 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b5c15d740 con 0x7f4b5c0ca2c0 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c15de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c15de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59450/0 (socket says 172.21.15.67:59450) 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 -- 172.21.15.67:0/3175881029 learned_addr learned my addr 172.21.15.67:0/3175881029 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b5bfff640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c06e610 0x7f4b5c157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b53fff640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b5c0ca2c0 0x7f4b5c14db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:33.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 -- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b5c0ca2c0 msgr2=0x7f4b5c14db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b5c0ca2c0 0x7f4b5c14db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 -- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c06e610 msgr2=0x7f4b5c157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c06e610 0x7f4b5c157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 -- 172.21.15.67:0/3175881029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b5c157cb0 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b609c6640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c15de80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400b9e0 tx=0x7f4b5400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b597fa640 1 -- 172.21.15.67:0/3175881029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b54003b80 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b597fa640 1 -- 172.21.15.67:0/3175881029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b54003d20 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b5c1f3f20 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b597fa640 1 -- 172.21.15.67:0/3175881029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400cb60 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.402+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b5c15f320 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.406+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b5c1f3c80 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.406+0000 7f4b597fa640 1 -- 172.21.15.67:0/3175881029 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4b54006d10 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.406+0000 7f4b597fa640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b3c06f400 0x7f4b3c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:33.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.406+0000 7f4b597fa640 1 -- 172.21.15.67:0/3175881029 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4b5408ed80 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.406+0000 7f4b5bfff640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b3c06f400 0x7f4b3c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:33.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.406+0000 7f4b5bfff640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b3c06f400 0x7f4b3c071910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c000f70 tx=0x7f4b4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:33.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.410+0000 7f4b597fa640 1 -- 172.21.15.67:0/3175881029 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4b5c1f3c80 con 0x7f4b5c0ca6d0 2024-04-25T15:50:33.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:33.610+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f4b5c0c6950 con 0x7f4b5c0ca6d0 2024-04-25T15:50:34.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.198+0000 7f4b597fa640 1 -- 172.21.15.67:0/3175881029 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) ==== 105+0+0 (secure 0 0 0) 0x7f4b5405bd00 con 0x7f4b5c0ca6d0 2024-04-25T15:50:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b3c06f400 msgr2=0x7f4b3c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b3c06f400 0x7f4b3c071910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c000f70 tx=0x7f4b4c009040 comp rx=0 tx=0).stop 2024-04-25T15:50:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 msgr2=0x7f4b5c15de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c15de80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400b9e0 tx=0x7f4b5400bea0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 shutdown_connections 2024-04-25T15:50:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4b3c06f400 0x7f4b3c071910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b5c0ca6d0 0x7f4b5c15de80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b5c0ca2c0 0x7f4b5c14db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 --2- 172.21.15.67:0/3175881029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c06e610 0x7f4b5c157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 >> 172.21.15.67:0/3175881029 conn(0x7f4b5c073c50 msgr2=0x7f4b5c15ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 shutdown_connections 2024-04-25T15:50:34.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.202+0000 7f4b62429640 1 -- 172.21.15.67:0/3175881029 wait complete. 2024-04-25T15:50:34.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T15:50:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad31e0640 1 Processor -- start 2024-04-25T15:50:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad31e0640 1 -- start start 2024-04-25T15:50:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad31e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc06c640 0x7f6acc06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad31e0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6acc0c8680 0x7f6acc0c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad31e0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6acc158d20 con 0x7f6acc152b00 2024-04-25T15:50:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad31e0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6acc0722c0 con 0x7f6acc0c8680 2024-04-25T15:50:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad31e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6acc158fe0 con 0x7f6acc06c640 2024-04-25T15:50:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6acbfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc06c640 0x7f6acc06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad177d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6acc0c8680 0x7f6acc0c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6acbfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc06c640 0x7f6acc06ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60326/0 (socket says 172.21.15.67:60326) 2024-04-25T15:50:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6acbfff640 1 -- 172.21.15.67:0/2624060056 learned_addr learned my addr 172.21.15.67:0/2624060056 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6ad0f7c640 1 --1- 172.21.15.67:0/2624060056 >> v1:172.21.15.176:6789/0 conn(0x7f6acc152b00 0x7f6acc0c4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:48830/0 (socket says 172.21.15.67:48830) 2024-04-25T15:50:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6acbfff640 1 -- 172.21.15.67:0/2624060056 >> v2:172.21.15.176:3300/0 conn(0x7f6acc0c8680 msgr2=0x7f6acc0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6acbfff640 1 --2- 172.21.15.67:0/2624060056 >> v2:172.21.15.176:3300/0 conn(0x7f6acc0c8680 0x7f6acc0c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6acbfff640 1 -- 172.21.15.67:0/2624060056 >> v1:172.21.15.176:6789/0 conn(0x7f6acc152b00 legacy=0x7f6acc0c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.390+0000 7f6acbfff640 1 -- 172.21.15.67:0/2624060056 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6acc1592c0 con 0x7f6acc06c640 2024-04-25T15:50:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6acbfff640 1 --2- 172.21.15.67:0/2624060056 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc06c640 0x7f6acc06ca50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009870 tx=0x7f6abc01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fc75ea5b511b6d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6acb7fe640 1 -- 172.21.15.67:0/2624060056 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc0b0070 con 0x7f6acc06c640 2024-04-25T15:50:34.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6acb7fe640 1 -- 172.21.15.67:0/2624060056 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6abc0ab600 con 0x7f6acc06c640 2024-04-25T15:50:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6acb7fe640 1 -- 172.21.15.67:0/2624060056 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc0ab9d0 con 0x7f6acc06c640 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 -- 172.21.15.67:0/2624060056 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc06c640 msgr2=0x7f6acc06ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/2624060056 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc06c640 0x7f6acc06ca50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009870 tx=0x7f6abc01c560 comp rx=0 tx=0).stop 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 -- 172.21.15.67:0/2624060056 shutdown_connections 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/2624060056 >> v2:172.21.15.176:3300/0 conn(0x7f6acc0c8680 0x7f6acc0c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/2624060056 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc06c640 0x7f6acc06ca50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 -- 172.21.15.67:0/2624060056 >> 172.21.15.67:0/2624060056 conn(0x7f6acc06fe70 msgr2=0x7f6acc0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 -- 172.21.15.67:0/2624060056 shutdown_connections 2024-04-25T15:50:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.394+0000 7f6ad31e0640 1 -- 172.21.15.67:0/2624060056 wait complete. 2024-04-25T15:50:34.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 Processor -- start 2024-04-25T15:50:34.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 -- start start 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6acc06c640 0x7f6acc155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc0c8680 0x7f6acc1f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6acc152b00 0x7f6acc148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6acc152f10 con 0x7f6acc152b00 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6acc0ca100 con 0x7f6acc06c640 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6acc1665d0 con 0x7f6acc0c8680 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6acc152b00 0x7f6acc148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6acc152b00 0x7f6acc148e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59456/0 (socket says 172.21.15.67:59456) 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 -- 172.21.15.67:0/1586812390 learned_addr learned my addr 172.21.15.67:0/1586812390 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6acbfff640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc0c8680 0x7f6acc1f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad0f7c640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6acc06c640 0x7f6acc155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:34.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 -- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc0c8680 msgr2=0x7f6acc1f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc0c8680 0x7f6acc1f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 -- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6acc06c640 msgr2=0x7f6acc155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6acc06c640 0x7f6acc155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 -- 172.21.15.67:0/1586812390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6acc1f6ca0 con 0x7f6acc152b00 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad177d640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6acc152b00 0x7f6acc148e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400ef90 tx=0x7f6ac400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ac97fa640 1 -- 172.21.15.67:0/1586812390 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4018020 con 0x7f6acc152b00 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ac97fa640 1 -- 172.21.15.67:0/1586812390 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac4014480 con 0x7f6acc152b00 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6acc1f4540 con 0x7f6acc152b00 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ac97fa640 1 -- 172.21.15.67:0/1586812390 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4009e60 con 0x7f6acc152b00 2024-04-25T15:50:34.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.398+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6acc0745a0 con 0x7f6acc152b00 2024-04-25T15:50:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.402+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6acc1f6e70 con 0x7f6acc152b00 2024-04-25T15:50:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.402+0000 7f6ac97fa640 1 -- 172.21.15.67:0/1586812390 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6ac401b020 con 0x7f6acc152b00 2024-04-25T15:50:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.402+0000 7f6ac97fa640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6ab806f400 0x7f6ab8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.402+0000 7f6ac97fa640 1 -- 172.21.15.67:0/1586812390 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6ac408ff50 con 0x7f6acc152b00 2024-04-25T15:50:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.402+0000 7f6ad0f7c640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6ab806f400 0x7f6ab8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:34.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.402+0000 7f6ad0f7c640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6ab806f400 0x7f6ab8071910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0000f70 tx=0x7f6ac0007c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:34.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.406+0000 7f6ac97fa640 1 -- 172.21.15.67:0/1586812390 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6ac4022030 con 0x7f6acc152b00 2024-04-25T15:50:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:34.606+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6acc074310 con 0x7f6acc152b00 2024-04-25T15:50:35.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.210+0000 7f6ac97fa640 1 -- 172.21.15.67:0/1586812390 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7f6ac4010040 con 0x7f6acc152b00 2024-04-25T15:50:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6ab806f400 msgr2=0x7f6ab8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6ab806f400 0x7f6ab8071910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0000f70 tx=0x7f6ac0007c90 comp rx=0 tx=0).stop 2024-04-25T15:50:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6acc152b00 msgr2=0x7f6acc148e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6acc152b00 0x7f6acc148e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400ef90 tx=0x7f6ac400c490 comp rx=0 tx=0).stop 2024-04-25T15:50:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 shutdown_connections 2024-04-25T15:50:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6ab806f400 0x7f6ab8071910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6acc152b00 0x7f6acc148e10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6acc0c8680 0x7f6acc1f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 --2- 172.21.15.67:0/1586812390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6acc06c640 0x7f6acc155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 >> 172.21.15.67:0/1586812390 conn(0x7f6acc06fe70 msgr2=0x7f6acc14e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:35.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 shutdown_connections 2024-04-25T15:50:35.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.214+0000 7f6ad31e0640 1 -- 172.21.15.67:0/1586812390 wait complete. 2024-04-25T15:50:35.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T15:50:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f9515640 1 Processor -- start 2024-04-25T15:50:35.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f9515640 1 -- start start 2024-04-25T15:50:35.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f9515640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f9515640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5f415e4a0 0x7fb5f40c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f9515640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5f40749d0 con 0x7fb5f415e8b0 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f9515640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb5f406e510 con 0x7fb5f415e4a0 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f9515640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f415dca0 con 0x7fb5f406c640 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5f415e8b0 0x7fb5f415a650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35952/0 (socket says 172.21.15.67:35952) 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f37fe640 1 -- 172.21.15.67:0/252530937 learned_addr learned my addr 172.21.15.67:0/252530937 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f27fc640 1 --2- 172.21.15.67:0/252530937 >> v2:172.21.15.67:3300/0 conn(0x7fb5f415e4a0 0x7fb5f40c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f2ffd640 1 --2- 172.21.15.67:0/252530937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f1ffb640 1 -- 172.21.15.67:0/252530937 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 979732524 0 0) 0x7fb5f40749d0 con 0x7fb5f415e8b0 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f1ffb640 1 -- 172.21.15.67:0/252530937 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5dc003530 con 0x7fb5f415e8b0 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f27fc640 1 -- 172.21.15.67:0/252530937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 msgr2=0x7fb5f4154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:35.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f27fc640 1 --2- 172.21.15.67:0/252530937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f27fc640 1 -- 172.21.15.67:0/252530937 >> v1:172.21.15.67:6789/0 conn(0x7fb5f415e8b0 legacy=0x7fb5f415a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:35.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.398+0000 7fb5f27fc640 1 -- 172.21.15.67:0/252530937 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f415df80 con 0x7fb5f415e4a0 2024-04-25T15:50:35.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.402+0000 7fb5f27fc640 1 --2- 172.21.15.67:0/252530937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e4a0 0x7fb5f40c7fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80097c0 tx=0x7fb5e801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12eafb8bb6a5af87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:35.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.402+0000 7fb5f1ffb640 1 -- 172.21.15.67:0/252530937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e80b0070 con 0x7fb5f415e4a0 2024-04-25T15:50:35.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.402+0000 7fb5f1ffb640 1 -- 172.21.15.67:0/252530937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5e80ab600 con 0x7fb5f415e4a0 2024-04-25T15:50:35.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.402+0000 7fb5f1ffb640 1 -- 172.21.15.67:0/252530937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e80ab9b0 con 0x7fb5f415e4a0 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- 172.21.15.67:0/252530937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e4a0 msgr2=0x7fb5f40c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 --2- 172.21.15.67:0/252530937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e4a0 0x7fb5f40c7fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80097c0 tx=0x7fb5e801c410 comp rx=0 tx=0).stop 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- 172.21.15.67:0/252530937 shutdown_connections 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 --2- 172.21.15.67:0/252530937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e4a0 0x7fb5f40c7fc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 --2- 172.21.15.67:0/252530937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- 172.21.15.67:0/252530937 >> 172.21.15.67:0/252530937 conn(0x7fb5f4070db0 msgr2=0x7fb5f40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- 172.21.15.67:0/252530937 shutdown_connections 2024-04-25T15:50:35.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- 172.21.15.67:0/252530937 wait complete. 2024-04-25T15:50:35.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 Processor -- start 2024-04-25T15:50:35.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- start start 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5f415e4a0 0x7fb5f4158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e8b0 0x7fb5f4155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f40cad60 con 0x7fb5f415e8b0 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4072bc0 con 0x7fb5f415e4a0 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.406+0000 7fb5f9515640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f41f7150 con 0x7fb5f406c640 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e8b0 0x7fb5f4155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e8b0 0x7fb5f4155d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59472/0 (socket says 172.21.15.67:59472) 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 -- 172.21.15.67:0/511255789 learned_addr learned my addr 172.21.15.67:0/511255789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f2ffd640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f27fc640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5f415e4a0 0x7fb5f4158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 -- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 msgr2=0x7fb5f4154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 -- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5f415e4a0 msgr2=0x7fb5f4158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5f415e4a0 0x7fb5f4158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 -- 172.21.15.67:0/511255789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f4157960 con 0x7fb5f415e8b0 2024-04-25T15:50:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f37fe640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e8b0 0x7fb5f4155d20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00caa0 tx=0x7fb5ec00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5db7fe640 1 -- 172.21.15.67:0/511255789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec017070 con 0x7fb5f415e8b0 2024-04-25T15:50:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5db7fe640 1 -- 172.21.15.67:0/511255789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ec00ac00 con 0x7fb5f415e8b0 2024-04-25T15:50:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f4157c00 con 0x7fb5f415e8b0 2024-04-25T15:50:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5db7fe640 1 -- 172.21.15.67:0/511255789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec012400 con 0x7fb5f415e8b0 2024-04-25T15:50:35.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f40ceba0 con 0x7fb5f415e8b0 2024-04-25T15:50:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f406a480 con 0x7fb5f415e8b0 2024-04-25T15:50:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5db7fe640 1 -- 172.21.15.67:0/511255789 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb5ec01a070 con 0x7fb5f415e8b0 2024-04-25T15:50:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5db7fe640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb5e006f430 0x7fb5e0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5db7fe640 1 -- 172.21.15.67:0/511255789 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb5ec08ee90 con 0x7fb5f415e8b0 2024-04-25T15:50:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.410+0000 7fb5f2ffd640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb5e006f430 0x7fb5e0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.414+0000 7fb5f2ffd640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb5e006f430 0x7fb5e0071940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009a50 tx=0x7fb5e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.418+0000 7fb5db7fe640 1 -- 172.21.15.67:0/511255789 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb5ec05b7c0 con 0x7fb5f415e8b0 2024-04-25T15:50:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:35.614+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb5f40740e0 con 0x7fb5f415e8b0 2024-04-25T15:50:36.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.226+0000 7fb5db7fe640 1 -- 172.21.15.67:0/511255789 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) ==== 115+0+0 (secure 0 0 0) 0x7fb5ec05fac0 con 0x7fb5f415e8b0 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb5e006f430 msgr2=0x7fb5e0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb5e006f430 0x7fb5e0071940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009a50 tx=0x7fb5e4008040 comp rx=0 tx=0).stop 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e8b0 msgr2=0x7fb5f4155d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e8b0 0x7fb5f4155d20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec00caa0 tx=0x7fb5ec00cf60 comp rx=0 tx=0).stop 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 shutdown_connections 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb5e006f430 0x7fb5e0071940 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5f415e8b0 0x7fb5f4155d20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5f415e4a0 0x7fb5f4158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 --2- 172.21.15.67:0/511255789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5f406c640 0x7fb5f4154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 >> 172.21.15.67:0/511255789 conn(0x7fb5f4070db0 msgr2=0x7fb5f414f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 shutdown_connections 2024-04-25T15:50:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.230+0000 7fb5f9515640 1 -- 172.21.15.67:0/511255789 wait complete. 2024-04-25T15:50:36.255 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T15:50:36.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:50:36.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c6adb640 1 Processor -- start 2024-04-25T15:50:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c6adb640 1 -- start start 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c6adb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3c0071b20 0x7fa3c015d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c6adb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c00c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c6adb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c00c6890 con 0x7fa3c015d710 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c6adb640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa3c015cd40 con 0x7fa3c0071b20 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c6adb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c00c6b70 con 0x7fa3c00c5bb0 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c4877640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3c0071b20 0x7fa3c015d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3bffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3c015d710 0x7fa3c00c7610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35968/0 (socket says 172.21.15.67:35968) 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3bffff640 1 -- 172.21.15.67:0/601848348 learned_addr learned my addr 172.21.15.67:0/601848348 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c5078640 1 --2- 172.21.15.67:0/601848348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c00c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c4877640 1 -- 172.21.15.67:0/601848348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 msgr2=0x7fa3c00c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c4877640 1 --2- 172.21.15.67:0/601848348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c00c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c4877640 1 -- 172.21.15.67:0/601848348 >> v1:172.21.15.67:6789/0 conn(0x7fa3c015d710 legacy=0x7fa3c00c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.410+0000 7fa3c4877640 1 -- 172.21.15.67:0/601848348 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c00c6e50 con 0x7fa3c0071b20 2024-04-25T15:50:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c4877640 1 --2- 172.21.15.67:0/601848348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c0071b20 0x7fa3c015d1d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097c0 tx=0x7fa3b001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae5bba0156d2434 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3bf7fe640 1 -- 172.21.15.67:0/601848348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00b1070 con 0x7fa3c0071b20 2024-04-25T15:50:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3bf7fe640 1 -- 172.21.15.67:0/601848348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b00ac410 con 0x7fa3c0071b20 2024-04-25T15:50:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3bf7fe640 1 -- 172.21.15.67:0/601848348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00ac7e0 con 0x7fa3c0071b20 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 -- 172.21.15.67:0/601848348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c0071b20 msgr2=0x7fa3c015d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/601848348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c0071b20 0x7fa3c015d1d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097c0 tx=0x7fa3b001c4a0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 -- 172.21.15.67:0/601848348 shutdown_connections 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/601848348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c00c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/601848348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c0071b20 0x7fa3c015d1d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 -- 172.21.15.67:0/601848348 >> 172.21.15.67:0/601848348 conn(0x7fa3c006fe70 msgr2=0x7fa3c00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 -- 172.21.15.67:0/601848348 shutdown_connections 2024-04-25T15:50:36.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.414+0000 7fa3c6adb640 1 -- 172.21.15.67:0/601848348 wait complete. 2024-04-25T15:50:36.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 Processor -- start 2024-04-25T15:50:36.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 -- start start 2024-04-25T15:50:36.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c0071b20 0x7fa3c014d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c01462f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c015d710 0x7fa3c0149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c0166580 con 0x7fa3c015d710 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c00c7d20 con 0x7fa3c0071b20 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c0166700 con 0x7fa3c00c5bb0 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c015d710 0x7fa3c0149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c015d710 0x7fa3c0149d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59478/0 (socket says 172.21.15.67:59478) 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 -- 172.21.15.67:0/2416051980 learned_addr learned my addr 172.21.15.67:0/2416051980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3bffff640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c01462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c4877640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c0071b20 0x7fa3c014d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 -- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 msgr2=0x7fa3c01462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c01462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 -- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c0071b20 msgr2=0x7fa3c014d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c0071b20 0x7fa3c014d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 -- 172.21.15.67:0/2416051980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c014d9f0 con 0x7fa3c015d710 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3bffff640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c01462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c4877640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c0071b20 0x7fa3c014d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c5078640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c015d710 0x7fa3c0149d40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8007db0 tx=0x7fa3b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3bd7fa640 1 -- 172.21.15.67:0/2416051980 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8014420 con 0x7fa3c015d710 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3bd7fa640 1 -- 172.21.15.67:0/2416051980 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b8019070 con 0x7fa3c015d710 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c0155ed0 con 0x7fa3c015d710 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3bd7fa640 1 -- 172.21.15.67:0/2416051980 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8014800 con 0x7fa3c015d710 2024-04-25T15:50:36.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c01454a0 con 0x7fa3c015d710 2024-04-25T15:50:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.418+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3c006a480 con 0x7fa3c015d710 2024-04-25T15:50:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.422+0000 7fa3bd7fa640 1 -- 172.21.15.67:0/2416051980 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa3b80149a0 con 0x7fa3c015d710 2024-04-25T15:50:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.422+0000 7fa3bd7fa640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa3ac06f430 0x7fa3ac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.422+0000 7fa3bd7fa640 1 -- 172.21.15.67:0/2416051980 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa3b808f9b0 con 0x7fa3c015d710 2024-04-25T15:50:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.422+0000 7fa3c4877640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa3ac06f430 0x7fa3ac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.422+0000 7fa3c4877640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa3ac06f430 0x7fa3ac071940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00ae000 tx=0x7fa3b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.426+0000 7fa3bd7fa640 1 -- 172.21.15.67:0/2416051980 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa3b805c2e0 con 0x7fa3c015d710 2024-04-25T15:50:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.626+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3c00c4700 con 0x7fa3c015d710 2024-04-25T15:50:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.626+0000 7fa3bd7fa640 1 -- 172.21.15.67:0/2416051980 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6822 (secure 0 0 0) 0x7fa3b80605e0 con 0x7fa3c015d710 2024-04-25T15:50:36.628 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:36.628 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/920422475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":920422475},{"type":"v1","addr":"172.21.15.67:6835","nonce":920422475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/144842397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":144842397},{"type":"v1","addr":"172.21.15.67:6834","nonce":144842397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4272,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6838/2023672746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2023672746},{"type":"v1","addr":"172.21.15.176:6838","nonce":2023672746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4288,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/2385091301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2385091301},{"type":"v1","addr":"172.21.15.176:6835","nonce":2385091301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:50:30.570677+0000","modified":"2024-04-25T15:50:35.618129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.176:6839/1886671623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6837","nonce":1886671623},{"type":"v1","addr":"172.21.15.176:6839","nonce":1886671623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-25T15:50:36.628 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa3ac06f430 msgr2=0x7fa3ac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa3ac06f430 0x7fa3ac071940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00ae000 tx=0x7fa3b0008040 comp rx=0 tx=0).stop 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c015d710 msgr2=0x7fa3c0149d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c015d710 0x7fa3c0149d40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8007db0 tx=0x7fa3b800e4b0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 shutdown_connections 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa3ac06f430 0x7fa3ac071940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3c015d710 0x7fa3c0149d40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3c00c5bb0 0x7fa3c01462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 --2- 172.21.15.67:0/2416051980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3c0071b20 0x7fa3c014d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 >> 172.21.15.67:0/2416051980 conn(0x7fa3c006fe70 msgr2=0x7fa3c0073f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 shutdown_connections 2024-04-25T15:50:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.630+0000 7fa3c6adb640 1 -- 172.21.15.67:0/2416051980 wait complete. 2024-04-25T15:50:36.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:36.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c9b08640 1 Processor -- start 2024-04-25T15:50:36.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c9b08640 1 -- start start 2024-04-25T15:50:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c9b08640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c4068590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c9b08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97c406c640 0x7f97c406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c9b08640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c40c8fa0 con 0x7f97c415c750 2024-04-25T15:50:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c9b08640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f97c415cb60 con 0x7f97c406c640 2024-04-25T15:50:36.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c9b08640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97c40c9600 con 0x7f97c4073a90 2024-04-25T15:50:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f97c415c750 0x7f97c41568a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35974/0 (socket says 172.21.15.67:35974) 2024-04-25T15:50:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c37fe640 1 -- 172.21.15.67:0/2408375679 learned_addr learned my addr 172.21.15.67:0/2408375679 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c3fff640 1 --2- 172.21.15.67:0/2408375679 >> v2:172.21.15.67:3300/0 conn(0x7f97c406c640 0x7f97c406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c2ffd640 1 --2- 172.21.15.67:0/2408375679 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c4068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c27fc640 1 -- 172.21.15.67:0/2408375679 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711286927 0 0) 0x7f97c40c8fa0 con 0x7f97c415c750 2024-04-25T15:50:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c27fc640 1 -- 172.21.15.67:0/2408375679 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97b0003530 con 0x7f97c415c750 2024-04-25T15:50:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c3fff640 1 -- 172.21.15.67:0/2408375679 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 msgr2=0x7f97c4068590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c3fff640 1 --2- 172.21.15.67:0/2408375679 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c4068590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c3fff640 1 -- 172.21.15.67:0/2408375679 >> v1:172.21.15.67:6789/0 conn(0x7f97c415c750 legacy=0x7f97c41568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.806+0000 7f97c3fff640 1 -- 172.21.15.67:0/2408375679 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c40c98e0 con 0x7f97c406c640 2024-04-25T15:50:36.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c3fff640 1 --2- 172.21.15.67:0/2408375679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c406ca50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097c0 tx=0x7f97bc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b271e44cdcc2297d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:36.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c27fc640 1 -- 172.21.15.67:0/2408375679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0b0070 con 0x7f97c406c640 2024-04-25T15:50:36.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c27fc640 1 -- 172.21.15.67:0/2408375679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97bc0ab600 con 0x7f97c406c640 2024-04-25T15:50:36.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c27fc640 1 -- 172.21.15.67:0/2408375679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0ab9b0 con 0x7f97c406c640 2024-04-25T15:50:36.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 -- 172.21.15.67:0/2408375679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 msgr2=0x7f97c406ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 --2- 172.21.15.67:0/2408375679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c406ca50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097c0 tx=0x7f97bc01c4a0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 -- 172.21.15.67:0/2408375679 shutdown_connections 2024-04-25T15:50:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 --2- 172.21.15.67:0/2408375679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c406ca50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 --2- 172.21.15.67:0/2408375679 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c4068590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 -- 172.21.15.67:0/2408375679 >> 172.21.15.67:0/2408375679 conn(0x7f97c4069220 msgr2=0x7f97c40cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 -- 172.21.15.67:0/2408375679 shutdown_connections 2024-04-25T15:50:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.810+0000 7f97c9b08640 1 -- 172.21.15.67:0/2408375679 wait complete. 2024-04-25T15:50:36.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 Processor -- start 2024-04-25T15:50:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 -- start start 2024-04-25T15:50:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c40c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c40ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97c415c750 0x7f97c41446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97c40c92b0 con 0x7f97c406c640 2024-04-25T15:50:36.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4068ad0 con 0x7f97c415c750 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c9b08640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97c41698f0 con 0x7f97c4073a90 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c40c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c40c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59494/0 (socket says 172.21.15.67:59494) 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 -- 172.21.15.67:0/829500523 learned_addr learned my addr 172.21.15.67:0/829500523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c2ffd640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c40ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c3fff640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97c415c750 0x7f97c41446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 -- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 msgr2=0x7f97c40ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c40ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 -- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97c415c750 msgr2=0x7f97c41446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97c415c750 0x7f97c41446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 -- 172.21.15.67:0/829500523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c40c6da0 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c2ffd640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c40ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c3fff640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97c415c750 0x7f97c41446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.814+0000 7f97c37fe640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c40c6860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c3c0 tx=0x7f97b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97bbfff640 1 -- 172.21.15.67:0/829500523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4017070 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97bbfff640 1 -- 172.21.15.67:0/829500523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b4005db0 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c41f5b10 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97bbfff640 1 -- 172.21.15.67:0/829500523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b40072f0 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c40c5d80 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c406a480 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97bbfff640 1 -- 172.21.15.67:0/829500523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f97b401a070 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97bbfff640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f97a406f400 0x7f97a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97bbfff640 1 -- 172.21.15.67:0/829500523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f97b408e6c0 con 0x7f97c406c640 2024-04-25T15:50:36.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97c2ffd640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f97a406f400 0x7f97a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.818+0000 7f97c2ffd640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f97a406f400 0x7f97a4071910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97c41473c0 tx=0x7f97ac00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:36.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.822+0000 7f97bbfff640 1 -- 172.21.15.67:0/829500523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f97b405b640 con 0x7f97c406c640 2024-04-25T15:50:36.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.986+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97c415a170 con 0x7f97c406c640 2024-04-25T15:50:36.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.990+0000 7f97bbfff640 1 -- 172.21.15.67:0/829500523 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f97b405af40 con 0x7f97c406c640 2024-04-25T15:50:36.992 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:32.143126+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f97a406f400 msgr2=0x7f97a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f97a406f400 0x7f97a4071910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97c41473c0 tx=0x7f97ac00b040 comp rx=0 tx=0).stop 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 msgr2=0x7f97c40c6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c40c6860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c3c0 tx=0x7f97b400c880 comp rx=0 tx=0).stop 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 shutdown_connections 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f97a406f400 0x7f97a4071910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97c415c750 0x7f97c41446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97c4073a90 0x7f97c40ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 --2- 172.21.15.67:0/829500523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c406c640 0x7f97c40c6860 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 >> 172.21.15.67:0/829500523 conn(0x7f97c4069220 msgr2=0x7f97c4153e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 shutdown_connections 2024-04-25T15:50:36.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:36.994+0000 7f97c9b08640 1 -- 172.21.15.67:0/829500523 wait complete. 2024-04-25T15:50:37.015 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:50:37.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:50:37.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a3324640 1 Processor -- start 2024-04-25T15:50:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a3324640 1 -- start start 2024-04-25T15:50:37.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a3324640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 0x7f469c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a3324640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469c06d780 con 0x7f469c06ca60 2024-04-25T15:50:37.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a3324640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469c160920 con 0x7f469c0b5d70 2024-04-25T15:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a3324640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f469c15d510 con 0x7f469c160510 2024-04-25T15:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a10c0640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f469c06ca60 0x7f469c0b5660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35982/0 (socket says 172.21.15.67:35982) 2024-04-25T15:50:37.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a10c0640 1 -- 172.21.15.67:0/3253772085 learned_addr learned my addr 172.21.15.67:0/3253772085 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:37.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a18c1640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 0x7f469c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900179510 0 0) 0x7f469c06d780 con 0x7f469c06ca60 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4688003530 con 0x7f469c06ca60 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 425746074 0 0) 0x7f4688003530 con 0x7f469c06ca60 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 msgr2=0x7f469c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 0x7f469c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 >> v1:172.21.15.176:6789/0 conn(0x7f469c0b5d70 legacy=0x7f469c15fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c160be0 con 0x7f469c06ca60 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f4698001a20 con 0x7f469c06ca60 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 >> v1:172.21.15.67:6789/0 conn(0x7f469c06ca60 legacy=0x7f469c0b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46880039f0 0x7f4688003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 0x7f4688004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4688004d50 0x7f4688005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f469c15d510 con 0x7f46880039f0 2024-04-25T15:50:37.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f468800bbc0 con 0x7f4688004d50 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f468800bdc0 con 0x7f469c160510 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a10c0640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46880039f0 0x7f4688003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.162+0000 7f46a18c1640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4688004d50 0x7f4688005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a08bf640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 0x7f4688004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a10c0640 1 -- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 msgr2=0x7f4688004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a10c0640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c160510 0x7f4688004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a10c0640 1 -- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4688004d50 msgr2=0x7f4688005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a10c0640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4688004d50 0x7f4688005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a10c0640 1 -- 172.21.15.67:0/3253772085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4688003530 con 0x7f46880039f0 2024-04-25T15:50:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a10c0640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46880039f0 0x7f4688003e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f469c06d950 tx=0x7f469801f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcfc0924598bc537 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469800a480 con 0x7f46880039f0 2024-04-25T15:50:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46980b1070 con 0x7f46880039f0 2024-04-25T15:50:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f4693fff640 1 -- 172.21.15.67:0/3253772085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46980ac890 con 0x7f46880039f0 2024-04-25T15:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a3324640 1 -- 172.21.15.67:0/3253772085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46880039f0 msgr2=0x7f4688003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a3324640 1 --2- 172.21.15.67:0/3253772085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46880039f0 0x7f4688003e60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f469c06d950 tx=0x7f469801f9a0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a18c1640 1 -- 172.21.15.67:0/3253772085 reap_dead start 2024-04-25T15:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a3324640 1 -- 172.21.15.67:0/3253772085 shutdown_connections 2024-04-25T15:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a3324640 1 -- 172.21.15.67:0/3253772085 >> 172.21.15.67:0/3253772085 conn(0x7f469c06e420 msgr2=0x7f469c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.166+0000 7f46a3324640 1 -- 172.21.15.67:0/3253772085 shutdown_connections 2024-04-25T15:50:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 -- 172.21.15.67:0/3253772085 wait complete. 2024-04-25T15:50:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 Processor -- start 2024-04-25T15:50:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 -- start start 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46880039f0 0x7f469c155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c1e83d0 0x7f469c1535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c155e90 0x7f469c14d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f469c15d1a0 con 0x7f469c155e90 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f469c15cec0 con 0x7f46880039f0 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a3324640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f469c15c890 con 0x7f469c1e83d0 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.170+0000 7f46a18c1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c155e90 0x7f469c14d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c155e90 0x7f469c14d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59514/0 (socket says 172.21.15.67:59514) 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 -- 172.21.15.67:0/647574153 learned_addr learned my addr 172.21.15.67:0/647574153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a10c0640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46880039f0 0x7f469c155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a08bf640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c1e83d0 0x7f469c1535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 -- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c1e83d0 msgr2=0x7f469c1535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c1e83d0 0x7f469c1535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 -- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46880039f0 msgr2=0x7f469c155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46880039f0 0x7f469c155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 -- 172.21.15.67:0/647574153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469c1e8a30 con 0x7f469c155e90 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a18c1640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c155e90 0x7f469c14d410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f469c06c6f0 tx=0x7f4694011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f4691ffb640 1 -- 172.21.15.67:0/647574153 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694003620 con 0x7f469c155e90 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f4691ffb640 1 -- 172.21.15.67:0/647574153 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f469401b070 con 0x7f469c155e90 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469c1f5f20 con 0x7f469c155e90 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f4691ffb640 1 -- 172.21.15.67:0/647574153 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694003a40 con 0x7f469c155e90 2024-04-25T15:50:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469c1502b0 con 0x7f469c155e90 2024-04-25T15:50:37.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f469c06a480 con 0x7f469c155e90 2024-04-25T15:50:37.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f4691ffb640 1 -- 172.21.15.67:0/647574153 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4694003be0 con 0x7f469c155e90 2024-04-25T15:50:37.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f4691ffb640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f468006f430 0x7f4680071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f4691ffb640 1 -- 172.21.15.67:0/647574153 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f469408f660 con 0x7f469c155e90 2024-04-25T15:50:37.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.174+0000 7f46a10c0640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f468006f430 0x7f4680071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.178+0000 7f46a10c0640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f468006f430 0x7f4680071940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f46980b0000 tx=0x7f469c0b53d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:37.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.178+0000 7f4691ffb640 1 -- 172.21.15.67:0/647574153 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4694024060 con 0x7f469c155e90 2024-04-25T15:50:37.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.346+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f469c0cd380 con 0x7f468006f430 2024-04-25T15:50:37.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.374+0000 7f4691ffb640 1 -- 172.21.15.67:0/647574153 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7f469c0cd380 con 0x7f468006f430 2024-04-25T15:50:37.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:37.381 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:50:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f468006f430 msgr2=0x7f4680071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f468006f430 0x7f4680071940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f46980b0000 tx=0x7f469c0b53d0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c155e90 msgr2=0x7f469c14d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c155e90 0x7f469c14d410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f469c06c6f0 tx=0x7f4694011920 comp rx=0 tx=0).stop 2024-04-25T15:50:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 shutdown_connections 2024-04-25T15:50:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f468006f430 0x7f4680071940 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f469c155e90 0x7f469c14d410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f469c1e83d0 0x7f469c1535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 --2- 172.21.15.67:0/647574153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46880039f0 0x7f469c155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 >> 172.21.15.67:0/647574153 conn(0x7f469c06e420 msgr2=0x7f469c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 shutdown_connections 2024-04-25T15:50:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.378+0000 7f46a3324640 1 -- 172.21.15.67:0/647574153 wait complete. 2024-04-25T15:50:37.401 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-04-25T15:50:35.798881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218920,"kb_used_data":2712,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755800,"statfs":{"total":773094113280,"available":772869939200,"internally_reserved":0,"allocated":2777088,"data_stored":1723008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002316"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750328+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:50:32.750328+0000","last_peered":"2024-04-25T15:50:32.750328+0000","last_clean":"2024-04-25T15:50:32.750328+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:50:32.750328+0000","last_undegraded":"2024-04-25T15:50:32.750328+0000","last_fullsized":"2024-04-25T15:50:32.750328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149188+0000","last_change":"2024-04-25T15:50:31.192224+0000","last_active":"2024-04-25T15:50:32.149188+0000","last_peered":"2024-04-25T15:50:32.149188+0000","last_clean":"2024-04-25T15:50:32.149188+0000","last_became_active":"2024-04-25T15:50:31.189311+0000","last_became_peered":"2024-04-25T15:50:31.189311+0000","last_unstale":"2024-04-25T15:50:32.149188+0000","last_undegraded":"2024-04-25T15:50:32.149188+0000","last_fullsized":"2024-04-25T15:50:32.149188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.837762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750252+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:50:32.750252+0000","last_peered":"2024-04-25T15:50:32.750252+0000","last_clean":"2024-04-25T15:50:32.750252+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:50:32.750252+0000","last_undegraded":"2024-04-25T15:50:32.750252+0000","last_fullsized":"2024-04-25T15:50:32.750252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152869+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:50:32.152869+0000","last_peered":"2024-04-25T15:50:32.152869+0000","last_clean":"2024-04-25T15:50:32.152869+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:50:32.152869+0000","last_undegraded":"2024-04-25T15:50:32.152869+0000","last_fullsized":"2024-04-25T15:50:32.152869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748897+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:50:32.748897+0000","last_peered":"2024-04-25T15:50:32.748897+0000","last_clean":"2024-04-25T15:50:32.748897+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:50:32.748897+0000","last_undegraded":"2024-04-25T15:50:32.748897+0000","last_fullsized":"2024-04-25T15:50:32.748897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151867+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:50:32.151867+0000","last_peered":"2024-04-25T15:50:32.151867+0000","last_clean":"2024-04-25T15:50:32.151867+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:50:32.151867+0000","last_undegraded":"2024-04-25T15:50:32.151867+0000","last_fullsized":"2024-04-25T15:50:32.151867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747896+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:50:32.747896+0000","last_peered":"2024-04-25T15:50:32.747896+0000","last_clean":"2024-04-25T15:50:32.747896+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:50:32.747896+0000","last_undegraded":"2024-04-25T15:50:32.747896+0000","last_fullsized":"2024-04-25T15:50:32.747896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246622+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:50:32.246622+0000","last_peered":"2024-04-25T15:50:32.246622+0000","last_clean":"2024-04-25T15:50:32.246622+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:50:32.246622+0000","last_undegraded":"2024-04-25T15:50:32.246622+0000","last_fullsized":"2024-04-25T15:50:32.246622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750587+0000","last_change":"2024-04-25T15:50:31.210825+0000","last_active":"2024-04-25T15:50:32.750587+0000","last_peered":"2024-04-25T15:50:32.750587+0000","last_clean":"2024-04-25T15:50:32.750587+0000","last_became_active":"2024-04-25T15:50:31.210510+0000","last_became_peered":"2024-04-25T15:50:31.210510+0000","last_unstale":"2024-04-25T15:50:32.750587+0000","last_undegraded":"2024-04-25T15:50:32.750587+0000","last_fullsized":"2024-04-25T15:50:32.750587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:23.810710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749861+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:50:32.749861+0000","last_peered":"2024-04-25T15:50:32.749861+0000","last_clean":"2024-04-25T15:50:32.749861+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:50:32.749861+0000","last_undegraded":"2024-04-25T15:50:32.749861+0000","last_fullsized":"2024-04-25T15:50:32.749861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750086+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:50:32.750086+0000","last_peered":"2024-04-25T15:50:32.750086+0000","last_clean":"2024-04-25T15:50:32.750086+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:50:32.750086+0000","last_undegraded":"2024-04-25T15:50:32.750086+0000","last_fullsized":"2024-04-25T15:50:32.750086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151449+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:50:32.151449+0000","last_peered":"2024-04-25T15:50:32.151449+0000","last_clean":"2024-04-25T15:50:32.151449+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:50:32.151449+0000","last_undegraded":"2024-04-25T15:50:32.151449+0000","last_fullsized":"2024-04-25T15:50:32.151449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247483+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:50:32.247483+0000","last_peered":"2024-04-25T15:50:32.247483+0000","last_clean":"2024-04-25T15:50:32.247483+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:50:32.247483+0000","last_undegraded":"2024-04-25T15:50:32.247483+0000","last_fullsized":"2024-04-25T15:50:32.247483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750096+0000","last_change":"2024-04-25T15:50:31.171300+0000","last_active":"2024-04-25T15:50:32.750096+0000","last_peered":"2024-04-25T15:50:32.750096+0000","last_clean":"2024-04-25T15:50:32.750096+0000","last_became_active":"2024-04-25T15:50:31.170451+0000","last_became_peered":"2024-04-25T15:50:31.170451+0000","last_unstale":"2024-04-25T15:50:32.750096+0000","last_undegraded":"2024-04-25T15:50:32.750096+0000","last_fullsized":"2024-04-25T15:50:32.750096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:52:45.807418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148500+0000","last_change":"2024-04-25T15:50:31.198817+0000","last_active":"2024-04-25T15:50:32.148500+0000","last_peered":"2024-04-25T15:50:32.148500+0000","last_clean":"2024-04-25T15:50:32.148500+0000","last_became_active":"2024-04-25T15:50:31.198165+0000","last_became_peered":"2024-04-25T15:50:31.198165+0000","last_unstale":"2024-04-25T15:50:32.148500+0000","last_undegraded":"2024-04-25T15:50:32.148500+0000","last_fullsized":"2024-04-25T15:50:32.148500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:05.893280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150141+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:50:32.150141+0000","last_peered":"2024-04-25T15:50:32.150141+0000","last_clean":"2024-04-25T15:50:32.150141+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:50:32.150141+0000","last_undegraded":"2024-04-25T15:50:32.150141+0000","last_fullsized":"2024-04-25T15:50:32.150141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154678+0000","last_change":"2024-04-25T15:50:31.198225+0000","last_active":"2024-04-25T15:50:32.154678+0000","last_peered":"2024-04-25T15:50:32.154678+0000","last_clean":"2024-04-25T15:50:32.154678+0000","last_became_active":"2024-04-25T15:50:31.196737+0000","last_became_peered":"2024-04-25T15:50:31.196737+0000","last_unstale":"2024-04-25T15:50:32.154678+0000","last_undegraded":"2024-04-25T15:50:32.154678+0000","last_fullsized":"2024-04-25T15:50:32.154678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:25.652638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748753+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:50:32.748753+0000","last_peered":"2024-04-25T15:50:32.748753+0000","last_clean":"2024-04-25T15:50:32.748753+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:50:32.748753+0000","last_undegraded":"2024-04-25T15:50:32.748753+0000","last_fullsized":"2024-04-25T15:50:32.748753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748982+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:50:32.748982+0000","last_peered":"2024-04-25T15:50:32.748982+0000","last_clean":"2024-04-25T15:50:32.748982+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:50:32.748982+0000","last_undegraded":"2024-04-25T15:50:32.748982+0000","last_fullsized":"2024-04-25T15:50:32.748982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749171+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:50:32.749171+0000","last_peered":"2024-04-25T15:50:32.749171+0000","last_clean":"2024-04-25T15:50:32.749171+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:50:32.749171+0000","last_undegraded":"2024-04-25T15:50:32.749171+0000","last_fullsized":"2024-04-25T15:50:32.749171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749435+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:50:32.749435+0000","last_peered":"2024-04-25T15:50:32.749435+0000","last_clean":"2024-04-25T15:50:32.749435+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:50:32.749435+0000","last_undegraded":"2024-04-25T15:50:32.749435+0000","last_fullsized":"2024-04-25T15:50:32.749435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152360+0000","last_change":"2024-04-25T15:50:31.198787+0000","last_active":"2024-04-25T15:50:32.152360+0000","last_peered":"2024-04-25T15:50:32.152360+0000","last_clean":"2024-04-25T15:50:32.152360+0000","last_became_active":"2024-04-25T15:50:31.197737+0000","last_became_peered":"2024-04-25T15:50:31.197737+0000","last_unstale":"2024-04-25T15:50:32.152360+0000","last_undegraded":"2024-04-25T15:50:32.152360+0000","last_fullsized":"2024-04-25T15:50:32.152360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:03:21.865150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246985+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:50:32.246985+0000","last_peered":"2024-04-25T15:50:32.246985+0000","last_clean":"2024-04-25T15:50:32.246985+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:50:32.246985+0000","last_undegraded":"2024-04-25T15:50:32.246985+0000","last_fullsized":"2024-04-25T15:50:32.246985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150206+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:50:32.150206+0000","last_peered":"2024-04-25T15:50:32.150206+0000","last_clean":"2024-04-25T15:50:32.150206+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:50:32.150206+0000","last_undegraded":"2024-04-25T15:50:32.150206+0000","last_fullsized":"2024-04-25T15:50:32.150206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247233+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:50:32.247233+0000","last_peered":"2024-04-25T15:50:32.247233+0000","last_clean":"2024-04-25T15:50:32.247233+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:50:32.247233+0000","last_undegraded":"2024-04-25T15:50:32.247233+0000","last_fullsized":"2024-04-25T15:50:32.247233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750349+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:50:32.750349+0000","last_peered":"2024-04-25T15:50:32.750349+0000","last_clean":"2024-04-25T15:50:32.750349+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:50:32.750349+0000","last_undegraded":"2024-04-25T15:50:32.750349+0000","last_fullsized":"2024-04-25T15:50:32.750349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149180+0000","last_change":"2024-04-25T15:50:31.192245+0000","last_active":"2024-04-25T15:50:32.149180+0000","last_peered":"2024-04-25T15:50:32.149180+0000","last_clean":"2024-04-25T15:50:32.149180+0000","last_became_active":"2024-04-25T15:50:31.190989+0000","last_became_peered":"2024-04-25T15:50:31.190989+0000","last_unstale":"2024-04-25T15:50:32.149180+0000","last_undegraded":"2024-04-25T15:50:32.149180+0000","last_fullsized":"2024-04-25T15:50:32.149180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:29:24.813097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247952+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:50:32.247952+0000","last_peered":"2024-04-25T15:50:32.247952+0000","last_clean":"2024-04-25T15:50:32.247952+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:50:32.247952+0000","last_undegraded":"2024-04-25T15:50:32.247952+0000","last_fullsized":"2024-04-25T15:50:32.247952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750022+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:50:32.750022+0000","last_peered":"2024-04-25T15:50:32.750022+0000","last_clean":"2024-04-25T15:50:32.750022+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:50:32.750022+0000","last_undegraded":"2024-04-25T15:50:32.750022+0000","last_fullsized":"2024-04-25T15:50:32.750022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153195+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:50:32.153195+0000","last_peered":"2024-04-25T15:50:32.153195+0000","last_clean":"2024-04-25T15:50:32.153195+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:50:32.153195+0000","last_undegraded":"2024-04-25T15:50:32.153195+0000","last_fullsized":"2024-04-25T15:50:32.153195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748323+0000","last_change":"2024-04-25T15:50:31.174913+0000","last_active":"2024-04-25T15:50:32.748323+0000","last_peered":"2024-04-25T15:50:32.748323+0000","last_clean":"2024-04-25T15:50:32.748323+0000","last_became_active":"2024-04-25T15:50:31.171499+0000","last_became_peered":"2024-04-25T15:50:31.171499+0000","last_unstale":"2024-04-25T15:50:32.748323+0000","last_undegraded":"2024-04-25T15:50:32.748323+0000","last_fullsized":"2024-04-25T15:50:32.748323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:34:06.773860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747841+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:50:32.747841+0000","last_peered":"2024-04-25T15:50:32.747841+0000","last_clean":"2024-04-25T15:50:32.747841+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:50:32.747841+0000","last_undegraded":"2024-04-25T15:50:32.747841+0000","last_fullsized":"2024-04-25T15:50:32.747841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150636+0000","last_change":"2024-04-25T15:50:31.174593+0000","last_active":"2024-04-25T15:50:32.150636+0000","last_peered":"2024-04-25T15:50:32.150636+0000","last_clean":"2024-04-25T15:50:32.150636+0000","last_became_active":"2024-04-25T15:50:31.173352+0000","last_became_peered":"2024-04-25T15:50:31.173352+0000","last_unstale":"2024-04-25T15:50:32.150636+0000","last_undegraded":"2024-04-25T15:50:32.150636+0000","last_fullsized":"2024-04-25T15:50:32.150636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:45.575622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247786+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:50:32.247786+0000","last_peered":"2024-04-25T15:50:32.247786+0000","last_clean":"2024-04-25T15:50:32.247786+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:50:32.247786+0000","last_undegraded":"2024-04-25T15:50:32.247786+0000","last_fullsized":"2024-04-25T15:50:32.247786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750532+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:50:32.750532+0000","last_peered":"2024-04-25T15:50:32.750532+0000","last_clean":"2024-04-25T15:50:32.750532+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:50:32.750532+0000","last_undegraded":"2024-04-25T15:50:32.750532+0000","last_fullsized":"2024-04-25T15:50:32.750532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749002+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:50:32.749002+0000","last_peered":"2024-04-25T15:50:32.749002+0000","last_clean":"2024-04-25T15:50:32.749002+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:50:32.749002+0000","last_undegraded":"2024-04-25T15:50:32.749002+0000","last_fullsized":"2024-04-25T15:50:32.749002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246857+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:50:32.246857+0000","last_peered":"2024-04-25T15:50:32.246857+0000","last_clean":"2024-04-25T15:50:32.246857+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:50:32.246857+0000","last_undegraded":"2024-04-25T15:50:32.246857+0000","last_fullsized":"2024-04-25T15:50:32.246857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246486+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:50:32.246486+0000","last_peered":"2024-04-25T15:50:32.246486+0000","last_clean":"2024-04-25T15:50:32.246486+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:50:32.246486+0000","last_undegraded":"2024-04-25T15:50:32.246486+0000","last_fullsized":"2024-04-25T15:50:32.246486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150445+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:50:32.150445+0000","last_peered":"2024-04-25T15:50:32.150445+0000","last_clean":"2024-04-25T15:50:32.150445+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:50:32.150445+0000","last_undegraded":"2024-04-25T15:50:32.150445+0000","last_fullsized":"2024-04-25T15:50:32.150445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148600+0000","last_change":"2024-04-25T15:50:31.182857+0000","last_active":"2024-04-25T15:50:32.148600+0000","last_peered":"2024-04-25T15:50:32.148600+0000","last_clean":"2024-04-25T15:50:32.148600+0000","last_became_active":"2024-04-25T15:50:31.181994+0000","last_became_peered":"2024-04-25T15:50:31.181994+0000","last_unstale":"2024-04-25T15:50:32.148600+0000","last_undegraded":"2024-04-25T15:50:32.148600+0000","last_fullsized":"2024-04-25T15:50:32.148600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:58.571421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.751547+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:50:32.751547+0000","last_peered":"2024-04-25T15:50:32.751547+0000","last_clean":"2024-04-25T15:50:32.751547+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:50:32.751547+0000","last_undegraded":"2024-04-25T15:50:32.751547+0000","last_fullsized":"2024-04-25T15:50:32.751547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154204+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:50:32.154204+0000","last_peered":"2024-04-25T15:50:32.154204+0000","last_clean":"2024-04-25T15:50:32.154204+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:50:32.154204+0000","last_undegraded":"2024-04-25T15:50:32.154204+0000","last_fullsized":"2024-04-25T15:50:32.154204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154135+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:50:32.154135+0000","last_peered":"2024-04-25T15:50:32.154135+0000","last_clean":"2024-04-25T15:50:32.154135+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:50:32.154135+0000","last_undegraded":"2024-04-25T15:50:32.154135+0000","last_fullsized":"2024-04-25T15:50:32.154135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749352+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:50:32.749352+0000","last_peered":"2024-04-25T15:50:32.749352+0000","last_clean":"2024-04-25T15:50:32.749352+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:50:32.749352+0000","last_undegraded":"2024-04-25T15:50:32.749352+0000","last_fullsized":"2024-04-25T15:50:32.749352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148621+0000","last_change":"2024-04-25T15:50:30.200105+0000","last_active":"2024-04-25T15:50:32.148621+0000","last_peered":"2024-04-25T15:50:32.148621+0000","last_clean":"2024-04-25T15:50:32.148621+0000","last_became_active":"2024-04-25T15:50:30.199328+0000","last_became_peered":"2024-04-25T15:50:30.199328+0000","last_unstale":"2024-04-25T15:50:32.148621+0000","last_undegraded":"2024-04-25T15:50:32.148621+0000","last_fullsized":"2024-04-25T15:50:32.148621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:59.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151895+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:50:32.151895+0000","last_peered":"2024-04-25T15:50:32.151895+0000","last_clean":"2024-04-25T15:50:32.151895+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:50:32.151895+0000","last_undegraded":"2024-04-25T15:50:32.151895+0000","last_fullsized":"2024-04-25T15:50:32.151895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749431+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:50:32.749431+0000","last_peered":"2024-04-25T15:50:32.749431+0000","last_clean":"2024-04-25T15:50:32.749431+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:50:32.749431+0000","last_undegraded":"2024-04-25T15:50:32.749431+0000","last_fullsized":"2024-04-25T15:50:32.749431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748150+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:50:32.748150+0000","last_peered":"2024-04-25T15:50:32.748150+0000","last_clean":"2024-04-25T15:50:32.748150+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:50:32.748150+0000","last_undegraded":"2024-04-25T15:50:32.748150+0000","last_fullsized":"2024-04-25T15:50:32.748150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750352+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:50:32.750352+0000","last_peered":"2024-04-25T15:50:32.750352+0000","last_clean":"2024-04-25T15:50:32.750352+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:50:32.750352+0000","last_undegraded":"2024-04-25T15:50:32.750352+0000","last_fullsized":"2024-04-25T15:50:32.750352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750368+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:50:32.750368+0000","last_peered":"2024-04-25T15:50:32.750368+0000","last_clean":"2024-04-25T15:50:32.750368+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:50:32.750368+0000","last_undegraded":"2024-04-25T15:50:32.750368+0000","last_fullsized":"2024-04-25T15:50:32.750368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152764+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:50:32.152764+0000","last_peered":"2024-04-25T15:50:32.152764+0000","last_clean":"2024-04-25T15:50:32.152764+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:50:32.152764+0000","last_undegraded":"2024-04-25T15:50:32.152764+0000","last_fullsized":"2024-04-25T15:50:32.152764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748438+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:50:32.748438+0000","last_peered":"2024-04-25T15:50:32.748438+0000","last_clean":"2024-04-25T15:50:32.748438+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:50:32.748438+0000","last_undegraded":"2024-04-25T15:50:32.748438+0000","last_fullsized":"2024-04-25T15:50:32.748438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748758+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:50:32.748758+0000","last_peered":"2024-04-25T15:50:32.748758+0000","last_clean":"2024-04-25T15:50:32.748758+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:50:32.748758+0000","last_undegraded":"2024-04-25T15:50:32.748758+0000","last_fullsized":"2024-04-25T15:50:32.748758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748532+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:50:32.748532+0000","last_peered":"2024-04-25T15:50:32.748532+0000","last_clean":"2024-04-25T15:50:32.748532+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:50:32.748532+0000","last_undegraded":"2024-04-25T15:50:32.748532+0000","last_fullsized":"2024-04-25T15:50:32.748532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747849+0000","last_change":"2024-04-25T15:50:30.179053+0000","last_active":"2024-04-25T15:50:32.747849+0000","last_peered":"2024-04-25T15:50:32.747849+0000","last_clean":"2024-04-25T15:50:32.747849+0000","last_became_active":"2024-04-25T15:50:30.178196+0000","last_became_peered":"2024-04-25T15:50:30.178196+0000","last_unstale":"2024-04-25T15:50:32.747849+0000","last_undegraded":"2024-04-25T15:50:32.747849+0000","last_fullsized":"2024-04-25T15:50:32.747849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:26.635717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748379+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:50:32.748379+0000","last_peered":"2024-04-25T15:50:32.748379+0000","last_clean":"2024-04-25T15:50:32.748379+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:50:32.748379+0000","last_undegraded":"2024-04-25T15:50:32.748379+0000","last_fullsized":"2024-04-25T15:50:32.748379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152210+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:50:32.152210+0000","last_peered":"2024-04-25T15:50:32.152210+0000","last_clean":"2024-04-25T15:50:32.152210+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:50:32.152210+0000","last_undegraded":"2024-04-25T15:50:32.152210+0000","last_fullsized":"2024-04-25T15:50:32.152210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749720+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:50:32.749720+0000","last_peered":"2024-04-25T15:50:32.749720+0000","last_clean":"2024-04-25T15:50:32.749720+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:50:32.749720+0000","last_undegraded":"2024-04-25T15:50:32.749720+0000","last_fullsized":"2024-04-25T15:50:32.749720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749054+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:50:32.749054+0000","last_peered":"2024-04-25T15:50:32.749054+0000","last_clean":"2024-04-25T15:50:32.749054+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:50:32.749054+0000","last_undegraded":"2024-04-25T15:50:32.749054+0000","last_fullsized":"2024-04-25T15:50:32.749054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750686+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:50:32.750686+0000","last_peered":"2024-04-25T15:50:32.750686+0000","last_clean":"2024-04-25T15:50:32.750686+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:50:32.750686+0000","last_undegraded":"2024-04-25T15:50:32.750686+0000","last_fullsized":"2024-04-25T15:50:32.750686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148745+0000","last_change":"2024-04-25T15:50:30.192595+0000","last_active":"2024-04-25T15:50:32.148745+0000","last_peered":"2024-04-25T15:50:32.148745+0000","last_clean":"2024-04-25T15:50:32.148745+0000","last_became_active":"2024-04-25T15:50:30.192044+0000","last_became_peered":"2024-04-25T15:50:30.192044+0000","last_unstale":"2024-04-25T15:50:32.148745+0000","last_undegraded":"2024-04-25T15:50:32.148745+0000","last_fullsized":"2024-04-25T15:50:32.148745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:08.837774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153693+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:50:32.153693+0000","last_peered":"2024-04-25T15:50:32.153693+0000","last_clean":"2024-04-25T15:50:32.153693+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:50:32.153693+0000","last_undegraded":"2024-04-25T15:50:32.153693+0000","last_fullsized":"2024-04-25T15:50:32.153693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749938+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:50:32.749938+0000","last_peered":"2024-04-25T15:50:32.749938+0000","last_clean":"2024-04-25T15:50:32.749938+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:50:32.749938+0000","last_undegraded":"2024-04-25T15:50:32.749938+0000","last_fullsized":"2024-04-25T15:50:32.749938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748511+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:50:32.748511+0000","last_peered":"2024-04-25T15:50:32.748511+0000","last_clean":"2024-04-25T15:50:32.748511+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:50:32.748511+0000","last_undegraded":"2024-04-25T15:50:32.748511+0000","last_fullsized":"2024-04-25T15:50:32.748511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747640+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:50:32.747640+0000","last_peered":"2024-04-25T15:50:32.747640+0000","last_clean":"2024-04-25T15:50:32.747640+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:50:32.747640+0000","last_undegraded":"2024-04-25T15:50:32.747640+0000","last_fullsized":"2024-04-25T15:50:32.747640+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152008+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:50:32.152008+0000","last_peered":"2024-04-25T15:50:32.152008+0000","last_clean":"2024-04-25T15:50:32.152008+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:50:32.152008+0000","last_undegraded":"2024-04-25T15:50:32.152008+0000","last_fullsized":"2024-04-25T15:50:32.152008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247247+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:50:32.247247+0000","last_peered":"2024-04-25T15:50:32.247247+0000","last_clean":"2024-04-25T15:50:32.247247+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:50:32.247247+0000","last_undegraded":"2024-04-25T15:50:32.247247+0000","last_fullsized":"2024-04-25T15:50:32.247247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747456+0000","last_change":"2024-04-25T15:50:30.197359+0000","last_active":"2024-04-25T15:50:32.747456+0000","last_peered":"2024-04-25T15:50:32.747456+0000","last_clean":"2024-04-25T15:50:32.747456+0000","last_became_active":"2024-04-25T15:50:30.196640+0000","last_became_peered":"2024-04-25T15:50:30.196640+0000","last_unstale":"2024-04-25T15:50:32.747456+0000","last_undegraded":"2024-04-25T15:50:32.747456+0000","last_fullsized":"2024-04-25T15:50:32.747456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.928809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750609+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:50:32.750609+0000","last_peered":"2024-04-25T15:50:32.750609+0000","last_clean":"2024-04-25T15:50:32.750609+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:50:32.750609+0000","last_undegraded":"2024-04-25T15:50:32.750609+0000","last_fullsized":"2024-04-25T15:50:32.750609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748372+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:50:32.748372+0000","last_peered":"2024-04-25T15:50:32.748372+0000","last_clean":"2024-04-25T15:50:32.748372+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:50:32.748372+0000","last_undegraded":"2024-04-25T15:50:32.748372+0000","last_fullsized":"2024-04-25T15:50:32.748372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148771+0000","last_change":"2024-04-25T15:50:31.171339+0000","last_active":"2024-04-25T15:50:32.148771+0000","last_peered":"2024-04-25T15:50:32.148771+0000","last_clean":"2024-04-25T15:50:32.148771+0000","last_became_active":"2024-04-25T15:50:31.170349+0000","last_became_peered":"2024-04-25T15:50:31.170349+0000","last_unstale":"2024-04-25T15:50:32.148771+0000","last_undegraded":"2024-04-25T15:50:32.148771+0000","last_fullsized":"2024-04-25T15:50:32.148771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:24.647984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153062+0000","last_change":"2024-04-25T15:50:30.195563+0000","last_active":"2024-04-25T15:50:32.153062+0000","last_peered":"2024-04-25T15:50:32.153062+0000","last_clean":"2024-04-25T15:50:32.153062+0000","last_became_active":"2024-04-25T15:50:30.194468+0000","last_became_peered":"2024-04-25T15:50:30.194468+0000","last_unstale":"2024-04-25T15:50:32.153062+0000","last_undegraded":"2024-04-25T15:50:32.153062+0000","last_fullsized":"2024-04-25T15:50:32.153062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:04.850096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151035+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:50:32.151035+0000","last_peered":"2024-04-25T15:50:32.151035+0000","last_clean":"2024-04-25T15:50:32.151035+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:50:32.151035+0000","last_undegraded":"2024-04-25T15:50:32.151035+0000","last_fullsized":"2024-04-25T15:50:32.151035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149161+0000","last_change":"2024-04-25T15:50:30.175727+0000","last_active":"2024-04-25T15:50:32.149161+0000","last_peered":"2024-04-25T15:50:32.149161+0000","last_clean":"2024-04-25T15:50:32.149161+0000","last_became_active":"2024-04-25T15:50:30.175136+0000","last_became_peered":"2024-04-25T15:50:30.175136+0000","last_unstale":"2024-04-25T15:50:32.149161+0000","last_undegraded":"2024-04-25T15:50:32.149161+0000","last_fullsized":"2024-04-25T15:50:32.149161+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:22.434474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153102+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:50:32.153102+0000","last_peered":"2024-04-25T15:50:32.153102+0000","last_clean":"2024-04-25T15:50:32.153102+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:50:32.153102+0000","last_undegraded":"2024-04-25T15:50:32.153102+0000","last_fullsized":"2024-04-25T15:50:32.153102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749240+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:50:32.749240+0000","last_peered":"2024-04-25T15:50:32.749240+0000","last_clean":"2024-04-25T15:50:32.749240+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:50:32.749240+0000","last_undegraded":"2024-04-25T15:50:32.749240+0000","last_fullsized":"2024-04-25T15:50:32.749240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149163+0000","last_change":"2024-04-25T15:50:31.177424+0000","last_active":"2024-04-25T15:50:32.149163+0000","last_peered":"2024-04-25T15:50:32.149163+0000","last_clean":"2024-04-25T15:50:32.149163+0000","last_became_active":"2024-04-25T15:50:31.176553+0000","last_became_peered":"2024-04-25T15:50:31.176553+0000","last_unstale":"2024-04-25T15:50:32.149163+0000","last_undegraded":"2024-04-25T15:50:32.149163+0000","last_fullsized":"2024-04-25T15:50:32.149163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:29.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149356+0000","last_change":"2024-04-25T15:50:30.191244+0000","last_active":"2024-04-25T15:50:32.149356+0000","last_peered":"2024-04-25T15:50:32.149356+0000","last_clean":"2024-04-25T15:50:32.149356+0000","last_became_active":"2024-04-25T15:50:30.190437+0000","last_became_peered":"2024-04-25T15:50:30.190437+0000","last_unstale":"2024-04-25T15:50:32.149356+0000","last_undegraded":"2024-04-25T15:50:32.149356+0000","last_fullsized":"2024-04-25T15:50:32.149356+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:55.506915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247702+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:50:32.247702+0000","last_peered":"2024-04-25T15:50:32.247702+0000","last_clean":"2024-04-25T15:50:32.247702+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:50:32.247702+0000","last_undegraded":"2024-04-25T15:50:32.247702+0000","last_fullsized":"2024-04-25T15:50:32.247702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.751545+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:50:32.751545+0000","last_peered":"2024-04-25T15:50:32.751545+0000","last_clean":"2024-04-25T15:50:32.751545+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:50:32.751545+0000","last_undegraded":"2024-04-25T15:50:32.751545+0000","last_fullsized":"2024-04-25T15:50:32.751545+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749239+0000","last_change":"2024-04-25T15:50:31.182607+0000","last_active":"2024-04-25T15:50:32.749239+0000","last_peered":"2024-04-25T15:50:32.749239+0000","last_clean":"2024-04-25T15:50:32.749239+0000","last_became_active":"2024-04-25T15:50:31.181641+0000","last_became_peered":"2024-04-25T15:50:31.181641+0000","last_unstale":"2024-04-25T15:50:32.749239+0000","last_undegraded":"2024-04-25T15:50:32.749239+0000","last_fullsized":"2024-04-25T15:50:32.749239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:49.636675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747485+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:50:32.747485+0000","last_peered":"2024-04-25T15:50:32.747485+0000","last_clean":"2024-04-25T15:50:32.747485+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:50:32.747485+0000","last_undegraded":"2024-04-25T15:50:32.747485+0000","last_fullsized":"2024-04-25T15:50:32.747485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151509+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:50:32.151509+0000","last_peered":"2024-04-25T15:50:32.151509+0000","last_clean":"2024-04-25T15:50:32.151509+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:50:32.151509+0000","last_undegraded":"2024-04-25T15:50:32.151509+0000","last_fullsized":"2024-04-25T15:50:32.151509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148622+0000","last_change":"2024-04-25T15:50:30.178078+0000","last_active":"2024-04-25T15:50:32.148622+0000","last_peered":"2024-04-25T15:50:32.148622+0000","last_clean":"2024-04-25T15:50:32.148622+0000","last_became_active":"2024-04-25T15:50:30.177718+0000","last_became_peered":"2024-04-25T15:50:30.177718+0000","last_unstale":"2024-04-25T15:50:32.148622+0000","last_undegraded":"2024-04-25T15:50:32.148622+0000","last_fullsized":"2024-04-25T15:50:32.148622+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246699+0000","last_change":"2024-04-25T15:50:31.192152+0000","last_active":"2024-04-25T15:50:32.246699+0000","last_peered":"2024-04-25T15:50:32.246699+0000","last_clean":"2024-04-25T15:50:32.246699+0000","last_became_active":"2024-04-25T15:50:31.179200+0000","last_became_peered":"2024-04-25T15:50:31.179200+0000","last_unstale":"2024-04-25T15:50:32.246699+0000","last_undegraded":"2024-04-25T15:50:32.246699+0000","last_fullsized":"2024-04-25T15:50:32.246699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:03.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148915+0000","last_change":"2024-04-25T15:50:30.184989+0000","last_active":"2024-04-25T15:50:32.148915+0000","last_peered":"2024-04-25T15:50:32.148915+0000","last_clean":"2024-04-25T15:50:32.148915+0000","last_became_active":"2024-04-25T15:50:30.184132+0000","last_became_peered":"2024-04-25T15:50:30.184132+0000","last_unstale":"2024-04-25T15:50:32.148915+0000","last_undegraded":"2024-04-25T15:50:32.148915+0000","last_fullsized":"2024-04-25T15:50:32.148915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:03.833303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247766+0000","last_change":"2024-04-25T15:50:31.190275+0000","last_active":"2024-04-25T15:50:32.247766+0000","last_peered":"2024-04-25T15:50:32.247766+0000","last_clean":"2024-04-25T15:50:32.247766+0000","last_became_active":"2024-04-25T15:50:31.189302+0000","last_became_peered":"2024-04-25T15:50:31.189302+0000","last_unstale":"2024-04-25T15:50:32.247766+0000","last_undegraded":"2024-04-25T15:50:32.247766+0000","last_fullsized":"2024-04-25T15:50:32.247766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:39.479706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748920+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:50:32.748920+0000","last_peered":"2024-04-25T15:50:32.748920+0000","last_clean":"2024-04-25T15:50:32.748920+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:50:32.748920+0000","last_undegraded":"2024-04-25T15:50:32.748920+0000","last_fullsized":"2024-04-25T15:50:32.748920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748648+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:50:32.748648+0000","last_peered":"2024-04-25T15:50:32.748648+0000","last_clean":"2024-04-25T15:50:32.748648+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:50:32.748648+0000","last_undegraded":"2024-04-25T15:50:32.748648+0000","last_fullsized":"2024-04-25T15:50:32.748648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153111+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:50:32.153111+0000","last_peered":"2024-04-25T15:50:32.153111+0000","last_clean":"2024-04-25T15:50:32.153111+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:50:32.153111+0000","last_undegraded":"2024-04-25T15:50:32.153111+0000","last_fullsized":"2024-04-25T15:50:32.153111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749776+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:50:32.749776+0000","last_peered":"2024-04-25T15:50:32.749776+0000","last_clean":"2024-04-25T15:50:32.749776+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:50:32.749776+0000","last_undegraded":"2024-04-25T15:50:32.749776+0000","last_fullsized":"2024-04-25T15:50:32.749776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153450+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:50:32.153450+0000","last_peered":"2024-04-25T15:50:32.153450+0000","last_clean":"2024-04-25T15:50:32.153450+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:50:32.153450+0000","last_undegraded":"2024-04-25T15:50:32.153450+0000","last_fullsized":"2024-04-25T15:50:32.153450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749103+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:50:32.749103+0000","last_peered":"2024-04-25T15:50:32.749103+0000","last_clean":"2024-04-25T15:50:32.749103+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:50:32.749103+0000","last_undegraded":"2024-04-25T15:50:32.749103+0000","last_fullsized":"2024-04-25T15:50:32.749103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749087+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:50:32.749087+0000","last_peered":"2024-04-25T15:50:32.749087+0000","last_clean":"2024-04-25T15:50:32.749087+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:50:32.749087+0000","last_undegraded":"2024-04-25T15:50:32.749087+0000","last_fullsized":"2024-04-25T15:50:32.749087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247277+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:50:32.247277+0000","last_peered":"2024-04-25T15:50:32.247277+0000","last_clean":"2024-04-25T15:50:32.247277+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:50:32.247277+0000","last_undegraded":"2024-04-25T15:50:32.247277+0000","last_fullsized":"2024-04-25T15:50:32.247277+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247337+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:50:32.247337+0000","last_peered":"2024-04-25T15:50:32.247337+0000","last_clean":"2024-04-25T15:50:32.247337+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:50:32.247337+0000","last_undegraded":"2024-04-25T15:50:32.247337+0000","last_fullsized":"2024-04-25T15:50:32.247337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749650+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:50:32.749650+0000","last_peered":"2024-04-25T15:50:32.749650+0000","last_clean":"2024-04-25T15:50:32.749650+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:50:32.749650+0000","last_undegraded":"2024-04-25T15:50:32.749650+0000","last_fullsized":"2024-04-25T15:50:32.749650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":655002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":655002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:50:37.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:50:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5ba875640 1 Processor -- start 2024-04-25T15:50:37.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5ba875640 1 -- start start 2024-04-25T15:50:37.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5ba875640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa5b40cbf30 0x7fa5b415c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5ba875640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa5b4071b50 0x7fa5b4071f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5ba875640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5b40cc340 con 0x7fa5b41609d0 2024-04-25T15:50:37.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5ba875640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b4160de0 con 0x7fa5b4071b50 2024-04-25T15:50:37.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5ba875640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b415d1a0 con 0x7fa5b40cbf30 2024-04-25T15:50:37.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b8e12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa5b4071b50 0x7fa5b4071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b8e12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa5b4071b50 0x7fa5b4071f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59516/0 (socket says 172.21.15.67:59516) 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b8e12640 1 -- 172.21.15.67:0/3561493471 learned_addr learned my addr 172.21.15.67:0/3561493471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b37fe640 1 --2- 172.21.15.67:0/3561493471 >> v2:172.21.15.176:3300/0 conn(0x7fa5b40cbf30 0x7fa5b415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b8e12640 1 -- 172.21.15.67:0/3561493471 >> v2:172.21.15.176:3300/0 conn(0x7fa5b40cbf30 msgr2=0x7fa5b415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b8e12640 1 --2- 172.21.15.67:0/3561493471 >> v2:172.21.15.176:3300/0 conn(0x7fa5b40cbf30 0x7fa5b415c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b8e12640 1 -- 172.21.15.67:0/3561493471 >> v1:172.21.15.67:6789/0 conn(0x7fa5b41609d0 legacy=0x7fa5b40ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b2ffd640 1 -- 172.21.15.67:0/3561493471 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450456642 0 0) 0x7fa5b40cc340 con 0x7fa5b41609d0 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b8e12640 1 -- 172.21.15.67:0/3561493471 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b415d480 con 0x7fa5b4071b50 2024-04-25T15:50:37.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.602+0000 7fa5b37fe640 1 --2- 172.21.15.67:0/3561493471 >> v2:172.21.15.176:3300/0 conn(0x7fa5b40cbf30 0x7fa5b415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:37.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5b8e12640 1 --2- 172.21.15.67:0/3561493471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b4071b50 0x7fa5b4071f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4073a20 tx=0x7fa5ac01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730822baea21794b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:37.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5b2ffd640 1 -- 172.21.15.67:0/3561493471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0a56c0 con 0x7fa5b4071b50 2024-04-25T15:50:37.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5b2ffd640 1 -- 172.21.15.67:0/3561493471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ac0a5860 con 0x7fa5b4071b50 2024-04-25T15:50:37.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5b2ffd640 1 -- 172.21.15.67:0/3561493471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0a5c30 con 0x7fa5b4071b50 2024-04-25T15:50:37.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 -- 172.21.15.67:0/3561493471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b4071b50 msgr2=0x7fa5b4071f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 --2- 172.21.15.67:0/3561493471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b4071b50 0x7fa5b4071f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4073a20 tx=0x7fa5ac01c9f0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 -- 172.21.15.67:0/3561493471 shutdown_connections 2024-04-25T15:50:37.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 --2- 172.21.15.67:0/3561493471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b4071b50 0x7fa5b4071f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 --2- 172.21.15.67:0/3561493471 >> v2:172.21.15.176:3300/0 conn(0x7fa5b40cbf30 0x7fa5b415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 -- 172.21.15.67:0/3561493471 >> 172.21.15.67:0/3561493471 conn(0x7fa5b40700b0 msgr2=0x7fa5b40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 -- 172.21.15.67:0/3561493471 shutdown_connections 2024-04-25T15:50:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.606+0000 7fa5ba875640 1 -- 172.21.15.67:0/3561493471 wait complete. 2024-04-25T15:50:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 Processor -- start 2024-04-25T15:50:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 -- start start 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5b4071b50 0x7fa5b414a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b40cbf30 0x7fa5b414b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5b41609d0 0x7fa5b4154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b415cdc0 con 0x7fa5b40cbf30 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b415d760 con 0x7fa5b4071b50 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b415cf40 con 0x7fa5b41609d0 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b40cbf30 0x7fa5b414b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b40cbf30 0x7fa5b414b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59524/0 (socket says 172.21.15.67:59524) 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 -- 172.21.15.67:0/2923208371 learned_addr learned my addr 172.21.15.67:0/2923208371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b3fff640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5b4071b50 0x7fa5b414a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b8e12640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5b41609d0 0x7fa5b4154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 -- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5b41609d0 msgr2=0x7fa5b4154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5b41609d0 0x7fa5b4154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 -- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5b4071b50 msgr2=0x7fa5b414a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5b4071b50 0x7fa5b414a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 -- 172.21.15.67:0/2923208371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b41f9670 con 0x7fa5b40cbf30 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b37fe640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b40cbf30 0x7fa5b414b5a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400d830 tx=0x7fa5a400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b0ff9640 1 -- 172.21.15.67:0/2923208371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a40096a0 con 0x7fa5b40cbf30 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b0ff9640 1 -- 172.21.15.67:0/2923208371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a4019070 con 0x7fa5b40cbf30 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5b0ff9640 1 -- 172.21.15.67:0/2923208371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4009a80 con 0x7fa5b40cbf30 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b414a7e0 con 0x7fa5b40cbf30 2024-04-25T15:50:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.610+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b414ada0 con 0x7fa5b40cbf30 2024-04-25T15:50:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.614+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b406a480 con 0x7fa5b40cbf30 2024-04-25T15:50:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.614+0000 7fa5b0ff9640 1 -- 172.21.15.67:0/2923208371 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa5a401e460 con 0x7fa5b40cbf30 2024-04-25T15:50:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.614+0000 7fa5b0ff9640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa5a006f430 0x7fa5a0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.614+0000 7fa5b0ff9640 1 -- 172.21.15.67:0/2923208371 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa5a408f910 con 0x7fa5b40cbf30 2024-04-25T15:50:37.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.614+0000 7fa5b3fff640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa5a006f430 0x7fa5a0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:37.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.614+0000 7fa5b3fff640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa5a006f430 0x7fa5a0071940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000f70 tx=0x7fa5a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.618+0000 7fa5b0ff9640 1 -- 172.21.15.67:0/2923208371 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa5a405c800 con 0x7fa5b40cbf30 2024-04-25T15:50:37.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.774+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5b41f9400 con 0x7fa5a006f430 2024-04-25T15:50:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.794+0000 7fa5b0ff9640 1 -- 172.21.15.67:0/2923208371 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251838 (secure 0 0 0) 0x7fa5b41f9400 con 0x7fa5a006f430 2024-04-25T15:50:37.797 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:37.799 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:50:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa5a006f430 msgr2=0x7fa5a0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa5a006f430 0x7fa5a0071940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8000f70 tx=0x7fa5a8009040 comp rx=0 tx=0).stop 2024-04-25T15:50:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b40cbf30 msgr2=0x7fa5b414b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b40cbf30 0x7fa5b414b5a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5a400d830 tx=0x7fa5a400dcf0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 shutdown_connections 2024-04-25T15:50:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa5a006f430 0x7fa5a0071940 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5b41609d0 0x7fa5b4154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b40cbf30 0x7fa5b414b5a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 --2- 172.21.15.67:0/2923208371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5b4071b50 0x7fa5b414a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:37.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 >> 172.21.15.67:0/2923208371 conn(0x7fa5b40700b0 msgr2=0x7fa5b41518a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:37.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 shutdown_connections 2024-04-25T15:50:37.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:37.798+0000 7fa5ba875640 1 -- 172.21.15.67:0/2923208371 wait complete. 2024-04-25T15:50:37.817 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-04-25T15:50:35.798881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218920,"kb_used_data":2712,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755800,"statfs":{"total":773094113280,"available":772869939200,"internally_reserved":0,"allocated":2777088,"data_stored":1723008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002316"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750328+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:50:32.750328+0000","last_peered":"2024-04-25T15:50:32.750328+0000","last_clean":"2024-04-25T15:50:32.750328+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:50:32.750328+0000","last_undegraded":"2024-04-25T15:50:32.750328+0000","last_fullsized":"2024-04-25T15:50:32.750328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149188+0000","last_change":"2024-04-25T15:50:31.192224+0000","last_active":"2024-04-25T15:50:32.149188+0000","last_peered":"2024-04-25T15:50:32.149188+0000","last_clean":"2024-04-25T15:50:32.149188+0000","last_became_active":"2024-04-25T15:50:31.189311+0000","last_became_peered":"2024-04-25T15:50:31.189311+0000","last_unstale":"2024-04-25T15:50:32.149188+0000","last_undegraded":"2024-04-25T15:50:32.149188+0000","last_fullsized":"2024-04-25T15:50:32.149188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.837762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750252+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:50:32.750252+0000","last_peered":"2024-04-25T15:50:32.750252+0000","last_clean":"2024-04-25T15:50:32.750252+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:50:32.750252+0000","last_undegraded":"2024-04-25T15:50:32.750252+0000","last_fullsized":"2024-04-25T15:50:32.750252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152869+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:50:32.152869+0000","last_peered":"2024-04-25T15:50:32.152869+0000","last_clean":"2024-04-25T15:50:32.152869+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:50:32.152869+0000","last_undegraded":"2024-04-25T15:50:32.152869+0000","last_fullsized":"2024-04-25T15:50:32.152869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748897+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:50:32.748897+0000","last_peered":"2024-04-25T15:50:32.748897+0000","last_clean":"2024-04-25T15:50:32.748897+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:50:32.748897+0000","last_undegraded":"2024-04-25T15:50:32.748897+0000","last_fullsized":"2024-04-25T15:50:32.748897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151867+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:50:32.151867+0000","last_peered":"2024-04-25T15:50:32.151867+0000","last_clean":"2024-04-25T15:50:32.151867+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:50:32.151867+0000","last_undegraded":"2024-04-25T15:50:32.151867+0000","last_fullsized":"2024-04-25T15:50:32.151867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747896+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:50:32.747896+0000","last_peered":"2024-04-25T15:50:32.747896+0000","last_clean":"2024-04-25T15:50:32.747896+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:50:32.747896+0000","last_undegraded":"2024-04-25T15:50:32.747896+0000","last_fullsized":"2024-04-25T15:50:32.747896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246622+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:50:32.246622+0000","last_peered":"2024-04-25T15:50:32.246622+0000","last_clean":"2024-04-25T15:50:32.246622+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:50:32.246622+0000","last_undegraded":"2024-04-25T15:50:32.246622+0000","last_fullsized":"2024-04-25T15:50:32.246622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750587+0000","last_change":"2024-04-25T15:50:31.210825+0000","last_active":"2024-04-25T15:50:32.750587+0000","last_peered":"2024-04-25T15:50:32.750587+0000","last_clean":"2024-04-25T15:50:32.750587+0000","last_became_active":"2024-04-25T15:50:31.210510+0000","last_became_peered":"2024-04-25T15:50:31.210510+0000","last_unstale":"2024-04-25T15:50:32.750587+0000","last_undegraded":"2024-04-25T15:50:32.750587+0000","last_fullsized":"2024-04-25T15:50:32.750587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:23.810710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749861+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:50:32.749861+0000","last_peered":"2024-04-25T15:50:32.749861+0000","last_clean":"2024-04-25T15:50:32.749861+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:50:32.749861+0000","last_undegraded":"2024-04-25T15:50:32.749861+0000","last_fullsized":"2024-04-25T15:50:32.749861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750086+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:50:32.750086+0000","last_peered":"2024-04-25T15:50:32.750086+0000","last_clean":"2024-04-25T15:50:32.750086+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:50:32.750086+0000","last_undegraded":"2024-04-25T15:50:32.750086+0000","last_fullsized":"2024-04-25T15:50:32.750086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151449+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:50:32.151449+0000","last_peered":"2024-04-25T15:50:32.151449+0000","last_clean":"2024-04-25T15:50:32.151449+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:50:32.151449+0000","last_undegraded":"2024-04-25T15:50:32.151449+0000","last_fullsized":"2024-04-25T15:50:32.151449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247483+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:50:32.247483+0000","last_peered":"2024-04-25T15:50:32.247483+0000","last_clean":"2024-04-25T15:50:32.247483+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:50:32.247483+0000","last_undegraded":"2024-04-25T15:50:32.247483+0000","last_fullsized":"2024-04-25T15:50:32.247483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750096+0000","last_change":"2024-04-25T15:50:31.171300+0000","last_active":"2024-04-25T15:50:32.750096+0000","last_peered":"2024-04-25T15:50:32.750096+0000","last_clean":"2024-04-25T15:50:32.750096+0000","last_became_active":"2024-04-25T15:50:31.170451+0000","last_became_peered":"2024-04-25T15:50:31.170451+0000","last_unstale":"2024-04-25T15:50:32.750096+0000","last_undegraded":"2024-04-25T15:50:32.750096+0000","last_fullsized":"2024-04-25T15:50:32.750096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:52:45.807418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148500+0000","last_change":"2024-04-25T15:50:31.198817+0000","last_active":"2024-04-25T15:50:32.148500+0000","last_peered":"2024-04-25T15:50:32.148500+0000","last_clean":"2024-04-25T15:50:32.148500+0000","last_became_active":"2024-04-25T15:50:31.198165+0000","last_became_peered":"2024-04-25T15:50:31.198165+0000","last_unstale":"2024-04-25T15:50:32.148500+0000","last_undegraded":"2024-04-25T15:50:32.148500+0000","last_fullsized":"2024-04-25T15:50:32.148500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:05.893280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150141+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:50:32.150141+0000","last_peered":"2024-04-25T15:50:32.150141+0000","last_clean":"2024-04-25T15:50:32.150141+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:50:32.150141+0000","last_undegraded":"2024-04-25T15:50:32.150141+0000","last_fullsized":"2024-04-25T15:50:32.150141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154678+0000","last_change":"2024-04-25T15:50:31.198225+0000","last_active":"2024-04-25T15:50:32.154678+0000","last_peered":"2024-04-25T15:50:32.154678+0000","last_clean":"2024-04-25T15:50:32.154678+0000","last_became_active":"2024-04-25T15:50:31.196737+0000","last_became_peered":"2024-04-25T15:50:31.196737+0000","last_unstale":"2024-04-25T15:50:32.154678+0000","last_undegraded":"2024-04-25T15:50:32.154678+0000","last_fullsized":"2024-04-25T15:50:32.154678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:25.652638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748753+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:50:32.748753+0000","last_peered":"2024-04-25T15:50:32.748753+0000","last_clean":"2024-04-25T15:50:32.748753+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:50:32.748753+0000","last_undegraded":"2024-04-25T15:50:32.748753+0000","last_fullsized":"2024-04-25T15:50:32.748753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748982+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:50:32.748982+0000","last_peered":"2024-04-25T15:50:32.748982+0000","last_clean":"2024-04-25T15:50:32.748982+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:50:32.748982+0000","last_undegraded":"2024-04-25T15:50:32.748982+0000","last_fullsized":"2024-04-25T15:50:32.748982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749171+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:50:32.749171+0000","last_peered":"2024-04-25T15:50:32.749171+0000","last_clean":"2024-04-25T15:50:32.749171+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:50:32.749171+0000","last_undegraded":"2024-04-25T15:50:32.749171+0000","last_fullsized":"2024-04-25T15:50:32.749171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749435+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:50:32.749435+0000","last_peered":"2024-04-25T15:50:32.749435+0000","last_clean":"2024-04-25T15:50:32.749435+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:50:32.749435+0000","last_undegraded":"2024-04-25T15:50:32.749435+0000","last_fullsized":"2024-04-25T15:50:32.749435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152360+0000","last_change":"2024-04-25T15:50:31.198787+0000","last_active":"2024-04-25T15:50:32.152360+0000","last_peered":"2024-04-25T15:50:32.152360+0000","last_clean":"2024-04-25T15:50:32.152360+0000","last_became_active":"2024-04-25T15:50:31.197737+0000","last_became_peered":"2024-04-25T15:50:31.197737+0000","last_unstale":"2024-04-25T15:50:32.152360+0000","last_undegraded":"2024-04-25T15:50:32.152360+0000","last_fullsized":"2024-04-25T15:50:32.152360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:03:21.865150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246985+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:50:32.246985+0000","last_peered":"2024-04-25T15:50:32.246985+0000","last_clean":"2024-04-25T15:50:32.246985+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:50:32.246985+0000","last_undegraded":"2024-04-25T15:50:32.246985+0000","last_fullsized":"2024-04-25T15:50:32.246985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150206+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:50:32.150206+0000","last_peered":"2024-04-25T15:50:32.150206+0000","last_clean":"2024-04-25T15:50:32.150206+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:50:32.150206+0000","last_undegraded":"2024-04-25T15:50:32.150206+0000","last_fullsized":"2024-04-25T15:50:32.150206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247233+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:50:32.247233+0000","last_peered":"2024-04-25T15:50:32.247233+0000","last_clean":"2024-04-25T15:50:32.247233+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:50:32.247233+0000","last_undegraded":"2024-04-25T15:50:32.247233+0000","last_fullsized":"2024-04-25T15:50:32.247233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750349+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:50:32.750349+0000","last_peered":"2024-04-25T15:50:32.750349+0000","last_clean":"2024-04-25T15:50:32.750349+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:50:32.750349+0000","last_undegraded":"2024-04-25T15:50:32.750349+0000","last_fullsized":"2024-04-25T15:50:32.750349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149180+0000","last_change":"2024-04-25T15:50:31.192245+0000","last_active":"2024-04-25T15:50:32.149180+0000","last_peered":"2024-04-25T15:50:32.149180+0000","last_clean":"2024-04-25T15:50:32.149180+0000","last_became_active":"2024-04-25T15:50:31.190989+0000","last_became_peered":"2024-04-25T15:50:31.190989+0000","last_unstale":"2024-04-25T15:50:32.149180+0000","last_undegraded":"2024-04-25T15:50:32.149180+0000","last_fullsized":"2024-04-25T15:50:32.149180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:29:24.813097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247952+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:50:32.247952+0000","last_peered":"2024-04-25T15:50:32.247952+0000","last_clean":"2024-04-25T15:50:32.247952+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:50:32.247952+0000","last_undegraded":"2024-04-25T15:50:32.247952+0000","last_fullsized":"2024-04-25T15:50:32.247952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750022+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:50:32.750022+0000","last_peered":"2024-04-25T15:50:32.750022+0000","last_clean":"2024-04-25T15:50:32.750022+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:50:32.750022+0000","last_undegraded":"2024-04-25T15:50:32.750022+0000","last_fullsized":"2024-04-25T15:50:32.750022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153195+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:50:32.153195+0000","last_peered":"2024-04-25T15:50:32.153195+0000","last_clean":"2024-04-25T15:50:32.153195+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:50:32.153195+0000","last_undegraded":"2024-04-25T15:50:32.153195+0000","last_fullsized":"2024-04-25T15:50:32.153195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748323+0000","last_change":"2024-04-25T15:50:31.174913+0000","last_active":"2024-04-25T15:50:32.748323+0000","last_peered":"2024-04-25T15:50:32.748323+0000","last_clean":"2024-04-25T15:50:32.748323+0000","last_became_active":"2024-04-25T15:50:31.171499+0000","last_became_peered":"2024-04-25T15:50:31.171499+0000","last_unstale":"2024-04-25T15:50:32.748323+0000","last_undegraded":"2024-04-25T15:50:32.748323+0000","last_fullsized":"2024-04-25T15:50:32.748323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:34:06.773860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747841+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:50:32.747841+0000","last_peered":"2024-04-25T15:50:32.747841+0000","last_clean":"2024-04-25T15:50:32.747841+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:50:32.747841+0000","last_undegraded":"2024-04-25T15:50:32.747841+0000","last_fullsized":"2024-04-25T15:50:32.747841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150636+0000","last_change":"2024-04-25T15:50:31.174593+0000","last_active":"2024-04-25T15:50:32.150636+0000","last_peered":"2024-04-25T15:50:32.150636+0000","last_clean":"2024-04-25T15:50:32.150636+0000","last_became_active":"2024-04-25T15:50:31.173352+0000","last_became_peered":"2024-04-25T15:50:31.173352+0000","last_unstale":"2024-04-25T15:50:32.150636+0000","last_undegraded":"2024-04-25T15:50:32.150636+0000","last_fullsized":"2024-04-25T15:50:32.150636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:45.575622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247786+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:50:32.247786+0000","last_peered":"2024-04-25T15:50:32.247786+0000","last_clean":"2024-04-25T15:50:32.247786+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:50:32.247786+0000","last_undegraded":"2024-04-25T15:50:32.247786+0000","last_fullsized":"2024-04-25T15:50:32.247786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750532+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:50:32.750532+0000","last_peered":"2024-04-25T15:50:32.750532+0000","last_clean":"2024-04-25T15:50:32.750532+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:50:32.750532+0000","last_undegraded":"2024-04-25T15:50:32.750532+0000","last_fullsized":"2024-04-25T15:50:32.750532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749002+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:50:32.749002+0000","last_peered":"2024-04-25T15:50:32.749002+0000","last_clean":"2024-04-25T15:50:32.749002+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:50:32.749002+0000","last_undegraded":"2024-04-25T15:50:32.749002+0000","last_fullsized":"2024-04-25T15:50:32.749002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246857+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:50:32.246857+0000","last_peered":"2024-04-25T15:50:32.246857+0000","last_clean":"2024-04-25T15:50:32.246857+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:50:32.246857+0000","last_undegraded":"2024-04-25T15:50:32.246857+0000","last_fullsized":"2024-04-25T15:50:32.246857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246486+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:50:32.246486+0000","last_peered":"2024-04-25T15:50:32.246486+0000","last_clean":"2024-04-25T15:50:32.246486+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:50:32.246486+0000","last_undegraded":"2024-04-25T15:50:32.246486+0000","last_fullsized":"2024-04-25T15:50:32.246486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150445+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:50:32.150445+0000","last_peered":"2024-04-25T15:50:32.150445+0000","last_clean":"2024-04-25T15:50:32.150445+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:50:32.150445+0000","last_undegraded":"2024-04-25T15:50:32.150445+0000","last_fullsized":"2024-04-25T15:50:32.150445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148600+0000","last_change":"2024-04-25T15:50:31.182857+0000","last_active":"2024-04-25T15:50:32.148600+0000","last_peered":"2024-04-25T15:50:32.148600+0000","last_clean":"2024-04-25T15:50:32.148600+0000","last_became_active":"2024-04-25T15:50:31.181994+0000","last_became_peered":"2024-04-25T15:50:31.181994+0000","last_unstale":"2024-04-25T15:50:32.148600+0000","last_undegraded":"2024-04-25T15:50:32.148600+0000","last_fullsized":"2024-04-25T15:50:32.148600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:58.571421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.751547+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:50:32.751547+0000","last_peered":"2024-04-25T15:50:32.751547+0000","last_clean":"2024-04-25T15:50:32.751547+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:50:32.751547+0000","last_undegraded":"2024-04-25T15:50:32.751547+0000","last_fullsized":"2024-04-25T15:50:32.751547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154204+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:50:32.154204+0000","last_peered":"2024-04-25T15:50:32.154204+0000","last_clean":"2024-04-25T15:50:32.154204+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:50:32.154204+0000","last_undegraded":"2024-04-25T15:50:32.154204+0000","last_fullsized":"2024-04-25T15:50:32.154204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154135+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:50:32.154135+0000","last_peered":"2024-04-25T15:50:32.154135+0000","last_clean":"2024-04-25T15:50:32.154135+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:50:32.154135+0000","last_undegraded":"2024-04-25T15:50:32.154135+0000","last_fullsized":"2024-04-25T15:50:32.154135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749352+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:50:32.749352+0000","last_peered":"2024-04-25T15:50:32.749352+0000","last_clean":"2024-04-25T15:50:32.749352+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:50:32.749352+0000","last_undegraded":"2024-04-25T15:50:32.749352+0000","last_fullsized":"2024-04-25T15:50:32.749352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148621+0000","last_change":"2024-04-25T15:50:30.200105+0000","last_active":"2024-04-25T15:50:32.148621+0000","last_peered":"2024-04-25T15:50:32.148621+0000","last_clean":"2024-04-25T15:50:32.148621+0000","last_became_active":"2024-04-25T15:50:30.199328+0000","last_became_peered":"2024-04-25T15:50:30.199328+0000","last_unstale":"2024-04-25T15:50:32.148621+0000","last_undegraded":"2024-04-25T15:50:32.148621+0000","last_fullsized":"2024-04-25T15:50:32.148621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:59.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151895+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:50:32.151895+0000","last_peered":"2024-04-25T15:50:32.151895+0000","last_clean":"2024-04-25T15:50:32.151895+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:50:32.151895+0000","last_undegraded":"2024-04-25T15:50:32.151895+0000","last_fullsized":"2024-04-25T15:50:32.151895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749431+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:50:32.749431+0000","last_peered":"2024-04-25T15:50:32.749431+0000","last_clean":"2024-04-25T15:50:32.749431+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:50:32.749431+0000","last_undegraded":"2024-04-25T15:50:32.749431+0000","last_fullsized":"2024-04-25T15:50:32.749431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748150+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:50:32.748150+0000","last_peered":"2024-04-25T15:50:32.748150+0000","last_clean":"2024-04-25T15:50:32.748150+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:50:32.748150+0000","last_undegraded":"2024-04-25T15:50:32.748150+0000","last_fullsized":"2024-04-25T15:50:32.748150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750352+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:50:32.750352+0000","last_peered":"2024-04-25T15:50:32.750352+0000","last_clean":"2024-04-25T15:50:32.750352+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:50:32.750352+0000","last_undegraded":"2024-04-25T15:50:32.750352+0000","last_fullsized":"2024-04-25T15:50:32.750352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750368+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:50:32.750368+0000","last_peered":"2024-04-25T15:50:32.750368+0000","last_clean":"2024-04-25T15:50:32.750368+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:50:32.750368+0000","last_undegraded":"2024-04-25T15:50:32.750368+0000","last_fullsized":"2024-04-25T15:50:32.750368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152764+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:50:32.152764+0000","last_peered":"2024-04-25T15:50:32.152764+0000","last_clean":"2024-04-25T15:50:32.152764+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:50:32.152764+0000","last_undegraded":"2024-04-25T15:50:32.152764+0000","last_fullsized":"2024-04-25T15:50:32.152764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748438+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:50:32.748438+0000","last_peered":"2024-04-25T15:50:32.748438+0000","last_clean":"2024-04-25T15:50:32.748438+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:50:32.748438+0000","last_undegraded":"2024-04-25T15:50:32.748438+0000","last_fullsized":"2024-04-25T15:50:32.748438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748758+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:50:32.748758+0000","last_peered":"2024-04-25T15:50:32.748758+0000","last_clean":"2024-04-25T15:50:32.748758+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:50:32.748758+0000","last_undegraded":"2024-04-25T15:50:32.748758+0000","last_fullsized":"2024-04-25T15:50:32.748758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748532+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:50:32.748532+0000","last_peered":"2024-04-25T15:50:32.748532+0000","last_clean":"2024-04-25T15:50:32.748532+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:50:32.748532+0000","last_undegraded":"2024-04-25T15:50:32.748532+0000","last_fullsized":"2024-04-25T15:50:32.748532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747849+0000","last_change":"2024-04-25T15:50:30.179053+0000","last_active":"2024-04-25T15:50:32.747849+0000","last_peered":"2024-04-25T15:50:32.747849+0000","last_clean":"2024-04-25T15:50:32.747849+0000","last_became_active":"2024-04-25T15:50:30.178196+0000","last_became_peered":"2024-04-25T15:50:30.178196+0000","last_unstale":"2024-04-25T15:50:32.747849+0000","last_undegraded":"2024-04-25T15:50:32.747849+0000","last_fullsized":"2024-04-25T15:50:32.747849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:26.635717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748379+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:50:32.748379+0000","last_peered":"2024-04-25T15:50:32.748379+0000","last_clean":"2024-04-25T15:50:32.748379+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:50:32.748379+0000","last_undegraded":"2024-04-25T15:50:32.748379+0000","last_fullsized":"2024-04-25T15:50:32.748379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152210+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:50:32.152210+0000","last_peered":"2024-04-25T15:50:32.152210+0000","last_clean":"2024-04-25T15:50:32.152210+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:50:32.152210+0000","last_undegraded":"2024-04-25T15:50:32.152210+0000","last_fullsized":"2024-04-25T15:50:32.152210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749720+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:50:32.749720+0000","last_peered":"2024-04-25T15:50:32.749720+0000","last_clean":"2024-04-25T15:50:32.749720+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:50:32.749720+0000","last_undegraded":"2024-04-25T15:50:32.749720+0000","last_fullsized":"2024-04-25T15:50:32.749720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749054+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:50:32.749054+0000","last_peered":"2024-04-25T15:50:32.749054+0000","last_clean":"2024-04-25T15:50:32.749054+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:50:32.749054+0000","last_undegraded":"2024-04-25T15:50:32.749054+0000","last_fullsized":"2024-04-25T15:50:32.749054+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750686+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:50:32.750686+0000","last_peered":"2024-04-25T15:50:32.750686+0000","last_clean":"2024-04-25T15:50:32.750686+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:50:32.750686+0000","last_undegraded":"2024-04-25T15:50:32.750686+0000","last_fullsized":"2024-04-25T15:50:32.750686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148745+0000","last_change":"2024-04-25T15:50:30.192595+0000","last_active":"2024-04-25T15:50:32.148745+0000","last_peered":"2024-04-25T15:50:32.148745+0000","last_clean":"2024-04-25T15:50:32.148745+0000","last_became_active":"2024-04-25T15:50:30.192044+0000","last_became_peered":"2024-04-25T15:50:30.192044+0000","last_unstale":"2024-04-25T15:50:32.148745+0000","last_undegraded":"2024-04-25T15:50:32.148745+0000","last_fullsized":"2024-04-25T15:50:32.148745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:08.837774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153693+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:50:32.153693+0000","last_peered":"2024-04-25T15:50:32.153693+0000","last_clean":"2024-04-25T15:50:32.153693+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:50:32.153693+0000","last_undegraded":"2024-04-25T15:50:32.153693+0000","last_fullsized":"2024-04-25T15:50:32.153693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749938+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:50:32.749938+0000","last_peered":"2024-04-25T15:50:32.749938+0000","last_clean":"2024-04-25T15:50:32.749938+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:50:32.749938+0000","last_undegraded":"2024-04-25T15:50:32.749938+0000","last_fullsized":"2024-04-25T15:50:32.749938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748511+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:50:32.748511+0000","last_peered":"2024-04-25T15:50:32.748511+0000","last_clean":"2024-04-25T15:50:32.748511+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:50:32.748511+0000","last_undegraded":"2024-04-25T15:50:32.748511+0000","last_fullsized":"2024-04-25T15:50:32.748511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747640+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:50:32.747640+0000","last_peered":"2024-04-25T15:50:32.747640+0000","last_clean":"2024-04-25T15:50:32.747640+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:50:32.747640+0000","last_undegraded":"2024-04-25T15:50:32.747640+0000","last_fullsized":"2024-04-25T15:50:32.747640+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152008+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:50:32.152008+0000","last_peered":"2024-04-25T15:50:32.152008+0000","last_clean":"2024-04-25T15:50:32.152008+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:50:32.152008+0000","last_undegraded":"2024-04-25T15:50:32.152008+0000","last_fullsized":"2024-04-25T15:50:32.152008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247247+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:50:32.247247+0000","last_peered":"2024-04-25T15:50:32.247247+0000","last_clean":"2024-04-25T15:50:32.247247+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:50:32.247247+0000","last_undegraded":"2024-04-25T15:50:32.247247+0000","last_fullsized":"2024-04-25T15:50:32.247247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747456+0000","last_change":"2024-04-25T15:50:30.197359+0000","last_active":"2024-04-25T15:50:32.747456+0000","last_peered":"2024-04-25T15:50:32.747456+0000","last_clean":"2024-04-25T15:50:32.747456+0000","last_became_active":"2024-04-25T15:50:30.196640+0000","last_became_peered":"2024-04-25T15:50:30.196640+0000","last_unstale":"2024-04-25T15:50:32.747456+0000","last_undegraded":"2024-04-25T15:50:32.747456+0000","last_fullsized":"2024-04-25T15:50:32.747456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.928809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750609+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:50:32.750609+0000","last_peered":"2024-04-25T15:50:32.750609+0000","last_clean":"2024-04-25T15:50:32.750609+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:50:32.750609+0000","last_undegraded":"2024-04-25T15:50:32.750609+0000","last_fullsized":"2024-04-25T15:50:32.750609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748372+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:50:32.748372+0000","last_peered":"2024-04-25T15:50:32.748372+0000","last_clean":"2024-04-25T15:50:32.748372+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:50:32.748372+0000","last_undegraded":"2024-04-25T15:50:32.748372+0000","last_fullsized":"2024-04-25T15:50:32.748372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148771+0000","last_change":"2024-04-25T15:50:31.171339+0000","last_active":"2024-04-25T15:50:32.148771+0000","last_peered":"2024-04-25T15:50:32.148771+0000","last_clean":"2024-04-25T15:50:32.148771+0000","last_became_active":"2024-04-25T15:50:31.170349+0000","last_became_peered":"2024-04-25T15:50:31.170349+0000","last_unstale":"2024-04-25T15:50:32.148771+0000","last_undegraded":"2024-04-25T15:50:32.148771+0000","last_fullsized":"2024-04-25T15:50:32.148771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:24.647984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153062+0000","last_change":"2024-04-25T15:50:30.195563+0000","last_active":"2024-04-25T15:50:32.153062+0000","last_peered":"2024-04-25T15:50:32.153062+0000","last_clean":"2024-04-25T15:50:32.153062+0000","last_became_active":"2024-04-25T15:50:30.194468+0000","last_became_peered":"2024-04-25T15:50:30.194468+0000","last_unstale":"2024-04-25T15:50:32.153062+0000","last_undegraded":"2024-04-25T15:50:32.153062+0000","last_fullsized":"2024-04-25T15:50:32.153062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:04.850096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151035+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:50:32.151035+0000","last_peered":"2024-04-25T15:50:32.151035+0000","last_clean":"2024-04-25T15:50:32.151035+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:50:32.151035+0000","last_undegraded":"2024-04-25T15:50:32.151035+0000","last_fullsized":"2024-04-25T15:50:32.151035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149161+0000","last_change":"2024-04-25T15:50:30.175727+0000","last_active":"2024-04-25T15:50:32.149161+0000","last_peered":"2024-04-25T15:50:32.149161+0000","last_clean":"2024-04-25T15:50:32.149161+0000","last_became_active":"2024-04-25T15:50:30.175136+0000","last_became_peered":"2024-04-25T15:50:30.175136+0000","last_unstale":"2024-04-25T15:50:32.149161+0000","last_undegraded":"2024-04-25T15:50:32.149161+0000","last_fullsized":"2024-04-25T15:50:32.149161+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:22.434474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153102+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:50:32.153102+0000","last_peered":"2024-04-25T15:50:32.153102+0000","last_clean":"2024-04-25T15:50:32.153102+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:50:32.153102+0000","last_undegraded":"2024-04-25T15:50:32.153102+0000","last_fullsized":"2024-04-25T15:50:32.153102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749240+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:50:32.749240+0000","last_peered":"2024-04-25T15:50:32.749240+0000","last_clean":"2024-04-25T15:50:32.749240+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:50:32.749240+0000","last_undegraded":"2024-04-25T15:50:32.749240+0000","last_fullsized":"2024-04-25T15:50:32.749240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149163+0000","last_change":"2024-04-25T15:50:31.177424+0000","last_active":"2024-04-25T15:50:32.149163+0000","last_peered":"2024-04-25T15:50:32.149163+0000","last_clean":"2024-04-25T15:50:32.149163+0000","last_became_active":"2024-04-25T15:50:31.176553+0000","last_became_peered":"2024-04-25T15:50:31.176553+0000","last_unstale":"2024-04-25T15:50:32.149163+0000","last_undegraded":"2024-04-25T15:50:32.149163+0000","last_fullsized":"2024-04-25T15:50:32.149163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:29.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149356+0000","last_change":"2024-04-25T15:50:30.191244+0000","last_active":"2024-04-25T15:50:32.149356+0000","last_peered":"2024-04-25T15:50:32.149356+0000","last_clean":"2024-04-25T15:50:32.149356+0000","last_became_active":"2024-04-25T15:50:30.190437+0000","last_became_peered":"2024-04-25T15:50:30.190437+0000","last_unstale":"2024-04-25T15:50:32.149356+0000","last_undegraded":"2024-04-25T15:50:32.149356+0000","last_fullsized":"2024-04-25T15:50:32.149356+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:55.506915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247702+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:50:32.247702+0000","last_peered":"2024-04-25T15:50:32.247702+0000","last_clean":"2024-04-25T15:50:32.247702+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:50:32.247702+0000","last_undegraded":"2024-04-25T15:50:32.247702+0000","last_fullsized":"2024-04-25T15:50:32.247702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.751545+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:50:32.751545+0000","last_peered":"2024-04-25T15:50:32.751545+0000","last_clean":"2024-04-25T15:50:32.751545+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:50:32.751545+0000","last_undegraded":"2024-04-25T15:50:32.751545+0000","last_fullsized":"2024-04-25T15:50:32.751545+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749239+0000","last_change":"2024-04-25T15:50:31.182607+0000","last_active":"2024-04-25T15:50:32.749239+0000","last_peered":"2024-04-25T15:50:32.749239+0000","last_clean":"2024-04-25T15:50:32.749239+0000","last_became_active":"2024-04-25T15:50:31.181641+0000","last_became_peered":"2024-04-25T15:50:31.181641+0000","last_unstale":"2024-04-25T15:50:32.749239+0000","last_undegraded":"2024-04-25T15:50:32.749239+0000","last_fullsized":"2024-04-25T15:50:32.749239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:49.636675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747485+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:50:32.747485+0000","last_peered":"2024-04-25T15:50:32.747485+0000","last_clean":"2024-04-25T15:50:32.747485+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:50:32.747485+0000","last_undegraded":"2024-04-25T15:50:32.747485+0000","last_fullsized":"2024-04-25T15:50:32.747485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151509+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:50:32.151509+0000","last_peered":"2024-04-25T15:50:32.151509+0000","last_clean":"2024-04-25T15:50:32.151509+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:50:32.151509+0000","last_undegraded":"2024-04-25T15:50:32.151509+0000","last_fullsized":"2024-04-25T15:50:32.151509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148622+0000","last_change":"2024-04-25T15:50:30.178078+0000","last_active":"2024-04-25T15:50:32.148622+0000","last_peered":"2024-04-25T15:50:32.148622+0000","last_clean":"2024-04-25T15:50:32.148622+0000","last_became_active":"2024-04-25T15:50:30.177718+0000","last_became_peered":"2024-04-25T15:50:30.177718+0000","last_unstale":"2024-04-25T15:50:32.148622+0000","last_undegraded":"2024-04-25T15:50:32.148622+0000","last_fullsized":"2024-04-25T15:50:32.148622+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246699+0000","last_change":"2024-04-25T15:50:31.192152+0000","last_active":"2024-04-25T15:50:32.246699+0000","last_peered":"2024-04-25T15:50:32.246699+0000","last_clean":"2024-04-25T15:50:32.246699+0000","last_became_active":"2024-04-25T15:50:31.179200+0000","last_became_peered":"2024-04-25T15:50:31.179200+0000","last_unstale":"2024-04-25T15:50:32.246699+0000","last_undegraded":"2024-04-25T15:50:32.246699+0000","last_fullsized":"2024-04-25T15:50:32.246699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:03.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148915+0000","last_change":"2024-04-25T15:50:30.184989+0000","last_active":"2024-04-25T15:50:32.148915+0000","last_peered":"2024-04-25T15:50:32.148915+0000","last_clean":"2024-04-25T15:50:32.148915+0000","last_became_active":"2024-04-25T15:50:30.184132+0000","last_became_peered":"2024-04-25T15:50:30.184132+0000","last_unstale":"2024-04-25T15:50:32.148915+0000","last_undegraded":"2024-04-25T15:50:32.148915+0000","last_fullsized":"2024-04-25T15:50:32.148915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:03.833303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247766+0000","last_change":"2024-04-25T15:50:31.190275+0000","last_active":"2024-04-25T15:50:32.247766+0000","last_peered":"2024-04-25T15:50:32.247766+0000","last_clean":"2024-04-25T15:50:32.247766+0000","last_became_active":"2024-04-25T15:50:31.189302+0000","last_became_peered":"2024-04-25T15:50:31.189302+0000","last_unstale":"2024-04-25T15:50:32.247766+0000","last_undegraded":"2024-04-25T15:50:32.247766+0000","last_fullsized":"2024-04-25T15:50:32.247766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:39.479706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748920+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:50:32.748920+0000","last_peered":"2024-04-25T15:50:32.748920+0000","last_clean":"2024-04-25T15:50:32.748920+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:50:32.748920+0000","last_undegraded":"2024-04-25T15:50:32.748920+0000","last_fullsized":"2024-04-25T15:50:32.748920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748648+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:50:32.748648+0000","last_peered":"2024-04-25T15:50:32.748648+0000","last_clean":"2024-04-25T15:50:32.748648+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:50:32.748648+0000","last_undegraded":"2024-04-25T15:50:32.748648+0000","last_fullsized":"2024-04-25T15:50:32.748648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153111+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:50:32.153111+0000","last_peered":"2024-04-25T15:50:32.153111+0000","last_clean":"2024-04-25T15:50:32.153111+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:50:32.153111+0000","last_undegraded":"2024-04-25T15:50:32.153111+0000","last_fullsized":"2024-04-25T15:50:32.153111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749776+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:50:32.749776+0000","last_peered":"2024-04-25T15:50:32.749776+0000","last_clean":"2024-04-25T15:50:32.749776+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:50:32.749776+0000","last_undegraded":"2024-04-25T15:50:32.749776+0000","last_fullsized":"2024-04-25T15:50:32.749776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153450+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:50:32.153450+0000","last_peered":"2024-04-25T15:50:32.153450+0000","last_clean":"2024-04-25T15:50:32.153450+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:50:32.153450+0000","last_undegraded":"2024-04-25T15:50:32.153450+0000","last_fullsized":"2024-04-25T15:50:32.153450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749103+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:50:32.749103+0000","last_peered":"2024-04-25T15:50:32.749103+0000","last_clean":"2024-04-25T15:50:32.749103+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:50:32.749103+0000","last_undegraded":"2024-04-25T15:50:32.749103+0000","last_fullsized":"2024-04-25T15:50:32.749103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749087+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:50:32.749087+0000","last_peered":"2024-04-25T15:50:32.749087+0000","last_clean":"2024-04-25T15:50:32.749087+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:50:32.749087+0000","last_undegraded":"2024-04-25T15:50:32.749087+0000","last_fullsized":"2024-04-25T15:50:32.749087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247277+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:50:32.247277+0000","last_peered":"2024-04-25T15:50:32.247277+0000","last_clean":"2024-04-25T15:50:32.247277+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:50:32.247277+0000","last_undegraded":"2024-04-25T15:50:32.247277+0000","last_fullsized":"2024-04-25T15:50:32.247277+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247337+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:50:32.247337+0000","last_peered":"2024-04-25T15:50:32.247337+0000","last_clean":"2024-04-25T15:50:32.247337+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:50:32.247337+0000","last_undegraded":"2024-04-25T15:50:32.247337+0000","last_fullsized":"2024-04-25T15:50:32.247337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749650+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:50:32.749650+0000","last_peered":"2024-04-25T15:50:32.749650+0000","last_clean":"2024-04-25T15:50:32.749650+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:50:32.749650+0000","last_undegraded":"2024-04-25T15:50:32.749650+0000","last_fullsized":"2024-04-25T15:50:32.749650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":655002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":655002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:50:37.820 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:50:37.821 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T15:50:37.821 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T15:50:37.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T15:50:37.821 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T15:50:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b351f640 1 Processor -- start 2024-04-25T15:50:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b351f640 1 -- start start 2024-04-25T15:50:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b351f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39ac156330 0x7f39ac06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b351f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f39ac06cb80 0x7f39ac0c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b351f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac06fa10 con 0x7f39ac0c4f30 2024-04-25T15:50:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b351f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f39ac068ca0 con 0x7f39ac156330 2024-04-25T15:50:38.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b351f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f39ac0ca6c0 con 0x7f39ac06cb80 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b12bb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39ac156330 0x7f39ac06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b12bb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39ac156330 0x7f39ac06c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59526/0 (socket says 172.21.15.67:59526) 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b12bb640 1 -- 172.21.15.67:0/3587460090 learned_addr learned my addr 172.21.15.67:0/3587460090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b0aba640 1 --2- 172.21.15.67:0/3587460090 >> v2:172.21.15.176:3300/0 conn(0x7f39ac06cb80 0x7f39ac0c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b12bb640 1 -- 172.21.15.67:0/3587460090 >> v2:172.21.15.176:3300/0 conn(0x7f39ac06cb80 msgr2=0x7f39ac0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b12bb640 1 --2- 172.21.15.67:0/3587460090 >> v2:172.21.15.176:3300/0 conn(0x7f39ac06cb80 0x7f39ac0c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b12bb640 1 -- 172.21.15.67:0/3587460090 >> v1:172.21.15.176:6789/0 conn(0x7f39ac0c4f30 legacy=0x7f39ac068590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:38.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.014+0000 7f39b12bb640 1 -- 172.21.15.67:0/3587460090 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac0ca8f0 con 0x7f39ac156330 2024-04-25T15:50:38.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b12bb640 1 --2- 172.21.15.67:0/3587460090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac156330 0x7f39ac06c640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0b1130 tx=0x7f399c01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733d1db315deed31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39abfff640 1 -- 172.21.15.67:0/3587460090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c0b4070 con 0x7f39ac156330 2024-04-25T15:50:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39abfff640 1 -- 172.21.15.67:0/3587460090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f399c0af400 con 0x7f39ac156330 2024-04-25T15:50:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39abfff640 1 -- 172.21.15.67:0/3587460090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c0af7d0 con 0x7f39ac156330 2024-04-25T15:50:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 -- 172.21.15.67:0/3587460090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac156330 msgr2=0x7f39ac06c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 --2- 172.21.15.67:0/3587460090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac156330 0x7f39ac06c640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0b1130 tx=0x7f399c01f1a0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 -- 172.21.15.67:0/3587460090 shutdown_connections 2024-04-25T15:50:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 --2- 172.21.15.67:0/3587460090 >> v2:172.21.15.176:3300/0 conn(0x7f39ac06cb80 0x7f39ac0c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 --2- 172.21.15.67:0/3587460090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac156330 0x7f39ac06c640 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 -- 172.21.15.67:0/3587460090 >> 172.21.15.67:0/3587460090 conn(0x7f39ac0ce1c0 msgr2=0x7f39ac0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 -- 172.21.15.67:0/3587460090 shutdown_connections 2024-04-25T15:50:38.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.018+0000 7f39b351f640 1 -- 172.21.15.67:0/3587460090 wait complete. 2024-04-25T15:50:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 Processor -- start 2024-04-25T15:50:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 -- start start 2024-04-25T15:50:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39ac06cb80 0x7f39ac0c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac0c4f30 0x7f39ac0c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39ac156330 0x7f39ac1e7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac0c43d0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac0c3d80 con 0x7f39ac06cb80 2024-04-25T15:50:38.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f39ac0c4060 con 0x7f39ac156330 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac0c4f30 0x7f39ac0c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac0c4f30 0x7f39ac0c6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59534/0 (socket says 172.21.15.67:59534) 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 -- 172.21.15.67:0/3337834837 learned_addr learned my addr 172.21.15.67:0/3337834837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b12bb640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39ac06cb80 0x7f39ac0c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b1abc640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39ac156330 0x7f39ac1e7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 -- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39ac156330 msgr2=0x7f39ac1e7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39ac156330 0x7f39ac1e7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 -- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39ac06cb80 msgr2=0x7f39ac0c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39ac06cb80 0x7f39ac0c7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 -- 172.21.15.67:0/3337834837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac0c69c0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b12bb640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39ac06cb80 0x7f39ac0c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b0aba640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac0c4f30 0x7f39ac0c6480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f399400d800 tx=0x7f399400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39a9ffb640 1 -- 172.21.15.67:0/3337834837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399400b6c0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39a9ffb640 1 -- 172.21.15.67:0/3337834837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3994019070 con 0x7f39ac0c4f30 2024-04-25T15:50:38.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ac0c53e0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39a9ffb640 1 -- 172.21.15.67:0/3337834837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399400baa0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ac0c59a0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.022+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ac06a480 con 0x7f39ac0c4f30 2024-04-25T15:50:38.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.026+0000 7f39a9ffb640 1 -- 172.21.15.67:0/3337834837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f399400bc40 con 0x7f39ac0c4f30 2024-04-25T15:50:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.026+0000 7f39a9ffb640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398c06f400 0x7f398c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.026+0000 7f39a9ffb640 1 -- 172.21.15.67:0/3337834837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f399408f6c0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.026+0000 7f39b12bb640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398c06f400 0x7f398c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.026+0000 7f39b12bb640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398c06f400 0x7f398c071910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f399c0b1000 tx=0x7f399c0aa000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.030+0000 7f39a9ffb640 1 -- 172.21.15.67:0/3337834837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f399405c640 con 0x7f39ac0c4f30 2024-04-25T15:50:38.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.234+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f39ac0ca4a0 con 0x7f39ac0c4f30 2024-04-25T15:50:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.242+0000 7f39a9ffb640 1 -- 172.21.15.67:0/3337834837 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+179549 (secure 0 0 0) 0x7f399405bf40 con 0x7f39ac0c4f30 2024-04-25T15:50:38.242 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398c06f400 msgr2=0x7f398c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398c06f400 0x7f398c071910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f399c0b1000 tx=0x7f399c0aa000 comp rx=0 tx=0).stop 2024-04-25T15:50:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac0c4f30 msgr2=0x7f39ac0c6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac0c4f30 0x7f39ac0c6480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f399400d800 tx=0x7f399400dcc0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 shutdown_connections 2024-04-25T15:50:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398c06f400 0x7f398c071910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39ac156330 0x7f39ac1e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39ac0c4f30 0x7f39ac0c6480 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 --2- 172.21.15.67:0/3337834837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39ac06cb80 0x7f39ac0c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 >> 172.21.15.67:0/3337834837 conn(0x7f39ac0ce1c0 msgr2=0x7f39ac1555e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 shutdown_connections 2024-04-25T15:50:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.246+0000 7f39b351f640 1 -- 172.21.15.67:0/3337834837 wait complete. 2024-04-25T15:50:38.264 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":2784768863},{"type":"v1","addr":"172.21.15.176:6833","nonce":2784768863}]},"active_addr":"172.21.15.176:6833/2784768863","active_change":"2024-04-25T15:50:09.784967+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":346871287}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":18783227}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3837082340}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3237483256}]}]} 2024-04-25T15:50:38.268 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T15:50:38.268 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T15:50:38.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:38.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.418+0000 7f8e91ebb640 1 Processor -- start 2024-04-25T15:50:38.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.418+0000 7f8e91ebb640 1 -- start start 2024-04-25T15:50:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e91ebb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 0x7f8e8c164d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e91ebb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e8c0c6890 con 0x7f8e8c0c7610 2024-04-25T15:50:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e91ebb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e8c0c6b70 con 0x7f8e8c071b20 2024-04-25T15:50:38.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e91ebb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e8c15cd40 con 0x7f8e8c0c7a20 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8e8c0c7610 0x7f8e8c0c5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35996/0 (socket says 172.21.15.67:35996) 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8affd640 1 -- 172.21.15.67:0/651033804 learned_addr learned my addr 172.21.15.67:0/651033804 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8bfff640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 0x7f8e8c164d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870696901 0 0) 0x7f8e8c0c6890 con 0x7f8e8c0c7610 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e78003530 con 0x7f8e8c0c7610 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1579953454 0 0) 0x7f8e78003530 con 0x7f8e8c0c7610 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 msgr2=0x7f8e8c164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 0x7f8e8c164d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 >> v1:172.21.15.176:6789/0 conn(0x7f8e8c071b20 legacy=0x7f8e8c15d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c0c6e50 con 0x7f8e8c0c7610 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8e7c001680 con 0x7f8e8c0c7610 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 >> v1:172.21.15.67:6789/0 conn(0x7f8e8c0c7610 legacy=0x7f8e8c0c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e780039f0 0x7f8e78003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 0x7f8e78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78004d50 0x7f8e78005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c15cd40 con 0x7f8e78004d50 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7800bb00 con 0x7f8e780039f0 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e7800bd00 con 0x7f8e8c0c7a20 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8bfff640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78004d50 0x7f8e78005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8affd640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 0x7f8e78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8b7fe640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e780039f0 0x7f8e78003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8bfff640 1 -- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 msgr2=0x7f8e78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8bfff640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 0x7f8e78004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8bfff640 1 -- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e780039f0 msgr2=0x7f8e78003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8bfff640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e780039f0 0x7f8e78003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8bfff640 1 -- 172.21.15.67:0/651033804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e78003530 con 0x7f8e78004d50 2024-04-25T15:50:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.422+0000 7f8e8affd640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c0c7a20 0x7f8e78004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e8bfff640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78004d50 0x7f8e78005160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009f90 tx=0x7f8e8401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe235da60f291c73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e840b2070 con 0x7f8e78004d50 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e840a5c70 con 0x7f8e78004d50 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e8a7fc640 1 -- 172.21.15.67:0/651033804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e84009c10 con 0x7f8e78004d50 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e91ebb640 1 -- 172.21.15.67:0/651033804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78004d50 msgr2=0x7f8e78005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e91ebb640 1 --2- 172.21.15.67:0/651033804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78004d50 0x7f8e78005160 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009f90 tx=0x7f8e8401cc60 comp rx=0 tx=0).stop 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e8bfff640 1 -- 172.21.15.67:0/651033804 reap_dead start 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e91ebb640 1 -- 172.21.15.67:0/651033804 shutdown_connections 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e91ebb640 1 -- 172.21.15.67:0/651033804 >> 172.21.15.67:0/651033804 conn(0x7f8e8c06fe70 msgr2=0x7f8e8c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e91ebb640 1 -- 172.21.15.67:0/651033804 shutdown_connections 2024-04-25T15:50:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e91ebb640 1 -- 172.21.15.67:0/651033804 wait complete. 2024-04-25T15:50:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.426+0000 7f8e91ebb640 1 Processor -- start 2024-04-25T15:50:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 -- start start 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e8c143fb0 0x7f8e8c15a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c1e4f40 0x7f8e8c1e2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e8c1e2d40 0x7f8e8c1433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c15d8e0 con 0x7f8e8c143fb0 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c15c9d0 con 0x7f8e8c1e2d40 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e8c0734f0 con 0x7f8e8c1e4f40 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e8c143fb0 0x7f8e8c15a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e8c143fb0 0x7f8e8c15a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59562/0 (socket says 172.21.15.67:59562) 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 -- 172.21.15.67:0/4122351638 learned_addr learned my addr 172.21.15.67:0/4122351638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8affd640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c1e4f40 0x7f8e8c1e2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8bfff640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e8c1e2d40 0x7f8e8c1433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 -- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c1e4f40 msgr2=0x7f8e8c1e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c1e4f40 0x7f8e8c1e2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 -- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e8c1e2d40 msgr2=0x7f8e8c1433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e8c1e2d40 0x7f8e8c1433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 -- 172.21.15.67:0/4122351638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c15ac90 con 0x7f8e8c143fb0 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e8b7fe640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e8c143fb0 0x7f8e8c15a5a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e470 tx=0x7f8e7400e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e83fff640 1 -- 172.21.15.67:0/4122351638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74019070 con 0x7f8e8c143fb0 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e83fff640 1 -- 172.21.15.67:0/4122351638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e740052e0 con 0x7f8e8c143fb0 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e83fff640 1 -- 172.21.15.67:0/4122351638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74003820 con 0x7f8e8c143fb0 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8c15af30 con 0x7f8e8c143fb0 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8c1efe00 con 0x7f8e8c143fb0 2024-04-25T15:50:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.430+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e8c06a480 con 0x7f8e8c143fb0 2024-04-25T15:50:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.434+0000 7f8e83fff640 1 -- 172.21.15.67:0/4122351638 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8e7401c070 con 0x7f8e8c143fb0 2024-04-25T15:50:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.434+0000 7f8e83fff640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e6c06f400 0x7f8e6c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.434+0000 7f8e83fff640 1 -- 172.21.15.67:0/4122351638 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8e7408ef80 con 0x7f8e8c143fb0 2024-04-25T15:50:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.434+0000 7f8e8affd640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e6c06f400 0x7f8e6c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.434+0000 7f8e8affd640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e6c06f400 0x7f8e6c071910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0ccff0 tx=0x7f8e7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.438+0000 7f8e83fff640 1 -- 172.21.15.67:0/4122351638 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8e7405b990 con 0x7f8e8c143fb0 2024-04-25T15:50:38.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.594+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e8c0cd590 con 0x7f8e8c143fb0 2024-04-25T15:50:38.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.594+0000 7f8e83fff640 1 -- 172.21.15.67:0/4122351638 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f8e74060e20 con 0x7f8e8c143fb0 2024-04-25T15:50:38.597 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:38.597 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:32.143126+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e6c06f400 msgr2=0x7f8e6c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e6c06f400 0x7f8e6c071910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c0ccff0 tx=0x7f8e7c009040 comp rx=0 tx=0).stop 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e8c143fb0 msgr2=0x7f8e8c15a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e8c143fb0 0x7f8e8c15a5a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e470 tx=0x7f8e7400e930 comp rx=0 tx=0).stop 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 shutdown_connections 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e6c06f400 0x7f8e6c071910 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e8c1e2d40 0x7f8e8c1433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e8c1e4f40 0x7f8e8c1e2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 --2- 172.21.15.67:0/4122351638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e8c143fb0 0x7f8e8c15a5a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 >> 172.21.15.67:0/4122351638 conn(0x7f8e8c06fe70 msgr2=0x7f8e8c1496c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 shutdown_connections 2024-04-25T15:50:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.598+0000 7f8e91ebb640 1 -- 172.21.15.67:0/4122351638 wait complete. 2024-04-25T15:50:38.616 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T15:50:38.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:50:38.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbecd1d2640 1 Processor -- start 2024-04-25T15:50:38.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbecd1d2640 1 -- start start 2024-04-25T15:50:38.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbecd1d2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbec8160800 0x7fbec806d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbecd1d2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbec80c92d0 0x7fbec8072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbecd1d2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec806d8e0 con 0x7fbec80c96e0 2024-04-25T15:50:38.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbecd1d2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbec806dbc0 con 0x7fbec8160800 2024-04-25T15:50:38.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbecd1d2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbec8073c70 con 0x7fbec80c92d0 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec6d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbec8160800 0x7fbec806d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec759e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbec80c96e0 0x7fbec815c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36006/0 (socket says 172.21.15.67:36006) 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec759e640 1 -- 172.21.15.67:0/3458314541 learned_addr learned my addr 172.21.15.67:0/3458314541 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec659c640 1 --2- 172.21.15.67:0/3458314541 >> v2:172.21.15.176:3300/0 conn(0x7fbec80c92d0 0x7fbec8072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec6d9d640 1 -- 172.21.15.67:0/3458314541 >> v2:172.21.15.176:3300/0 conn(0x7fbec80c92d0 msgr2=0x7fbec8072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec6d9d640 1 --2- 172.21.15.67:0/3458314541 >> v2:172.21.15.176:3300/0 conn(0x7fbec80c92d0 0x7fbec8072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec6d9d640 1 -- 172.21.15.67:0/3458314541 >> v1:172.21.15.67:6789/0 conn(0x7fbec80c96e0 legacy=0x7fbec815c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec6d9d640 1 -- 172.21.15.67:0/3458314541 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec8073f50 con 0x7fbec8160800 2024-04-25T15:50:38.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.762+0000 7fbec5d9b640 1 -- 172.21.15.67:0/3458314541 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301428625 0 0) 0x7fbec806d8e0 con 0x7fbec80c96e0 2024-04-25T15:50:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbec6d9d640 1 --2- 172.21.15.67:0/3458314541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec806d030 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbec806dd40 tx=0x7fbeb801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec07c905b621b055 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbec5d9b640 1 -- 172.21.15.67:0/3458314541 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb80b2070 con 0x7fbec8160800 2024-04-25T15:50:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbec5d9b640 1 -- 172.21.15.67:0/3458314541 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb80a5760 con 0x7fbec8160800 2024-04-25T15:50:38.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbec5d9b640 1 -- 172.21.15.67:0/3458314541 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb80a5b30 con 0x7fbec8160800 2024-04-25T15:50:38.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 -- 172.21.15.67:0/3458314541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 msgr2=0x7fbec806d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/3458314541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec806d030 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbec806dd40 tx=0x7fbeb801c8a0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 -- 172.21.15.67:0/3458314541 shutdown_connections 2024-04-25T15:50:38.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/3458314541 >> v2:172.21.15.176:3300/0 conn(0x7fbec80c92d0 0x7fbec8072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/3458314541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec806d030 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 -- 172.21.15.67:0/3458314541 >> 172.21.15.67:0/3458314541 conn(0x7fbec8070db0 msgr2=0x7fbec80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:38.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 -- 172.21.15.67:0/3458314541 shutdown_connections 2024-04-25T15:50:38.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.766+0000 7fbecd1d2640 1 -- 172.21.15.67:0/3458314541 wait complete. 2024-04-25T15:50:38.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 Processor -- start 2024-04-25T15:50:38.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 -- start start 2024-04-25T15:50:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbec80c92d0 0x7fbec814ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbec80c96e0 0x7fbec815c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec814f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbec8160c10 con 0x7fbec8160800 2024-04-25T15:50:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbec806d570 con 0x7fbec80c92d0 2024-04-25T15:50:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbec8160d90 con 0x7fbec80c96e0 2024-04-25T15:50:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec814f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec814f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59580/0 (socket says 172.21.15.67:59580) 2024-04-25T15:50:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 -- 172.21.15.67:0/2403883458 learned_addr learned my addr 172.21.15.67:0/2403883458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec659c640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbec80c96e0 0x7fbec815c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec6d9d640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbec80c92d0 0x7fbec814ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 -- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbec80c96e0 msgr2=0x7fbec815c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbec80c96e0 0x7fbec815c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 -- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbec80c92d0 msgr2=0x7fbec814ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbec80c92d0 0x7fbec814ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 -- 172.21.15.67:0/2403883458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec806fec0 con 0x7fbec8160800 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbec759e640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec814f960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbec000c410 tx=0x7fbec000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2403883458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec00074e0 con 0x7fbec8160800 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2403883458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec00047f0 con 0x7fbec8160800 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2403883458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0007680 con 0x7fbec8160800 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec80c8270 con 0x7fbec8160800 2024-04-25T15:50:38.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec814dc70 con 0x7fbec8160800 2024-04-25T15:50:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.770+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbec80c6a60 con 0x7fbec8160800 2024-04-25T15:50:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.774+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2403883458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbec0011070 con 0x7fbec8160800 2024-04-25T15:50:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.774+0000 7fbebf7fe640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbea806f400 0x7fbea8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.774+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2403883458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbec008ed70 con 0x7fbec8160800 2024-04-25T15:50:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.774+0000 7fbec6d9d640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbea806f400 0x7fbea8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:38.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.774+0000 7fbec6d9d640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbea806f400 0x7fbea8071910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80af000 tx=0x7fbeb80a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:38.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.778+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2403883458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbec80c6a60 con 0x7fbec8160800 2024-04-25T15:50:38.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.934+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbec814ded0 con 0x7fbec8160800 2024-04-25T15:50:38.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbebf7fe640 1 -- 172.21.15.67:0/2403883458 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7fbec005bcf0 con 0x7fbec8160800 2024-04-25T15:50:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:38.939 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:32.143126+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbea806f400 msgr2=0x7fbea8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbea806f400 0x7fbea8071910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80af000 tx=0x7fbeb80a7070 comp rx=0 tx=0).stop 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 msgr2=0x7fbec814f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec814f960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbec000c410 tx=0x7fbec000c8d0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 shutdown_connections 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbea806f400 0x7fbea8071910 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbec8160800 0x7fbec814f960 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbec80c96e0 0x7fbec815c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 --2- 172.21.15.67:0/2403883458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbec80c92d0 0x7fbec814ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:38.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 >> 172.21.15.67:0/2403883458 conn(0x7fbec8070db0 msgr2=0x7fbec81560d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:38.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 shutdown_connections 2024-04-25T15:50:38.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:38.938+0000 7fbecd1d2640 1 -- 172.21.15.67:0/2403883458 wait complete. 2024-04-25T15:50:38.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T15:50:38.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T15:50:38.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T15:50:38.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T15:50:38.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T15:50:38.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T15:50:38.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T15:50:38.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T15:50:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feee1f08640 1 Processor -- start 2024-04-25T15:50:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feee1f08640 1 -- start start 2024-04-25T15:50:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7ff45105f640 1 Processor -- start 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feee1f08640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feedc0b3b70 0x7feedc068590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feee1f08640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc06f980 0x7feedc06fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feee1f08640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feedc0c3f00 con 0x7feedc1610f0 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feee1f08640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7feedc160c60 con 0x7feedc0b3b70 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feee1f08640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feedc0c41e0 con 0x7feedc06f980 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feedb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feedc0b3b70 0x7feedc068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feedb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feedc0b3b70 0x7feedc068590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59594/0 (socket says 172.21.15.67:59594) 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7feedb7fe640 1 -- 172.21.15.67:0/4265483293 learned_addr learned my addr 172.21.15.67:0/4265483293 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.182+0000 7ff45105f640 1 -- start start 2024-04-25T15:50:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efeceaae640 1 Processor -- start 2024-04-25T15:50:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7feedaffd640 1 --2- 172.21.15.67:0/4265483293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc06f980 0x7feedc06fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7feedb7fe640 1 -- 172.21.15.67:0/4265483293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc06f980 msgr2=0x7feedc06fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7feedb7fe640 1 --2- 172.21.15.67:0/4265483293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc06f980 0x7feedc06fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7feedb7fe640 1 -- 172.21.15.67:0/4265483293 >> v1:172.21.15.176:6789/0 conn(0x7feedc1610f0 legacy=0x7feedc161500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:50:39.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7feedb7fe640 1 -- 172.21.15.67:0/4265483293 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc0c44c0 con 0x7feedc0b3b70 2024-04-25T15:50:39.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efeceaae640 1 -- start start 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff45105f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff44c161440 0x7ff44c0c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff45105f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff44c0c93d0 0x7ff44c06ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff45105f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff44c0c5570 con 0x7ff44c06f350 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff45105f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff44c0c4d80 con 0x7ff44c161440 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff45105f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff44c0c80d0 con 0x7ff44c0c93d0 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff44c161440 0x7ff44c0c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff44c161440 0x7ff44c0c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59610/0 (socket says 172.21.15.67:59610) 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44ad9d640 1 -- 172.21.15.67:0/1214686927 learned_addr learned my addr 172.21.15.67:0/1214686927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44a59c640 1 --2- 172.21.15.67:0/1214686927 >> v2:172.21.15.176:3300/0 conn(0x7ff44c0c93d0 0x7ff44c06ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44ad9d640 1 -- 172.21.15.67:0/1214686927 >> v2:172.21.15.176:3300/0 conn(0x7ff44c0c93d0 msgr2=0x7ff44c06ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44ad9d640 1 --2- 172.21.15.67:0/1214686927 >> v2:172.21.15.176:3300/0 conn(0x7ff44c0c93d0 0x7ff44c06ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44ad9d640 1 -- 172.21.15.67:0/1214686927 >> v1:172.21.15.176:6789/0 conn(0x7ff44c06f350 legacy=0x7ff44c164000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff44ad9d640 1 -- 172.21.15.67:0/1214686927 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44c0c83b0 con 0x7ff44c161440 2024-04-25T15:50:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff38bfff640 1 Processor -- start 2024-04-25T15:50:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efeceaae640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7efec8160730 0x7efec80c3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efeceaae640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec80c4500 con 0x7efec806d4d0 2024-04-25T15:50:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efeceaae640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec80c47e0 con 0x7efec80c8860 2024-04-25T15:50:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efeceaae640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7efec80c42e0 con 0x7efec8160730 2024-04-25T15:50:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efecd04b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7efec80c8860 0x7efec80c8c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:48866/0 (socket says 172.21.15.67:48866) 2024-04-25T15:50:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efecd04b640 1 -- 172.21.15.67:0/3094344306 learned_addr learned my addr 172.21.15.67:0/3094344306 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7efec7fff640 1 --2- 172.21.15.67:0/3094344306 >> v2:172.21.15.176:3300/0 conn(0x7efec8160730 0x7efec80c3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.186+0000 7ff38bfff640 1 -- start start 2024-04-25T15:50:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1738767289 0 0) 0x7efec80c47e0 con 0x7efec80c8860 2024-04-25T15:50:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeb8003530 con 0x7efec80c8860 2024-04-25T15:50:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7feedb7fe640 1 --2- 172.21.15.67:0/4265483293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc0b3b70 0x7feedc068590 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feed4009e50 tx=0x7feed401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adab1e0d43da1616 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3498599220 0 0) 0x7efec80c4500 con 0x7efec806d4d0 2024-04-25T15:50:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efec80c47e0 con 0x7efec806d4d0 2024-04-25T15:50:39.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff44ad9d640 1 --2- 172.21.15.67:0/1214686927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c161440 0x7ff44c0c8e90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff44c070a10 tx=0x7ff43c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c7b3fd2ad5230e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f8cc15a5640 1 Processor -- start 2024-04-25T15:50:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f8cc15a5640 1 -- start start 2024-04-25T15:50:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff449d9b640 1 -- 172.21.15.67:0/1214686927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0b5070 con 0x7ff44c161440 2024-04-25T15:50:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff449d9b640 1 -- 172.21.15.67:0/1214686927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43c0a89d0 con 0x7ff44c161440 2024-04-25T15:50:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff449d9b640 1 -- 172.21.15.67:0/1214686927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0a8da0 con 0x7ff44c161440 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff38c0c51d0 0x7ff38c15acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c06efb0 0x7ff38c06f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38bfff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff38c0c6b60 con 0x7ff38c0c99c0 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38bfff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff38c0c77d0 con 0x7ff38c0c51d0 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38bfff640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff38c161510 con 0x7ff38c06efb0 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1129218054 0 0) 0x7efeb8003530 con 0x7efec80c8860 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 >> v2:172.21.15.176:3300/0 conn(0x7efec8160730 msgr2=0x7efec80c3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 --2- 172.21.15.67:0/3094344306 >> v2:172.21.15.176:3300/0 conn(0x7efec8160730 0x7efec80c3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 >> v1:172.21.15.67:6789/0 conn(0x7efec806d4d0 legacy=0x7efec806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec815fc50 con 0x7efec80c8860 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7efec0001a00 con 0x7efec80c8860 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 >> v1:172.21.15.176:6789/0 conn(0x7efec80c8860 legacy=0x7efec80c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb8003b50 0x7efeb8003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb8004810 0x7efeb8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efeb800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efec80c42e0 con 0x7efeb80051b0 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efeb800c150 con 0x7efeb8003b50 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efeb800c350 con 0x7efeb8004810 2024-04-25T15:50:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec7fff640 1 --2- 172.21.15.67:0/3094344306 >> v2:172.21.15.176:3300/0 conn(0x7efec8160730 0x7efec80c3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff38c0c51d0 0x7ff38c15acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff38c0c51d0 0x7ff38c15acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59624/0 (socket says 172.21.15.67:59624) 2024-04-25T15:50:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38affd640 1 -- 172.21.15.67:0/2870729802 learned_addr learned my addr 172.21.15.67:0/2870729802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7fb2db400640 1 Processor -- start 2024-04-25T15:50:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38b7fe640 1 --2- 172.21.15.67:0/2870729802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c06efb0 0x7ff38c06f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7fb2db400640 1 -- start start 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7feeda7fc640 1 -- 172.21.15.67:0/4265483293 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40a5560 con 0x7feedc0b3b70 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7feeda7fc640 1 -- 172.21.15.67:0/4265483293 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed40a5700 con 0x7feedc0b3b70 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7feeda7fc640 1 -- 172.21.15.67:0/4265483293 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed40a5ad0 con 0x7feedc0b3b70 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7feee1f08640 1 -- 172.21.15.67:0/4265483293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc0b3b70 msgr2=0x7feedc068590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7feee1f08640 1 --2- 172.21.15.67:0/4265483293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc0b3b70 0x7feedc068590 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feed4009e50 tx=0x7feed401c9f0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 -- 172.21.15.67:0/4265483293 shutdown_connections 2024-04-25T15:50:39.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 --2- 172.21.15.67:0/4265483293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc06f980 0x7feedc06fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 --2- 172.21.15.67:0/4265483293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc0b3b70 0x7feedc068590 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 -- 172.21.15.67:0/4265483293 >> 172.21.15.67:0/4265483293 conn(0x7feedc0ca9d0 msgr2=0x7feedc0cade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 -- 172.21.15.67:0/1214686927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c161440 msgr2=0x7ff44c0c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 --2- 172.21.15.67:0/1214686927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c161440 0x7ff44c0c8e90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff44c070a10 tx=0x7ff43c01ecd0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 -- 172.21.15.67:0/1214686927 shutdown_connections 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 --2- 172.21.15.67:0/1214686927 >> v2:172.21.15.176:3300/0 conn(0x7ff44c0c93d0 0x7ff44c06ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 --2- 172.21.15.67:0/1214686927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c161440 0x7ff44c0c8e90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 -- 172.21.15.67:0/1214686927 >> 172.21.15.67:0/1214686927 conn(0x7ff44c0ca050 msgr2=0x7ff44c0ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 -- 172.21.15.67:0/1214686927 shutdown_connections 2024-04-25T15:50:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff45105f640 1 -- 172.21.15.67:0/1214686927 wait complete. 2024-04-25T15:50:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38affd640 1 -- 172.21.15.67:0/2870729802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c06efb0 msgr2=0x7ff38c06f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38affd640 1 --2- 172.21.15.67:0/2870729802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c06efb0 0x7ff38c06f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38affd640 1 -- 172.21.15.67:0/2870729802 >> v1:172.21.15.67:6789/0 conn(0x7ff38c0c99c0 legacy=0x7ff38c0c6450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:50:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7ff38affd640 1 -- 172.21.15.67:0/2870729802 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38c1617f0 con 0x7ff38c0c51d0 2024-04-25T15:50:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f8cc15a5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cbc15c8c0 0x7f8cbc0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f8cc15a5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc153070 con 0x7f8cbc072cf0 2024-04-25T15:50:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f8cc15a5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cbc0c8110 con 0x7f8cbc15d240 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f8cc15a5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8cbc0c7ec0 con 0x7f8cbc15c8c0 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efecd04b640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efeb800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efec7fff640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb8004810 0x7efeb8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efecc84a640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb8003b50 0x7efeb8003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efecd04b640 1 -- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb8004810 msgr2=0x7efeb8004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efecd04b640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb8004810 0x7efeb8004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efecd04b640 1 -- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb8003b50 msgr2=0x7efeb8003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efecd04b640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb8003b50 0x7efeb8003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7efecd04b640 1 -- 172.21.15.67:0/3094344306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec815fc50 con 0x7efeb80051b0 2024-04-25T15:50:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 -- 172.21.15.67:0/4265483293 shutdown_connections 2024-04-25T15:50:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 -- 172.21.15.67:0/4265483293 wait complete. 2024-04-25T15:50:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8cbc072cf0 0x7f8cbc073100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36020/0 (socket says 172.21.15.67:36020) 2024-04-25T15:50:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbb7fe640 1 -- 172.21.15.67:0/2565170728 learned_addr learned my addr 172.21.15.67:0/2565170728 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3470985606 0 0) 0x7f8cbc153070 con 0x7f8cbc072cf0 2024-04-25T15:50:39.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ca4003530 con 0x7f8cbc072cf0 2024-04-25T15:50:39.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7ff45105f640 1 Processor -- start 2024-04-25T15:50:39.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbbfff640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cbc15c8c0 0x7f8cbc0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1314192785 0 0) 0x7f8ca4003530 con 0x7f8cbc072cf0 2024-04-25T15:50:39.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cbc15c8c0 msgr2=0x7f8cbc0c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cbc15c8c0 0x7f8cbc0c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 >> v1:172.21.15.176:6789/0 conn(0x7f8cbc15d240 legacy=0x7f8cbc15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc0c83f0 con 0x7f8cbc072cf0 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8cb0001680 con 0x7f8cbc072cf0 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 >> v1:172.21.15.67:6789/0 conn(0x7f8cbc072cf0 legacy=0x7f8cbc073100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca40039f0 0x7f8ca4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca4004810 0x7f8ca4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca4005160 0x7f8ca40145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc0c7ec0 con 0x7f8ca40039f0 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca4014b10 con 0x7f8ca4004810 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca4014c90 con 0x7f8ca4005160 2024-04-25T15:50:39.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7efecd04b640 1 -- 172.21.15.67:0/3094344306 reap_dead start 2024-04-25T15:50:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7ff45105f640 1 -- start start 2024-04-25T15:50:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cc0da4640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca4005160 0x7f8ca40145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f17adde9640 1 Processor -- start 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.190+0000 7f17adde9640 1 -- start start 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17adde9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17a80c99c0 0x7f17a8169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17adde9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c9240 0x7f17a80c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17adde9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17a80c7ec0 con 0x7f17a80c4940 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17adde9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f17a80c81a0 con 0x7f17a80c99c0 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17adde9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17a80c8480 con 0x7f17a80c9240 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17ad5e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c9240 0x7f17a80c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17ad5e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c9240 0x7f17a80c51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60512/0 (socket says 172.21.15.67:60512) 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17ad5e8640 1 -- 172.21.15.67:0/808650945 learned_addr learned my addr 172.21.15.67:0/808650945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbb7fe640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca4004810 0x7f8ca4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7efecd04b640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efeb800b9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efeb80057f0 tx=0x7efec001fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267433e8942afaf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2db400640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc047a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 Processor -- start 2024-04-25T15:50:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2db400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2cc0dc110 0x7fb2cc04c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2db400640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2cc0d5df0 con 0x7fb2cc04c900 2024-04-25T15:50:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2db400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb2cc046070 con 0x7fb2cc0dc110 2024-04-25T15:50:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2db400640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc04d200 con 0x7fb2cc046ad0 2024-04-25T15:50:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17a77fe640 1 -- 172.21.15.67:0/808650945 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595686912 0 0) 0x7f17a80c7ec0 con 0x7f17a80c4940 2024-04-25T15:50:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17a77fe640 1 -- 172.21.15.67:0/808650945 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1790003530 con 0x7f17a80c4940 2024-04-25T15:50:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7feee1f08640 1 -- start start 2024-04-25T15:50:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec000a4e0 con 0x7efeb80051b0 2024-04-25T15:50:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec00072d0 con 0x7efeb80051b0 2024-04-25T15:50:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7efec77fe640 1 -- 172.21.15.67:0/3094344306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec00076c0 con 0x7efeb80051b0 2024-04-25T15:50:39.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17acde7640 1 --2- 172.21.15.67:0/808650945 >> v2:172.21.15.67:3300/0 conn(0x7f17a80c99c0 0x7f17a8169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2d9bfd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2cc0dc110 0x7fb2cc04c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2d9bfd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2cc0dc110 0x7fb2cc04c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59660/0 (socket says 172.21.15.67:59660) 2024-04-25T15:50:39.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7fb2d9bfd640 1 -- 172.21.15.67:0/31270253 learned_addr learned my addr 172.21.15.67:0/31270253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7ff45105f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06f350 0x7ff44c1eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f8cbbfff640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca40039f0 0x7f8ca4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff45105f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c0c93d0 0x7ff44c154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff45105f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c161440 0x7ff44c1508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7fb2da3fe640 1 --2- 172.21.15.67:0/31270253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc047a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06f350 0x7ff44c1eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06f350 0x7ff44c1eb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59668/0 (socket says 172.21.15.67:59668) 2024-04-25T15:50:39.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff38affd640 1 --2- 172.21.15.67:0/2870729802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c0c51d0 0x7ff38c15acc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff38c06db30 tx=0x7ff38401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c0ea3575a0715a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 -- 172.21.15.67:0/3094344306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 msgr2=0x7efeb800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efeb800b9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efeb80057f0 tx=0x7efec001fa40 comp rx=0 tx=0).stop 2024-04-25T15:50:39.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7fb2d9bfd640 1 -- 172.21.15.67:0/31270253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 msgr2=0x7fb2cc047a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7fb2d9bfd640 1 --2- 172.21.15.67:0/31270253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc047a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7fb2d9bfd640 1 -- 172.21.15.67:0/31270253 >> v1:172.21.15.176:6789/0 conn(0x7fb2cc04c900 legacy=0x7fb2cc0decf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7fb2d9bfd640 1 -- 172.21.15.67:0/31270253 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc046690 con 0x7fb2cc0dc110 2024-04-25T15:50:39.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c0c93d0 0x7ff44c154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c0c93d0 0x7ff44c154d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60532/0 (socket says 172.21.15.67:60532) 2024-04-25T15:50:39.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff45105f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c0c9890 con 0x7ff44c06f350 2024-04-25T15:50:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff45105f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c0c8970 con 0x7ff44c161440 2024-04-25T15:50:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff45105f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff44c0c4590 con 0x7ff44c0c93d0 2024-04-25T15:50:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7f8cbbfff640 1 -- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca4005160 msgr2=0x7f8ca40145d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7f8cbbfff640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca4005160 0x7f8ca40145d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7f8cbbfff640 1 -- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca4004810 msgr2=0x7f8ca4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7f8cbbfff640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca4004810 0x7f8ca4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7f8cbbfff640 1 -- 172.21.15.67:0/2565170728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc0c83f0 con 0x7f8ca40039f0 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 -- 172.21.15.67:0/3094344306 shutdown_connections 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 --2- 172.21.15.67:0/3094344306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efeb800b9d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 -- 172.21.15.67:0/3094344306 >> 172.21.15.67:0/3094344306 conn(0x7efec80cde80 msgr2=0x7efec80ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 -- 172.21.15.67:0/3094344306 shutdown_connections 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17ad5e8640 1 -- 172.21.15.67:0/808650945 >> v2:172.21.15.67:3300/0 conn(0x7f17a80c99c0 msgr2=0x7f17a8169ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17a77fe640 1 -- 172.21.15.67:0/808650945 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1375520567 0 0) 0x7f1790003530 con 0x7f17a80c4940 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17ad5e8640 1 --2- 172.21.15.67:0/808650945 >> v2:172.21.15.67:3300/0 conn(0x7f17a80c99c0 0x7f17a8169ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17ad5e8640 1 -- 172.21.15.67:0/808650945 >> v1:172.21.15.176:6789/0 conn(0x7f17a80c4940 legacy=0x7f17a806efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17ad5e8640 1 -- 172.21.15.67:0/808650945 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a80c8760 con 0x7f17a80c9240 2024-04-25T15:50:39.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.194+0000 7f17acde7640 1 --2- 172.21.15.67:0/808650945 >> v2:172.21.15.67:3300/0 conn(0x7f17a80c99c0 0x7f17a8169ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:39.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 -- 172.21.15.67:0/3094344306 wait complete. 2024-04-25T15:50:39.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c161440 0x7ff44c1508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff389ffb640 1 -- 172.21.15.67:0/2870729802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3840b4070 con 0x7ff38c0c51d0 2024-04-25T15:50:39.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff389ffb640 1 -- 172.21.15.67:0/2870729802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3840af400 con 0x7ff38c0c51d0 2024-04-25T15:50:39.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff389ffb640 1 -- 172.21.15.67:0/2870729802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3840af810 con 0x7ff38c0c51d0 2024-04-25T15:50:39.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c161440 0x7ff44c1508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39878/0 (socket says 172.21.15.67:39878) 2024-04-25T15:50:39.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 -- 172.21.15.67:0/1186898224 learned_addr learned my addr 172.21.15.67:0/1186898224 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 -- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c0c93d0 msgr2=0x7ff44c154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c0c93d0 0x7ff44c154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 -- 172.21.15.67:0/1186898224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06f350 msgr2=0x7ff44c1eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff44c06f350 0x7ff44c1eb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 -- 172.21.15.67:0/1186898224 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44c1eb840 con 0x7ff44c161440 2024-04-25T15:50:39.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff44b59e640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c161440 0x7ff44c1508c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff44400b990 tx=0x7ff44400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feee1f08640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc06f980 0x7feedc0c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444019070 con 0x7ff44c161440 2024-04-25T15:50:39.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 Processor -- start 2024-04-25T15:50:39.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feee1f08640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc0b3b70 0x7feedc15b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44400cba0 con 0x7ff44c161440 2024-04-25T15:50:39.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444014440 con 0x7ff44c161440 2024-04-25T15:50:39.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feee1f08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc1610f0 0x7feedc15fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7fb2d9bfd640 1 --2- 172.21.15.67:0/31270253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc0dc110 0x7fb2cc04c3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0d5fc0 tx=0x7fb2c40211a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f86bc69af5d270b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7efeceaae640 1 -- start start 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7f8cbbfff640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca40039f0 0x7f8ca4003e40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0ce560 tx=0x7f8cb401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c951ec249d884f0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feee1f08640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feedc16dd40 con 0x7feedc1610f0 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feee1f08640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feedc16dbc0 con 0x7feedc06f980 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feee1f08640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feedc16dec0 con 0x7feedc0b3b70 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feedbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc1610f0 0x7feedc15fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feedbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc1610f0 0x7feedc15fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59682/0 (socket says 172.21.15.67:59682) 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feedbfff640 1 -- 172.21.15.67:0/3560485695 learned_addr learned my addr 172.21.15.67:0/3560485695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedaffd640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc0b3b70 0x7feedc15b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7feedb7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc06f980 0x7feedc0c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedbfff640 1 -- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc0b3b70 msgr2=0x7feedc15b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedbfff640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feedc0b3b70 0x7feedc15b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedbfff640 1 -- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc06f980 msgr2=0x7feedc0c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedbfff640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc06f980 0x7feedc0c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedbfff640 1 -- 172.21.15.67:0/3560485695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feedc15bd50 con 0x7feedc1610f0 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 -- 172.21.15.67:0/2870729802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c0c51d0 msgr2=0x7ff38c15acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 --2- 172.21.15.67:0/2870729802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c0c51d0 0x7ff38c15acc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff38c06db30 tx=0x7ff38401f1e0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44c1f8070 con 0x7ff44c161440 2024-04-25T15:50:39.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff44c161eb0 con 0x7ff44c161440 2024-04-25T15:50:39.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedb7fe640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feedc06f980 0x7feedc0c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:39.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.198+0000 7f17ad5e8640 1 --2- 172.21.15.67:0/808650945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c9240 0x7f17a80c51d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1798007eb0 tx=0x7f179801ddd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=506f70a846e0f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17a77fe640 1 -- 172.21.15.67:0/808650945 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17980b3070 con 0x7f17a80c9240 2024-04-25T15:50:39.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17a77fe640 1 -- 172.21.15.67:0/808650945 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17980a6da0 con 0x7f17a80c9240 2024-04-25T15:50:39.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17a77fe640 1 -- 172.21.15.67:0/808650945 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17980ae660 con 0x7f17a80c9240 2024-04-25T15:50:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2d93fc640 1 -- 172.21.15.67:0/31270253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4004030 con 0x7fb2cc0dc110 2024-04-25T15:50:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2d93fc640 1 -- 172.21.15.67:0/31270253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c40041d0 con 0x7fb2cc0dc110 2024-04-25T15:50:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2d93fc640 1 -- 172.21.15.67:0/31270253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4003620 con 0x7fb2cc0dc110 2024-04-25T15:50:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feedbfff640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc1610f0 0x7feedc15fb50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feed000b440 tx=0x7feed000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff44c0c6fe0 con 0x7ff44c161440 2024-04-25T15:50:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40b0070 con 0x7f8ca40039f0 2024-04-25T15:50:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb40ab600 con 0x7f8ca40039f0 2024-04-25T15:50:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f8cbaffd640 1 -- 172.21.15.67:0/2565170728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4007a90 con 0x7f8ca40039f0 2024-04-25T15:50:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efec80c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec80b0890 0x7efec80b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efec80b11f0 0x7efec8159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efec8160b40 con 0x7efeb80051b0 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efec80c4ac0 con 0x7efec80b11f0 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efec8160210 con 0x7efec80b0890 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efec80c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efec80c7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59698/0 (socket says 172.21.15.67:59698) 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 -- 172.21.15.67:0/2684854125 learned_addr learned my addr 172.21.15.67:0/2684854125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2565170728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca40039f0 msgr2=0x7f8ca4003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f8cc15a5640 1 --2- 172.21.15.67:0/2565170728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ca40039f0 0x7f8ca4003e40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0ce560 tx=0x7f8cb401ca30 comp rx=0 tx=0).stop 2024-04-25T15:50:39.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f8cc0da4640 1 -- 172.21.15.67:0/2565170728 reap_dead start 2024-04-25T15:50:39.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 -- 172.21.15.67:0/2870729802 shutdown_connections 2024-04-25T15:50:39.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 -- 172.21.15.67:0/808650945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c9240 msgr2=0x7f17a80c51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 --2- 172.21.15.67:0/808650945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c9240 0x7f17a80c51d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1798007eb0 tx=0x7f179801ddd0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0015020 con 0x7feedc1610f0 2024-04-25T15:50:39.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 --2- 172.21.15.67:0/2870729802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c06efb0 0x7ff38c06f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 --2- 172.21.15.67:0/2870729802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c0c51d0 0x7ff38c15acc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 -- 172.21.15.67:0/2870729802 >> 172.21.15.67:0/2870729802 conn(0x7ff38c0c9fc0 msgr2=0x7ff38c0ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 -- 172.21.15.67:0/808650945 shutdown_connections 2024-04-25T15:50:39.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 --2- 172.21.15.67:0/808650945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c9240 0x7f17a80c51d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed0011850 con 0x7feedc1610f0 2024-04-25T15:50:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed0011c40 con 0x7feedc1610f0 2024-04-25T15:50:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feedc159ee0 con 0x7feedc1610f0 2024-04-25T15:50:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feedc15a460 con 0x7feedc1610f0 2024-04-25T15:50:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 --2- 172.21.15.67:0/808650945 >> v2:172.21.15.67:3300/0 conn(0x7f17a80c99c0 0x7f17a8169ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 -- 172.21.15.67:0/808650945 >> 172.21.15.67:0/808650945 conn(0x7f17a80c9fc0 msgr2=0x7f17a80ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 -- 172.21.15.67:0/2870729802 shutdown_connections 2024-04-25T15:50:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2db400640 1 -- 172.21.15.67:0/31270253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc0dc110 msgr2=0x7fb2cc04c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2db400640 1 --2- 172.21.15.67:0/31270253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc0dc110 0x7fb2cc04c3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0d5fc0 tx=0x7fb2c40211a0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efec7fff640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec80b0890 0x7efec80b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 -- 172.21.15.67:0/808650945 shutdown_connections 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecd04b640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efec80b11f0 0x7efec8159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 -- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec80b0890 msgr2=0x7efec80b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec80b0890 0x7efec80b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 -- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efec80b11f0 msgr2=0x7efec8159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efec80b11f0 0x7efec8159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 -- 172.21.15.67:0/2684854125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec81f2a70 con 0x7efeb80051b0 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2db400640 1 -- 172.21.15.67:0/31270253 shutdown_connections 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2db400640 1 --2- 172.21.15.67:0/31270253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc0dc110 0x7fb2cc04c3c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2db400640 1 --2- 172.21.15.67:0/31270253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc047a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7fb2db400640 1 -- 172.21.15.67:0/31270253 >> 172.21.15.67:0/31270253 conn(0x7fb2cc050c50 msgr2=0x7fb2cc051060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efecc84a640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efec80c7d90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efebc00e580 tx=0x7efebc00ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc0052e0 con 0x7efeb80051b0 2024-04-25T15:50:39.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efebc018070 con 0x7efeb80051b0 2024-04-25T15:50:39.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc0054a0 con 0x7efeb80051b0 2024-04-25T15:50:39.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec81f2d50 con 0x7efeb80051b0 2024-04-25T15:50:39.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec81f3310 con 0x7efeb80051b0 2024-04-25T15:50:39.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feebc001040 con 0x7feedc1610f0 2024-04-25T15:50:39.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff44401c070 con 0x7ff44c161440 2024-04-25T15:50:39.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efec8050750 con 0x7efeb80051b0 2024-04-25T15:50:39.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7f17adde9640 1 -- 172.21.15.67:0/808650945 wait complete. 2024-04-25T15:50:39.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7fb2db400640 1 -- 172.21.15.67:0/31270253 shutdown_connections 2024-04-25T15:50:39.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f17adde9640 1 Processor -- start 2024-04-25T15:50:39.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.202+0000 7ff38bfff640 1 -- 172.21.15.67:0/2870729802 wait complete. 2024-04-25T15:50:39.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff38bfff640 1 Processor -- start 2024-04-25T15:50:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2565170728 shutdown_connections 2024-04-25T15:50:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2565170728 >> 172.21.15.67:0/2565170728 conn(0x7f8cbc0c9fc0 msgr2=0x7f8cbc0ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f17adde9640 1 -- start start 2024-04-25T15:50:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2565170728 shutdown_connections 2024-04-25T15:50:39.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff38bfff640 1 -- start start 2024-04-25T15:50:39.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7feed0018020 con 0x7feedc1610f0 2024-04-25T15:50:39.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7feec3fff640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feec806f430 0x7feec8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7feed008ec70 con 0x7feedc1610f0 2024-04-25T15:50:39.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7feec3fff640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7feec8077680 0x7feec8079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feec807a1a0 con 0x7feec8077680 2024-04-25T15:50:39.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7feedb7fe640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feec806f430 0x7feec8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7feedaffd640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7feec8077680 0x7feec8079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efebc003840 con 0x7efeb80051b0 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efec57fa640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeac06f500 0x7efeac071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efebc08fb00 con 0x7efeb80051b0 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efec57fa640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7efeac0777d0 0x7efeac079c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efeac07a2f0 con 0x7efeac0777d0 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7efebc08fef0 con 0x7efeb80051b0 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efec7fff640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeac06f500 0x7efeac071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff4337fe640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff43806f400 0x7ff438071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff44408f840 con 0x7ff44c161440 2024-04-25T15:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff4337fe640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff438077650 0x7ff438079aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff444020e50 con 0x7ff438077650 2024-04-25T15:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff44408fbf0 con 0x7ff44c161440 2024-04-25T15:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff44ad9d640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff43806f400 0x7ff438071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff44ad9d640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff43806f400 0x7ff438071910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0b2000 tx=0x7ff43c0aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f17adde9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c4940 0x7f17a8203320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7fb2db400640 1 -- 172.21.15.67:0/31270253 wait complete. 2024-04-25T15:50:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f17adde9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a80c9240 0x7f17a8089df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f17adde9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a80c99c0 0x7f17a80cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff38bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c06efb0 0x7ff38c15da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2565170728 wait complete. 2024-04-25T15:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7efecd04b640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7efeac0777d0 0x7efeac079c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17acde7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c4940 0x7f17a8203320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17acde7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c4940 0x7f17a8203320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60548/0 (socket says 172.21.15.67:60548) 2024-04-25T15:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17acde7640 1 -- 172.21.15.67:0/181328738 learned_addr learned my addr 172.21.15.67:0/181328738 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a7fff640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a80c9240 0x7f17a8089df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7feed0009170 con 0x7feedc1610f0 2024-04-25T15:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7ff44a59c640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff438077650 0x7ff438079aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c0c51d0 0x7ff38c1ea2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff38c0c99c0 0x7ff38c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17adde9640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17a81764a0 con 0x7f17a80c9240 2024-04-25T15:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c06efb0 0x7ff38c15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c06efb0 0x7ff38c15da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59728/0 (socket says 172.21.15.67:59728) 2024-04-25T15:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38affd640 1 -- 172.21.15.67:0/2045848945 learned_addr learned my addr 172.21.15.67:0/2045848945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17ad5e8640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a80c99c0 0x7f17a80cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 Processor -- start 2024-04-25T15:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 Processor -- start 2024-04-25T15:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 -- start start 2024-04-25T15:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17adde9640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f17a8176320 con 0x7f17a80c99c0 2024-04-25T15:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17adde9640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17a8176620 con 0x7f17a80c4940 2024-04-25T15:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7feedb7fe640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feec806f430 0x7feec8071940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feed40ae000 tx=0x7feed40a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7feedaffd640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7feec8077680 0x7feec8079ad0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff44a59c640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff438077650 0x7ff438079aa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38a7fc640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c0c51d0 0x7ff38c1ea2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38b7fe640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff38c0c99c0 0x7ff38c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 -- start start 2024-04-25T15:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a7fff640 1 -- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c4940 msgr2=0x7f17a8203320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a7fff640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17a80c4940 0x7f17a8203320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a7fff640 1 -- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a80c99c0 msgr2=0x7f17a80cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a7fff640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17a80c99c0 0x7f17a80cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a7fff640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a808a330 con 0x7f17a80c9240 2024-04-25T15:50:39.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7efec7fff640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeac06f500 0x7efeac071a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efec81e6d10 tx=0x7efeb4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feec807a1a0 con 0x7feec8077680 2024-04-25T15:50:39.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f006ad0e640 1 Processor -- start 2024-04-25T15:50:39.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f006ad0e640 1 -- start start 2024-04-25T15:50:39.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f006ad0e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f005c045da0 0x7f005c0457e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f006ad0e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f005c0e7c20 con 0x7f005c0060a0 2024-04-25T15:50:39.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f006ad0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f005c0e7960 con 0x7f005c0064b0 2024-04-25T15:50:39.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.206+0000 7f006ad0e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f005c0e3860 con 0x7f005c045da0 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f0068aaa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f005c045da0 0x7f005c0457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f0068aaa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f005c045da0 0x7f005c0457e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59712/0 (socket says 172.21.15.67:59712) 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f0068aaa640 1 -- 172.21.15.67:0/4126122557 learned_addr learned my addr 172.21.15.67:0/4126122557 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f0063fff640 1 --1- 172.21.15.67:0/4126122557 >> v1:172.21.15.67:6789/0 conn(0x7f005c0060a0 0x7f005c04b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36036/0 (socket says 172.21.15.67:36036) 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f0068aaa640 1 -- 172.21.15.67:0/4126122557 >> v1:172.21.15.176:6789/0 conn(0x7f005c0064b0 legacy=0x7f005c0ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f0068aaa640 1 -- 172.21.15.67:0/4126122557 >> v1:172.21.15.67:6789/0 conn(0x7f005c0060a0 legacy=0x7f005c04b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f0068aaa640 1 -- 172.21.15.67:0/4126122557 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c0e3b40 con 0x7f005c045da0 2024-04-25T15:50:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7efecd04b640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7efeac0777d0 0x7efeac079c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c0c5f30 con 0x7ff38c06efb0 2024-04-25T15:50:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff38c0c5c50 con 0x7ff38c0c99c0 2024-04-25T15:50:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff38c0c7ec0 con 0x7ff38c0c51d0 2024-04-25T15:50:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a7fff640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a80c9240 0x7f17a8089df0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17a80cb3f0 tx=0x7f179c00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff444020e50 con 0x7ff438077650 2024-04-25T15:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca40039f0 0x7f8cbc154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cbc0c64e0 0x7f8cbc0c5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cbc1e4f50 0x7f8cbc14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc0c8c90 con 0x7f8cbc0c64e0 2024-04-25T15:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8cbc0c89b0 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc15a5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8cbc15ccd0 con 0x7f8ca40039f0 2024-04-25T15:50:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc0da4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cbc1e4f50 0x7f8cbc14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38b7fe640 1 -- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c0c51d0 msgr2=0x7ff38c1ea2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38b7fe640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c0c51d0 0x7ff38c1ea2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38b7fe640 1 -- 172.21.15.67:0/2045848945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c06efb0 msgr2=0x7ff38c15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38b7fe640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff38c06efb0 0x7ff38c15da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38b7fe640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38c15e1c0 con 0x7ff38c0c99c0 2024-04-25T15:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38a7fc640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff38c0c51d0 0x7ff38c1ea2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc0da4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cbc1e4f50 0x7f8cbc14be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39914/0 (socket says 172.21.15.67:39914) 2024-04-25T15:50:39.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc0da4640 1 -- 172.21.15.67:0/2992060309 learned_addr learned my addr 172.21.15.67:0/2992060309 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc182fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38b7fe640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff38c0c99c0 0x7ff38c157b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e7c0 tx=0x7ff38000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc04c900 0x7fb2cc0d3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0dc110 0x7fb2cc0d1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c017070 con 0x7f17a80c9240 2024-04-25T15:50:39.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f179c003600 con 0x7f17a80c9240 2024-04-25T15:50:39.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c0039f0 con 0x7f17a80c9240 2024-04-25T15:50:39.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc0dbbf0 con 0x7fb2cc04c900 2024-04-25T15:50:39.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc0db910 con 0x7fb2cc0dc110 2024-04-25T15:50:39.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7fb2db400640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc033800 con 0x7fb2cc046ad0 2024-04-25T15:50:39.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380017070 con 0x7ff38c0c99c0 2024-04-25T15:50:39.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cbb7fe640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cbc0c64e0 0x7f8cbc0c5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f17adde9640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a8088b00 con 0x7f17a80c9240 2024-04-25T15:50:39.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc0da4640 1 -- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca40039f0 msgr2=0x7f8cbc154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7f8cc0da4640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca40039f0 0x7f8cbc154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cc0da4640 1 -- 172.21.15.67:0/2992060309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cbc0c64e0 msgr2=0x7f8cbc0c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cc0da4640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cbc0c64e0 0x7f8cbc0c5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc182fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc182fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60574/0 (socket says 172.21.15.67:60574) 2024-04-25T15:50:39.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 -- 172.21.15.67:0/1750539756 learned_addr learned my addr 172.21.15.67:0/1750539756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cc0da4640 1 -- 172.21.15.67:0/2992060309 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cbc155320 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f0068aaa640 1 --2- 172.21.15.67:0/4126122557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c045da0 0x7f005c0457e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0058009870 tx=0x7f005801ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d1a7ddd782e145 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.210+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38c1ebcf0 con 0x7ff38c0c99c0 2024-04-25T15:50:39.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38c15ce60 con 0x7ff38c0c99c0 2024-04-25T15:50:39.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cc0da4640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cbc1e4f50 0x7f8cbc14be30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc071980 tx=0x7f8cac00f910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2d9bfd640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc04c900 0x7fb2cc0d3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff38000c730 con 0x7ff38c0c99c0 2024-04-25T15:50:39.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f17adde9640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a80895f0 con 0x7f17a80c9240 2024-04-25T15:50:39.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38000cb00 con 0x7ff38c0c99c0 2024-04-25T15:50:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efeac07a2f0 con 0x7efeac0777d0 2024-04-25T15:50:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 -- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0dc110 msgr2=0x7fb2cc0d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2cc0dc110 0x7fb2cc0d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 -- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc04c900 msgr2=0x7fb2cc0d3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2cc04c900 0x7fb2cc0d3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 -- 172.21.15.67:0/1750539756 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc183500 con 0x7fb2cc046ad0 2024-04-25T15:50:39.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1780001040 con 0x7f17a80c9240 2024-04-25T15:50:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2da3fe640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc182fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000d5f0 tx=0x7fb2d000dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00125f0 con 0x7fb2cc046ad0 2024-04-25T15:50:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac003a80 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2d0019070 con 0x7fb2cc046ad0 2024-04-25T15:50:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2cc04ef80 con 0x7fb2cc046ad0 2024-04-25T15:50:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2cc04f540 con 0x7fb2cc046ad0 2024-04-25T15:50:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00129b0 con 0x7fb2cc046ad0 2024-04-25T15:50:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cbc0c6d90 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cbc154300 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8cbc050750 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2bc001040 con 0x7fb2cc046ad0 2024-04-25T15:50:39.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff38c15a850 con 0x7ff38c0c99c0 2024-04-25T15:50:39.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f00637fe640 1 -- 172.21.15.67:0/4126122557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00580b5070 con 0x7f005c045da0 2024-04-25T15:50:39.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f00637fe640 1 -- 172.21.15.67:0/4126122557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00580a8a50 con 0x7f005c045da0 2024-04-25T15:50:39.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f00637fe640 1 -- 172.21.15.67:0/4126122557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00580a8e20 con 0x7f005c045da0 2024-04-25T15:50:39.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 -- 172.21.15.67:0/4126122557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c045da0 msgr2=0x7f005c0457e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 --2- 172.21.15.67:0/4126122557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c045da0 0x7f005c0457e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0058009870 tx=0x7f005801ed70 comp rx=0 tx=0).stop 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 -- 172.21.15.67:0/4126122557 shutdown_connections 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 --2- 172.21.15.67:0/4126122557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c045da0 0x7f005c0457e0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0058009870 tx=0x7f005801ed70 comp rx=0 tx=0).stop 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 -- 172.21.15.67:0/4126122557 >> 172.21.15.67:0/4126122557 conn(0x7f005c0503c0 msgr2=0x7f005c0507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff38001a070 con 0x7ff38c0c99c0 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.214+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f179c01a070 con 0x7f17a80c9240 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f17a57fa640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f179406f450 0x7f1794071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f179c08eb60 con 0x7f17a80c9240 2024-04-25T15:50:39.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f17a57fa640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1794077720 0x7f1794079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 -- 172.21.15.67:0/4126122557 shutdown_connections 2024-04-25T15:50:39.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f179407a240 con 0x7f1794077720 2024-04-25T15:50:39.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f179c08ef10 con 0x7f17a80c9240 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff3737fe640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff37806f430 0x7ff378071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff38008f2d0 con 0x7ff38c0c99c0 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff38affd640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff37806f430 0x7ff378071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff3737fe640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff378077700 0x7ff378079b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff380015e50 con 0x7ff378077700 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff380096050 con 0x7ff38c0c99c0 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff38a7fc640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff378077700 0x7ff378079b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff38affd640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff37806f430 0x7ff378071940 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff38401f220 tx=0x7ff3840aa000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb2d0012b50 con 0x7fb2cc046ad0 2024-04-25T15:50:39.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7fb2caffd640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2b806f4d0 0x7fb2b80719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb2d00900c0 con 0x7fb2cc046ad0 2024-04-25T15:50:39.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f17ad5e8640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1794077720 0x7f1794079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f17acde7640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f179406f450 0x7f1794071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7ff38a7fc640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff378077700 0x7ff378079b50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cac018070 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac004030 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 -- 172.21.15.67:0/4126122557 wait complete. 2024-04-25T15:50:39.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7f006ad0e640 1 Processor -- start 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7fb2caffd640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb2b80777a0 0x7fb2b8079bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2b807a2c0 con 0x7fb2b80777a0 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7fb2d9bfd640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2b806f4d0 0x7fb2b80719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.218+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb2d00904b0 con 0x7fb2cc046ad0 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7fb2dabff640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb2b80777a0 0x7fb2b8079bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f17ad5e8640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1794077720 0x7f1794079b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f17acde7640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f179406f450 0x7f1794071960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f17a0006fd0 tx=0x7f17a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7fb2d9bfd640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2b806f4d0 0x7fb2b80719e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2c40b4000 tx=0x7fb2c40ac000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7fb2dabff640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb2b80777a0 0x7fb2b8079bf0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f006ad0e640 1 -- start start 2024-04-25T15:50:39.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f006ad0e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f005c0060a0 0x7f005c0db9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8cac0041d0 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f006ad0e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f005c0064b0 0x7f005c04f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f006ad0e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c0e2b70 0x7f005c035610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2b807a2c0 con 0x7fb2b80777a0 2024-04-25T15:50:39.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f179407a240 con 0x7f1794077720 2024-04-25T15:50:39.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cb8ff9640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8ca806f430 0x7f8ca8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f0063fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f005c0064b0 0x7f005c04f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f0063fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f005c0064b0 0x7f005c04f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60586/0 (socket says 172.21.15.67:60586) 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f0063fff640 1 -- 172.21.15.67:0/1737189930 learned_addr learned my addr 172.21.15.67:0/1737189930 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f00692ab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c0e2b70 0x7f005c035610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f0068aaa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f005c0060a0 0x7f005c0db9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f005c0e7670 con 0x7f005c0e2b70 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f005c0e4100 con 0x7f005c0060a0 2024-04-25T15:50:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f005c0e4620 con 0x7f005c0064b0 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8cac08f880 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cb8ff9640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8ca8077680 0x7f8ca8079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ca807a1a0 con 0x7f8ca8077680 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f8cac08fd10 con 0x7f8cbc1e4f50 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cbbfff640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8ca806f430 0x7f8ca8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.222+0000 7f8cbb7fe640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8ca8077680 0x7f8ca8079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f8cbbfff640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8ca806f430 0x7f8ca8071940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40ad000 tx=0x7f8cb4004ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f8cbb7fe640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8ca8077680 0x7f8ca8079ad0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00692ab640 1 -- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f005c0064b0 msgr2=0x7f005c04f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00692ab640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f005c0064b0 0x7f005c04f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00692ab640 1 -- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f005c0060a0 msgr2=0x7f005c0db9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00692ab640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f005c0060a0 0x7f005c0db9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00692ab640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c0d3520 con 0x7f005c0e2b70 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00692ab640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c0e2b70 0x7f005c035610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f006406c430 tx=0x7f00640771f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f0068aaa640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f005c0060a0 0x7f005c0db9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0064092070 con 0x7f005c0e2b70 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f006406b040 con 0x7f005c0e2b70 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00640751c0 con 0x7f005c0e2b70 2024-04-25T15:50:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005c0d0d40 con 0x7f005c0e2b70 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feebc002de0 con 0x7feec8077680 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feec3fff640 1 -- 172.21.15.67:0/3560485695 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feebc002de0 con 0x7feec8077680 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7feec8077680 msgr2=0x7feec8079ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7feec8077680 0x7feec8079ad0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feec806f430 msgr2=0x7feec8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feec806f430 0x7feec8071940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7feed40ae000 tx=0x7feed40a7070 comp rx=0 tx=0).stop 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc1610f0 msgr2=0x7feedc15fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 --2- 172.21.15.67:0/3560485695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feedc1610f0 0x7feedc15fb50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feed000b440 tx=0x7feed000b900 comp rx=0 tx=0).stop 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feedbfff640 1 -- 172.21.15.67:0/3560485695 reap_dead start 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 shutdown_connections 2024-04-25T15:50:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 >> 172.21.15.67:0/3560485695 conn(0x7feedc0ca9d0 msgr2=0x7feedc0c8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 shutdown_connections 2024-04-25T15:50:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7feee1f08640 1 -- 172.21.15.67:0/3560485695 wait complete. 2024-04-25T15:50:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.226+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ca807a1a0 con 0x7f8ca8077680 2024-04-25T15:50:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efec80ca240 con 0x7efeac0777d0 2024-04-25T15:50:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005c0d3aa0 con 0x7f005c0e2b70 2024-04-25T15:50:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.230+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f005c005580 con 0x7f005c0e2b70 2024-04-25T15:50:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efec57fa640 1 -- 172.21.15.67:0/2684854125 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efec80ca240 con 0x7efeac0777d0 2024-04-25T15:50:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7efeac0777d0 msgr2=0x7efeac079c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7efeac0777d0 0x7efeac079c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeac06f500 msgr2=0x7efeac071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeac06f500 0x7efeac071a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efec81e6d10 tx=0x7efeb4008040 comp rx=0 tx=0).stop 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 msgr2=0x7efec80c7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 --2- 172.21.15.67:0/2684854125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb80051b0 0x7efec80c7d90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efebc00e580 tx=0x7efebc00ea40 comp rx=0 tx=0).stop 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efecd04b640 1 -- 172.21.15.67:0/2684854125 reap_dead start 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 shutdown_connections 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 >> 172.21.15.67:0/2684854125 conn(0x7efec80cde80 msgr2=0x7efec80ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0064075360 con 0x7f005c0e2b70 2024-04-25T15:50:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7f00617fa640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f004806f400 0x7f0048071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 shutdown_connections 2024-04-25T15:50:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7efeceaae640 1 -- 172.21.15.67:0/2684854125 wait complete. 2024-04-25T15:50:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.234+0000 7f0068aaa640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f004806f400 0x7f0048071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f006410b240 con 0x7f005c0e2b70 2024-04-25T15:50:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f00617fa640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0048077650 0x7f0048079aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0064096e50 con 0x7f0048077650 2024-04-25T15:50:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f0064110060 con 0x7f005c0e2b70 2024-04-25T15:50:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f0068aaa640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f004806f400 0x7f0048071910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f00580b2000 tx=0x7f00580aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f0063fff640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0048077650 0x7f0048079aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff44c1f7e20 con 0x7ff438077650 2024-04-25T15:50:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff4337fe640 1 -- 172.21.15.67:0/1186898224 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff44c1f7e20 con 0x7ff438077650 2024-04-25T15:50:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f0063fff640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0048077650 0x7f0048079aa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff438077650 msgr2=0x7ff438079aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff438077650 0x7ff438079aa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff43806f400 msgr2=0x7ff438071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff43806f400 0x7ff438071910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0b2000 tx=0x7ff43c0aa070 comp rx=0 tx=0).stop 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c161440 msgr2=0x7ff44c1508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 --2- 172.21.15.67:0/1186898224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c161440 0x7ff44c1508c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff44400b990 tx=0x7ff44400be50 comp rx=0 tx=0).stop 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff44b59e640 1 -- 172.21.15.67:0/1186898224 reap_dead start 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 shutdown_connections 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 >> 172.21.15.67:0/1186898224 conn(0x7ff44c0ca050 msgr2=0x7ff44c0cb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 shutdown_connections 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7ff45105f640 1 -- 172.21.15.67:0/1186898224 wait complete. 2024-04-25T15:50:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.238+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0064096e50 con 0x7f0048077650 2024-04-25T15:50:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb2bc002d60 con 0x7fb2b80777a0 2024-04-25T15:50:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2caffd640 1 -- 172.21.15.67:0/1750539756 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb2bc002d60 con 0x7fb2b80777a0 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb2b80777a0 msgr2=0x7fb2b8079bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb2b80777a0 0x7fb2b8079bf0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2b806f4d0 msgr2=0x7fb2b80719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2b806f4d0 0x7fb2b80719e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2c40b4000 tx=0x7fb2c40ac000 comp rx=0 tx=0).stop 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 msgr2=0x7fb2cc182fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 --2- 172.21.15.67:0/1750539756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2cc046ad0 0x7fb2cc182fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000d5f0 tx=0x7fb2d000dab0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2dabff640 1 -- 172.21.15.67:0/1750539756 reap_dead start 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 shutdown_connections 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 >> 172.21.15.67:0/1750539756 conn(0x7fb2cc050c50 msgr2=0x7fb2cc035680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 shutdown_connections 2024-04-25T15:50:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.250+0000 7fb2db400640 1 -- 172.21.15.67:0/1750539756 wait complete. 2024-04-25T15:50:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1780002d60 con 0x7f1794077720 2024-04-25T15:50:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f17a57fa640 1 -- 172.21.15.67:0/181328738 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1780002d60 con 0x7f1794077720 2024-04-25T15:50:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1794077720 msgr2=0x7f1794079b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1794077720 0x7f1794079b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f179406f450 msgr2=0x7f1794071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f179406f450 0x7f1794071960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f17a0006fd0 tx=0x7f17a0008040 comp rx=0 tx=0).stop 2024-04-25T15:50:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a80c9240 msgr2=0x7f17a8089df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 --2- 172.21.15.67:0/181328738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17a80c9240 0x7f17a8089df0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f17a80cb3f0 tx=0x7f179c00c760 comp rx=0 tx=0).stop 2024-04-25T15:50:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f17ad5e8640 1 -- 172.21.15.67:0/181328738 reap_dead start 2024-04-25T15:50:39.283 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:39.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:50:39.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 shutdown_connections 2024-04-25T15:50:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 >> 172.21.15.67:0/181328738 conn(0x7f17a80c9fc0 msgr2=0x7f17a80733e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 shutdown_connections 2024-04-25T15:50:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.254+0000 7f178effd640 1 -- 172.21.15.67:0/181328738 wait complete. 2024-04-25T15:50:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8cbc0c6b40 con 0x7f8ca8077680 2024-04-25T15:50:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cb8ff9640 1 -- 172.21.15.67:0/2992060309 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8cbc0c6b40 con 0x7f8ca8077680 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8ca8077680 msgr2=0x7f8ca8079ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8ca8077680 0x7f8ca8079ad0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8ca806f430 msgr2=0x7f8ca8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8ca806f430 0x7f8ca8071940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40ad000 tx=0x7f8cb4004ac0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cbc1e4f50 msgr2=0x7f8cbc14be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 --2- 172.21.15.67:0/2992060309 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cbc1e4f50 0x7f8cbc14be30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc071980 tx=0x7f8cac00f910 comp rx=0 tx=0).stop 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc0da4640 1 -- 172.21.15.67:0/2992060309 reap_dead start 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 shutdown_connections 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 >> 172.21.15.67:0/2992060309 conn(0x7f8cbc0c9fc0 msgr2=0x7f8cbc06eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 shutdown_connections 2024-04-25T15:50:39.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.262+0000 7f8cc15a5640 1 -- 172.21.15.67:0/2992060309 wait complete. 2024-04-25T15:50:39.286 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:39.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:50:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f005c0d0aa0 con 0x7f0048077650 2024-04-25T15:50:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f00617fa640 1 -- 172.21.15.67:0/1737189930 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f005c0d0aa0 con 0x7f0048077650 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0048077650 msgr2=0x7f0048079aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0048077650 0x7f0048079aa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f004806f400 msgr2=0x7f0048071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f004806f400 0x7f0048071910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f00580b2000 tx=0x7f00580aa070 comp rx=0 tx=0).stop 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c0e2b70 msgr2=0x7f005c035610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 --2- 172.21.15.67:0/1737189930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f005c0e2b70 0x7f005c035610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f006406c430 tx=0x7f00640771f0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f00692ab640 1 -- 172.21.15.67:0/1737189930 reap_dead start 2024-04-25T15:50:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 shutdown_connections 2024-04-25T15:50:39.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 >> 172.21.15.67:0/1737189930 conn(0x7f005c0503c0 msgr2=0x7f005c04bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 shutdown_connections 2024-04-25T15:50:39.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.266+0000 7f006ad0e640 1 -- 172.21.15.67:0/1737189930 wait complete. 2024-04-25T15:50:39.289 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:39.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:50:39.290 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:39.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:50:39.292 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:39.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:50:39.293 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:39.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:50:39.295 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:50:39.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:50:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fd5257640 1 Processor -- start 2024-04-25T15:50:39.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fd5257640 1 -- start start 2024-04-25T15:50:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fd5257640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 0x7f0fd00ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fd5257640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0fd0164300 0x7f0fd0164710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fd5257640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd0074720 con 0x7f0fd00733f0 2024-04-25T15:50:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fd5257640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd006d050 con 0x7f0fd0164300 2024-04-25T15:50:39.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fd5257640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd0074990 con 0x7f0fd00c99e0 2024-04-25T15:50:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fcf59e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0fd0164300 0x7f0fd0164710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fced9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f0fd00733f0 0x7f0fd00c92d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:48904/0 (socket says 172.21.15.67:48904) 2024-04-25T15:50:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fced9d640 1 -- 172.21.15.67:0/2407739143 learned_addr learned my addr 172.21.15.67:0/2407739143 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:39.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fcf59e640 1 -- 172.21.15.67:0/2407739143 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 msgr2=0x7f0fd00ca780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:50:39.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fcf59e640 1 --2- 172.21.15.67:0/2407739143 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 0x7f0fd00ca780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fcf59e640 1 -- 172.21.15.67:0/2407739143 >> v1:172.21.15.176:6789/0 conn(0x7f0fd00733f0 legacy=0x7f0fd00c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.434+0000 7f0fcf59e640 1 -- 172.21.15.67:0/2407739143 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd006d2a0 con 0x7f0fd0164300 2024-04-25T15:50:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fcf59e640 1 --2- 172.21.15.67:0/2407739143 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 0x7f0fd0164710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009d90 tx=0x7f0fc801c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebc11badf7094b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fcdd9b640 1 -- 172.21.15.67:0/2407739143 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc80b2070 con 0x7f0fd0164300 2024-04-25T15:50:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fcdd9b640 1 -- 172.21.15.67:0/2407739143 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc80a5940 con 0x7f0fd0164300 2024-04-25T15:50:39.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fcdd9b640 1 -- 172.21.15.67:0/2407739143 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc80a5d10 con 0x7f0fd0164300 2024-04-25T15:50:39.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 -- 172.21.15.67:0/2407739143 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 msgr2=0x7f0fd0164710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2407739143 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 0x7f0fd0164710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009d90 tx=0x7f0fc801c990 comp rx=0 tx=0).stop 2024-04-25T15:50:39.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 -- 172.21.15.67:0/2407739143 shutdown_connections 2024-04-25T15:50:39.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2407739143 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 0x7f0fd0164710 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2407739143 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 0x7f0fd00ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 -- 172.21.15.67:0/2407739143 >> 172.21.15.67:0/2407739143 conn(0x7f0fd0070db0 msgr2=0x7f0fd00ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 -- 172.21.15.67:0/2407739143 shutdown_connections 2024-04-25T15:50:39.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.438+0000 7f0fd5257640 1 -- 172.21.15.67:0/2407739143 wait complete. 2024-04-25T15:50:39.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 Processor -- start 2024-04-25T15:50:39.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 -- start start 2024-04-25T15:50:39.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd00733f0 0x7f0fd01fdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fced9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd00733f0 0x7f0fd01fdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fced9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd00733f0 0x7f0fd01fdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59760/0 (socket says 172.21.15.67:59760) 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fced9d640 1 -- 172.21.15.67:0/2903515929 learned_addr learned my addr 172.21.15.67:0/2903515929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 0x7f0fd014eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 0x7f0fd014c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd016de50 con 0x7f0fd00733f0 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd0165270 con 0x7f0fd0164300 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd016dfd0 con 0x7f0fd00c99e0 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.442+0000 7f0fce59c640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 0x7f0fd014eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.446+0000 7f0fcf59e640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 0x7f0fd014c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.446+0000 7f0fced9d640 1 -- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 msgr2=0x7f0fd014eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.446+0000 7f0fced9d640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 0x7f0fd014eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.446+0000 7f0fced9d640 1 -- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 msgr2=0x7f0fd014c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.446+0000 7f0fced9d640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 0x7f0fd014c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.446+0000 7f0fced9d640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd00c78b0 con 0x7f0fd00733f0 2024-04-25T15:50:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.446+0000 7f0fced9d640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd00733f0 0x7f0fd01fdc90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c3a0 tx=0x7f0fc000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.450+0000 7f0fc77fe640 1 -- 172.21.15.67:0/2903515929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0017070 con 0x7f0fd00733f0 2024-04-25T15:50:39.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.450+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd015d730 con 0x7f0fd00733f0 2024-04-25T15:50:39.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.450+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd015d110 con 0x7f0fd00733f0 2024-04-25T15:50:39.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.450+0000 7f0fc77fe640 1 -- 172.21.15.67:0/2903515929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc0005cf0 con 0x7f0fd00733f0 2024-04-25T15:50:39.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.454+0000 7f0fc77fe640 1 -- 172.21.15.67:0/2903515929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00072f0 con 0x7f0fd00733f0 2024-04-25T15:50:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.454+0000 7f0fc77fe640 1 -- 172.21.15.67:0/2903515929 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0fc001a070 con 0x7f0fd00733f0 2024-04-25T15:50:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.454+0000 7f0fc77fe640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fb006f4d0 0x7f0fb00719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.454+0000 7f0fc77fe640 1 -- 172.21.15.67:0/2903515929 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0fc008e540 con 0x7f0fd00733f0 2024-04-25T15:50:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.454+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb40052a0 con 0x7f0fd00733f0 2024-04-25T15:50:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.454+0000 7f0fce59c640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fb006f4d0 0x7f0fb00719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.458+0000 7f0fce59c640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fb006f4d0 0x7f0fb00719e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00c8860 tx=0x7f0fb8009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.466+0000 7f0fc77fe640 1 -- 172.21.15.67:0/2903515929 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0fc005ae50 con 0x7f0fd00733f0 2024-04-25T15:50:39.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.474+0000 7fde5b59e640 1 Processor -- start 2024-04-25T15:50:39.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.474+0000 7fde5b59e640 1 -- start start 2024-04-25T15:50:39.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.474+0000 7fee489ea640 1 Processor -- start 2024-04-25T15:50:39.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.474+0000 7fee489ea640 1 -- start start 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5b59e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fde540d6220 0x7fde540df560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5b59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540092d0 0x7fde54006cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5b59e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde54046890 con 0x7fde540dfaa0 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5b59e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fde540e0470 con 0x7fde540d6220 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5b59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde540e0190 con 0x7fde540092d0 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde59d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fde540dfaa0 0x7fde54008bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36052/0 (socket says 172.21.15.67:36052) 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde59d9b640 1 -- 172.21.15.67:0/558593880 learned_addr learned my addr 172.21.15.67:0/558593880 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5a59c640 1 --2- 172.21.15.67:0/558593880 >> v2:172.21.15.176:3300/0 conn(0x7fde540d6220 0x7fde540df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5ad9d640 1 --2- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540092d0 0x7fde54006cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5959a640 1 -- 172.21.15.67:0/558593880 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4207603018 0 0) 0x7fde54046890 con 0x7fde540dfaa0 2024-04-25T15:50:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5959a640 1 -- 172.21.15.67:0/558593880 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde44003530 con 0x7fde540dfaa0 2024-04-25T15:50:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5a59c640 1 -- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540092d0 msgr2=0x7fde54006cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee4259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee44071bc0 0x7fee440cbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36054/0 (socket says 172.21.15.67:36054) 2024-04-25T15:50:39.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee4259c640 1 -- 172.21.15.67:0/2510097313 learned_addr learned my addr 172.21.15.67:0/2510097313 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:39.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee489ea640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4415a720 0x7fee4415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4415de10 con 0x7fee44071bc0 2024-04-25T15:50:39.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee44072390 con 0x7fee440ca540 2024-04-25T15:50:39.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee4415ebb0 con 0x7fee4415a720 2024-04-25T15:50:39.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805757384 0 0) 0x7fee4415de10 con 0x7fee44071bc0 2024-04-25T15:50:39.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee2c003530 con 0x7fee44071bc0 2024-04-25T15:50:39.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7f0235c17640 1 Processor -- start 2024-04-25T15:50:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee41d9b640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4415a720 0x7fee4415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460204026 0 0) 0x7fee44072390 con 0x7fee440ca540 2024-04-25T15:50:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee4415de10 con 0x7fee440ca540 2024-04-25T15:50:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3242687618 0 0) 0x7fee2c003530 con 0x7fee44071bc0 2024-04-25T15:50:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4415a720 msgr2=0x7fee4415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4415a720 0x7fee4415ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 >> v1:172.21.15.176:6789/0 conn(0x7fee440ca540 legacy=0x7fee44164fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4415e0d0 con 0x7fee44071bc0 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fee38001680 con 0x7fee44071bc0 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 >> v1:172.21.15.67:6789/0 conn(0x7fee44071bc0 legacy=0x7fee440cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee2c003b80 0x7fee2c003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee2c004810 0x7fee2c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c005310 0x7fee2c0145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fee4415ebb0 con 0x7fee2c003b80 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c014b10 con 0x7fee2c005310 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee2c014c90 con 0x7fee2c004810 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee41d9b640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4415a720 0x7fee4415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4259c640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee2c003b80 0x7fee2c003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee42d9d640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c005310 0x7fee2c0145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5a59c640 1 --2- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540092d0 0x7fde54006cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5a59c640 1 -- 172.21.15.67:0/558593880 >> v1:172.21.15.67:6789/0 conn(0x7fde540dfaa0 legacy=0x7fde54008bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7fde5a59c640 1 -- 172.21.15.67:0/558593880 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5404cd40 con 0x7fde540d6220 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4259c640 1 -- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee2c004810 msgr2=0x7fee2c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4259c640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee2c004810 0x7fee2c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4259c640 1 -- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c005310 msgr2=0x7fee2c0145d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4259c640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c005310 0x7fee2c0145d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fee4259c640 1 -- 172.21.15.67:0/2510097313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c003530 con 0x7fee2c003b80 2024-04-25T15:50:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5a59c640 1 --2- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540d6220 0x7fde540df560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fde500097c0 tx=0x7fde5001c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c993062a80aae70a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5959a640 1 -- 172.21.15.67:0/558593880 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500b0070 con 0x7fde540d6220 2024-04-25T15:50:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5959a640 1 -- 172.21.15.67:0/558593880 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde500ab400 con 0x7fde540d6220 2024-04-25T15:50:39.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5959a640 1 -- 172.21.15.67:0/558593880 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500ab7b0 con 0x7fde540d6220 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.478+0000 7f0235c17640 1 -- start start 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f0235c17640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f02300c6c10 0x7f0230156c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f0235c17640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f02300d2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f0235c17640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023015cc60 con 0x7f02300d2540 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f0235c17640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f023015cf40 con 0x7f02300c6c10 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f0235c17640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f023015d220 con 0x7f02300c9bc0 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022effd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f02300d2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022effd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f02300d2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60636/0 (socket says 172.21.15.67:60636) 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022effd640 1 -- 172.21.15.67:0/3201607828 learned_addr learned my addr 172.21.15.67:0/3201607828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022f7fe640 1 --2- 172.21.15.67:0/3201607828 >> v2:172.21.15.176:3300/0 conn(0x7f02300c6c10 0x7f0230156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5b59e640 1 -- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540d6220 msgr2=0x7fde540df560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5b59e640 1 --2- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540d6220 0x7fde540df560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fde500097c0 tx=0x7fde5001c2e0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022effd640 1 -- 172.21.15.67:0/3201607828 >> v2:172.21.15.176:3300/0 conn(0x7f02300c6c10 msgr2=0x7f0230156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022effd640 1 --2- 172.21.15.67:0/3201607828 >> v2:172.21.15.176:3300/0 conn(0x7f02300c6c10 0x7f0230156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022effd640 1 -- 172.21.15.67:0/3201607828 >> v1:172.21.15.176:6789/0 conn(0x7f02300d2540 legacy=0x7f02300c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7f022effd640 1 -- 172.21.15.67:0/3201607828 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023015d500 con 0x7f02300c9bc0 2024-04-25T15:50:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5b59e640 1 -- 172.21.15.67:0/558593880 shutdown_connections 2024-04-25T15:50:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5b59e640 1 --2- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540092d0 0x7fde54006cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5b59e640 1 --2- 172.21.15.67:0/558593880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540d6220 0x7fde540df560 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.482+0000 7fde5b59e640 1 -- 172.21.15.67:0/558593880 >> 172.21.15.67:0/558593880 conn(0x7fde540514d0 msgr2=0x7fde540518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fde5b59e640 1 -- 172.21.15.67:0/558593880 shutdown_connections 2024-04-25T15:50:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee4259c640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee2c003b80 0x7fee2c003fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c0e0 tx=0x7fee3801cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b6811598d1a049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38007bf0 con 0x7fee2c003b80 2024-04-25T15:50:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee380a9070 con 0x7fee2c003b80 2024-04-25T15:50:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee4159a640 1 -- 172.21.15.67:0/2510097313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38005d40 con 0x7fee2c003b80 2024-04-25T15:50:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee2c003b80 msgr2=0x7fee2c003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee489ea640 1 --2- 172.21.15.67:0/2510097313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee2c003b80 0x7fee2c003fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c0e0 tx=0x7fee3801cb40 comp rx=0 tx=0).stop 2024-04-25T15:50:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee42d9d640 1 -- 172.21.15.67:0/2510097313 reap_dead start 2024-04-25T15:50:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 shutdown_connections 2024-04-25T15:50:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 >> 172.21.15.67:0/2510097313 conn(0x7fee440700b0 msgr2=0x7fee440704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7f022effd640 1 --2- 172.21.15.67:0/3201607828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f02300d2000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0224009870 tx=0x7f022401ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ff7a9b5396fb3e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 shutdown_connections 2024-04-25T15:50:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fee489ea640 1 -- 172.21.15.67:0/2510097313 wait complete. 2024-04-25T15:50:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fde5b59e640 1 -- 172.21.15.67:0/558593880 wait complete. 2024-04-25T15:50:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fde5b59e640 1 Processor -- start 2024-04-25T15:50:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7f022e7fc640 1 -- 172.21.15.67:0/3201607828 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02240b5070 con 0x7f02300c9bc0 2024-04-25T15:50:39.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7fde5b59e640 1 -- start start 2024-04-25T15:50:39.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7f022e7fc640 1 -- 172.21.15.67:0/3201607828 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02240a8a50 con 0x7f02300c9bc0 2024-04-25T15:50:39.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.486+0000 7f022e7fc640 1 -- 172.21.15.67:0/3201607828 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02240a8e20 con 0x7f02300c9bc0 2024-04-25T15:50:39.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 -- 172.21.15.67:0/3201607828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 msgr2=0x7f02300d2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 --2- 172.21.15.67:0/3201607828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f02300d2000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0224009870 tx=0x7f022401ed50 comp rx=0 tx=0).stop 2024-04-25T15:50:39.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde540092d0 0x7fde540d6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540d6220 0x7fde540c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde540092d0 0x7fde540d6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde540092d0 0x7fde540d6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59788/0 (socket says 172.21.15.67:59788) 2024-04-25T15:50:39.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fee489ea640 1 Processor -- start 2024-04-25T15:50:39.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 -- 172.21.15.67:0/2351707669 learned_addr learned my addr 172.21.15.67:0/2351707669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540dfaa0 0x7fde540c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fee489ea640 1 -- start start 2024-04-25T15:50:39.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 -- 172.21.15.67:0/3201607828 shutdown_connections 2024-04-25T15:50:39.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 --2- 172.21.15.67:0/3201607828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f02300d2000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 --2- 172.21.15.67:0/3201607828 >> v2:172.21.15.176:3300/0 conn(0x7f02300c6c10 0x7f0230156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 -- 172.21.15.67:0/3201607828 >> 172.21.15.67:0/3201607828 conn(0x7f0230068760 msgr2=0x7f0230068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde540465f0 con 0x7fde540092d0 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fde54046470 con 0x7fde540dfaa0 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde59d9b640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540d6220 0x7fde540c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde54034060 con 0x7fde540d6220 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 -- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540d6220 msgr2=0x7fde540c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540d6220 0x7fde540c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 -- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540dfaa0 msgr2=0x7fde540c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540dfaa0 0x7fde540c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde54185590 con 0x7fde540092d0 2024-04-25T15:50:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5a59c640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde540092d0 0x7fde540d6860 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde50000c10 tx=0x7fde5001ab20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 -- 172.21.15.67:0/3201607828 shutdown_connections 2024-04-25T15:50:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7f0235c17640 1 -- 172.21.15.67:0/3201607828 wait complete. 2024-04-25T15:50:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde4affd640 1 -- 172.21.15.67:0/2351707669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500b0050 con 0x7fde540092d0 2024-04-25T15:50:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde4affd640 1 -- 172.21.15.67:0/2351707669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde500bc070 con 0x7fde540092d0 2024-04-25T15:50:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde541852f0 con 0x7fde540092d0 2024-04-25T15:50:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.490+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde540d0b20 con 0x7fde540092d0 2024-04-25T15:50:39.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c003b80 0x7fee4415cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4414fe00 0x7fee4414b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4414e400 0x7fee4414ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c003b80 0x7fee4415cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c003b80 0x7fee4415cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39986/0 (socket says 172.21.15.67:39986) 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 -- 172.21.15.67:0/2902920414 learned_addr learned my addr 172.21.15.67:0/2902920414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fee440cad60 con 0x7fee4414fe00 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fee4415e720 con 0x7fee2c003b80 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee41d9b640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4414fe00 0x7fee4414b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee42d9d640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4414e400 0x7fee4414ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee440cc2d0 con 0x7fee4414e400 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fde4affd640 1 -- 172.21.15.67:0/2351707669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500b7870 con 0x7fde540092d0 2024-04-25T15:50:39.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7f0235c17640 1 Processor -- start 2024-04-25T15:50:39.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 -- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4414e400 msgr2=0x7fee4414ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4414e400 0x7fee4414ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 -- 172.21.15.67:0/2902920414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4414fe00 msgr2=0x7fee4414b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4414fe00 0x7fee4414b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee44150af0 con 0x7fee2c003b80 2024-04-25T15:50:39.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7f0235c17640 1 -- start start 2024-04-25T15:50:39.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee4259c640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c003b80 0x7fee4415cde0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c0e0 tx=0x7fee2c005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fde4affd640 1 -- 172.21.15.67:0/2351707669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde500b7a10 con 0x7fde540092d0 2024-04-25T15:50:39.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee41d9b640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4414fe00 0x7fee4414b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:39.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee42d9d640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4414e400 0x7fee4414ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:39.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde540059d0 con 0x7fde540092d0 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee32ffd640 1 -- 172.21.15.67:0/2902920414 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38005d20 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee32ffd640 1 -- 172.21.15.67:0/2902920414 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee38001480 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee32ffd640 1 -- 172.21.15.67:0/2902920414 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38007240 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4414e140 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee44150dd0 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee280052a0 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.502+0000 7fee32ffd640 1 -- 172.21.15.67:0/2902920414 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fee380073e0 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.502+0000 7fee32ffd640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee2406f480 0x7fee24071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.502+0000 7fee32ffd640 1 -- 172.21.15.67:0/2902920414 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fee381289a0 con 0x7fee2c003b80 2024-04-25T15:50:39.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.502+0000 7fee41d9b640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee2406f480 0x7fee24071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.502+0000 7fee41d9b640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee2406f480 0x7fee24071990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee4414c260 tx=0x7fee34008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fde4affd640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde3c06f430 0x7fde3c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.494+0000 7fde4affd640 1 -- 172.21.15.67:0/2351707669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fde50129660 con 0x7fde540092d0 2024-04-25T15:50:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7fde59d9b640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde3c06f430 0x7fde3c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f0235c17640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02300c6c10 0x7f023015ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02300c6c10 0x7f023015ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02300c6c10 0x7f023015ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59806/0 (socket says 172.21.15.67:59806) 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 -- 172.21.15.67:0/705223688 learned_addr learned my addr 172.21.15.67:0/705223688 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f023015b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02300d2540 0x7f0230148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f023015c8b0 con 0x7f02300c6c10 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02300c7ce0 con 0x7f02300d2540 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f02301e7290 con 0x7f02300c9bc0 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022effd640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f023015b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 -- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 msgr2=0x7f023015b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f023015b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 -- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02300d2540 msgr2=0x7f0230148540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02300d2540 0x7f0230148540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02301e8580 con 0x7f02300c6c10 2024-04-25T15:50:39.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.498+0000 7f022f7fe640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02300c6c10 0x7f023015ab90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f022000e8a0 tx=0x7f022000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.506+0000 7fde59d9b640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde3c06f430 0x7fde3c071940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fde4c007fd0 tx=0x7fde4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.506+0000 7fde4affd640 1 -- 172.21.15.67:0/2351707669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fde50015ff0 con 0x7fde540092d0 2024-04-25T15:50:39.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.506+0000 7f0217fff640 1 -- 172.21.15.67:0/705223688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220017070 con 0x7f02300c6c10 2024-04-25T15:50:39.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.506+0000 7f0217fff640 1 -- 172.21.15.67:0/705223688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f022000c8e0 con 0x7f02300c6c10 2024-04-25T15:50:39.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.506+0000 7f0217fff640 1 -- 172.21.15.67:0/705223688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022000cc90 con 0x7f02300c6c10 2024-04-25T15:50:39.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.506+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02301e74e0 con 0x7f02300c6c10 2024-04-25T15:50:39.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.506+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02301e7aa0 con 0x7f02300c6c10 2024-04-25T15:50:39.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.510+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023006a480 con 0x7f02300c6c10 2024-04-25T15:50:39.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.510+0000 7f0217fff640 1 -- 172.21.15.67:0/705223688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f022001a070 con 0x7f02300c6c10 2024-04-25T15:50:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.510+0000 7f0217fff640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f021c06f400 0x7f021c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.510+0000 7f0217fff640 1 -- 172.21.15.67:0/705223688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f022008f7e0 con 0x7f02300c6c10 2024-04-25T15:50:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.510+0000 7f022effd640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f021c06f400 0x7f021c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.514+0000 7f022effd640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f021c06f400 0x7f021c071910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f02240b2000 tx=0x7f0224009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.514+0000 7fee32ffd640 1 -- 172.21.15.67:0/2902920414 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fee38016850 con 0x7fee2c003b80 2024-04-25T15:50:39.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.518+0000 7f0217fff640 1 -- 172.21.15.67:0/705223688 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f022005c180 con 0x7f02300c6c10 2024-04-25T15:50:39.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.530+0000 7f75f8878640 1 Processor -- start 2024-04-25T15:50:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.530+0000 7f75f8878640 1 -- start start 2024-04-25T15:50:39.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f9609e47640 1 Processor -- start 2024-04-25T15:50:39.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f9609e47640 1 -- start start 2024-04-25T15:50:39.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f75f8878640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f40caf60 0x7f75f415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f75f8878640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f75f4073bf0 0x7f75f4163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f75f8878640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f40c77a0 con 0x7f75f415d790 2024-04-25T15:50:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f75f8878640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f75f40cab60 con 0x7f75f40caf60 2024-04-25T15:50:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f75f8878640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f75f4159620 con 0x7f75f4073bf0 2024-04-25T15:50:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f75f1d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75f415d790 0x7f75f406d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36066/0 (socket says 172.21.15.67:36066) 2024-04-25T15:50:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f75f1d9b640 1 -- 172.21.15.67:0/3171483460 learned_addr learned my addr 172.21.15.67:0/3171483460 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f9609e47640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c7610 0x7f96040c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f9609e47640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f96040c5bb0 0x7f96040c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f9609e47640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96040c6890 con 0x7f9604071b20 2024-04-25T15:50:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f9609e47640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f960415cd40 con 0x7f96040c5bb0 2024-04-25T15:50:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.534+0000 7f9609e47640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96040c6b70 con 0x7f96040c7610 2024-04-25T15:50:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f75f2d9d640 1 --2- 172.21.15.67:0/3171483460 >> v2:172.21.15.176:3300/0 conn(0x7f75f4073bf0 0x7f75f4163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f96037fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9604071b20 0x7f960415d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:48928/0 (socket says 172.21.15.67:48928) 2024-04-25T15:50:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f96037fe640 1 -- 172.21.15.67:0/2530062112 learned_addr learned my addr 172.21.15.67:0/2530062112 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f9603fff640 1 --2- 172.21.15.67:0/2530062112 >> v2:172.21.15.176:3300/0 conn(0x7f96040c5bb0 0x7f96040c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f96027fc640 1 -- 172.21.15.67:0/2530062112 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 453077306 0 0) 0x7f96040c6890 con 0x7f9604071b20 2024-04-25T15:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f96027fc640 1 -- 172.21.15.67:0/2530062112 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95ec003530 con 0x7f9604071b20 2024-04-25T15:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f9603fff640 1 -- 172.21.15.67:0/2530062112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c7610 msgr2=0x7f96040c7a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f9603fff640 1 --2- 172.21.15.67:0/2530062112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c7610 0x7f96040c7a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f9603fff640 1 -- 172.21.15.67:0/2530062112 >> v1:172.21.15.176:6789/0 conn(0x7f9604071b20 legacy=0x7f960415d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f9603fff640 1 -- 172.21.15.67:0/2530062112 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96040c6e50 con 0x7f96040c5bb0 2024-04-25T15:50:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f75f2d9d640 1 -- 172.21.15.67:0/3171483460 >> v2:172.21.15.67:3300/0 conn(0x7f75f40caf60 msgr2=0x7f75f415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f75f2d9d640 1 --2- 172.21.15.67:0/3171483460 >> v2:172.21.15.67:3300/0 conn(0x7f75f40caf60 0x7f75f415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f75f2d9d640 1 -- 172.21.15.67:0/3171483460 >> v1:172.21.15.67:6789/0 conn(0x7f75f415d790 legacy=0x7f75f406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.538+0000 7f75f2d9d640 1 -- 172.21.15.67:0/3171483460 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f40c7a60 con 0x7f75f4073bf0 2024-04-25T15:50:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f2d9d640 1 --2- 172.21.15.67:0/3171483460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f4073bf0 0x7f75f4163ec0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75f40c61d0 tx=0x7f75ec01eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36ee586dd8269571 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9603fff640 1 --2- 172.21.15.67:0/2530062112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c5bb0 0x7f96040c5fc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009d90 tx=0x7f95fc01c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f60a31e7d67ff393 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f159a640 1 -- 172.21.15.67:0/3171483460 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0b4070 con 0x7f75f4073bf0 2024-04-25T15:50:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f159a640 1 -- 172.21.15.67:0/3171483460 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec0a7950 con 0x7f75f4073bf0 2024-04-25T15:50:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f159a640 1 -- 172.21.15.67:0/3171483460 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0a7d80 con 0x7f75f4073bf0 2024-04-25T15:50:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f96027fc640 1 -- 172.21.15.67:0/2530062112 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0b2070 con 0x7f96040c5bb0 2024-04-25T15:50:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f96027fc640 1 -- 172.21.15.67:0/2530062112 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95fc0a5960 con 0x7f96040c5bb0 2024-04-25T15:50:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f96027fc640 1 -- 172.21.15.67:0/2530062112 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0a5d30 con 0x7f96040c5bb0 2024-04-25T15:50:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 -- 172.21.15.67:0/3171483460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f4073bf0 msgr2=0x7f75f4163ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 --2- 172.21.15.67:0/3171483460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f4073bf0 0x7f75f4163ec0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f75f40c61d0 tx=0x7f75ec01eb80 comp rx=0 tx=0).stop 2024-04-25T15:50:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 -- 172.21.15.67:0/2530062112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c5bb0 msgr2=0x7f96040c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 --2- 172.21.15.67:0/2530062112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c5bb0 0x7f96040c5fc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009d90 tx=0x7f95fc01c990 comp rx=0 tx=0).stop 2024-04-25T15:50:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 -- 172.21.15.67:0/3171483460 shutdown_connections 2024-04-25T15:50:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 --2- 172.21.15.67:0/3171483460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f4073bf0 0x7f75f4163ec0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 --2- 172.21.15.67:0/3171483460 >> v2:172.21.15.67:3300/0 conn(0x7f75f40caf60 0x7f75f415d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 -- 172.21.15.67:0/2530062112 shutdown_connections 2024-04-25T15:50:39.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 --2- 172.21.15.67:0/2530062112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c5bb0 0x7f96040c5fc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 --2- 172.21.15.67:0/2530062112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c7610 0x7f96040c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 -- 172.21.15.67:0/2530062112 >> 172.21.15.67:0/2530062112 conn(0x7f960406fe70 msgr2=0x7f96040cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 -- 172.21.15.67:0/3171483460 >> 172.21.15.67:0/3171483460 conn(0x7f75f4070db0 msgr2=0x7f75f40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 -- 172.21.15.67:0/2530062112 shutdown_connections 2024-04-25T15:50:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 -- 172.21.15.67:0/3171483460 shutdown_connections 2024-04-25T15:50:39.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f75f8878640 1 -- 172.21.15.67:0/3171483460 wait complete. 2024-04-25T15:50:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.542+0000 7f9609e47640 1 -- 172.21.15.67:0/2530062112 wait complete. 2024-04-25T15:50:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.546+0000 7f9609e47640 1 Processor -- start 2024-04-25T15:50:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.546+0000 7f9609e47640 1 -- start start 2024-04-25T15:50:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.546+0000 7f75f8878640 1 Processor -- start 2024-04-25T15:50:39.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.546+0000 7f75f8878640 1 -- start start 2024-04-25T15:50:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f9609e47640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9604071b20 0x7f96042064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f8878640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f4073bf0 0x7f75f41569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f8878640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f40caf60 0x7f75f41f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f8878640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f415d790 0x7f75f40cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f8878640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75f415a570 con 0x7f75f4073bf0 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f8878640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f75f415a3f0 con 0x7f75f415d790 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f4073bf0 0x7f75f41569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f8878640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f75f406ce50 con 0x7f75f40caf60 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f4073bf0 0x7f75f41569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59830/0 (socket says 172.21.15.67:59830) 2024-04-25T15:50:39.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 -- 172.21.15.67:0/3306767532 learned_addr learned my addr 172.21.15.67:0/3306767532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9604071b20 0x7f96042064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9604071b20 0x7f96042064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59844/0 (socket says 172.21.15.67:59844) 2024-04-25T15:50:39.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f1d9b640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f40caf60 0x7f75f41f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 -- 172.21.15.67:0/2146512343 learned_addr learned my addr 172.21.15.67:0/2146512343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 -- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f40caf60 msgr2=0x7f75f41f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f40caf60 0x7f75f41f36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 -- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f415d790 msgr2=0x7f75f40cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f415d790 0x7f75f40cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 -- 172.21.15.67:0/3306767532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f41f3c00 con 0x7f75f4073bf0 2024-04-25T15:50:39.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c5bb0 0x7f9604152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f259c640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f4073bf0 0x7f75f41569d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b440 tx=0x7f75e800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c7610 0x7f960414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9604166580 con 0x7f9604071b20 2024-04-25T15:50:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f96040734f0 con 0x7f96040c7610 2024-04-25T15:50:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75e2ffd640 1 -- 172.21.15.67:0/3306767532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8015020 con 0x7f75f4073bf0 2024-04-25T15:50:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9604166700 con 0x7f96040c5bb0 2024-04-25T15:50:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75e2ffd640 1 -- 172.21.15.67:0/3306767532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e80118b0 con 0x7f75f4073bf0 2024-04-25T15:50:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75e2ffd640 1 -- 172.21.15.67:0/3306767532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8011ca0 con 0x7f75f4073bf0 2024-04-25T15:50:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f75f8878640 1 -- 172.21.15.67:0/3306767532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f414ee50 con 0x7f75f4073bf0 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d2e2f640 1 Processor -- start 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d2e2f640 1 -- start start 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f9603fff640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c7610 0x7f960414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 -- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c5bb0 msgr2=0x7f9604152910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c5bb0 0x7f9604152910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 -- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c7610 msgr2=0x7f960414d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c7610 0x7f960414d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9604206a00 con 0x7f9604071b20 2024-04-25T15:50:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.550+0000 7f96037fe640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9604071b20 0x7f96042064c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c430 tx=0x7f95f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f95ebfff640 1 -- 172.21.15.67:0/2146512343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4017070 con 0x7f9604071b20 2024-04-25T15:50:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f95ebfff640 1 -- 172.21.15.67:0/2146512343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f4005cf0 con 0x7f9604071b20 2024-04-25T15:50:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f95ebfff640 1 -- 172.21.15.67:0/2146512343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f40072d0 con 0x7f9604071b20 2024-04-25T15:50:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960415a2d0 con 0x7f9604071b20 2024-04-25T15:50:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f75e2ffd640 1 -- 172.21.15.67:0/3306767532 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f75e8018020 con 0x7f75f4073bf0 2024-04-25T15:50:39.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f75e2ffd640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75dc073960 0x7f75dc075e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d2e2f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68c40d6220 0x7f68c40df560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d2e2f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c4008bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f75f8878640 1 -- 172.21.15.67:0/3306767532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f414f410 con 0x7f75f4073bf0 2024-04-25T15:50:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d2e2f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68c4046890 con 0x7f68c4009100 2024-04-25T15:50:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d2e2f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f68c40e0470 con 0x7f68c40d6220 2024-04-25T15:50:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d162c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c4008bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d162c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c4008bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60696/0 (socket says 172.21.15.67:60696) 2024-04-25T15:50:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d162c640 1 -- 172.21.15.67:0/3226506078 learned_addr learned my addr 172.21.15.67:0/3226506078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d2e2f640 1 -- 172.21.15.67:0/3226506078 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f68c40e0190 con 0x7f68c40dfaa0 2024-04-25T15:50:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d1e2d640 1 --2- 172.21.15.67:0/3226506078 >> v2:172.21.15.67:3300/0 conn(0x7f68c40d6220 0x7f68c40df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f75f1d9b640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75dc073960 0x7f75dc075e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f75e2ffd640 1 -- 172.21.15.67:0/3306767532 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f75e808caa0 con 0x7f75f4073bf0 2024-04-25T15:50:39.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f95ebfff640 1 -- 172.21.15.67:0/2146512343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f95f401a070 con 0x7f9604071b20 2024-04-25T15:50:39.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f95ebfff640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f95f006f480 0x7f95f0071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9604158970 con 0x7f9604071b20 2024-04-25T15:50:39.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f9602ffd640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f95f006f480 0x7f95f0071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f75f1d9b640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75dc073960 0x7f75dc075e70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75e4000f70 tx=0x7f75e4007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f95ebfff640 1 -- 172.21.15.67:0/2146512343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f95f4056fe0 con 0x7f9604071b20 2024-04-25T15:50:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f9602ffd640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f95f006f480 0x7f95f0071990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006fd0 tx=0x7f95f8006b40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f960406a480 con 0x7f9604071b20 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d162c640 1 -- 172.21.15.67:0/3226506078 >> v2:172.21.15.67:3300/0 conn(0x7f68c40d6220 msgr2=0x7f68c40df560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d162c640 1 --2- 172.21.15.67:0/3226506078 >> v2:172.21.15.67:3300/0 conn(0x7f68c40d6220 0x7f68c40df560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d162c640 1 -- 172.21.15.67:0/3226506078 >> v1:172.21.15.67:6789/0 conn(0x7f68c4009100 legacy=0x7f68c40dbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.554+0000 7f68d162c640 1 -- 172.21.15.67:0/3226506078 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c404cd40 con 0x7f68c40dfaa0 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f68d162c640 1 --2- 172.21.15.67:0/3226506078 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c4008bc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009870 tx=0x7f68c801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a521f4b01c307d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f68d0e2b640 1 -- 172.21.15.67:0/3226506078 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80b0070 con 0x7f68c40dfaa0 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f68d0e2b640 1 -- 172.21.15.67:0/3226506078 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c80ab600 con 0x7f68c40dfaa0 2024-04-25T15:50:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.558+0000 7f68d0e2b640 1 -- 172.21.15.67:0/3226506078 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80ab9d0 con 0x7f68c40dfaa0 2024-04-25T15:50:39.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.562+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d80052a0 con 0x7f75f4073bf0 2024-04-25T15:50:39.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.562+0000 7f68d2e2f640 1 -- 172.21.15.67:0/3226506078 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 msgr2=0x7f68c4008bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.562+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/3226506078 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c4008bc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009870 tx=0x7f68c801c440 comp rx=0 tx=0).stop 2024-04-25T15:50:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 -- 172.21.15.67:0/3226506078 shutdown_connections 2024-04-25T15:50:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/3226506078 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c4008bc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/3226506078 >> v2:172.21.15.67:3300/0 conn(0x7f68c40d6220 0x7f68c40df560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 -- 172.21.15.67:0/3226506078 >> 172.21.15.67:0/3226506078 conn(0x7f68c40514d0 msgr2=0x7f68c40518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 -- 172.21.15.67:0/3226506078 shutdown_connections 2024-04-25T15:50:39.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 -- 172.21.15.67:0/3226506078 wait complete. 2024-04-25T15:50:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 Processor -- start 2024-04-25T15:50:39.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.566+0000 7f68d2e2f640 1 -- start start 2024-04-25T15:50:39.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f95ebfff640 1 -- 172.21.15.67:0/2146512343 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f95f405ad50 con 0x7f9604071b20 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d2e2f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68c4009100 0x7f68c40d9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d2e2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68c40d6220 0x7f68c40de810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d2e2f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c40db9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d2e2f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f68c404d5e0 con 0x7f68c40d6220 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d2e2f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f68c404d460 con 0x7f68c4009100 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d2e2f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f68c4046470 con 0x7f68c40dfaa0 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68c40d6220 0x7f68c40de810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68c40d6220 0x7f68c40de810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59854/0 (socket says 172.21.15.67:59854) 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 -- 172.21.15.67:0/161855388 learned_addr learned my addr 172.21.15.67:0/161855388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:39.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d262e640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c40db9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 -- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 msgr2=0x7f68c40db9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c40db9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 -- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68c4009100 msgr2=0x7f68c40d9de0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:50:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68c4009100 0x7f68c40d9de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 -- 172.21.15.67:0/161855388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c40df770 con 0x7f68c40d6220 2024-04-25T15:50:39.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.570+0000 7f68d162c640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68c40d6220 0x7f68c40de810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68c80ae040 tx=0x7f68c80191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.582+0000 7f75e2ffd640 1 -- 172.21.15.67:0/3306767532 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f75e8071020 con 0x7f75f4073bf0 2024-04-25T15:50:39.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.582+0000 7f68c27fc640 1 -- 172.21.15.67:0/161855388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80b0040 con 0x7f68c40d6220 2024-04-25T15:50:39.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.582+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c4178a10 con 0x7f68c40d6220 2024-04-25T15:50:39.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.582+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c40df130 con 0x7f68c40d6220 2024-04-25T15:50:39.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.582+0000 7f68c27fc640 1 -- 172.21.15.67:0/161855388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c80b4640 con 0x7f68c40d6220 2024-04-25T15:50:39.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.582+0000 7f68c27fc640 1 -- 172.21.15.67:0/161855388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80b4a30 con 0x7f68c40d6220 2024-04-25T15:50:39.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.582+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c40059d0 con 0x7f68c40d6220 2024-04-25T15:50:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.586+0000 7f68c27fc640 1 -- 172.21.15.67:0/161855388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f68c80ab4c0 con 0x7f68c40d6220 2024-04-25T15:50:39.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.586+0000 7f68c27fc640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f68b006f400 0x7f68b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.586+0000 7f68c27fc640 1 -- 172.21.15.67:0/161855388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f68c8128ee0 con 0x7f68c40d6220 2024-04-25T15:50:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.586+0000 7f68d1e2d640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f68b006f400 0x7f68b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:39.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.586+0000 7f68d1e2d640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f68b006f400 0x7f68b0071910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00b7f0 tx=0x7f68bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:39.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.594+0000 7f68c27fc640 1 -- 172.21.15.67:0/161855388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f68c8016150 con 0x7f68c40d6220 2024-04-25T15:50:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.702+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fde540c9660 con 0x7fde540092d0 2024-04-25T15:50:39.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde4affd640 1 -- 172.21.15.67:0/2351707669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fde500f6560 con 0x7fde540092d0 2024-04-25T15:50:39.705 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde3c06f430 msgr2=0x7fde3c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde3c06f430 0x7fde3c071940 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fde4c007fd0 tx=0x7fde4c009040 comp rx=0 tx=0).stop 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde540092d0 msgr2=0x7fde540d6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde540092d0 0x7fde540d6860 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde50000c10 tx=0x7fde5001ab20 comp rx=0 tx=0).stop 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 shutdown_connections 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde3c06f430 0x7fde3c071940 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde540dfaa0 0x7fde540c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde540d6220 0x7fde540c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 --2- 172.21.15.67:0/2351707669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde540092d0 0x7fde540d6860 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 >> 172.21.15.67:0/2351707669 conn(0x7fde540514d0 msgr2=0x7fde54050730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 shutdown_connections 2024-04-25T15:50:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.706+0000 7fde5b59e640 1 -- 172.21.15.67:0/2351707669 wait complete. 2024-04-25T15:50:39.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.3 2024-04-25T15:50:39.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.761+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0fb4005590 con 0x7f0fd00733f0 2024-04-25T15:50:39.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.761+0000 7f0fc77fe640 1 -- 172.21.15.67:0/2903515929 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0fc005f150 con 0x7f0fd00733f0 2024-04-25T15:50:39.763 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2024-04-25T15:50:39.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.761+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fb006f4d0 msgr2=0x7f0fb00719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.761+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fb006f4d0 0x7f0fb00719e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00c8860 tx=0x7f0fb8009060 comp rx=0 tx=0).stop 2024-04-25T15:50:39.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.761+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd00733f0 msgr2=0x7f0fd01fdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.761+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd00733f0 0x7f0fd01fdc90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c3a0 tx=0x7f0fc000c860 comp rx=0 tx=0).stop 2024-04-25T15:50:39.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 shutdown_connections 2024-04-25T15:50:39.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fb006f4d0 0x7f0fb00719e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fd0164300 0x7f0fd014c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fd00c99e0 0x7f0fd014eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 --2- 172.21.15.67:0/2903515929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fd00733f0 0x7f0fd01fdc90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 >> 172.21.15.67:0/2903515929 conn(0x7f0fd0070db0 msgr2=0x7f0fd0150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 shutdown_connections 2024-04-25T15:50:39.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.765+0000 7f0fd5257640 1 -- 172.21.15.67:0/2903515929 wait complete. 2024-04-25T15:50:39.783 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.0 2024-04-25T15:50:39.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.817+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f023014f420 con 0x7f02300c6c10 2024-04-25T15:50:39.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.817+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fee28005860 con 0x7fee2c003b80 2024-04-25T15:50:39.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.817+0000 7f0217fff640 1 -- 172.21.15.67:0/705223688 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0220060480 con 0x7f02300c6c10 2024-04-25T15:50:39.821 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:39.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee32ffd640 1 -- 172.21.15.67:0/2902920414 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fee380f5a00 con 0x7fee2c003b80 2024-04-25T15:50:39.822 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f021c06f400 msgr2=0x7f021c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f021c06f400 0x7f021c071910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f02240b2000 tx=0x7f0224009040 comp rx=0 tx=0).stop 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02300c6c10 msgr2=0x7f023015ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02300c6c10 0x7f023015ab90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f022000e8a0 tx=0x7f022000eca0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 shutdown_connections 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f021c06f400 0x7f021c071910 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02300d2540 0x7f0230148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02300c9bc0 0x7f023015b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 --2- 172.21.15.67:0/705223688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02300c6c10 0x7f023015ab90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 >> 172.21.15.67:0/705223688 conn(0x7f0230068760 msgr2=0x7f023014ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 shutdown_connections 2024-04-25T15:50:39.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7f0235c17640 1 -- 172.21.15.67:0/705223688 wait complete. 2024-04-25T15:50:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee2406f480 msgr2=0x7fee24071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee2406f480 0x7fee24071990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee4414c260 tx=0x7fee34008040 comp rx=0 tx=0).stop 2024-04-25T15:50:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c003b80 msgr2=0x7fee4415cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c003b80 0x7fee4415cde0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee3800c0e0 tx=0x7fee2c005840 comp rx=0 tx=0).stop 2024-04-25T15:50:39.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 shutdown_connections 2024-04-25T15:50:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee2406f480 0x7fee24071990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee4414e400 0x7fee4414ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee4414fe00 0x7fee4414b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 --2- 172.21.15.67:0/2902920414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee2c003b80 0x7fee4415cde0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 >> 172.21.15.67:0/2902920414 conn(0x7fee440700b0 msgr2=0x7fee440704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 shutdown_connections 2024-04-25T15:50:39.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.821+0000 7fee489ea640 1 -- 172.21.15.67:0/2902920414 wait complete. 2024-04-25T15:50:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.825+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f96040c4700 con 0x7f9604071b20 2024-04-25T15:50:39.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.825+0000 7f95ebfff640 1 -- 172.21.15.67:0/2146512343 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f95f405f050 con 0x7f9604071b20 2024-04-25T15:50:39.829 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f95f006f480 msgr2=0x7f95f0071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f95f006f480 0x7f95f0071990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006fd0 tx=0x7f95f8006b40 comp rx=0 tx=0).stop 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9604071b20 msgr2=0x7f96042064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9604071b20 0x7f96042064c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f95f400c430 tx=0x7f95f400c8f0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 shutdown_connections 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f95f006f480 0x7f95f0071990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96040c7610 0x7f960414d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96040c5bb0 0x7f9604152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 --2- 172.21.15.67:0/2146512343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9604071b20 0x7f96042064c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 >> 172.21.15.67:0/2146512343 conn(0x7f960406fe70 msgr2=0x7f96040c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 shutdown_connections 2024-04-25T15:50:39.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.829+0000 7f9609e47640 1 -- 172.21.15.67:0/2146512343 wait complete. 2024-04-25T15:50:39.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.6 2024-04-25T15:50:39.856 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.2 2024-04-25T15:50:39.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.857+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f75d8005860 con 0x7f75f4073bf0 2024-04-25T15:50:39.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.857+0000 7f75e2ffd640 1 -- 172.21.15.67:0/3306767532 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f75e800d040 con 0x7f75f4073bf0 2024-04-25T15:50:39.860 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2024-04-25T15:50:39.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.7 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75dc073960 msgr2=0x7f75dc075e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75dc073960 0x7f75dc075e70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f75e4000f70 tx=0x7f75e4007e90 comp rx=0 tx=0).stop 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f4073bf0 msgr2=0x7f75f41569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f4073bf0 0x7f75f41569d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b440 tx=0x7f75e800b900 comp rx=0 tx=0).stop 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 shutdown_connections 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75dc073960 0x7f75dc075e70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f415d790 0x7f75f40cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f40caf60 0x7f75f41f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 --2- 172.21.15.67:0/3306767532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f4073bf0 0x7f75f41569d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 >> 172.21.15.67:0/3306767532 conn(0x7f75f4070db0 msgr2=0x7f75f4158880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 shutdown_connections 2024-04-25T15:50:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.861+0000 7f75e0ff9640 1 -- 172.21.15.67:0/3306767532 wait complete. 2024-04-25T15:50:39.879 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.5 2024-04-25T15:50:39.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f68c4036cb0 con 0x7f68c40d6220 2024-04-25T15:50:39.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68c27fc640 1 -- 172.21.15.67:0/161855388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f68c80f57f0 con 0x7f68c40d6220 2024-04-25T15:50:39.887 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f68b006f400 msgr2=0x7f68b0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f68b006f400 0x7f68b0071910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00b7f0 tx=0x7f68bc008040 comp rx=0 tx=0).stop 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68c40d6220 msgr2=0x7f68c40de810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68c40d6220 0x7f68c40de810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68c80ae040 tx=0x7f68c80191e0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 shutdown_connections 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f68b006f400 0x7f68b0071910 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68c40dfaa0 0x7f68c40db9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68c40d6220 0x7f68c40de810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 --2- 172.21.15.67:0/161855388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68c4009100 0x7f68c40d9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:39.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 >> 172.21.15.67:0/161855388 conn(0x7f68c40514d0 msgr2=0x7f68c40d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:39.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 shutdown_connections 2024-04-25T15:50:39.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:39.885+0000 7f68d2e2f640 1 -- 172.21.15.67:0/161855388 wait complete. 2024-04-25T15:50:39.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.1 2024-04-25T15:50:40.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.157+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff380015e50 con 0x7ff378077700 2024-04-25T15:50:40.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.173+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff38c1e9ed0 con 0x7ff378077700 2024-04-25T15:50:40.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.173+0000 7ff3737fe640 1 -- 172.21.15.67:0/2045848945 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff38c1e9ed0 con 0x7ff378077700 2024-04-25T15:50:40.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.173+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff378077700 msgr2=0x7ff378079b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.173+0000 7ff38bfff640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff378077700 0x7ff378079b50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff37806f430 msgr2=0x7ff378071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff37806f430 0x7ff378071940 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff38401f220 tx=0x7ff3840aa000 comp rx=0 tx=0).stop 2024-04-25T15:50:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff38c0c99c0 msgr2=0x7ff38c157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 --2- 172.21.15.67:0/2045848945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff38c0c99c0 0x7ff38c157b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e7c0 tx=0x7ff38000ec80 comp rx=0 tx=0).stop 2024-04-25T15:50:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38b7fe640 1 -- 172.21.15.67:0/2045848945 reap_dead start 2024-04-25T15:50:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 shutdown_connections 2024-04-25T15:50:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 >> 172.21.15.67:0/2045848945 conn(0x7ff38c0c9fc0 msgr2=0x7ff38c0c8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 shutdown_connections 2024-04-25T15:50:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.177+0000 7ff38bfff640 1 -- 172.21.15.67:0/2045848945 wait complete. 2024-04-25T15:50:40.202 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:50:40.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:50:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.357+0000 7f75965ec640 1 Processor -- start 2024-04-25T15:50:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.357+0000 7f75965ec640 1 -- start start 2024-04-25T15:50:40.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f75965ec640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7590071bc0 0x7f75900cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f75965ec640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75900cc100 0x7f759015a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f75965ec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f759015ac60 0x7f75900ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f75965ec640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f759015ebb0 con 0x7f759015ac60 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f75965ec640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f759015e720 con 0x7f7590071bc0 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f75965ec640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f759015de10 con 0x7f75900cc100 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f759015ac60 0x7f75900ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f759015ac60 0x7f75900ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59870/0 (socket says 172.21.15.67:59870) 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 -- 172.21.15.67:0/2436683229 learned_addr learned my addr 172.21.15.67:0/2436683229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f758f7fe640 1 --2- 172.21.15.67:0/2436683229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75900cc100 0x7f759015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f758ffff640 1 --2- 172.21.15.67:0/2436683229 >> v2:172.21.15.176:3300/0 conn(0x7f7590071bc0 0x7f75900cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 -- 172.21.15.67:0/2436683229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75900cc100 msgr2=0x7f759015a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 --2- 172.21.15.67:0/2436683229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75900cc100 0x7f759015a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 -- 172.21.15.67:0/2436683229 >> v2:172.21.15.176:3300/0 conn(0x7f7590071bc0 msgr2=0x7f75900cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:40.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 --2- 172.21.15.67:0/2436683229 >> v2:172.21.15.176:3300/0 conn(0x7f7590071bc0 0x7f75900cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 -- 172.21.15.67:0/2436683229 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7590072390 con 0x7f759015ac60 2024-04-25T15:50:40.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f758f7fe640 1 --2- 172.21.15.67:0/2436683229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75900cc100 0x7f759015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:40.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f7594b89640 1 --2- 172.21.15.67:0/2436683229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759015ac60 0x7f75900ca540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f759015e8a0 tx=0x7f758801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be82abff6fcc46f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f758effd640 1 -- 172.21.15.67:0/2436683229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880a5780 con 0x7f759015ac60 2024-04-25T15:50:40.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f758effd640 1 -- 172.21.15.67:0/2436683229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75880a5920 con 0x7f759015ac60 2024-04-25T15:50:40.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.361+0000 7f758effd640 1 -- 172.21.15.67:0/2436683229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880a5cf0 con 0x7f759015ac60 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 -- 172.21.15.67:0/2436683229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759015ac60 msgr2=0x7f75900ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 --2- 172.21.15.67:0/2436683229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759015ac60 0x7f75900ca540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f759015e8a0 tx=0x7f758801c9f0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 -- 172.21.15.67:0/2436683229 shutdown_connections 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 --2- 172.21.15.67:0/2436683229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f759015ac60 0x7f75900ca540 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 --2- 172.21.15.67:0/2436683229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75900cc100 0x7f759015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 --2- 172.21.15.67:0/2436683229 >> v2:172.21.15.176:3300/0 conn(0x7f7590071bc0 0x7f75900cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 -- 172.21.15.67:0/2436683229 >> 172.21.15.67:0/2436683229 conn(0x7f75900700b0 msgr2=0x7f75900704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 -- 172.21.15.67:0/2436683229 shutdown_connections 2024-04-25T15:50:40.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 -- 172.21.15.67:0/2436683229 wait complete. 2024-04-25T15:50:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 Processor -- start 2024-04-25T15:50:40.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.365+0000 7f75965ec640 1 -- start start 2024-04-25T15:50:40.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7590071bc0 0x7f75901f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900cc100 0x7f75900ce410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f759015ac60 0x7f759014af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7590072150 con 0x7f7590071bc0 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7590071fd0 con 0x7f75900cc100 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f75900d02e0 con 0x7f759015ac60 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7590071bc0 0x7f75901f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7590071bc0 0x7f75901f68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59882/0 (socket says 172.21.15.67:59882) 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 -- 172.21.15.67:0/1626298863 learned_addr learned my addr 172.21.15.67:0/1626298863 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758f7fe640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900cc100 0x7f75900ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f7594b89640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f759015ac60 0x7f759014af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 -- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f759015ac60 msgr2=0x7f759014af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f759015ac60 0x7f759014af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 -- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900cc100 msgr2=0x7f75900ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900cc100 0x7f75900ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 -- 172.21.15.67:0/1626298863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75900cf140 con 0x7f7590071bc0 2024-04-25T15:50:40.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758f7fe640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900cc100 0x7f75900ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758ffff640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7590071bc0 0x7f75901f68b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f758000c620 tx=0x7f758000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758cff9640 1 -- 172.21.15.67:0/1626298863 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580015070 con 0x7f7590071bc0 2024-04-25T15:50:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758cff9640 1 -- 172.21.15.67:0/1626298863 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7580011800 con 0x7f7590071bc0 2024-04-25T15:50:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758cff9640 1 -- 172.21.15.67:0/1626298863 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580011bd0 con 0x7f7590071bc0 2024-04-25T15:50:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75901f5df0 con 0x7f7590071bc0 2024-04-25T15:50:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75900cf6c0 con 0x7f7590071bc0 2024-04-25T15:50:40.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759006a480 con 0x7f7590071bc0 2024-04-25T15:50:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.369+0000 7f758cff9640 1 -- 172.21.15.67:0/1626298863 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7580018070 con 0x7f7590071bc0 2024-04-25T15:50:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.373+0000 7f758cff9640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f757c06f430 0x7f757c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.373+0000 7f758cff9640 1 -- 172.21.15.67:0/1626298863 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f758008ef50 con 0x7f7590071bc0 2024-04-25T15:50:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.373+0000 7f758f7fe640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f757c06f430 0x7f757c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.373+0000 7f758f7fe640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f757c06f430 0x7f757c071940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7590076240 tx=0x7f7584008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.377+0000 7f758cff9640 1 -- 172.21.15.67:0/1626298863 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f758005bf60 con 0x7f7590071bc0 2024-04-25T15:50:40.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f759014a460 con 0x7f7590071bc0 2024-04-25T15:50:40.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f758cff9640 1 -- 172.21.15.67:0/1626298863 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f758005b860 con 0x7f7590071bc0 2024-04-25T15:50:40.534 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f757c06f430 msgr2=0x7f757c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f757c06f430 0x7f757c071940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7590076240 tx=0x7f7584008040 comp rx=0 tx=0).stop 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7590071bc0 msgr2=0x7f75901f68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7590071bc0 0x7f75901f68b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f758000c620 tx=0x7f758000cae0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 shutdown_connections 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f757c06f430 0x7f757c071940 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f759015ac60 0x7f759014af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900cc100 0x7f75900ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 --2- 172.21.15.67:0/1626298863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7590071bc0 0x7f75901f68b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 >> 172.21.15.67:0/1626298863 conn(0x7f75900700b0 msgr2=0x7f75900ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 shutdown_connections 2024-04-25T15:50:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.533+0000 7f75965ec640 1 -- 172.21.15.67:0/1626298863 wait complete. 2024-04-25T15:50:40.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2024-04-25T15:50:40.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:50:40.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:50:40.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:50:40.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:50:40.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:50:40.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 Processor -- start 2024-04-25T15:50:40.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 -- start start 2024-04-25T15:50:40.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd406c640 0x7f6cd4154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cd415e4a0 0x7f6cd40c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6cd415e8b0 0x7f6cd40c8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd40749d0 con 0x7f6cd415e4a0 2024-04-25T15:50:40.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd406e510 con 0x7f6cd415e8b0 2024-04-25T15:50:40.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cdbb97640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd415dca0 con 0x7f6cd406c640 2024-04-25T15:50:40.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cd415e4a0 0x7f6cd40c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cd415e4a0 0x7f6cd40c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59886/0 (socket says 172.21.15.67:59886) 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 -- 172.21.15.67:0/4289175067 learned_addr learned my addr 172.21.15.67:0/4289175067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9933640 1 --2- 172.21.15.67:0/4289175067 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd406c640 0x7f6cd4154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cda134640 1 --2- 172.21.15.67:0/4289175067 >> v2:172.21.15.176:3300/0 conn(0x7f6cd415e8b0 0x7f6cd40c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 -- 172.21.15.67:0/4289175067 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd406c640 msgr2=0x7f6cd4154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 --2- 172.21.15.67:0/4289175067 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd406c640 0x7f6cd4154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 -- 172.21.15.67:0/4289175067 >> v2:172.21.15.176:3300/0 conn(0x7f6cd415e8b0 msgr2=0x7f6cd40c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 --2- 172.21.15.67:0/4289175067 >> v2:172.21.15.176:3300/0 conn(0x7f6cd415e8b0 0x7f6cd40c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.877+0000 7f6cd9132640 1 -- 172.21.15.67:0/4289175067 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd415df80 con 0x7f6cd415e4a0 2024-04-25T15:50:40.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cd9132640 1 --2- 172.21.15.67:0/4289175067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e4a0 0x7f6cd40c7fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40cf6c0 tx=0x7f6cc401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eaddb0c955ac2f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cd8931640 1 -- 172.21.15.67:0/4289175067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40b1070 con 0x7f6cd415e4a0 2024-04-25T15:50:40.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cd8931640 1 -- 172.21.15.67:0/4289175067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc40ac600 con 0x7f6cd415e4a0 2024-04-25T15:50:40.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cd8931640 1 -- 172.21.15.67:0/4289175067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40ac9d0 con 0x7f6cd415e4a0 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 -- 172.21.15.67:0/4289175067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e4a0 msgr2=0x7f6cd40c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/4289175067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e4a0 0x7f6cd40c7fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40cf6c0 tx=0x7f6cc401c490 comp rx=0 tx=0).stop 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 -- 172.21.15.67:0/4289175067 shutdown_connections 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/4289175067 >> v2:172.21.15.176:3300/0 conn(0x7f6cd415e8b0 0x7f6cd40c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/4289175067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e4a0 0x7f6cd40c7fc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/4289175067 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd406c640 0x7f6cd4154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 -- 172.21.15.67:0/4289175067 >> 172.21.15.67:0/4289175067 conn(0x7f6cd4070db0 msgr2=0x7f6cd40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:40.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 -- 172.21.15.67:0/4289175067 shutdown_connections 2024-04-25T15:50:40.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.881+0000 7f6cdbb97640 1 -- 172.21.15.67:0/4289175067 wait complete. 2024-04-25T15:50:40.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 Processor -- start 2024-04-25T15:50:40.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 -- start start 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6cd406c640 0x7f6cd40cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd415e4a0 0x7f6cd40c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e8b0 0x7f6cd40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd40cad60 con 0x7f6cd415e8b0 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd4072bc0 con 0x7f6cd406c640 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cdbb97640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd4148a80 con 0x7f6cd415e4a0 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.885+0000 7f6cda134640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e8b0 0x7f6cd40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e8b0 0x7f6cd40cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59900/0 (socket says 172.21.15.67:59900) 2024-04-25T15:50:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 -- 172.21.15.67:0/3020641170 learned_addr learned my addr 172.21.15.67:0/3020641170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cd9132640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd415e4a0 0x7f6cd40c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cd9933640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6cd406c640 0x7f6cd40cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 -- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd415e4a0 msgr2=0x7f6cd40c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd415e4a0 0x7f6cd40c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 -- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6cd406c640 msgr2=0x7f6cd40cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6cd406c640 0x7f6cd40cdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 -- 172.21.15.67:0/3020641170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd40ce4e0 con 0x7f6cd415e8b0 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cda134640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e8b0 0x7f6cd40cd030 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009fd0 tx=0x7f6ccc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cca7fc640 1 -- 172.21.15.67:0/3020641170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc014420 con 0x7f6cd415e8b0 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cca7fc640 1 -- 172.21.15.67:0/3020641170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ccc019070 con 0x7f6cd415e8b0 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cca7fc640 1 -- 172.21.15.67:0/3020641170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc014800 con 0x7f6cd415e8b0 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd4149660 con 0x7f6cd415e8b0 2024-04-25T15:50:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd4149c20 con 0x7f6cd415e8b0 2024-04-25T15:50:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.889+0000 7f6cca7fc640 1 -- 172.21.15.67:0/3020641170 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6ccc0149a0 con 0x7f6cd415e8b0 2024-04-25T15:50:40.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.893+0000 7f6cca7fc640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6cb806f400 0x7f6cb8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.893+0000 7f6cca7fc640 1 -- 172.21.15.67:0/3020641170 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6ccc08f890 con 0x7f6cd415e8b0 2024-04-25T15:50:40.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.893+0000 7f6cd9933640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6cb806f400 0x7f6cb8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.893+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd4148d20 con 0x7f6cd415e8b0 2024-04-25T15:50:40.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.893+0000 7f6cd9933640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6cb806f400 0x7f6cb8071910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00079c0 tx=0x7f6cd0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.897+0000 7f6cca7fc640 1 -- 172.21.15.67:0/3020641170 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6ccc05c250 con 0x7f6cd415e8b0 2024-04-25T15:50:40.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:50:40.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.961+0000 7eff65836640 1 Processor -- start 2024-04-25T15:50:40.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.961+0000 7eff65836640 1 -- start start 2024-04-25T15:50:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff65836640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7eff600c35a0 0x7eff600c9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff65836640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff6006c640 0x7eff6006ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff65836640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff60068bf0 con 0x7eff60161740 2024-04-25T15:50:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff65836640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7eff60068630 con 0x7eff6006c640 2024-04-25T15:50:40.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff65836640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7eff60068910 con 0x7eff600c35a0 2024-04-25T15:50:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff6006c640 0x7eff6006ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff6006c640 0x7eff6006ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59902/0 (socket says 172.21.15.67:59902) 2024-04-25T15:50:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5f7fe640 1 -- 172.21.15.67:0/3396612170 learned_addr learned my addr 172.21.15.67:0/3396612170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5effd640 1 --2- 172.21.15.67:0/3396612170 >> v2:172.21.15.176:3300/0 conn(0x7eff600c35a0 0x7eff600c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5f7fe640 1 -- 172.21.15.67:0/3396612170 >> v2:172.21.15.176:3300/0 conn(0x7eff600c35a0 msgr2=0x7eff600c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5f7fe640 1 --2- 172.21.15.67:0/3396612170 >> v2:172.21.15.176:3300/0 conn(0x7eff600c35a0 0x7eff600c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5f7fe640 1 -- 172.21.15.67:0/3396612170 >> v1:172.21.15.176:6789/0 conn(0x7eff60161740 legacy=0x7eff6015ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.965+0000 7eff5f7fe640 1 -- 172.21.15.67:0/3396612170 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff600c8c60 con 0x7eff6006c640 2024-04-25T15:50:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff5f7fe640 1 --2- 172.21.15.67:0/3396612170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff6006c640 0x7eff6006ca50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff600cbd80 tx=0x7eff5801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed73d580e9e738f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff5dffb640 1 -- 172.21.15.67:0/3396612170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff580a5540 con 0x7eff6006c640 2024-04-25T15:50:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff5dffb640 1 -- 172.21.15.67:0/3396612170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff580a56e0 con 0x7eff6006c640 2024-04-25T15:50:40.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff5dffb640 1 -- 172.21.15.67:0/3396612170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff580a5ab0 con 0x7eff6006c640 2024-04-25T15:50:40.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 -- 172.21.15.67:0/3396612170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff6006c640 msgr2=0x7eff6006ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 --2- 172.21.15.67:0/3396612170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff6006c640 0x7eff6006ca50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7eff600cbd80 tx=0x7eff5801c7f0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 -- 172.21.15.67:0/3396612170 shutdown_connections 2024-04-25T15:50:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 --2- 172.21.15.67:0/3396612170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff6006c640 0x7eff6006ca50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 --2- 172.21.15.67:0/3396612170 >> v2:172.21.15.176:3300/0 conn(0x7eff600c35a0 0x7eff600c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 -- 172.21.15.67:0/3396612170 >> 172.21.15.67:0/3396612170 conn(0x7eff600ce1c0 msgr2=0x7eff600ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 -- 172.21.15.67:0/3396612170 shutdown_connections 2024-04-25T15:50:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.969+0000 7eff65836640 1 -- 172.21.15.67:0/3396612170 wait complete. 2024-04-25T15:50:40.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.973+0000 7eff65836640 1 Processor -- start 2024-04-25T15:50:40.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.973+0000 7eff65836640 1 -- start start 2024-04-25T15:50:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff65836640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff6006c640 0x7eff601fe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff65836640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff600c35a0 0x7eff600c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff65836640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff60161740 0x7eff60153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff65836640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7eff600caaf0 con 0x7eff600c35a0 2024-04-25T15:50:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff600c35a0 0x7eff600c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff600c35a0 0x7eff600c75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59912/0 (socket says 172.21.15.67:59912) 2024-04-25T15:50:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 -- 172.21.15.67:0/1707644045 learned_addr learned my addr 172.21.15.67:0/1707644045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff65836640 1 -- 172.21.15.67:0/1707644045 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff600ca970 con 0x7eff6006c640 2024-04-25T15:50:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff65836640 1 -- 172.21.15.67:0/1707644045 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff6016dda0 con 0x7eff60161740 2024-04-25T15:50:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5f7fe640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff60161740 0x7eff60153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5effd640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff6006c640 0x7eff601fe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 -- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff60161740 msgr2=0x7eff60153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff60161740 0x7eff60153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 -- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff6006c640 msgr2=0x7eff601fe0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff6006c640 0x7eff601fe0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 -- 172.21.15.67:0/1707644045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff600b4340 con 0x7eff600c35a0 2024-04-25T15:50:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.977+0000 7eff5e7fc640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff600c35a0 0x7eff600c75c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff5000c580 tx=0x7eff5000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.981+0000 7eff47fff640 1 -- 172.21.15.67:0/1707644045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50017070 con 0x7eff600c35a0 2024-04-25T15:50:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.981+0000 7eff47fff640 1 -- 172.21.15.67:0/1707644045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff50003650 con 0x7eff600c35a0 2024-04-25T15:50:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.981+0000 7eff65836640 1 -- 172.21.15.67:0/1707644045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff601fe5e0 con 0x7eff600c35a0 2024-04-25T15:50:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.981+0000 7eff65836640 1 -- 172.21.15.67:0/1707644045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6015bab0 con 0x7eff600c35a0 2024-04-25T15:50:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.981+0000 7eff47fff640 1 -- 172.21.15.67:0/1707644045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50003a20 con 0x7eff600c35a0 2024-04-25T15:50:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.985+0000 7eff47fff640 1 -- 172.21.15.67:0/1707644045 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7eff5001a070 con 0x7eff600c35a0 2024-04-25T15:50:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.985+0000 7eff47fff640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff4c06f500 0x7eff4c071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.985+0000 7eff47fff640 1 -- 172.21.15.67:0/1707644045 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7eff5008e730 con 0x7eff600c35a0 2024-04-25T15:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.985+0000 7eff5effd640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff4c06f500 0x7eff4c071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.985+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff400052a0 con 0x7eff600c35a0 2024-04-25T15:50:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.985+0000 7eff5effd640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff4c06f500 0x7eff4c071a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff540079c0 tx=0x7eff54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:40.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:40.997+0000 7eff47fff640 1 -- 172.21.15.67:0/1707644045 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7eff5005b0d0 con 0x7eff600c35a0 2024-04-25T15:50:41.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b413b1640 1 Processor -- start 2024-04-25T15:50:41.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b413b1640 1 -- start start 2024-04-25T15:50:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b413b1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0b3c0c35a0 0x7f0b3c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b413b1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b3c06c640 0x7f0b3c06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b413b1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b3c068bf0 con 0x7f0b3c161740 2024-04-25T15:50:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b413b1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0b3c068630 con 0x7f0b3c06c640 2024-04-25T15:50:41.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b413b1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0b3c068910 con 0x7f0b3c0c35a0 2024-04-25T15:50:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b3c06c640 0x7f0b3c06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b3c06c640 0x7f0b3c06ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59926/0 (socket says 172.21.15.67:59926) 2024-04-25T15:50:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3b7fe640 1 -- 172.21.15.67:0/4252116989 learned_addr learned my addr 172.21.15.67:0/4252116989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3affd640 1 --2- 172.21.15.67:0/4252116989 >> v2:172.21.15.176:3300/0 conn(0x7f0b3c0c35a0 0x7f0b3c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3b7fe640 1 -- 172.21.15.67:0/4252116989 >> v2:172.21.15.176:3300/0 conn(0x7f0b3c0c35a0 msgr2=0x7f0b3c0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3b7fe640 1 --2- 172.21.15.67:0/4252116989 >> v2:172.21.15.176:3300/0 conn(0x7f0b3c0c35a0 0x7f0b3c0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3b7fe640 1 -- 172.21.15.67:0/4252116989 >> v1:172.21.15.67:6789/0 conn(0x7f0b3c161740 legacy=0x7f0b3c15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3b7fe640 1 -- 172.21.15.67:0/4252116989 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c0c8c60 con 0x7f0b3c06c640 2024-04-25T15:50:41.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.053+0000 7f0b3affd640 1 --2- 172.21.15.67:0/4252116989 >> v2:172.21.15.176:3300/0 conn(0x7f0b3c0c35a0 0x7f0b3c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:41.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b3b7fe640 1 --2- 172.21.15.67:0/4252116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c06ca50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0cbd80 tx=0x7f0b3401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9018fd62710d068 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b39ffb640 1 -- 172.21.15.67:0/4252116989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b340a5730 con 0x7f0b3c06c640 2024-04-25T15:50:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b39ffb640 1 -- 172.21.15.67:0/4252116989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b340a58d0 con 0x7f0b3c06c640 2024-04-25T15:50:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 -- 172.21.15.67:0/4252116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 msgr2=0x7f0b3c06ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 --2- 172.21.15.67:0/4252116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c06ca50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0cbd80 tx=0x7f0b3401c980 comp rx=0 tx=0).stop 2024-04-25T15:50:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 -- 172.21.15.67:0/4252116989 shutdown_connections 2024-04-25T15:50:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 --2- 172.21.15.67:0/4252116989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c06ca50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 --2- 172.21.15.67:0/4252116989 >> v2:172.21.15.176:3300/0 conn(0x7f0b3c0c35a0 0x7f0b3c0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 -- 172.21.15.67:0/4252116989 >> 172.21.15.67:0/4252116989 conn(0x7f0b3c0ce1c0 msgr2=0x7f0b3c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 -- 172.21.15.67:0/4252116989 shutdown_connections 2024-04-25T15:50:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.057+0000 7f0b413b1640 1 -- 172.21.15.67:0/4252116989 wait complete. 2024-04-25T15:50:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 Processor -- start 2024-04-25T15:50:41.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 -- start start 2024-04-25T15:50:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b3c0c35a0 0x7f0b3c1fda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b3c161740 0x7f0b3c0cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c0caaf0 con 0x7f0b3c06c640 2024-04-25T15:50:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c0ca970 con 0x7f0b3c0c35a0 2024-04-25T15:50:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0b3c16bd90 con 0x7f0b3c161740 2024-04-25T15:50:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c152790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59942/0 (socket says 172.21.15.67:59942) 2024-04-25T15:50:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 -- 172.21.15.67:0/370786703 learned_addr learned my addr 172.21.15.67:0/370786703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3b7fe640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b3c161740 0x7f0b3c0cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 -- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b3c161740 msgr2=0x7f0b3c0cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b3c161740 0x7f0b3c0cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 -- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b3c0c35a0 msgr2=0x7f0b3c1fda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b3c0c35a0 0x7f0b3c1fda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 -- 172.21.15.67:0/370786703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c152cd0 con 0x7f0b3c06c640 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3b7fe640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b3c161740 0x7f0b3c0cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b3affd640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c152790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400c7f0 tx=0x7f0b2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b337fe640 1 -- 172.21.15.67:0/370786703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24015070 con 0x7f0b3c06c640 2024-04-25T15:50:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b337fe640 1 -- 172.21.15.67:0/370786703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b24007820 con 0x7f0b3c06c640 2024-04-25T15:50:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b337fe640 1 -- 172.21.15.67:0/370786703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24007c10 con 0x7f0b3c06c640 2024-04-25T15:50:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.061+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3c1532a0 con 0x7f0b3c06c640 2024-04-25T15:50:41.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.065+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3c0cd8d0 con 0x7f0b3c06c640 2024-04-25T15:50:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.065+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b3c06a480 con 0x7f0b3c06c640 2024-04-25T15:50:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.065+0000 7f0b337fe640 1 -- 172.21.15.67:0/370786703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0b24018070 con 0x7f0b3c06c640 2024-04-25T15:50:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.065+0000 7f0b337fe640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0b1c06f430 0x7f0b1c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.065+0000 7f0b3a7fc640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0b1c06f430 0x7f0b1c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.069+0000 7f0b337fe640 1 -- 172.21.15.67:0/370786703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0b24092070 con 0x7f0b3c06c640 2024-04-25T15:50:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.069+0000 7f0b3a7fc640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0b1c06f430 0x7f0b1c071940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c007fd0 tx=0x7f0b2c007980 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.069+0000 7efc1e373640 1 Processor -- start 2024-04-25T15:50:41.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.069+0000 7efc1e373640 1 -- start start 2024-04-25T15:50:41.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc1e373640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc1806c640 0x7efc1815b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc1e373640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc180c92d0 con 0x7efc18074780 2024-04-25T15:50:41.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc1e373640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc1815b560 con 0x7efc18074370 2024-04-25T15:50:41.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc1e373640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7efc18074b90 con 0x7efc1806c640 2024-04-25T15:50:41.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc17fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc1806c640 0x7efc1815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc17fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc1806c640 0x7efc1815b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59952/0 (socket says 172.21.15.67:59952) 2024-04-25T15:50:41.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc17fff640 1 -- 172.21.15.67:0/1039974225 learned_addr learned my addr 172.21.15.67:0/1039974225 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc17fff640 1 -- 172.21.15.67:0/1039974225 >> v1:172.21.15.176:6789/0 conn(0x7efc18074370 legacy=0x7efc180c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc17fff640 1 -- 172.21.15.67:0/1039974225 >> v1:172.21.15.67:6789/0 conn(0x7efc18074780 legacy=0x7efc18164f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.073+0000 7efc17fff640 1 -- 172.21.15.67:0/1039974225 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1806e210 con 0x7efc1806c640 2024-04-25T15:50:41.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc17fff640 1 --2- 172.21.15.67:0/1039974225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1806c640 0x7efc1815b020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc08009870 tx=0x7efc0801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d36d7b78268b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7f0b337fe640 1 -- 172.21.15.67:0/370786703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0b2405b520 con 0x7f0b3c06c640 2024-04-25T15:50:41.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc16ffd640 1 -- 172.21.15.67:0/1039974225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc080b5070 con 0x7efc1806c640 2024-04-25T15:50:41.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc16ffd640 1 -- 172.21.15.67:0/1039974225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc080a8a50 con 0x7efc1806c640 2024-04-25T15:50:41.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc16ffd640 1 -- 172.21.15.67:0/1039974225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc080a8e20 con 0x7efc1806c640 2024-04-25T15:50:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc1e373640 1 -- 172.21.15.67:0/1039974225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1806c640 msgr2=0x7efc1815b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc1e373640 1 --2- 172.21.15.67:0/1039974225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1806c640 0x7efc1815b020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc08009870 tx=0x7efc0801ed50 comp rx=0 tx=0).stop 2024-04-25T15:50:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc1e373640 1 -- 172.21.15.67:0/1039974225 shutdown_connections 2024-04-25T15:50:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc1e373640 1 --2- 172.21.15.67:0/1039974225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1806c640 0x7efc1815b020 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc1e373640 1 -- 172.21.15.67:0/1039974225 >> 172.21.15.67:0/1039974225 conn(0x7efc18070db0 msgr2=0x7efc180ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7ff999433640 1 Processor -- start 2024-04-25T15:50:41.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7efc1e373640 1 -- 172.21.15.67:0/1039974225 shutdown_connections 2024-04-25T15:50:41.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.077+0000 7ff999433640 1 -- start start 2024-04-25T15:50:41.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff999433640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c0d6220 0x7ff98c0df560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff999433640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff98c0dfaa0 0x7ff98c008bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff999433640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff98c046890 con 0x7ff98c009100 2024-04-25T15:50:41.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff999433640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff98c0e0470 con 0x7ff98c0dfaa0 2024-04-25T15:50:41.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7efc1e373640 1 -- 172.21.15.67:0/1039974225 wait complete. 2024-04-25T15:50:41.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff999433640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff98c0e0190 con 0x7ff98c0d6220 2024-04-25T15:50:41.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff9937fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff98c0dfaa0 0x7ff98c008bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff9937fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff98c0dfaa0 0x7ff98c008bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59954/0 (socket says 172.21.15.67:59954) 2024-04-25T15:50:41.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff9937fe640 1 -- 172.21.15.67:0/3329955925 learned_addr learned my addr 172.21.15.67:0/3329955925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff993fff640 1 --2- 172.21.15.67:0/3329955925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c0d6220 0x7ff98c0df560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff9937fe640 1 -- 172.21.15.67:0/3329955925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c0d6220 msgr2=0x7ff98c0df560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff9937fe640 1 --2- 172.21.15.67:0/3329955925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c0d6220 0x7ff98c0df560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff9937fe640 1 -- 172.21.15.67:0/3329955925 >> v1:172.21.15.176:6789/0 conn(0x7ff98c009100 legacy=0x7ff98c0dbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.081+0000 7ff9937fe640 1 -- 172.21.15.67:0/3329955925 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c04cd40 con 0x7ff98c0dfaa0 2024-04-25T15:50:41.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7efc1e373640 1 Processor -- start 2024-04-25T15:50:41.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7f45d3281640 1 Processor -- start 2024-04-25T15:50:41.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7ff9937fe640 1 --2- 172.21.15.67:0/3329955925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0dfaa0 0x7ff98c008bc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff98800f9b0 tx=0x7ff988024b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517d42a2a751904 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7ff992ffd640 1 -- 172.21.15.67:0/3329955925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9880ba070 con 0x7ff98c0dfaa0 2024-04-25T15:50:41.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7ff992ffd640 1 -- 172.21.15.67:0/3329955925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9880ad9b0 con 0x7ff98c0dfaa0 2024-04-25T15:50:41.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7ff992ffd640 1 -- 172.21.15.67:0/3329955925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9880add80 con 0x7ff98c0dfaa0 2024-04-25T15:50:41.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7efc1e373640 1 -- start start 2024-04-25T15:50:41.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7efc1e373640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc1806c640 0x7efc181558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.085+0000 7f45d3281640 1 -- start start 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d3281640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc0caf60 0x7f45cc15d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d3281640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f45cc073bf0 0x7f45cc163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d3281640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45cc0c77a0 con 0x7f45cc15d790 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d3281640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f45cc0cab60 con 0x7f45cc073bf0 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d3281640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc159620 con 0x7f45cc0caf60 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d081c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f45cc15d790 0x7f45cc06d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36096/0 (socket says 172.21.15.67:36096) 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d081c640 1 -- 172.21.15.67:0/3530123 learned_addr learned my addr 172.21.15.67:0/3530123 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 -- 172.21.15.67:0/3329955925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0dfaa0 msgr2=0x7ff98c008bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 --2- 172.21.15.67:0/3329955925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0dfaa0 0x7ff98c008bc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff98800f9b0 tx=0x7ff988024b20 comp rx=0 tx=0).stop 2024-04-25T15:50:41.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d101d640 1 --2- 172.21.15.67:0/3530123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc0caf60 0x7f45cc15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d181e640 1 --2- 172.21.15.67:0/3530123 >> v2:172.21.15.176:3300/0 conn(0x7f45cc073bf0 0x7f45cc163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45c3fff640 1 -- 172.21.15.67:0/3530123 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075895881 0 0) 0x7f45cc0c77a0 con 0x7f45cc15d790 2024-04-25T15:50:41.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45c3fff640 1 -- 172.21.15.67:0/3530123 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45b8003530 con 0x7f45cc15d790 2024-04-25T15:50:41.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d101d640 1 -- 172.21.15.67:0/3530123 >> v2:172.21.15.176:3300/0 conn(0x7f45cc073bf0 msgr2=0x7f45cc163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d101d640 1 --2- 172.21.15.67:0/3530123 >> v2:172.21.15.176:3300/0 conn(0x7f45cc073bf0 0x7f45cc163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d101d640 1 -- 172.21.15.67:0/3530123 >> v1:172.21.15.67:6789/0 conn(0x7f45cc15d790 legacy=0x7f45cc06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7f45d101d640 1 -- 172.21.15.67:0/3530123 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc0c7a60 con 0x7f45cc0caf60 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1e373640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc18074370 0x7efc18159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1e373640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc18074780 0x7efc18150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1e373640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efc18073890 con 0x7efc18074780 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1e373640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efc1806e7b0 con 0x7efc18074370 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1e373640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efc18073b70 con 0x7efc1806c640 2024-04-25T15:50:41.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc17fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc1806c640 0x7efc181558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1c910640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc18074780 0x7efc18150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1c910640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc18074780 0x7efc18150e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59960/0 (socket says 172.21.15.67:59960) 2024-04-25T15:50:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1c910640 1 -- 172.21.15.67:0/142630978 learned_addr learned my addr 172.21.15.67:0/142630978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc177fe640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc18074370 0x7efc18159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 -- 172.21.15.67:0/3329955925 shutdown_connections 2024-04-25T15:50:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 --2- 172.21.15.67:0/3329955925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0dfaa0 0x7ff98c008bc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 --2- 172.21.15.67:0/3329955925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c0d6220 0x7ff98c0df560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 -- 172.21.15.67:0/3329955925 >> 172.21.15.67:0/3329955925 conn(0x7ff98c0514d0 msgr2=0x7ff98c0518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 -- 172.21.15.67:0/3329955925 shutdown_connections 2024-04-25T15:50:41.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc17fff640 1 -- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc18074370 msgr2=0x7efc18159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc17fff640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc18074370 0x7efc18159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc17fff640 1 -- 172.21.15.67:0/142630978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc18074780 msgr2=0x7efc18150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc17fff640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc18074780 0x7efc18150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc17fff640 1 -- 172.21.15.67:0/142630978 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc18155e10 con 0x7efc1806c640 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc1c910640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc18074780 0x7efc18150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc17fff640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc1806c640 0x7efc181558d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc080af000 tx=0x7efc0801baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc14ff9640 1 -- 172.21.15.67:0/142630978 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc080c0070 con 0x7efc1806c640 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc14ff9640 1 -- 172.21.15.67:0/142630978 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc080b9550 con 0x7efc1806c640 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7efc14ff9640 1 -- 172.21.15.67:0/142630978 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc080b5070 con 0x7efc1806c640 2024-04-25T15:50:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.089+0000 7ff999433640 1 -- 172.21.15.67:0/3329955925 wait complete. 2024-04-25T15:50:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699d545640 1 Processor -- start 2024-04-25T15:50:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699d545640 1 -- start start 2024-04-25T15:50:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45d101d640 1 --2- 172.21.15.67:0/3530123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc0caf60 0x7f45cc15d250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45c800b870 tx=0x7f45c801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de9a654fda7cadf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc181597d0 con 0x7efc1806c640 2024-04-25T15:50:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc18156660 con 0x7efc1806c640 2024-04-25T15:50:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45c3fff640 1 -- 172.21.15.67:0/3530123 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80a66b0 con 0x7f45cc0caf60 2024-04-25T15:50:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45c3fff640 1 -- 172.21.15.67:0/3530123 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c80a6850 con 0x7f45cc0caf60 2024-04-25T15:50:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45c3fff640 1 -- 172.21.15.67:0/3530123 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80a6c00 con 0x7f45cc0caf60 2024-04-25T15:50:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7ff999433640 1 Processor -- start 2024-04-25T15:50:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7ff999433640 1 -- start start 2024-04-25T15:50:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc1806a480 con 0x7efc1806c640 2024-04-25T15:50:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45d3281640 1 -- 172.21.15.67:0/3530123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc0caf60 msgr2=0x7f45cc15d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45d3281640 1 --2- 172.21.15.67:0/3530123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc0caf60 0x7f45cc15d250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45c800b870 tx=0x7f45c801c920 comp rx=0 tx=0).stop 2024-04-25T15:50:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45d3281640 1 -- 172.21.15.67:0/3530123 shutdown_connections 2024-04-25T15:50:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45d3281640 1 --2- 172.21.15.67:0/3530123 >> v2:172.21.15.176:3300/0 conn(0x7f45cc073bf0 0x7f45cc163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45d3281640 1 --2- 172.21.15.67:0/3530123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc0caf60 0x7f45cc15d250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f45d3281640 1 -- 172.21.15.67:0/3530123 >> 172.21.15.67:0/3530123 conn(0x7f45cc070db0 msgr2=0x7f45cc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699d545640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69900e2910 0x7f69900e2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699d545640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6990046e60 con 0x7f69900e3260 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699d545640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6990047140 con 0x7f699004bee0 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699d545640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6990008560 con 0x7f69900e2910 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699cd44640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f69900e3260 0x7f69900e9ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36100/0 (socket says 172.21.15.67:36100) 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f699cd44640 1 -- 172.21.15.67:0/1943919880 learned_addr learned my addr 172.21.15.67:0/1943919880 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341274253 0 0) 0x7f6990046e60 con 0x7f69900e3260 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f69977fe640 1 --2- 172.21.15.67:0/1943919880 >> v2:172.21.15.67:3300/0 conn(0x7f69900e2910 0x7f69900e2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6988003530 con 0x7f69900e3260 2024-04-25T15:50:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7efc14ff9640 1 -- 172.21.15.67:0/142630978 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efc080b9940 con 0x7efc1806c640 2024-04-25T15:50:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282833889 0 0) 0x7f6990047140 con 0x7f699004bee0 2024-04-25T15:50:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6990046e60 con 0x7f699004bee0 2024-04-25T15:50:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7efc14ff9640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efc0406f400 0x7efc04071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7efc14ff9640 1 -- 172.21.15.67:0/142630978 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efc0812d090 con 0x7efc1806c640 2024-04-25T15:50:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1699294105 0 0) 0x7f6988003530 con 0x7f69900e3260 2024-04-25T15:50:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 >> v2:172.21.15.67:3300/0 conn(0x7f69900e2910 msgr2=0x7f69900e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 --2- 172.21.15.67:0/1943919880 >> v2:172.21.15.67:3300/0 conn(0x7f69900e2910 0x7f69900e2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 >> v1:172.21.15.176:6789/0 conn(0x7f699004bee0 legacy=0x7f699004c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69900081f0 con 0x7f69900e3260 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f6998050750 con 0x7f69900e3260 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 >> v1:172.21.15.67:6789/0 conn(0x7f69900e3260 legacy=0x7f69900e9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988003b50 0x7f6988003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6988004810 0x7f6988004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f698800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6990008560 con 0x7f6988003b50 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f698800bfc0 con 0x7f6988005160 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.093+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f698800c1c0 con 0x7f6988004810 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f45d3281640 1 -- 172.21.15.67:0/3530123 shutdown_connections 2024-04-25T15:50:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f45d3281640 1 -- 172.21.15.67:0/3530123 wait complete. 2024-04-25T15:50:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f69977fe640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6988004810 0x7f6988004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f698800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7efc177fe640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efc0406f400 0x7efc04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f6997fff640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988003b50 0x7f6988003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 -- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6988004810 msgr2=0x7f6988004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6988004810 0x7f6988004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 -- 172.21.15.67:0/1943919880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988003b50 msgr2=0x7f6988003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6988003b50 0x7f6988003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7efc177fe640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efc0406f400 0x7efc04071910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc180cf750 tx=0x7efc0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff999433640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c009100 0x7ff98c0ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff999433640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0d6220 0x7ff98c0de9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff999433640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff98c0dfaa0 0x7ff98c0c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0d6220 0x7ff98c0de9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0d6220 0x7ff98c0de9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59990/0 (socket says 172.21.15.67:59990) 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 -- 172.21.15.67:0/2300498236 learned_addr learned my addr 172.21.15.67:0/2300498236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff999433640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c04d5e0 con 0x7ff98c0d6220 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff993fff640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c009100 0x7ff98c0ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff998c32640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff98c0dfaa0 0x7ff98c0c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff999433640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c04d460 con 0x7ff98c0dfaa0 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff999433640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff98c046470 con 0x7ff98c009100 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 -- 172.21.15.67:0/1943919880 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6988003530 con 0x7f6988005160 2024-04-25T15:50:41.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 -- 172.21.15.67:0/1943919880 reap_dead start 2024-04-25T15:50:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 -- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c009100 msgr2=0x7ff98c0ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c009100 0x7ff98c0ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 -- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff98c0dfaa0 msgr2=0x7ff98c0c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff98c0dfaa0 0x7ff98c0c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c0ca820 con 0x7ff98c0d6220 2024-04-25T15:50:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f699cd44640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f698800b9d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6998077670 tx=0x7f699809a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e9d76f90d668217 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f45d3281640 1 Processor -- start 2024-04-25T15:50:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff9937fe640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0d6220 0x7ff98c0de9c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff988022750 tx=0x7ff9880218c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998069720 con 0x7f6988005160 2024-04-25T15:50:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6998129660 con 0x7f6988005160 2024-04-25T15:50:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7f6996ffd640 1 -- 172.21.15.67:0/1943919880 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998129a50 con 0x7f6988005160 2024-04-25T15:50:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff990ff9640 1 -- 172.21.15.67:0/2300498236 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9880ba040 con 0x7ff98c0d6220 2024-04-25T15:50:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff990ff9640 1 -- 172.21.15.67:0/2300498236 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9880b64d0 con 0x7ff98c0d6220 2024-04-25T15:50:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.097+0000 7ff990ff9640 1 -- 172.21.15.67:0/2300498236 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9880b68e0 con 0x7ff98c0d6220 2024-04-25T15:50:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 -- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 msgr2=0x7f698800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f698800b9d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6998077670 tx=0x7f699809a340 comp rx=0 tx=0).stop 2024-04-25T15:50:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 -- 172.21.15.67:0/1943919880 shutdown_connections 2024-04-25T15:50:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 --2- 172.21.15.67:0/1943919880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f698800b9d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 -- 172.21.15.67:0/1943919880 >> 172.21.15.67:0/1943919880 conn(0x7f69900514d0 msgr2=0x7f69900518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7ff999433640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98c174c50 con 0x7ff98c0d6220 2024-04-25T15:50:41.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 -- 172.21.15.67:0/1943919880 shutdown_connections 2024-04-25T15:50:41.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 -- 172.21.15.67:0/1943919880 wait complete. 2024-04-25T15:50:41.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7ff999433640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98c175210 con 0x7ff98c0d6220 2024-04-25T15:50:41.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff96c0052a0 con 0x7ff98c0d6220 2024-04-25T15:50:41.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 Processor -- start 2024-04-25T15:50:41.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.101+0000 7f699d545640 1 -- start start 2024-04-25T15:50:41.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d3281640 1 -- start start 2024-04-25T15:50:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699d545640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f6990175a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699d545640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d8260 0x7f6990049110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699d545640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6990049650 0x7f69900cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699d545640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f699004d780 con 0x7f6990049650 2024-04-25T15:50:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699d545640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f699004d600 con 0x7f6988005160 2024-04-25T15:50:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699d545640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6990047b60 con 0x7f69900d8260 2024-04-25T15:50:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f6997fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f6990175a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f6997fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f6990175a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:40170/0 (socket says 172.21.15.67:40170) 2024-04-25T15:50:41.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f6997fff640 1 -- 172.21.15.67:0/1331211785 learned_addr learned my addr 172.21.15.67:0/1331211785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f69977fe640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d8260 0x7f6990049110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7ff990ff9640 1 -- 172.21.15.67:0/2300498236 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff9880c5030 con 0x7ff98c0d6220 2024-04-25T15:50:41.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7ff990ff9640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff98006f430 0x7ff980071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699cd44640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6990049650 0x7f69900cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d3281640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc073bf0 0x7f45cc156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d3281640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0caf60 0x7f45cc15fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d3281640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45cc15d790 0x7f45cc15e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d3281640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc15a570 con 0x7f45cc0caf60 2024-04-25T15:50:41.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d3281640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc15a3f0 con 0x7f45cc15d790 2024-04-25T15:50:41.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d3281640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f45cc06ce50 con 0x7f45cc073bf0 2024-04-25T15:50:41.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7ff990ff9640 1 -- 172.21.15.67:0/2300498236 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff988130e90 con 0x7ff98c0d6220 2024-04-25T15:50:41.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7ff993fff640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff98006f430 0x7ff980071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0caf60 0x7f45cc15fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0caf60 0x7f45cc15fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60002/0 (socket says 172.21.15.67:60002) 2024-04-25T15:50:41.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 -- 172.21.15.67:0/3611706151 learned_addr learned my addr 172.21.15.67:0/3611706151 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699cd44640 1 -- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d8260 msgr2=0x7f6990049110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699cd44640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d8260 0x7f6990049110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699cd44640 1 -- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 msgr2=0x7f6990175a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699cd44640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f6990175a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699cd44640 1 -- 172.21.15.67:0/1331211785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6990175fd0 con 0x7f6990049650 2024-04-25T15:50:41.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f699cd44640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6990049650 0x7f69900cc040 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f699807dc80 tx=0x7f6988005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d101d640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc073bf0 0x7f45cc156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 -- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc073bf0 msgr2=0x7f45cc156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc073bf0 0x7f45cc156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 -- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45cc15d790 msgr2=0x7f45cc15e910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:50:41.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45cc15d790 0x7f45cc15e910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 -- 172.21.15.67:0/3611706151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc15aba0 con 0x7f45cc0caf60 2024-04-25T15:50:41.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7ff993fff640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff98006f430 0x7ff980071940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff9840079c0 tx=0x7ff984008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.105+0000 7f45d081c640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0caf60 0x7f45cc15fd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00c410 tx=0x7f45bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f6994ff9640 1 -- 172.21.15.67:0/1331211785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998050750 con 0x7f6990049650 2024-04-25T15:50:41.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f6994ff9640 1 -- 172.21.15.67:0/1331211785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699813b070 con 0x7f6990049650 2024-04-25T15:50:41.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f6994ff9640 1 -- 172.21.15.67:0/1331211785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998129d70 con 0x7f6990049650 2024-04-25T15:50:41.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f45c1ffb640 1 -- 172.21.15.67:0/3611706151 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0074e0 con 0x7f45cc0caf60 2024-04-25T15:50:41.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69900d0790 con 0x7f6990049650 2024-04-25T15:50:41.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc15ae60 con 0x7f45cc0caf60 2024-04-25T15:50:41.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45cc15f570 con 0x7f45cc0caf60 2024-04-25T15:50:41.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69900d0d50 con 0x7f6990049650 2024-04-25T15:50:41.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7ff990ff9640 1 -- 172.21.15.67:0/2300498236 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff9880cc060 con 0x7ff98c0d6220 2024-04-25T15:50:41.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f45c1ffb640 1 -- 172.21.15.67:0/3611706151 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45bc0047f0 con 0x7f45cc0caf60 2024-04-25T15:50:41.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7f45c1ffb640 1 -- 172.21.15.67:0/3611706151 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0074e0 con 0x7f45cc0caf60 2024-04-25T15:50:41.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.113+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ac0052a0 con 0x7f45cc0caf60 2024-04-25T15:50:41.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.109+0000 7efc14ff9640 1 -- 172.21.15.67:0/142630978 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efc080ca020 con 0x7efc1806c640 2024-04-25T15:50:41.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.113+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69900059d0 con 0x7f6990049650 2024-04-25T15:50:41.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.113+0000 7f6994ff9640 1 -- 172.21.15.67:0/1331211785 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6998140460 con 0x7f6990049650 2024-04-25T15:50:41.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.113+0000 7f6994ff9640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f697c06f430 0x7f697c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.113+0000 7f6994ff9640 1 -- 172.21.15.67:0/1331211785 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f69981a7a70 con 0x7f6990049650 2024-04-25T15:50:41.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.113+0000 7f6997fff640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f697c06f430 0x7f697c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.117+0000 7f6997fff640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f697c06f430 0x7f697c071940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f69840014d0 tx=0x7f69840013e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.125+0000 7f6994ff9640 1 -- 172.21.15.67:0/1331211785 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6984001d10 con 0x7f6990049650 2024-04-25T15:50:41.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.129+0000 7f45c1ffb640 1 -- 172.21.15.67:0/3611706151 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f45bc011070 con 0x7f45cc0caf60 2024-04-25T15:50:41.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.129+0000 7f45c1ffb640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f45b006f630 0x7f45b0071b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.129+0000 7f45c1ffb640 1 -- 172.21.15.67:0/3611706151 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f45bc08e840 con 0x7f45cc0caf60 2024-04-25T15:50:41.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.129+0000 7f45d101d640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f45b006f630 0x7f45b0071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.129+0000 7f45d101d640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f45b006f630 0x7f45b0071b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f45c80af000 tx=0x7f45c80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.141+0000 7f45c1ffb640 1 -- 172.21.15.67:0/3611706151 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f45bc05b7c0 con 0x7f45cc0caf60 2024-04-25T15:50:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.193+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6cd406a480 con 0x7f6cd415e8b0 2024-04-25T15:50:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.193+0000 7f6cca7fc640 1 -- 172.21.15.67:0/3020641170 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6ccc060550 con 0x7f6cd415e8b0 2024-04-25T15:50:41.196 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6cb806f400 msgr2=0x7f6cb8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6cb806f400 0x7f6cb8071910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00079c0 tx=0x7f6cd0008040 comp rx=0 tx=0).stop 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e8b0 msgr2=0x7f6cd40cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e8b0 0x7f6cd40cd030 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009fd0 tx=0x7f6ccc00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 shutdown_connections 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6cb806f400 0x7f6cb8071910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cd415e8b0 0x7f6cd40cd030 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6cd415e4a0 0x7f6cd40c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 --2- 172.21.15.67:0/3020641170 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6cd406c640 0x7f6cd40cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 >> 172.21.15.67:0/3020641170 conn(0x7f6cd4070db0 msgr2=0x7f6cd406d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 shutdown_connections 2024-04-25T15:50:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.197+0000 7f6cdbb97640 1 -- 172.21.15.67:0/3020641170 wait complete. 2024-04-25T15:50:41.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-04-25T15:50:41.227 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:41.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.285+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7efc180cd3f0 con 0x7efc1806c640 2024-04-25T15:50:41.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.285+0000 7efc14ff9640 1 -- 172.21.15.67:0/142630978 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efc080188b0 con 0x7efc1806c640 2024-04-25T15:50:41.288 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efc0406f400 msgr2=0x7efc04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efc0406f400 0x7efc04071910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc180cf750 tx=0x7efc0c009040 comp rx=0 tx=0).stop 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc1806c640 msgr2=0x7efc181558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc1806c640 0x7efc181558d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc080af000 tx=0x7efc0801baf0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 shutdown_connections 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efc0406f400 0x7efc04071910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc18074780 0x7efc18150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc18074370 0x7efc18159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 --2- 172.21.15.67:0/142630978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc1806c640 0x7efc181558d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 >> 172.21.15.67:0/142630978 conn(0x7efc18070db0 msgr2=0x7efc180cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 shutdown_connections 2024-04-25T15:50:41.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.289+0000 7efc1e373640 1 -- 172.21.15.67:0/142630978 wait complete. 2024-04-25T15:50:41.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-04-25T15:50:41.308 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:41.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.341+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f45ac005860 con 0x7f45cc0caf60 2024-04-25T15:50:41.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.341+0000 7f45c1ffb640 1 -- 172.21.15.67:0/3611706151 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f45bc05b0c0 con 0x7f45cc0caf60 2024-04-25T15:50:41.344 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f45b006f630 msgr2=0x7f45b0071b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f45b006f630 0x7f45b0071b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f45c80af000 tx=0x7f45c80a7000 comp rx=0 tx=0).stop 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0caf60 msgr2=0x7f45cc15fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0caf60 0x7f45cc15fd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00c410 tx=0x7f45bc00c8d0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 shutdown_connections 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f45b006f630 0x7f45b0071b40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45cc15d790 0x7f45cc15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45cc0caf60 0x7f45cc15fd70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 --2- 172.21.15.67:0/3611706151 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45cc073bf0 0x7f45cc156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 >> 172.21.15.67:0/3611706151 conn(0x7f45cc070db0 msgr2=0x7f45cc15c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 shutdown_connections 2024-04-25T15:50:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.345+0000 7f45d3281640 1 -- 172.21.15.67:0/3611706151 wait complete. 2024-04-25T15:50:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.353+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7eff40005590 con 0x7eff600c35a0 2024-04-25T15:50:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.353+0000 7eff47fff640 1 -- 172.21.15.67:0/1707644045 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7eff5005f3d0 con 0x7eff600c35a0 2024-04-25T15:50:41.356 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff4c06f500 msgr2=0x7eff4c071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff4c06f500 0x7eff4c071a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7eff540079c0 tx=0x7eff54008040 comp rx=0 tx=0).stop 2024-04-25T15:50:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff600c35a0 msgr2=0x7eff600c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff600c35a0 0x7eff600c75c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff5000c580 tx=0x7eff5000ca40 comp rx=0 tx=0).stop 2024-04-25T15:50:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 shutdown_connections 2024-04-25T15:50:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff4c06f500 0x7eff4c071a10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff60161740 0x7eff60153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff600c35a0 0x7eff600c75c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 --2- 172.21.15.67:0/1707644045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff6006c640 0x7eff601fe0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 >> 172.21.15.67:0/1707644045 conn(0x7eff600ce1c0 msgr2=0x7eff600c96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 shutdown_connections 2024-04-25T15:50:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.357+0000 7eff45ffb640 1 -- 172.21.15.67:0/1707644045 wait complete. 2024-04-25T15:50:41.364 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2024-04-25T15:50:41.364 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:41.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-04-25T15:50:41.384 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.409+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0b3c0ca460 con 0x7f0b3c06c640 2024-04-25T15:50:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.409+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff96c005590 con 0x7ff98c0d6220 2024-04-25T15:50:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.409+0000 7f0b337fe640 1 -- 172.21.15.67:0/370786703 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0b2405ae20 con 0x7f0b3c06c640 2024-04-25T15:50:41.411 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:41.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.409+0000 7ff990ff9640 1 -- 172.21.15.67:0/2300498236 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff98801e830 con 0x7ff98c0d6220 2024-04-25T15:50:41.412 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-25T15:50:41.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff98006f430 msgr2=0x7ff980071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff98006f430 0x7ff980071940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff9840079c0 tx=0x7ff984008040 comp rx=0 tx=0).stop 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0d6220 msgr2=0x7ff98c0de9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0d6220 0x7ff98c0de9c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff988022750 tx=0x7ff9880218c0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 shutdown_connections 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff98006f430 0x7ff980071940 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff98c0dfaa0 0x7ff98c0c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c0d6220 0x7ff98c0de9c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 --2- 172.21.15.67:0/2300498236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff98c009100 0x7ff98c0ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 >> 172.21.15.67:0/2300498236 conn(0x7ff98c0514d0 msgr2=0x7ff98c046aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 shutdown_connections 2024-04-25T15:50:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7ff97a7fc640 1 -- 172.21.15.67:0/2300498236 wait complete. 2024-04-25T15:50:41.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0b1c06f430 msgr2=0x7f0b1c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0b1c06f430 0x7f0b1c071940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c007fd0 tx=0x7f0b2c007980 comp rx=0 tx=0).stop 2024-04-25T15:50:41.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 msgr2=0x7f0b3c152790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c152790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400c7f0 tx=0x7f0b2400ccb0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 shutdown_connections 2024-04-25T15:50:41.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0b1c06f430 0x7f0b1c071940 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0b3c161740 0x7f0b3c0cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0b3c0c35a0 0x7f0b3c1fda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 --2- 172.21.15.67:0/370786703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b3c06c640 0x7f0b3c152790 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 >> 172.21.15.67:0/370786703 conn(0x7f0b3c0ce1c0 msgr2=0x7f0b3c0c9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 shutdown_connections 2024-04-25T15:50:41.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.413+0000 7f0b413b1640 1 -- 172.21.15.67:0/370786703 wait complete. 2024-04-25T15:50:41.432 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-04-25T15:50:41.432 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:41.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.433+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6990179a20 con 0x7f6990049650 2024-04-25T15:50:41.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.433+0000 7f6994ff9640 1 -- 172.21.15.67:0/1331211785 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6998173d90 con 0x7f6990049650 2024-04-25T15:50:41.435 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f697c06f430 msgr2=0x7f697c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f697c06f430 0x7f697c071940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f69840014d0 tx=0x7f69840013e0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6990049650 msgr2=0x7f69900cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6990049650 0x7f69900cc040 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f699807dc80 tx=0x7f6988005840 comp rx=0 tx=0).stop 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 shutdown_connections 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f697c06f430 0x7f697c071940 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6990049650 0x7f69900cc040 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69900d8260 0x7f6990049110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 --2- 172.21.15.67:0/1331211785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6988005160 0x7f6990175a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 >> 172.21.15.67:0/1331211785 conn(0x7f69900514d0 msgr2=0x7f69900d5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 shutdown_connections 2024-04-25T15:50:41.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.437+0000 7f699d545640 1 -- 172.21.15.67:0/1331211785 wait complete. 2024-04-25T15:50:41.441 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-04-25T15:50:41.441 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:41.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-04-25T15:50:41.455 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:41.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:50:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.705+0000 7f67ac05a640 1 Processor -- start 2024-04-25T15:50:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.705+0000 7f67ac05a640 1 -- start start 2024-04-25T15:50:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67ac05a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a40c9740 0x7f67a4161740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67ac05a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a406c640 0x7f67a406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67ac05a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67a40c9b50 con 0x7f67a40c35a0 2024-04-25T15:50:41.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67ac05a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f67a4161c80 con 0x7f67a40c9740 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67ac05a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f67a4068bf0 con 0x7f67a406c640 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a95f5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f67a40c9740 0x7f67a4161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a9df6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f67a40c35a0 0x7f67a415ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36112/0 (socket says 172.21.15.67:36112) 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a9df6640 1 -- 172.21.15.67:0/4103102834 learned_addr learned my addr 172.21.15.67:0/4103102834 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67aa5f7640 1 --2- 172.21.15.67:0/4103102834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a406c640 0x7f67a406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a95f5640 1 -- 172.21.15.67:0/4103102834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a406c640 msgr2=0x7f67a406ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a95f5640 1 --2- 172.21.15.67:0/4103102834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a406c640 0x7f67a406ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a95f5640 1 -- 172.21.15.67:0/4103102834 >> v1:172.21.15.67:6789/0 conn(0x7f67a40c35a0 legacy=0x7f67a415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a8df4640 1 -- 172.21.15.67:0/4103102834 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3165845550 0 0) 0x7f67a40c9b50 con 0x7f67a40c35a0 2024-04-25T15:50:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a95f5640 1 -- 172.21.15.67:0/4103102834 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a4068630 con 0x7f67a40c9740 2024-04-25T15:50:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a95f5640 1 --2- 172.21.15.67:0/4103102834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a40c9740 0x7f67a4161740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f67940097a0 tx=0x7f679401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e601d4b3ec635619 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a8df4640 1 -- 172.21.15.67:0/4103102834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67940b0070 con 0x7f67a40c9740 2024-04-25T15:50:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a8df4640 1 -- 172.21.15.67:0/4103102834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67940ab600 con 0x7f67a40c9740 2024-04-25T15:50:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.709+0000 7f67a8df4640 1 -- 172.21.15.67:0/4103102834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67940ab9d0 con 0x7f67a40c9740 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- 172.21.15.67:0/4103102834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a40c9740 msgr2=0x7f67a4161740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 --2- 172.21.15.67:0/4103102834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a40c9740 0x7f67a4161740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f67940097a0 tx=0x7f679401c3f0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- 172.21.15.67:0/4103102834 shutdown_connections 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 --2- 172.21.15.67:0/4103102834 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a406c640 0x7f67a406ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 --2- 172.21.15.67:0/4103102834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a40c9740 0x7f67a4161740 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- 172.21.15.67:0/4103102834 >> 172.21.15.67:0/4103102834 conn(0x7f67a40ce1c0 msgr2=0x7f67a40ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- 172.21.15.67:0/4103102834 shutdown_connections 2024-04-25T15:50:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- 172.21.15.67:0/4103102834 wait complete. 2024-04-25T15:50:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 Processor -- start 2024-04-25T15:50:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- start start 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a406c640 0x7f67a415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67a40c35a0 0x7f67a40c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a40c9740 0x7f67a41fcc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f67a40ca970 con 0x7f67a406c640 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f67a40c9220 con 0x7f67a40c35a0 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67ac05a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f67a40caaf0 con 0x7f67a40c9740 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67a9df6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a406c640 0x7f67a415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67a9df6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a406c640 0x7f67a415b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60008/0 (socket says 172.21.15.67:60008) 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.713+0000 7f67a9df6640 1 -- 172.21.15.67:0/1228930583 learned_addr learned my addr 172.21.15.67:0/1228930583 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a95f5640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67a40c35a0 0x7f67a40c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67aa5f7640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a40c9740 0x7f67a41fcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a9df6640 1 -- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a40c9740 msgr2=0x7f67a41fcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a9df6640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a40c9740 0x7f67a41fcc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a9df6640 1 -- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67a40c35a0 msgr2=0x7f67a40c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a9df6640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67a40c35a0 0x7f67a40c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a9df6640 1 -- 172.21.15.67:0/1228930583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a415bce0 con 0x7f67a406c640 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a95f5640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67a40c35a0 0x7f67a40c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a9df6640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a406c640 0x7f67a415b7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c4b0 tx=0x7f67a000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f679a7fc640 1 -- 172.21.15.67:0/1228930583 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0018070 con 0x7f67a406c640 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f679a7fc640 1 -- 172.21.15.67:0/1228930583 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a0007500 con 0x7f67a406c640 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a4156b40 con 0x7f67a406c640 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f679a7fc640 1 -- 172.21.15.67:0/1228930583 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0004a30 con 0x7f67a406c640 2024-04-25T15:50:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a40c77c0 con 0x7f67a406c640 2024-04-25T15:50:41.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a406a480 con 0x7f67a406c640 2024-04-25T15:50:41.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f679a7fc640 1 -- 172.21.15.67:0/1228930583 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f67a001b070 con 0x7f67a406c640 2024-04-25T15:50:41.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f679a7fc640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f678806f400 0x7f6788071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:41.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f679a7fc640 1 -- 172.21.15.67:0/1228930583 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f67a008e5b0 con 0x7f67a406c640 2024-04-25T15:50:41.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.717+0000 7f67a95f5640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f678806f400 0x7f6788071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:41.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.721+0000 7f67a95f5640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f678806f400 0x7f6788071910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67940ad000 tx=0x7f6794008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:41.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.721+0000 7f679a7fc640 1 -- 172.21.15.67:0/1228930583 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f67a005b4a0 con 0x7f67a406c640 2024-04-25T15:50:41.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f67a4074fc0 con 0x7f67a406c640 2024-04-25T15:50:41.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f679a7fc640 1 -- 172.21.15.67:0/1228930583 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f67a005ada0 con 0x7f67a406c640 2024-04-25T15:50:41.882 INFO:teuthology.orchestra.run.smithi067.stdout:55834574855 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f678806f400 msgr2=0x7f6788071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f678806f400 0x7f6788071910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67940ad000 tx=0x7f6794008040 comp rx=0 tx=0).stop 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a406c640 msgr2=0x7f67a415b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a406c640 0x7f67a415b7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f67a000c4b0 tx=0x7f67a000c970 comp rx=0 tx=0).stop 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 shutdown_connections 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f678806f400 0x7f6788071910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67a40c9740 0x7f67a41fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67a40c35a0 0x7f67a40c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 --2- 172.21.15.67:0/1228930583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67a406c640 0x7f67a415b7a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 >> 172.21.15.67:0/1228930583 conn(0x7f67a40ce1c0 msgr2=0x7f67a415eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 shutdown_connections 2024-04-25T15:50:41.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:41.881+0000 7f67ac05a640 1 -- 172.21.15.67:0/1228930583 wait complete. 2024-04-25T15:50:41.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2024-04-25T15:50:42.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:50:43.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.065+0000 7f79fa2c0640 1 Processor -- start 2024-04-25T15:50:43.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.065+0000 7f79fa2c0640 1 -- start start 2024-04-25T15:50:43.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f4074790 0x7f79f415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79f415e020 0x7f79f40723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f4072dc0 con 0x7f79f40728e0 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f79f415b430 con 0x7f79f415e020 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79f415a670 con 0x7f79f4074790 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79f415e020 0x7f79f40723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79f415e020 0x7f79f40723a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36834/0 (socket says 172.21.15.67:36834) 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f885d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79f40728e0 0x7f79f40c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49846/0 (socket says 172.21.15.67:49846) 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 -- 172.21.15.67:0/2748055285 learned_addr learned my addr 172.21.15.67:0/2748055285 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f3fff640 1 --2- 172.21.15.67:0/2748055285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f4074790 0x7f79f415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 -- 172.21.15.67:0/2748055285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f4074790 msgr2=0x7f79f415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 --2- 172.21.15.67:0/2748055285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f4074790 0x7f79f415dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 -- 172.21.15.67:0/2748055285 >> v1:172.21.15.67:6789/0 conn(0x7f79f40728e0 legacy=0x7f79f40c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 -- 172.21.15.67:0/2748055285 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f415a950 con 0x7f79f415e020 2024-04-25T15:50:43.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f37fe640 1 --2- 172.21.15.67:0/2748055285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f415e020 0x7f79f40723a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009870 tx=0x7f79dc01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8830cca4570d106 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f2ffd640 1 -- 172.21.15.67:0/2748055285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc0b5070 con 0x7f79f415e020 2024-04-25T15:50:43.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f2ffd640 1 -- 172.21.15.67:0/2748055285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79dc0a8a50 con 0x7f79f415e020 2024-04-25T15:50:43.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79f2ffd640 1 -- 172.21.15.67:0/2748055285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc0a8e20 con 0x7f79f415e020 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- 172.21.15.67:0/2748055285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f415e020 msgr2=0x7f79f40723a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/2748055285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f415e020 0x7f79f40723a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009870 tx=0x7f79dc01ed50 comp rx=0 tx=0).stop 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- 172.21.15.67:0/2748055285 shutdown_connections 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/2748055285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f415e020 0x7f79f40723a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/2748055285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f4074790 0x7f79f415dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- 172.21.15.67:0/2748055285 >> 172.21.15.67:0/2748055285 conn(0x7f79f406fe70 msgr2=0x7f79f40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- 172.21.15.67:0/2748055285 shutdown_connections 2024-04-25T15:50:43.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.069+0000 7f79fa2c0640 1 -- 172.21.15.67:0/2748055285 wait complete. 2024-04-25T15:50:43.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 Processor -- start 2024-04-25T15:50:43.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 -- start start 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f40728e0 0x7f79f41521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79f4074790 0x7f79f414ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f415e020 0x7f79f4148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4073080 con 0x7f79f40728e0 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f79f415af10 con 0x7f79f4074790 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79f4074ba0 con 0x7f79f415e020 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f40728e0 0x7f79f41521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f40728e0 0x7f79f41521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36844/0 (socket says 172.21.15.67:36844) 2024-04-25T15:50:43.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 -- 172.21.15.67:0/1547327087 learned_addr learned my addr 172.21.15.67:0/1547327087 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 -- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f415e020 msgr2=0x7f79f4148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f415e020 0x7f79f4148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 -- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79f4074790 msgr2=0x7f79f414ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79f4074790 0x7f79f414ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 -- 172.21.15.67:0/1547327087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f41490f0 con 0x7f79f40728e0 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f3fff640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f40728e0 0x7f79f41521f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f79e400cd00 tx=0x7f79e400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f0ff9640 1 -- 172.21.15.67:0/1547327087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e40039a0 con 0x7f79f40728e0 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f0ff9640 1 -- 172.21.15.67:0/1547327087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e4003b40 con 0x7f79f40728e0 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79f0ff9640 1 -- 172.21.15.67:0/1547327087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4004020 con 0x7f79f40728e0 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f4149390 con 0x7f79f40728e0 2024-04-25T15:50:43.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.073+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f41f1080 con 0x7f79f40728e0 2024-04-25T15:50:43.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.077+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f406a480 con 0x7f79f40728e0 2024-04-25T15:50:43.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.077+0000 7f79f0ff9640 1 -- 172.21.15.67:0/1547327087 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f79e40041c0 con 0x7f79f40728e0 2024-04-25T15:50:43.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.077+0000 7f79f0ff9640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f79d406f430 0x7f79d4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.077+0000 7f79f0ff9640 1 -- 172.21.15.67:0/1547327087 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f79e408f720 con 0x7f79f40728e0 2024-04-25T15:50:43.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.077+0000 7f79f37fe640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f79d406f430 0x7f79d4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.077+0000 7f79f37fe640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f79d406f430 0x7f79d4071940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0b2000 tx=0x7f79dc004740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.081+0000 7f79f0ff9640 1 -- 172.21.15.67:0/1547327087 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f79e4014030 con 0x7f79f40728e0 2024-04-25T15:50:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.237+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f79f40c8e90 con 0x7f79f40728e0 2024-04-25T15:50:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.237+0000 7f79f0ff9640 1 -- 172.21.15.67:0/1547327087 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f79e405c610 con 0x7f79f40728e0 2024-04-25T15:50:43.241 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f79d406f430 msgr2=0x7f79d4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f79d406f430 0x7f79d4071940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0b2000 tx=0x7f79dc004740 comp rx=0 tx=0).stop 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f40728e0 msgr2=0x7f79f41521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f40728e0 0x7f79f41521f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f79e400cd00 tx=0x7f79e400a5c0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 shutdown_connections 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f79d406f430 0x7f79d4071940 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f79f415e020 0x7f79f4148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79f4074790 0x7f79f414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 --2- 172.21.15.67:0/1547327087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79f40728e0 0x7f79f41521f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 >> 172.21.15.67:0/1547327087 conn(0x7f79f406fe70 msgr2=0x7f79f414e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 shutdown_connections 2024-04-25T15:50:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.241+0000 7f79fa2c0640 1 -- 172.21.15.67:0/1547327087 wait complete. 2024-04-25T15:50:43.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-04-25T15:50:43.262 DEBUG:teuthology.parallel:result is None 2024-04-25T15:50:43.262 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T15:50:43.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:50:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7aa1a0640 1 Processor -- start 2024-04-25T15:50:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7aa1a0640 1 -- start start 2024-04-25T15:50:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7aa1a0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a4074370 0x7fb7a4074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7aa1a0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7a40c7fc0 0x7fb7a40c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7aa1a0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7a40c92d0 con 0x7fb7a406c640 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7aa1a0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb7a406e210 con 0x7fb7a40c7fc0 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7aa1a0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a406e4d0 con 0x7fb7a4074370 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7a40c7fc0 0x7fb7a40c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7a40c7fc0 0x7fb7a40c83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36856/0 (socket says 172.21.15.67:36856) 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a3fff640 1 -- 172.21.15.67:0/3623793074 learned_addr learned my addr 172.21.15.67:0/3623793074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a2ffd640 1 --2- 172.21.15.67:0/3623793074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a4074370 0x7fb7a4074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a3fff640 1 -- 172.21.15.67:0/3623793074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a4074370 msgr2=0x7fb7a4074780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a3fff640 1 --2- 172.21.15.67:0/3623793074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a4074370 0x7fb7a4074780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a3fff640 1 -- 172.21.15.67:0/3623793074 >> v1:172.21.15.176:6789/0 conn(0x7fb7a406c640 legacy=0x7fb7a415b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.401+0000 7fb7a3fff640 1 -- 172.21.15.67:0/3623793074 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a406e7b0 con 0x7fb7a40c7fc0 2024-04-25T15:50:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7a3fff640 1 --2- 172.21.15.67:0/3623793074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40c7fc0 0x7fb7a40c83d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb794009870 tx=0x7fb79401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57fd2e45fdde9316 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7a27fc640 1 -- 172.21.15.67:0/3623793074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7940b5070 con 0x7fb7a40c7fc0 2024-04-25T15:50:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7a27fc640 1 -- 172.21.15.67:0/3623793074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7940a89d0 con 0x7fb7a40c7fc0 2024-04-25T15:50:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7a27fc640 1 -- 172.21.15.67:0/3623793074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7940a8da0 con 0x7fb7a40c7fc0 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/3623793074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40c7fc0 msgr2=0x7fb7a40c83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/3623793074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40c7fc0 0x7fb7a40c83d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb794009870 tx=0x7fb79401ed10 comp rx=0 tx=0).stop 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/3623793074 shutdown_connections 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/3623793074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a40c7fc0 0x7fb7a40c83d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/3623793074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a4074370 0x7fb7a4074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/3623793074 >> 172.21.15.67:0/3623793074 conn(0x7fb7a4070db0 msgr2=0x7fb7a40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/3623793074 shutdown_connections 2024-04-25T15:50:43.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.405+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/3623793074 wait complete. 2024-04-25T15:50:43.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 Processor -- start 2024-04-25T15:50:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 -- start start 2024-04-25T15:50:43.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a406c640 0x7fb7a41e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4074370 0x7fb7a414e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a40c7fc0 0x7fb7a41e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a4073e50 con 0x7fb7a4074370 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7a4073b70 con 0x7fb7a406c640 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7aa1a0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7a4073420 con 0x7fb7a40c7fc0 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4074370 0x7fb7a414e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4074370 0x7fb7a414e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36870/0 (socket says 172.21.15.67:36870) 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 -- 172.21.15.67:0/1485729412 learned_addr learned my addr 172.21.15.67:0/1485729412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a3fff640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a40c7fc0 0x7fb7a41e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a37fe640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a406c640 0x7fb7a41e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 -- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a40c7fc0 msgr2=0x7fb7a41e3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a40c7fc0 0x7fb7a41e3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 -- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a406c640 msgr2=0x7fb7a41e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a406c640 0x7fb7a41e5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 -- 172.21.15.67:0/1485729412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7a41e6720 con 0x7fb7a4074370 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb7a2ffd640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4074370 0x7fb7a414e510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb79800d650 tx=0x7fb79800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.409+0000 7fb78bfff640 1 -- 172.21.15.67:0/1485729412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79800b660 con 0x7fb7a4074370 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb78bfff640 1 -- 172.21.15.67:0/1485729412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb798019070 con 0x7fb7a4074370 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7a41f12b0 con 0x7fb7a4074370 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb78bfff640 1 -- 172.21.15.67:0/1485729412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79800ba20 con 0x7fb7a4074370 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7a41f1870 con 0x7fb7a4074370 2024-04-25T15:50:43.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7a41e6490 con 0x7fb7a4074370 2024-04-25T15:50:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb78bfff640 1 -- 172.21.15.67:0/1485729412 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb79800bbc0 con 0x7fb7a4074370 2024-04-25T15:50:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb78bfff640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb79006f430 0x7fb790071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb78bfff640 1 -- 172.21.15.67:0/1485729412 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb79808f5f0 con 0x7fb7a4074370 2024-04-25T15:50:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb7a37fe640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb79006f430 0x7fb790071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.413+0000 7fb7a37fe640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb79006f430 0x7fb790071940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb7a406dea0 tx=0x7fb79c007240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.417+0000 7fb78bfff640 1 -- 172.21.15.67:0/1485729412 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb79805c4e0 con 0x7fb7a4074370 2024-04-25T15:50:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.569+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7a406a480 con 0x7fb79006f430 2024-04-25T15:50:43.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.589+0000 7fb78bfff640 1 -- 172.21.15.67:0/1485729412 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7fb7a406a480 con 0x7fb79006f430 2024-04-25T15:50:43.593 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:43.595 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:50:43.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb79006f430 msgr2=0x7fb790071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb79006f430 0x7fb790071940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb7a406dea0 tx=0x7fb79c007240 comp rx=0 tx=0).stop 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4074370 msgr2=0x7fb7a414e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4074370 0x7fb7a414e510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb79800d650 tx=0x7fb79800db10 comp rx=0 tx=0).stop 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 shutdown_connections 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb79006f430 0x7fb790071940 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7a40c7fc0 0x7fb7a41e3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7a4074370 0x7fb7a414e510 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 --2- 172.21.15.67:0/1485729412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a406c640 0x7fb7a41e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 >> 172.21.15.67:0/1485729412 conn(0x7fb7a4070db0 msgr2=0x7fb7a40cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 shutdown_connections 2024-04-25T15:50:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.593+0000 7fb7aa1a0640 1 -- 172.21.15.67:0/1485729412 wait complete. 2024-04-25T15:50:43.613 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-25T15:50:41.801491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218952,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755768,"statfs":{"total":773094113280,"available":772869906432,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004232"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750328+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:50:32.750328+0000","last_peered":"2024-04-25T15:50:32.750328+0000","last_clean":"2024-04-25T15:50:32.750328+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:50:32.750328+0000","last_undegraded":"2024-04-25T15:50:32.750328+0000","last_fullsized":"2024-04-25T15:50:32.750328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149188+0000","last_change":"2024-04-25T15:50:31.192224+0000","last_active":"2024-04-25T15:50:32.149188+0000","last_peered":"2024-04-25T15:50:32.149188+0000","last_clean":"2024-04-25T15:50:32.149188+0000","last_became_active":"2024-04-25T15:50:31.189311+0000","last_became_peered":"2024-04-25T15:50:31.189311+0000","last_unstale":"2024-04-25T15:50:32.149188+0000","last_undegraded":"2024-04-25T15:50:32.149188+0000","last_fullsized":"2024-04-25T15:50:32.149188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.837762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750252+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:50:32.750252+0000","last_peered":"2024-04-25T15:50:32.750252+0000","last_clean":"2024-04-25T15:50:32.750252+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:50:32.750252+0000","last_undegraded":"2024-04-25T15:50:32.750252+0000","last_fullsized":"2024-04-25T15:50:32.750252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152869+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:50:32.152869+0000","last_peered":"2024-04-25T15:50:32.152869+0000","last_clean":"2024-04-25T15:50:32.152869+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:50:32.152869+0000","last_undegraded":"2024-04-25T15:50:32.152869+0000","last_fullsized":"2024-04-25T15:50:32.152869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748897+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:50:32.748897+0000","last_peered":"2024-04-25T15:50:32.748897+0000","last_clean":"2024-04-25T15:50:32.748897+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:50:32.748897+0000","last_undegraded":"2024-04-25T15:50:32.748897+0000","last_fullsized":"2024-04-25T15:50:32.748897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151867+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:50:32.151867+0000","last_peered":"2024-04-25T15:50:32.151867+0000","last_clean":"2024-04-25T15:50:32.151867+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:50:32.151867+0000","last_undegraded":"2024-04-25T15:50:32.151867+0000","last_fullsized":"2024-04-25T15:50:32.151867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747896+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:50:32.747896+0000","last_peered":"2024-04-25T15:50:32.747896+0000","last_clean":"2024-04-25T15:50:32.747896+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:50:32.747896+0000","last_undegraded":"2024-04-25T15:50:32.747896+0000","last_fullsized":"2024-04-25T15:50:32.747896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246622+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:50:32.246622+0000","last_peered":"2024-04-25T15:50:32.246622+0000","last_clean":"2024-04-25T15:50:32.246622+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:50:32.246622+0000","last_undegraded":"2024-04-25T15:50:32.246622+0000","last_fullsized":"2024-04-25T15:50:32.246622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750587+0000","last_change":"2024-04-25T15:50:31.210825+0000","last_active":"2024-04-25T15:50:32.750587+0000","last_peered":"2024-04-25T15:50:32.750587+0000","last_clean":"2024-04-25T15:50:32.750587+0000","last_became_active":"2024-04-25T15:50:31.210510+0000","last_became_peered":"2024-04-25T15:50:31.210510+0000","last_unstale":"2024-04-25T15:50:32.750587+0000","last_undegraded":"2024-04-25T15:50:32.750587+0000","last_fullsized":"2024-04-25T15:50:32.750587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:23.810710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749861+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:50:32.749861+0000","last_peered":"2024-04-25T15:50:32.749861+0000","last_clean":"2024-04-25T15:50:32.749861+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:50:32.749861+0000","last_undegraded":"2024-04-25T15:50:32.749861+0000","last_fullsized":"2024-04-25T15:50:32.749861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750086+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:50:32.750086+0000","last_peered":"2024-04-25T15:50:32.750086+0000","last_clean":"2024-04-25T15:50:32.750086+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:50:32.750086+0000","last_undegraded":"2024-04-25T15:50:32.750086+0000","last_fullsized":"2024-04-25T15:50:32.750086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151449+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:50:32.151449+0000","last_peered":"2024-04-25T15:50:32.151449+0000","last_clean":"2024-04-25T15:50:32.151449+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:50:32.151449+0000","last_undegraded":"2024-04-25T15:50:32.151449+0000","last_fullsized":"2024-04-25T15:50:32.151449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247483+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:50:32.247483+0000","last_peered":"2024-04-25T15:50:32.247483+0000","last_clean":"2024-04-25T15:50:32.247483+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:50:32.247483+0000","last_undegraded":"2024-04-25T15:50:32.247483+0000","last_fullsized":"2024-04-25T15:50:32.247483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750096+0000","last_change":"2024-04-25T15:50:31.171300+0000","last_active":"2024-04-25T15:50:32.750096+0000","last_peered":"2024-04-25T15:50:32.750096+0000","last_clean":"2024-04-25T15:50:32.750096+0000","last_became_active":"2024-04-25T15:50:31.170451+0000","last_became_peered":"2024-04-25T15:50:31.170451+0000","last_unstale":"2024-04-25T15:50:32.750096+0000","last_undegraded":"2024-04-25T15:50:32.750096+0000","last_fullsized":"2024-04-25T15:50:32.750096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:52:45.807418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148500+0000","last_change":"2024-04-25T15:50:31.198817+0000","last_active":"2024-04-25T15:50:32.148500+0000","last_peered":"2024-04-25T15:50:32.148500+0000","last_clean":"2024-04-25T15:50:32.148500+0000","last_became_active":"2024-04-25T15:50:31.198165+0000","last_became_peered":"2024-04-25T15:50:31.198165+0000","last_unstale":"2024-04-25T15:50:32.148500+0000","last_undegraded":"2024-04-25T15:50:32.148500+0000","last_fullsized":"2024-04-25T15:50:32.148500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:05.893280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150141+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:50:32.150141+0000","last_peered":"2024-04-25T15:50:32.150141+0000","last_clean":"2024-04-25T15:50:32.150141+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:50:32.150141+0000","last_undegraded":"2024-04-25T15:50:32.150141+0000","last_fullsized":"2024-04-25T15:50:32.150141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154678+0000","last_change":"2024-04-25T15:50:31.198225+0000","last_active":"2024-04-25T15:50:32.154678+0000","last_peered":"2024-04-25T15:50:32.154678+0000","last_clean":"2024-04-25T15:50:32.154678+0000","last_became_active":"2024-04-25T15:50:31.196737+0000","last_became_peered":"2024-04-25T15:50:31.196737+0000","last_unstale":"2024-04-25T15:50:32.154678+0000","last_undegraded":"2024-04-25T15:50:32.154678+0000","last_fullsized":"2024-04-25T15:50:32.154678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:25.652638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748753+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:50:32.748753+0000","last_peered":"2024-04-25T15:50:32.748753+0000","last_clean":"2024-04-25T15:50:32.748753+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:50:32.748753+0000","last_undegraded":"2024-04-25T15:50:32.748753+0000","last_fullsized":"2024-04-25T15:50:32.748753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748982+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:50:32.748982+0000","last_peered":"2024-04-25T15:50:32.748982+0000","last_clean":"2024-04-25T15:50:32.748982+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:50:32.748982+0000","last_undegraded":"2024-04-25T15:50:32.748982+0000","last_fullsized":"2024-04-25T15:50:32.748982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749171+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:50:32.749171+0000","last_peered":"2024-04-25T15:50:32.749171+0000","last_clean":"2024-04-25T15:50:32.749171+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:50:32.749171+0000","last_undegraded":"2024-04-25T15:50:32.749171+0000","last_fullsized":"2024-04-25T15:50:32.749171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749435+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:50:32.749435+0000","last_peered":"2024-04-25T15:50:32.749435+0000","last_clean":"2024-04-25T15:50:32.749435+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:50:32.749435+0000","last_undegraded":"2024-04-25T15:50:32.749435+0000","last_fullsized":"2024-04-25T15:50:32.749435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152360+0000","last_change":"2024-04-25T15:50:31.198787+0000","last_active":"2024-04-25T15:50:32.152360+0000","last_peered":"2024-04-25T15:50:32.152360+0000","last_clean":"2024-04-25T15:50:32.152360+0000","last_became_active":"2024-04-25T15:50:31.197737+0000","last_became_peered":"2024-04-25T15:50:31.197737+0000","last_unstale":"2024-04-25T15:50:32.152360+0000","last_undegraded":"2024-04-25T15:50:32.152360+0000","last_fullsized":"2024-04-25T15:50:32.152360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:03:21.865150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246985+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:50:32.246985+0000","last_peered":"2024-04-25T15:50:32.246985+0000","last_clean":"2024-04-25T15:50:32.246985+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:50:32.246985+0000","last_undegraded":"2024-04-25T15:50:32.246985+0000","last_fullsized":"2024-04-25T15:50:32.246985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150206+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:50:32.150206+0000","last_peered":"2024-04-25T15:50:32.150206+0000","last_clean":"2024-04-25T15:50:32.150206+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:50:32.150206+0000","last_undegraded":"2024-04-25T15:50:32.150206+0000","last_fullsized":"2024-04-25T15:50:32.150206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247233+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:50:32.247233+0000","last_peered":"2024-04-25T15:50:32.247233+0000","last_clean":"2024-04-25T15:50:32.247233+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:50:32.247233+0000","last_undegraded":"2024-04-25T15:50:32.247233+0000","last_fullsized":"2024-04-25T15:50:32.247233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750349+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:50:32.750349+0000","last_peered":"2024-04-25T15:50:32.750349+0000","last_clean":"2024-04-25T15:50:32.750349+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:50:32.750349+0000","last_undegraded":"2024-04-25T15:50:32.750349+0000","last_fullsized":"2024-04-25T15:50:32.750349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149180+0000","last_change":"2024-04-25T15:50:31.192245+0000","last_active":"2024-04-25T15:50:32.149180+0000","last_peered":"2024-04-25T15:50:32.149180+0000","last_clean":"2024-04-25T15:50:32.149180+0000","last_became_active":"2024-04-25T15:50:31.190989+0000","last_became_peered":"2024-04-25T15:50:31.190989+0000","last_unstale":"2024-04-25T15:50:32.149180+0000","last_undegraded":"2024-04-25T15:50:32.149180+0000","last_fullsized":"2024-04-25T15:50:32.149180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:29:24.813097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247952+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:50:32.247952+0000","last_peered":"2024-04-25T15:50:32.247952+0000","last_clean":"2024-04-25T15:50:32.247952+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:50:32.247952+0000","last_undegraded":"2024-04-25T15:50:32.247952+0000","last_fullsized":"2024-04-25T15:50:32.247952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750022+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:50:32.750022+0000","last_peered":"2024-04-25T15:50:32.750022+0000","last_clean":"2024-04-25T15:50:32.750022+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:50:32.750022+0000","last_undegraded":"2024-04-25T15:50:32.750022+0000","last_fullsized":"2024-04-25T15:50:32.750022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153195+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:50:32.153195+0000","last_peered":"2024-04-25T15:50:32.153195+0000","last_clean":"2024-04-25T15:50:32.153195+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:50:32.153195+0000","last_undegraded":"2024-04-25T15:50:32.153195+0000","last_fullsized":"2024-04-25T15:50:32.153195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748323+0000","last_change":"2024-04-25T15:50:31.174913+0000","last_active":"2024-04-25T15:50:32.748323+0000","last_peered":"2024-04-25T15:50:32.748323+0000","last_clean":"2024-04-25T15:50:32.748323+0000","last_became_active":"2024-04-25T15:50:31.171499+0000","last_became_peered":"2024-04-25T15:50:31.171499+0000","last_unstale":"2024-04-25T15:50:32.748323+0000","last_undegraded":"2024-04-25T15:50:32.748323+0000","last_fullsized":"2024-04-25T15:50:32.748323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:34:06.773860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747841+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:50:32.747841+0000","last_peered":"2024-04-25T15:50:32.747841+0000","last_clean":"2024-04-25T15:50:32.747841+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:50:32.747841+0000","last_undegraded":"2024-04-25T15:50:32.747841+0000","last_fullsized":"2024-04-25T15:50:32.747841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150636+0000","last_change":"2024-04-25T15:50:31.174593+0000","last_active":"2024-04-25T15:50:32.150636+0000","last_peered":"2024-04-25T15:50:32.150636+0000","last_clean":"2024-04-25T15:50:32.150636+0000","last_became_active":"2024-04-25T15:50:31.173352+0000","last_became_peered":"2024-04-25T15:50:31.173352+0000","last_unstale":"2024-04-25T15:50:32.150636+0000","last_undegraded":"2024-04-25T15:50:32.150636+0000","last_fullsized":"2024-04-25T15:50:32.150636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:45.575622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.848556+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:50:32.848556+0000","last_peered":"2024-04-25T15:50:32.848556+0000","last_clean":"2024-04-25T15:50:32.848556+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:50:32.848556+0000","last_undegraded":"2024-04-25T15:50:32.848556+0000","last_fullsized":"2024-04-25T15:50:32.848556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750532+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:50:32.750532+0000","last_peered":"2024-04-25T15:50:32.750532+0000","last_clean":"2024-04-25T15:50:32.750532+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:50:32.750532+0000","last_undegraded":"2024-04-25T15:50:32.750532+0000","last_fullsized":"2024-04-25T15:50:32.750532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.847543+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:50:32.847543+0000","last_peered":"2024-04-25T15:50:32.847543+0000","last_clean":"2024-04-25T15:50:32.847543+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:50:32.847543+0000","last_undegraded":"2024-04-25T15:50:32.847543+0000","last_fullsized":"2024-04-25T15:50:32.847543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246857+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:50:32.246857+0000","last_peered":"2024-04-25T15:50:32.246857+0000","last_clean":"2024-04-25T15:50:32.246857+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:50:32.246857+0000","last_undegraded":"2024-04-25T15:50:32.246857+0000","last_fullsized":"2024-04-25T15:50:32.246857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246486+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:50:32.246486+0000","last_peered":"2024-04-25T15:50:32.246486+0000","last_clean":"2024-04-25T15:50:32.246486+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:50:32.246486+0000","last_undegraded":"2024-04-25T15:50:32.246486+0000","last_fullsized":"2024-04-25T15:50:32.246486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150445+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:50:32.150445+0000","last_peered":"2024-04-25T15:50:32.150445+0000","last_clean":"2024-04-25T15:50:32.150445+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:50:32.150445+0000","last_undegraded":"2024-04-25T15:50:32.150445+0000","last_fullsized":"2024-04-25T15:50:32.150445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148600+0000","last_change":"2024-04-25T15:50:31.182857+0000","last_active":"2024-04-25T15:50:32.148600+0000","last_peered":"2024-04-25T15:50:32.148600+0000","last_clean":"2024-04-25T15:50:32.148600+0000","last_became_active":"2024-04-25T15:50:31.181994+0000","last_became_peered":"2024-04-25T15:50:31.181994+0000","last_unstale":"2024-04-25T15:50:32.148600+0000","last_undegraded":"2024-04-25T15:50:32.148600+0000","last_fullsized":"2024-04-25T15:50:32.148600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:58.571421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.751547+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:50:32.751547+0000","last_peered":"2024-04-25T15:50:32.751547+0000","last_clean":"2024-04-25T15:50:32.751547+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:50:32.751547+0000","last_undegraded":"2024-04-25T15:50:32.751547+0000","last_fullsized":"2024-04-25T15:50:32.751547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154204+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:50:32.154204+0000","last_peered":"2024-04-25T15:50:32.154204+0000","last_clean":"2024-04-25T15:50:32.154204+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:50:32.154204+0000","last_undegraded":"2024-04-25T15:50:32.154204+0000","last_fullsized":"2024-04-25T15:50:32.154204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154135+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:50:32.154135+0000","last_peered":"2024-04-25T15:50:32.154135+0000","last_clean":"2024-04-25T15:50:32.154135+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:50:32.154135+0000","last_undegraded":"2024-04-25T15:50:32.154135+0000","last_fullsized":"2024-04-25T15:50:32.154135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749352+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:50:32.749352+0000","last_peered":"2024-04-25T15:50:32.749352+0000","last_clean":"2024-04-25T15:50:32.749352+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:50:32.749352+0000","last_undegraded":"2024-04-25T15:50:32.749352+0000","last_fullsized":"2024-04-25T15:50:32.749352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.225021+0000","last_change":"2024-04-25T15:50:30.200105+0000","last_active":"2024-04-25T15:50:36.225021+0000","last_peered":"2024-04-25T15:50:36.225021+0000","last_clean":"2024-04-25T15:50:36.225021+0000","last_became_active":"2024-04-25T15:50:30.199328+0000","last_became_peered":"2024-04-25T15:50:30.199328+0000","last_unstale":"2024-04-25T15:50:36.225021+0000","last_undegraded":"2024-04-25T15:50:36.225021+0000","last_fullsized":"2024-04-25T15:50:36.225021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:59.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151895+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:50:32.151895+0000","last_peered":"2024-04-25T15:50:32.151895+0000","last_clean":"2024-04-25T15:50:32.151895+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:50:32.151895+0000","last_undegraded":"2024-04-25T15:50:32.151895+0000","last_fullsized":"2024-04-25T15:50:32.151895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.225508+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:50:36.225508+0000","last_peered":"2024-04-25T15:50:36.225508+0000","last_clean":"2024-04-25T15:50:36.225508+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:50:36.225508+0000","last_undegraded":"2024-04-25T15:50:36.225508+0000","last_fullsized":"2024-04-25T15:50:36.225508+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748150+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:50:32.748150+0000","last_peered":"2024-04-25T15:50:32.748150+0000","last_clean":"2024-04-25T15:50:32.748150+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:50:32.748150+0000","last_undegraded":"2024-04-25T15:50:32.748150+0000","last_fullsized":"2024-04-25T15:50:32.748150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.221598+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:50:36.221598+0000","last_peered":"2024-04-25T15:50:36.221598+0000","last_clean":"2024-04-25T15:50:36.221598+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:50:36.221598+0000","last_undegraded":"2024-04-25T15:50:36.221598+0000","last_fullsized":"2024-04-25T15:50:36.221598+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750368+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:50:32.750368+0000","last_peered":"2024-04-25T15:50:32.750368+0000","last_clean":"2024-04-25T15:50:32.750368+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:50:32.750368+0000","last_undegraded":"2024-04-25T15:50:32.750368+0000","last_fullsized":"2024-04-25T15:50:32.750368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152764+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:50:32.152764+0000","last_peered":"2024-04-25T15:50:32.152764+0000","last_clean":"2024-04-25T15:50:32.152764+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:50:32.152764+0000","last_undegraded":"2024-04-25T15:50:32.152764+0000","last_fullsized":"2024-04-25T15:50:32.152764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748438+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:50:32.748438+0000","last_peered":"2024-04-25T15:50:32.748438+0000","last_clean":"2024-04-25T15:50:32.748438+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:50:32.748438+0000","last_undegraded":"2024-04-25T15:50:32.748438+0000","last_fullsized":"2024-04-25T15:50:32.748438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748758+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:50:32.748758+0000","last_peered":"2024-04-25T15:50:32.748758+0000","last_clean":"2024-04-25T15:50:32.748758+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:50:32.748758+0000","last_undegraded":"2024-04-25T15:50:32.748758+0000","last_fullsized":"2024-04-25T15:50:32.748758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748532+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:50:32.748532+0000","last_peered":"2024-04-25T15:50:32.748532+0000","last_clean":"2024-04-25T15:50:32.748532+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:50:32.748532+0000","last_undegraded":"2024-04-25T15:50:32.748532+0000","last_fullsized":"2024-04-25T15:50:32.748532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.228444+0000","last_change":"2024-04-25T15:50:30.179053+0000","last_active":"2024-04-25T15:50:36.228444+0000","last_peered":"2024-04-25T15:50:36.228444+0000","last_clean":"2024-04-25T15:50:36.228444+0000","last_became_active":"2024-04-25T15:50:30.178196+0000","last_became_peered":"2024-04-25T15:50:30.178196+0000","last_unstale":"2024-04-25T15:50:36.228444+0000","last_undegraded":"2024-04-25T15:50:36.228444+0000","last_fullsized":"2024-04-25T15:50:36.228444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:26.635717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748379+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:50:32.748379+0000","last_peered":"2024-04-25T15:50:32.748379+0000","last_clean":"2024-04-25T15:50:32.748379+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:50:32.748379+0000","last_undegraded":"2024-04-25T15:50:32.748379+0000","last_fullsized":"2024-04-25T15:50:32.748379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.830157+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:50:32.830157+0000","last_peered":"2024-04-25T15:50:32.830157+0000","last_clean":"2024-04-25T15:50:32.830157+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:50:32.830157+0000","last_undegraded":"2024-04-25T15:50:32.830157+0000","last_fullsized":"2024-04-25T15:50:32.830157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749720+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:50:32.749720+0000","last_peered":"2024-04-25T15:50:32.749720+0000","last_clean":"2024-04-25T15:50:32.749720+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:50:32.749720+0000","last_undegraded":"2024-04-25T15:50:32.749720+0000","last_fullsized":"2024-04-25T15:50:32.749720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.223834+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:50:36.223834+0000","last_peered":"2024-04-25T15:50:36.223834+0000","last_clean":"2024-04-25T15:50:36.223834+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:50:36.223834+0000","last_undegraded":"2024-04-25T15:50:36.223834+0000","last_fullsized":"2024-04-25T15:50:36.223834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750686+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:50:32.750686+0000","last_peered":"2024-04-25T15:50:32.750686+0000","last_clean":"2024-04-25T15:50:32.750686+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:50:32.750686+0000","last_undegraded":"2024-04-25T15:50:32.750686+0000","last_fullsized":"2024-04-25T15:50:32.750686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.846975+0000","last_change":"2024-04-25T15:50:30.192595+0000","last_active":"2024-04-25T15:50:32.846975+0000","last_peered":"2024-04-25T15:50:32.846975+0000","last_clean":"2024-04-25T15:50:32.846975+0000","last_became_active":"2024-04-25T15:50:30.192044+0000","last_became_peered":"2024-04-25T15:50:30.192044+0000","last_unstale":"2024-04-25T15:50:32.846975+0000","last_undegraded":"2024-04-25T15:50:32.846975+0000","last_fullsized":"2024-04-25T15:50:32.846975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:08.837774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153693+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:50:32.153693+0000","last_peered":"2024-04-25T15:50:32.153693+0000","last_clean":"2024-04-25T15:50:32.153693+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:50:32.153693+0000","last_undegraded":"2024-04-25T15:50:32.153693+0000","last_fullsized":"2024-04-25T15:50:32.153693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749938+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:50:32.749938+0000","last_peered":"2024-04-25T15:50:32.749938+0000","last_clean":"2024-04-25T15:50:32.749938+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:50:32.749938+0000","last_undegraded":"2024-04-25T15:50:32.749938+0000","last_fullsized":"2024-04-25T15:50:32.749938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748511+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:50:32.748511+0000","last_peered":"2024-04-25T15:50:32.748511+0000","last_clean":"2024-04-25T15:50:32.748511+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:50:32.748511+0000","last_undegraded":"2024-04-25T15:50:32.748511+0000","last_fullsized":"2024-04-25T15:50:32.748511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747640+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:50:32.747640+0000","last_peered":"2024-04-25T15:50:32.747640+0000","last_clean":"2024-04-25T15:50:32.747640+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:50:32.747640+0000","last_undegraded":"2024-04-25T15:50:32.747640+0000","last_fullsized":"2024-04-25T15:50:32.747640+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152008+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:50:32.152008+0000","last_peered":"2024-04-25T15:50:32.152008+0000","last_clean":"2024-04-25T15:50:32.152008+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:50:32.152008+0000","last_undegraded":"2024-04-25T15:50:32.152008+0000","last_fullsized":"2024-04-25T15:50:32.152008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247247+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:50:32.247247+0000","last_peered":"2024-04-25T15:50:32.247247+0000","last_clean":"2024-04-25T15:50:32.247247+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:50:32.247247+0000","last_undegraded":"2024-04-25T15:50:32.247247+0000","last_fullsized":"2024-04-25T15:50:32.247247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747456+0000","last_change":"2024-04-25T15:50:30.197359+0000","last_active":"2024-04-25T15:50:32.747456+0000","last_peered":"2024-04-25T15:50:32.747456+0000","last_clean":"2024-04-25T15:50:32.747456+0000","last_became_active":"2024-04-25T15:50:30.196640+0000","last_became_peered":"2024-04-25T15:50:30.196640+0000","last_unstale":"2024-04-25T15:50:32.747456+0000","last_undegraded":"2024-04-25T15:50:32.747456+0000","last_fullsized":"2024-04-25T15:50:32.747456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.928809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750609+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:50:32.750609+0000","last_peered":"2024-04-25T15:50:32.750609+0000","last_clean":"2024-04-25T15:50:32.750609+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:50:32.750609+0000","last_undegraded":"2024-04-25T15:50:32.750609+0000","last_fullsized":"2024-04-25T15:50:32.750609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748372+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:50:32.748372+0000","last_peered":"2024-04-25T15:50:32.748372+0000","last_clean":"2024-04-25T15:50:32.748372+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:50:32.748372+0000","last_undegraded":"2024-04-25T15:50:32.748372+0000","last_fullsized":"2024-04-25T15:50:32.748372+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148771+0000","last_change":"2024-04-25T15:50:31.171339+0000","last_active":"2024-04-25T15:50:32.148771+0000","last_peered":"2024-04-25T15:50:32.148771+0000","last_clean":"2024-04-25T15:50:32.148771+0000","last_became_active":"2024-04-25T15:50:31.170349+0000","last_became_peered":"2024-04-25T15:50:31.170349+0000","last_unstale":"2024-04-25T15:50:32.148771+0000","last_undegraded":"2024-04-25T15:50:32.148771+0000","last_fullsized":"2024-04-25T15:50:32.148771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:24.647984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153062+0000","last_change":"2024-04-25T15:50:30.195563+0000","last_active":"2024-04-25T15:50:32.153062+0000","last_peered":"2024-04-25T15:50:32.153062+0000","last_clean":"2024-04-25T15:50:32.153062+0000","last_became_active":"2024-04-25T15:50:30.194468+0000","last_became_peered":"2024-04-25T15:50:30.194468+0000","last_unstale":"2024-04-25T15:50:32.153062+0000","last_undegraded":"2024-04-25T15:50:32.153062+0000","last_fullsized":"2024-04-25T15:50:32.153062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:04.850096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151035+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:50:32.151035+0000","last_peered":"2024-04-25T15:50:32.151035+0000","last_clean":"2024-04-25T15:50:32.151035+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:50:32.151035+0000","last_undegraded":"2024-04-25T15:50:32.151035+0000","last_fullsized":"2024-04-25T15:50:32.151035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149161+0000","last_change":"2024-04-25T15:50:30.175727+0000","last_active":"2024-04-25T15:50:32.149161+0000","last_peered":"2024-04-25T15:50:32.149161+0000","last_clean":"2024-04-25T15:50:32.149161+0000","last_became_active":"2024-04-25T15:50:30.175136+0000","last_became_peered":"2024-04-25T15:50:30.175136+0000","last_unstale":"2024-04-25T15:50:32.149161+0000","last_undegraded":"2024-04-25T15:50:32.149161+0000","last_fullsized":"2024-04-25T15:50:32.149161+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:22.434474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153102+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:50:32.153102+0000","last_peered":"2024-04-25T15:50:32.153102+0000","last_clean":"2024-04-25T15:50:32.153102+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:50:32.153102+0000","last_undegraded":"2024-04-25T15:50:32.153102+0000","last_fullsized":"2024-04-25T15:50:32.153102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749240+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:50:32.749240+0000","last_peered":"2024-04-25T15:50:32.749240+0000","last_clean":"2024-04-25T15:50:32.749240+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:50:32.749240+0000","last_undegraded":"2024-04-25T15:50:32.749240+0000","last_fullsized":"2024-04-25T15:50:32.749240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149163+0000","last_change":"2024-04-25T15:50:31.177424+0000","last_active":"2024-04-25T15:50:32.149163+0000","last_peered":"2024-04-25T15:50:32.149163+0000","last_clean":"2024-04-25T15:50:32.149163+0000","last_became_active":"2024-04-25T15:50:31.176553+0000","last_became_peered":"2024-04-25T15:50:31.176553+0000","last_unstale":"2024-04-25T15:50:32.149163+0000","last_undegraded":"2024-04-25T15:50:32.149163+0000","last_fullsized":"2024-04-25T15:50:32.149163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:29.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149356+0000","last_change":"2024-04-25T15:50:30.191244+0000","last_active":"2024-04-25T15:50:32.149356+0000","last_peered":"2024-04-25T15:50:32.149356+0000","last_clean":"2024-04-25T15:50:32.149356+0000","last_became_active":"2024-04-25T15:50:30.190437+0000","last_became_peered":"2024-04-25T15:50:30.190437+0000","last_unstale":"2024-04-25T15:50:32.149356+0000","last_undegraded":"2024-04-25T15:50:32.149356+0000","last_fullsized":"2024-04-25T15:50:32.149356+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:55.506915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247702+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:50:32.247702+0000","last_peered":"2024-04-25T15:50:32.247702+0000","last_clean":"2024-04-25T15:50:32.247702+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:50:32.247702+0000","last_undegraded":"2024-04-25T15:50:32.247702+0000","last_fullsized":"2024-04-25T15:50:32.247702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.220622+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:50:36.220622+0000","last_peered":"2024-04-25T15:50:36.220622+0000","last_clean":"2024-04-25T15:50:36.220622+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:50:36.220622+0000","last_undegraded":"2024-04-25T15:50:36.220622+0000","last_fullsized":"2024-04-25T15:50:36.220622+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749239+0000","last_change":"2024-04-25T15:50:31.182607+0000","last_active":"2024-04-25T15:50:32.749239+0000","last_peered":"2024-04-25T15:50:32.749239+0000","last_clean":"2024-04-25T15:50:32.749239+0000","last_became_active":"2024-04-25T15:50:31.181641+0000","last_became_peered":"2024-04-25T15:50:31.181641+0000","last_unstale":"2024-04-25T15:50:32.749239+0000","last_undegraded":"2024-04-25T15:50:32.749239+0000","last_fullsized":"2024-04-25T15:50:32.749239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:49.636675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747485+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:50:32.747485+0000","last_peered":"2024-04-25T15:50:32.747485+0000","last_clean":"2024-04-25T15:50:32.747485+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:50:32.747485+0000","last_undegraded":"2024-04-25T15:50:32.747485+0000","last_fullsized":"2024-04-25T15:50:32.747485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151509+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:50:32.151509+0000","last_peered":"2024-04-25T15:50:32.151509+0000","last_clean":"2024-04-25T15:50:32.151509+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:50:32.151509+0000","last_undegraded":"2024-04-25T15:50:32.151509+0000","last_fullsized":"2024-04-25T15:50:32.151509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148622+0000","last_change":"2024-04-25T15:50:30.178078+0000","last_active":"2024-04-25T15:50:32.148622+0000","last_peered":"2024-04-25T15:50:32.148622+0000","last_clean":"2024-04-25T15:50:32.148622+0000","last_became_active":"2024-04-25T15:50:30.177718+0000","last_became_peered":"2024-04-25T15:50:30.177718+0000","last_unstale":"2024-04-25T15:50:32.148622+0000","last_undegraded":"2024-04-25T15:50:32.148622+0000","last_fullsized":"2024-04-25T15:50:32.148622+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246699+0000","last_change":"2024-04-25T15:50:31.192152+0000","last_active":"2024-04-25T15:50:32.246699+0000","last_peered":"2024-04-25T15:50:32.246699+0000","last_clean":"2024-04-25T15:50:32.246699+0000","last_became_active":"2024-04-25T15:50:31.179200+0000","last_became_peered":"2024-04-25T15:50:31.179200+0000","last_unstale":"2024-04-25T15:50:32.246699+0000","last_undegraded":"2024-04-25T15:50:32.246699+0000","last_fullsized":"2024-04-25T15:50:32.246699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:03.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148915+0000","last_change":"2024-04-25T15:50:30.184989+0000","last_active":"2024-04-25T15:50:32.148915+0000","last_peered":"2024-04-25T15:50:32.148915+0000","last_clean":"2024-04-25T15:50:32.148915+0000","last_became_active":"2024-04-25T15:50:30.184132+0000","last_became_peered":"2024-04-25T15:50:30.184132+0000","last_unstale":"2024-04-25T15:50:32.148915+0000","last_undegraded":"2024-04-25T15:50:32.148915+0000","last_fullsized":"2024-04-25T15:50:32.148915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:03.833303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247766+0000","last_change":"2024-04-25T15:50:31.190275+0000","last_active":"2024-04-25T15:50:32.247766+0000","last_peered":"2024-04-25T15:50:32.247766+0000","last_clean":"2024-04-25T15:50:32.247766+0000","last_became_active":"2024-04-25T15:50:31.189302+0000","last_became_peered":"2024-04-25T15:50:31.189302+0000","last_unstale":"2024-04-25T15:50:32.247766+0000","last_undegraded":"2024-04-25T15:50:32.247766+0000","last_fullsized":"2024-04-25T15:50:32.247766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:39.479706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.217530+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:50:36.217530+0000","last_peered":"2024-04-25T15:50:36.217530+0000","last_clean":"2024-04-25T15:50:36.217530+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:50:36.217530+0000","last_undegraded":"2024-04-25T15:50:36.217530+0000","last_fullsized":"2024-04-25T15:50:36.217530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748648+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:50:32.748648+0000","last_peered":"2024-04-25T15:50:32.748648+0000","last_clean":"2024-04-25T15:50:32.748648+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:50:32.748648+0000","last_undegraded":"2024-04-25T15:50:32.748648+0000","last_fullsized":"2024-04-25T15:50:32.748648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153111+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:50:32.153111+0000","last_peered":"2024-04-25T15:50:32.153111+0000","last_clean":"2024-04-25T15:50:32.153111+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:50:32.153111+0000","last_undegraded":"2024-04-25T15:50:32.153111+0000","last_fullsized":"2024-04-25T15:50:32.153111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.214957+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:50:36.214957+0000","last_peered":"2024-04-25T15:50:36.214957+0000","last_clean":"2024-04-25T15:50:36.214957+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:50:36.214957+0000","last_undegraded":"2024-04-25T15:50:36.214957+0000","last_fullsized":"2024-04-25T15:50:36.214957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153450+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:50:32.153450+0000","last_peered":"2024-04-25T15:50:32.153450+0000","last_clean":"2024-04-25T15:50:32.153450+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:50:32.153450+0000","last_undegraded":"2024-04-25T15:50:32.153450+0000","last_fullsized":"2024-04-25T15:50:32.153450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.220304+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:50:36.220304+0000","last_peered":"2024-04-25T15:50:36.220304+0000","last_clean":"2024-04-25T15:50:36.220304+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:50:36.220304+0000","last_undegraded":"2024-04-25T15:50:36.220304+0000","last_fullsized":"2024-04-25T15:50:36.220304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749087+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:50:32.749087+0000","last_peered":"2024-04-25T15:50:32.749087+0000","last_clean":"2024-04-25T15:50:32.749087+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:50:32.749087+0000","last_undegraded":"2024-04-25T15:50:32.749087+0000","last_fullsized":"2024-04-25T15:50:32.749087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.848640+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:50:32.848640+0000","last_peered":"2024-04-25T15:50:32.848640+0000","last_clean":"2024-04-25T15:50:32.848640+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:50:32.848640+0000","last_undegraded":"2024-04-25T15:50:32.848640+0000","last_fullsized":"2024-04-25T15:50:32.848640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247337+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:50:32.247337+0000","last_peered":"2024-04-25T15:50:32.247337+0000","last_clean":"2024-04-25T15:50:32.247337+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:50:32.247337+0000","last_undegraded":"2024-04-25T15:50:32.247337+0000","last_fullsized":"2024-04-25T15:50:32.247337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.226700+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:50:36.226700+0000","last_peered":"2024-04-25T15:50:36.226700+0000","last_clean":"2024-04-25T15:50:36.226700+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:50:36.226700+0000","last_undegraded":"2024-04-25T15:50:36.226700+0000","last_fullsized":"2024-04-25T15:50:36.226700+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:50:43.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:50:43.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3f321640 1 Processor -- start 2024-04-25T15:50:43.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3f321640 1 -- start start 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3f321640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8e380c7610 0x7f8e380c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3f321640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e380c6890 con 0x7f8e380c5bb0 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3f321640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e380c6b70 con 0x7f8e38071b20 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3f321640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8e3815cd40 con 0x7f8e380c7610 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3d8be640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8e380c5bb0 0x7f8e38163d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49854/0 (socket says 172.21.15.67:49854) 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3d8be640 1 -- 172.21.15.67:0/253028927 learned_addr learned my addr 172.21.15.67:0/253028927 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292789646 0 0) 0x7f8e380c6890 con 0x7f8e380c5bb0 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e24003530 con 0x7f8e380c5bb0 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e3c8bc640 1 --2- 172.21.15.67:0/253028927 >> v2:172.21.15.176:3300/0 conn(0x7f8e380c7610 0x7f8e380c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1182033115 0 0) 0x7f8e24003530 con 0x7f8e380c5bb0 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 >> v2:172.21.15.176:3300/0 conn(0x7f8e380c7610 msgr2=0x7f8e380c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e2ffff640 1 --2- 172.21.15.67:0/253028927 >> v2:172.21.15.176:3300/0 conn(0x7f8e380c7610 0x7f8e380c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 >> v1:172.21.15.176:6789/0 conn(0x7f8e38071b20 legacy=0x7f8e3815d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.805+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e380c6e50 con 0x7f8e380c5bb0 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8e30001a00 con 0x7f8e380c5bb0 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 >> v1:172.21.15.67:6789/0 conn(0x7f8e380c5bb0 legacy=0x7f8e38163d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240039c0 0x7f8e24003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e24004810 0x7f8e24004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e240051d0 0x7f8e2400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3815cd40 con 0x7f8e240051d0 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2400c170 con 0x7f8e240039c0 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e2400c370 con 0x7f8e24004810 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e240051d0 0x7f8e2400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d0bd640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240039c0 0x7f8e24003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3c8bc640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e24004810 0x7f8e24004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 -- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e24004810 msgr2=0x7f8e24004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e24004810 0x7f8e24004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 -- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240039c0 msgr2=0x7f8e24003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240039c0 0x7f8e24003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 -- 172.21.15.67:0/253028927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e24003530 con 0x7f8e240051d0 2024-04-25T15:50:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 -- 172.21.15.67:0/253028927 reap_dead start 2024-04-25T15:50:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3d8be640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e240051d0 0x7f8e2400b9d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000a810 tx=0x7f8e3001d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44463b97ad07c67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000a460 con 0x7f8e240051d0 2024-04-25T15:50:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e300ac640 con 0x7f8e240051d0 2024-04-25T15:50:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e2ffff640 1 -- 172.21.15.67:0/253028927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e300a5e60 con 0x7f8e240051d0 2024-04-25T15:50:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3f321640 1 -- 172.21.15.67:0/253028927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e240051d0 msgr2=0x7f8e2400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3f321640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e240051d0 0x7f8e2400b9d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000a810 tx=0x7f8e3001d280 comp rx=0 tx=0).stop 2024-04-25T15:50:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3f321640 1 -- 172.21.15.67:0/253028927 shutdown_connections 2024-04-25T15:50:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3f321640 1 --2- 172.21.15.67:0/253028927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e240051d0 0x7f8e2400b9d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3f321640 1 -- 172.21.15.67:0/253028927 >> 172.21.15.67:0/253028927 conn(0x7f8e3806fe70 msgr2=0x7f8e380cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3f321640 1 -- 172.21.15.67:0/253028927 shutdown_connections 2024-04-25T15:50:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.809+0000 7f8e3f321640 1 -- 172.21.15.67:0/253028927 wait complete. 2024-04-25T15:50:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 Processor -- start 2024-04-25T15:50:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 -- start start 2024-04-25T15:50:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240051d0 0x7f8e381f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3815c250 0x7f8e381e2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e38155180 0x7f8e38155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3815c9d0 con 0x7f8e3815c250 2024-04-25T15:50:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8e380c66f0 con 0x7f8e240051d0 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8e380734f0 con 0x7f8e38155180 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3815c250 0x7f8e381e2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3815c250 0x7f8e381e2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36900/0 (socket says 172.21.15.67:36900) 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 -- 172.21.15.67:0/4218547183 learned_addr learned my addr 172.21.15.67:0/4218547183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3d8be640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e38155180 0x7f8e38155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3d0bd640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240051d0 0x7f8e381f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 -- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e38155180 msgr2=0x7f8e38155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e38155180 0x7f8e38155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 -- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240051d0 msgr2=0x7f8e381f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240051d0 0x7f8e381f38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 -- 172.21.15.67:0/4218547183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e38159e00 con 0x7f8e3815c250 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3d8be640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e38155180 0x7f8e38155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3c8bc640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3815c250 0x7f8e381e2530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e280107e0 tx=0x7f8e28010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e2dffb640 1 -- 172.21.15.67:0/4218547183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800f040 con 0x7f8e3815c250 2024-04-25T15:50:43.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e2dffb640 1 -- 172.21.15.67:0/4218547183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e2800eb60 con 0x7f8e3815c250 2024-04-25T15:50:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3815a0e0 con 0x7f8e3815c250 2024-04-25T15:50:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e2dffb640 1 -- 172.21.15.67:0/4218547183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2801c400 con 0x7f8e3815c250 2024-04-25T15:50:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.813+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e3815a5a0 con 0x7f8e3815c250 2024-04-25T15:50:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.817+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e3806a480 con 0x7f8e3815c250 2024-04-25T15:50:43.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.817+0000 7f8e2dffb640 1 -- 172.21.15.67:0/4218547183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8e2801c5a0 con 0x7f8e3815c250 2024-04-25T15:50:43.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.817+0000 7f8e2dffb640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e1c06f400 0x7f8e1c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:43.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.817+0000 7f8e2dffb640 1 -- 172.21.15.67:0/4218547183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8e2808f930 con 0x7f8e3815c250 2024-04-25T15:50:43.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.817+0000 7f8e3d0bd640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e1c06f400 0x7f8e1c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:43.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.817+0000 7f8e3d0bd640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e1c06f400 0x7f8e1c071910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8e380c5960 tx=0x7f8e34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:43.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.821+0000 7f8e2dffb640 1 -- 172.21.15.67:0/4218547183 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8e28018030 con 0x7f8e3815c250 2024-04-25T15:50:43.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:43.981+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8e38077130 con 0x7f8e1c06f400 2024-04-25T15:50:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.001+0000 7f8e2dffb640 1 -- 172.21.15.67:0/4218547183 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251964 (secure 0 0 0) 0x7f8e38077130 con 0x7f8e1c06f400 2024-04-25T15:50:44.005 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:44.007 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e1c06f400 msgr2=0x7f8e1c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e1c06f400 0x7f8e1c071910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8e380c5960 tx=0x7f8e34009040 comp rx=0 tx=0).stop 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3815c250 msgr2=0x7f8e381e2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3815c250 0x7f8e381e2530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e280107e0 tx=0x7f8e28010ca0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 shutdown_connections 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8e1c06f400 0x7f8e1c071910 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8e38155180 0x7f8e38155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3815c250 0x7f8e381e2530 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 --2- 172.21.15.67:0/4218547183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8e240051d0 0x7f8e381f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 >> 172.21.15.67:0/4218547183 conn(0x7f8e3806fe70 msgr2=0x7f8e380ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 shutdown_connections 2024-04-25T15:50:44.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.005+0000 7f8e3f321640 1 -- 172.21.15.67:0/4218547183 wait complete. 2024-04-25T15:50:44.027 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-25T15:50:43.802311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594094,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":184,"num_write_kb":1653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218988,"kb_used_data":2860,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755732,"statfs":{"total":773094113280,"available":772869869568,"internally_reserved":0,"allocated":2928640,"data_stored":1755078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3726,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.656246"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750328+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:50:32.750328+0000","last_peered":"2024-04-25T15:50:32.750328+0000","last_clean":"2024-04-25T15:50:32.750328+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:50:32.750328+0000","last_undegraded":"2024-04-25T15:50:32.750328+0000","last_fullsized":"2024-04-25T15:50:32.750328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149188+0000","last_change":"2024-04-25T15:50:31.192224+0000","last_active":"2024-04-25T15:50:32.149188+0000","last_peered":"2024-04-25T15:50:32.149188+0000","last_clean":"2024-04-25T15:50:32.149188+0000","last_became_active":"2024-04-25T15:50:31.189311+0000","last_became_peered":"2024-04-25T15:50:31.189311+0000","last_unstale":"2024-04-25T15:50:32.149188+0000","last_undegraded":"2024-04-25T15:50:32.149188+0000","last_fullsized":"2024-04-25T15:50:32.149188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.837762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750252+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:50:32.750252+0000","last_peered":"2024-04-25T15:50:32.750252+0000","last_clean":"2024-04-25T15:50:32.750252+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:50:32.750252+0000","last_undegraded":"2024-04-25T15:50:32.750252+0000","last_fullsized":"2024-04-25T15:50:32.750252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152869+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:50:32.152869+0000","last_peered":"2024-04-25T15:50:32.152869+0000","last_clean":"2024-04-25T15:50:32.152869+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:50:32.152869+0000","last_undegraded":"2024-04-25T15:50:32.152869+0000","last_fullsized":"2024-04-25T15:50:32.152869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748897+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:50:32.748897+0000","last_peered":"2024-04-25T15:50:32.748897+0000","last_clean":"2024-04-25T15:50:32.748897+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:50:32.748897+0000","last_undegraded":"2024-04-25T15:50:32.748897+0000","last_fullsized":"2024-04-25T15:50:32.748897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151867+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:50:32.151867+0000","last_peered":"2024-04-25T15:50:32.151867+0000","last_clean":"2024-04-25T15:50:32.151867+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:50:32.151867+0000","last_undegraded":"2024-04-25T15:50:32.151867+0000","last_fullsized":"2024-04-25T15:50:32.151867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747896+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:50:32.747896+0000","last_peered":"2024-04-25T15:50:32.747896+0000","last_clean":"2024-04-25T15:50:32.747896+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:50:32.747896+0000","last_undegraded":"2024-04-25T15:50:32.747896+0000","last_fullsized":"2024-04-25T15:50:32.747896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246622+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:50:32.246622+0000","last_peered":"2024-04-25T15:50:32.246622+0000","last_clean":"2024-04-25T15:50:32.246622+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:50:32.246622+0000","last_undegraded":"2024-04-25T15:50:32.246622+0000","last_fullsized":"2024-04-25T15:50:32.246622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750587+0000","last_change":"2024-04-25T15:50:31.210825+0000","last_active":"2024-04-25T15:50:32.750587+0000","last_peered":"2024-04-25T15:50:32.750587+0000","last_clean":"2024-04-25T15:50:32.750587+0000","last_became_active":"2024-04-25T15:50:31.210510+0000","last_became_peered":"2024-04-25T15:50:31.210510+0000","last_unstale":"2024-04-25T15:50:32.750587+0000","last_undegraded":"2024-04-25T15:50:32.750587+0000","last_fullsized":"2024-04-25T15:50:32.750587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:23.810710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749861+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:50:32.749861+0000","last_peered":"2024-04-25T15:50:32.749861+0000","last_clean":"2024-04-25T15:50:32.749861+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:50:32.749861+0000","last_undegraded":"2024-04-25T15:50:32.749861+0000","last_fullsized":"2024-04-25T15:50:32.749861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750086+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:50:32.750086+0000","last_peered":"2024-04-25T15:50:32.750086+0000","last_clean":"2024-04-25T15:50:32.750086+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:50:32.750086+0000","last_undegraded":"2024-04-25T15:50:32.750086+0000","last_fullsized":"2024-04-25T15:50:32.750086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151449+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:50:32.151449+0000","last_peered":"2024-04-25T15:50:32.151449+0000","last_clean":"2024-04-25T15:50:32.151449+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:50:32.151449+0000","last_undegraded":"2024-04-25T15:50:32.151449+0000","last_fullsized":"2024-04-25T15:50:32.151449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247483+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:50:32.247483+0000","last_peered":"2024-04-25T15:50:32.247483+0000","last_clean":"2024-04-25T15:50:32.247483+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:50:32.247483+0000","last_undegraded":"2024-04-25T15:50:32.247483+0000","last_fullsized":"2024-04-25T15:50:32.247483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750096+0000","last_change":"2024-04-25T15:50:31.171300+0000","last_active":"2024-04-25T15:50:32.750096+0000","last_peered":"2024-04-25T15:50:32.750096+0000","last_clean":"2024-04-25T15:50:32.750096+0000","last_became_active":"2024-04-25T15:50:31.170451+0000","last_became_peered":"2024-04-25T15:50:31.170451+0000","last_unstale":"2024-04-25T15:50:32.750096+0000","last_undegraded":"2024-04-25T15:50:32.750096+0000","last_fullsized":"2024-04-25T15:50:32.750096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:52:45.807418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148500+0000","last_change":"2024-04-25T15:50:31.198817+0000","last_active":"2024-04-25T15:50:32.148500+0000","last_peered":"2024-04-25T15:50:32.148500+0000","last_clean":"2024-04-25T15:50:32.148500+0000","last_became_active":"2024-04-25T15:50:31.198165+0000","last_became_peered":"2024-04-25T15:50:31.198165+0000","last_unstale":"2024-04-25T15:50:32.148500+0000","last_undegraded":"2024-04-25T15:50:32.148500+0000","last_fullsized":"2024-04-25T15:50:32.148500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:05.893280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150141+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:50:32.150141+0000","last_peered":"2024-04-25T15:50:32.150141+0000","last_clean":"2024-04-25T15:50:32.150141+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:50:32.150141+0000","last_undegraded":"2024-04-25T15:50:32.150141+0000","last_fullsized":"2024-04-25T15:50:32.150141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154678+0000","last_change":"2024-04-25T15:50:31.198225+0000","last_active":"2024-04-25T15:50:32.154678+0000","last_peered":"2024-04-25T15:50:32.154678+0000","last_clean":"2024-04-25T15:50:32.154678+0000","last_became_active":"2024-04-25T15:50:31.196737+0000","last_became_peered":"2024-04-25T15:50:31.196737+0000","last_unstale":"2024-04-25T15:50:32.154678+0000","last_undegraded":"2024-04-25T15:50:32.154678+0000","last_fullsized":"2024-04-25T15:50:32.154678+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:25.652638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748753+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:50:32.748753+0000","last_peered":"2024-04-25T15:50:32.748753+0000","last_clean":"2024-04-25T15:50:32.748753+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:50:32.748753+0000","last_undegraded":"2024-04-25T15:50:32.748753+0000","last_fullsized":"2024-04-25T15:50:32.748753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748982+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:50:32.748982+0000","last_peered":"2024-04-25T15:50:32.748982+0000","last_clean":"2024-04-25T15:50:32.748982+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:50:32.748982+0000","last_undegraded":"2024-04-25T15:50:32.748982+0000","last_fullsized":"2024-04-25T15:50:32.748982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749171+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:50:32.749171+0000","last_peered":"2024-04-25T15:50:32.749171+0000","last_clean":"2024-04-25T15:50:32.749171+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:50:32.749171+0000","last_undegraded":"2024-04-25T15:50:32.749171+0000","last_fullsized":"2024-04-25T15:50:32.749171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749435+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:50:32.749435+0000","last_peered":"2024-04-25T15:50:32.749435+0000","last_clean":"2024-04-25T15:50:32.749435+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:50:32.749435+0000","last_undegraded":"2024-04-25T15:50:32.749435+0000","last_fullsized":"2024-04-25T15:50:32.749435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.152360+0000","last_change":"2024-04-25T15:50:31.198787+0000","last_active":"2024-04-25T15:50:32.152360+0000","last_peered":"2024-04-25T15:50:32.152360+0000","last_clean":"2024-04-25T15:50:32.152360+0000","last_became_active":"2024-04-25T15:50:31.197737+0000","last_became_peered":"2024-04-25T15:50:31.197737+0000","last_unstale":"2024-04-25T15:50:32.152360+0000","last_undegraded":"2024-04-25T15:50:32.152360+0000","last_fullsized":"2024-04-25T15:50:32.152360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:03:21.865150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246985+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:50:32.246985+0000","last_peered":"2024-04-25T15:50:32.246985+0000","last_clean":"2024-04-25T15:50:32.246985+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:50:32.246985+0000","last_undegraded":"2024-04-25T15:50:32.246985+0000","last_fullsized":"2024-04-25T15:50:32.246985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150206+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:50:32.150206+0000","last_peered":"2024-04-25T15:50:32.150206+0000","last_clean":"2024-04-25T15:50:32.150206+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:50:32.150206+0000","last_undegraded":"2024-04-25T15:50:32.150206+0000","last_fullsized":"2024-04-25T15:50:32.150206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247233+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:50:32.247233+0000","last_peered":"2024-04-25T15:50:32.247233+0000","last_clean":"2024-04-25T15:50:32.247233+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:50:32.247233+0000","last_undegraded":"2024-04-25T15:50:32.247233+0000","last_fullsized":"2024-04-25T15:50:32.247233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750349+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:50:32.750349+0000","last_peered":"2024-04-25T15:50:32.750349+0000","last_clean":"2024-04-25T15:50:32.750349+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:50:32.750349+0000","last_undegraded":"2024-04-25T15:50:32.750349+0000","last_fullsized":"2024-04-25T15:50:32.750349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149180+0000","last_change":"2024-04-25T15:50:31.192245+0000","last_active":"2024-04-25T15:50:32.149180+0000","last_peered":"2024-04-25T15:50:32.149180+0000","last_clean":"2024-04-25T15:50:32.149180+0000","last_became_active":"2024-04-25T15:50:31.190989+0000","last_became_peered":"2024-04-25T15:50:31.190989+0000","last_unstale":"2024-04-25T15:50:32.149180+0000","last_undegraded":"2024-04-25T15:50:32.149180+0000","last_fullsized":"2024-04-25T15:50:32.149180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:29:24.813097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247952+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:50:32.247952+0000","last_peered":"2024-04-25T15:50:32.247952+0000","last_clean":"2024-04-25T15:50:32.247952+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:50:32.247952+0000","last_undegraded":"2024-04-25T15:50:32.247952+0000","last_fullsized":"2024-04-25T15:50:32.247952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750022+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:50:32.750022+0000","last_peered":"2024-04-25T15:50:32.750022+0000","last_clean":"2024-04-25T15:50:32.750022+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:50:32.750022+0000","last_undegraded":"2024-04-25T15:50:32.750022+0000","last_fullsized":"2024-04-25T15:50:32.750022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153195+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:50:32.153195+0000","last_peered":"2024-04-25T15:50:32.153195+0000","last_clean":"2024-04-25T15:50:32.153195+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:50:32.153195+0000","last_undegraded":"2024-04-25T15:50:32.153195+0000","last_fullsized":"2024-04-25T15:50:32.153195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748323+0000","last_change":"2024-04-25T15:50:31.174913+0000","last_active":"2024-04-25T15:50:32.748323+0000","last_peered":"2024-04-25T15:50:32.748323+0000","last_clean":"2024-04-25T15:50:32.748323+0000","last_became_active":"2024-04-25T15:50:31.171499+0000","last_became_peered":"2024-04-25T15:50:31.171499+0000","last_unstale":"2024-04-25T15:50:32.748323+0000","last_undegraded":"2024-04-25T15:50:32.748323+0000","last_fullsized":"2024-04-25T15:50:32.748323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:34:06.773860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747841+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:50:32.747841+0000","last_peered":"2024-04-25T15:50:32.747841+0000","last_clean":"2024-04-25T15:50:32.747841+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:50:32.747841+0000","last_undegraded":"2024-04-25T15:50:32.747841+0000","last_fullsized":"2024-04-25T15:50:32.747841+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.150636+0000","last_change":"2024-04-25T15:50:31.174593+0000","last_active":"2024-04-25T15:50:32.150636+0000","last_peered":"2024-04-25T15:50:32.150636+0000","last_clean":"2024-04-25T15:50:32.150636+0000","last_became_active":"2024-04-25T15:50:31.173352+0000","last_became_peered":"2024-04-25T15:50:31.173352+0000","last_unstale":"2024-04-25T15:50:32.150636+0000","last_undegraded":"2024-04-25T15:50:32.150636+0000","last_fullsized":"2024-04-25T15:50:32.150636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:45.575622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.263469+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:50:42.263469+0000","last_peered":"2024-04-25T15:50:42.263469+0000","last_clean":"2024-04-25T15:50:42.263469+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:50:42.263469+0000","last_undegraded":"2024-04-25T15:50:42.263469+0000","last_fullsized":"2024-04-25T15:50:42.263469+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750532+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:50:32.750532+0000","last_peered":"2024-04-25T15:50:32.750532+0000","last_clean":"2024-04-25T15:50:32.750532+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:50:32.750532+0000","last_undegraded":"2024-04-25T15:50:32.750532+0000","last_fullsized":"2024-04-25T15:50:32.750532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.847543+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:50:32.847543+0000","last_peered":"2024-04-25T15:50:32.847543+0000","last_clean":"2024-04-25T15:50:32.847543+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:50:32.847543+0000","last_undegraded":"2024-04-25T15:50:32.847543+0000","last_fullsized":"2024-04-25T15:50:32.847543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246857+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:50:32.246857+0000","last_peered":"2024-04-25T15:50:32.246857+0000","last_clean":"2024-04-25T15:50:32.246857+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:50:32.246857+0000","last_undegraded":"2024-04-25T15:50:32.246857+0000","last_fullsized":"2024-04-25T15:50:32.246857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246486+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:50:32.246486+0000","last_peered":"2024-04-25T15:50:32.246486+0000","last_clean":"2024-04-25T15:50:32.246486+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:50:32.246486+0000","last_undegraded":"2024-04-25T15:50:32.246486+0000","last_fullsized":"2024-04-25T15:50:32.246486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.829016+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:50:32.829016+0000","last_peered":"2024-04-25T15:50:32.829016+0000","last_clean":"2024-04-25T15:50:32.829016+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:50:32.829016+0000","last_undegraded":"2024-04-25T15:50:32.829016+0000","last_fullsized":"2024-04-25T15:50:32.829016+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148600+0000","last_change":"2024-04-25T15:50:31.182857+0000","last_active":"2024-04-25T15:50:32.148600+0000","last_peered":"2024-04-25T15:50:32.148600+0000","last_clean":"2024-04-25T15:50:32.148600+0000","last_became_active":"2024-04-25T15:50:31.181994+0000","last_became_peered":"2024-04-25T15:50:31.181994+0000","last_unstale":"2024-04-25T15:50:32.148600+0000","last_undegraded":"2024-04-25T15:50:32.148600+0000","last_fullsized":"2024-04-25T15:50:32.148600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:58.571421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.259243+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:50:42.259243+0000","last_peered":"2024-04-25T15:50:42.259243+0000","last_clean":"2024-04-25T15:50:42.259243+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:50:42.259243+0000","last_undegraded":"2024-04-25T15:50:42.259243+0000","last_fullsized":"2024-04-25T15:50:42.259243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.154204+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:50:32.154204+0000","last_peered":"2024-04-25T15:50:32.154204+0000","last_clean":"2024-04-25T15:50:32.154204+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:50:32.154204+0000","last_undegraded":"2024-04-25T15:50:32.154204+0000","last_fullsized":"2024-04-25T15:50:32.154204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.259513+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:50:42.259513+0000","last_peered":"2024-04-25T15:50:42.259513+0000","last_clean":"2024-04-25T15:50:42.259513+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:50:42.259513+0000","last_undegraded":"2024-04-25T15:50:42.259513+0000","last_fullsized":"2024-04-25T15:50:42.259513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749352+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:50:32.749352+0000","last_peered":"2024-04-25T15:50:32.749352+0000","last_clean":"2024-04-25T15:50:32.749352+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:50:32.749352+0000","last_undegraded":"2024-04-25T15:50:32.749352+0000","last_fullsized":"2024-04-25T15:50:32.749352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.225492+0000","last_change":"2024-04-25T15:50:30.200105+0000","last_active":"2024-04-25T15:50:36.225492+0000","last_peered":"2024-04-25T15:50:36.225492+0000","last_clean":"2024-04-25T15:50:36.225492+0000","last_became_active":"2024-04-25T15:50:30.199328+0000","last_became_peered":"2024-04-25T15:50:30.199328+0000","last_unstale":"2024-04-25T15:50:36.225492+0000","last_undegraded":"2024-04-25T15:50:36.225492+0000","last_fullsized":"2024-04-25T15:50:36.225492+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:59.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151895+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:50:32.151895+0000","last_peered":"2024-04-25T15:50:32.151895+0000","last_clean":"2024-04-25T15:50:32.151895+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:50:32.151895+0000","last_undegraded":"2024-04-25T15:50:32.151895+0000","last_fullsized":"2024-04-25T15:50:32.151895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.253397+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:50:42.253397+0000","last_peered":"2024-04-25T15:50:42.253397+0000","last_clean":"2024-04-25T15:50:42.253397+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:50:42.253397+0000","last_undegraded":"2024-04-25T15:50:42.253397+0000","last_fullsized":"2024-04-25T15:50:42.253397+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748150+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:50:32.748150+0000","last_peered":"2024-04-25T15:50:32.748150+0000","last_clean":"2024-04-25T15:50:32.748150+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:50:32.748150+0000","last_undegraded":"2024-04-25T15:50:32.748150+0000","last_fullsized":"2024-04-25T15:50:32.748150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.260422+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:50:42.260422+0000","last_peered":"2024-04-25T15:50:42.260422+0000","last_clean":"2024-04-25T15:50:42.260422+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:50:42.260422+0000","last_undegraded":"2024-04-25T15:50:42.260422+0000","last_fullsized":"2024-04-25T15:50:42.260422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750368+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:50:32.750368+0000","last_peered":"2024-04-25T15:50:32.750368+0000","last_clean":"2024-04-25T15:50:32.750368+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:50:32.750368+0000","last_undegraded":"2024-04-25T15:50:32.750368+0000","last_fullsized":"2024-04-25T15:50:32.750368+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.254265+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:50:42.254265+0000","last_peered":"2024-04-25T15:50:42.254265+0000","last_clean":"2024-04-25T15:50:42.254265+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:50:42.254265+0000","last_undegraded":"2024-04-25T15:50:42.254265+0000","last_fullsized":"2024-04-25T15:50:42.254265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748438+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:50:32.748438+0000","last_peered":"2024-04-25T15:50:32.748438+0000","last_clean":"2024-04-25T15:50:32.748438+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:50:32.748438+0000","last_undegraded":"2024-04-25T15:50:32.748438+0000","last_fullsized":"2024-04-25T15:50:32.748438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748758+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:50:32.748758+0000","last_peered":"2024-04-25T15:50:32.748758+0000","last_clean":"2024-04-25T15:50:32.748758+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:50:32.748758+0000","last_undegraded":"2024-04-25T15:50:32.748758+0000","last_fullsized":"2024-04-25T15:50:32.748758+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748532+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:50:32.748532+0000","last_peered":"2024-04-25T15:50:32.748532+0000","last_clean":"2024-04-25T15:50:32.748532+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:50:32.748532+0000","last_undegraded":"2024-04-25T15:50:32.748532+0000","last_fullsized":"2024-04-25T15:50:32.748532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'10","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.253184+0000","last_change":"2024-04-25T15:50:30.179053+0000","last_active":"2024-04-25T15:50:42.253184+0000","last_peered":"2024-04-25T15:50:42.253184+0000","last_clean":"2024-04-25T15:50:42.253184+0000","last_became_active":"2024-04-25T15:50:30.178196+0000","last_became_peered":"2024-04-25T15:50:30.178196+0000","last_unstale":"2024-04-25T15:50:42.253184+0000","last_undegraded":"2024-04-25T15:50:42.253184+0000","last_fullsized":"2024-04-25T15:50:42.253184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:26.635717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748379+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:50:32.748379+0000","last_peered":"2024-04-25T15:50:32.748379+0000","last_clean":"2024-04-25T15:50:32.748379+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:50:32.748379+0000","last_undegraded":"2024-04-25T15:50:32.748379+0000","last_fullsized":"2024-04-25T15:50:32.748379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.830157+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:50:32.830157+0000","last_peered":"2024-04-25T15:50:32.830157+0000","last_clean":"2024-04-25T15:50:32.830157+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:50:32.830157+0000","last_undegraded":"2024-04-25T15:50:32.830157+0000","last_fullsized":"2024-04-25T15:50:32.830157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749720+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:50:32.749720+0000","last_peered":"2024-04-25T15:50:32.749720+0000","last_clean":"2024-04-25T15:50:32.749720+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:50:32.749720+0000","last_undegraded":"2024-04-25T15:50:32.749720+0000","last_fullsized":"2024-04-25T15:50:32.749720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.223834+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:50:36.223834+0000","last_peered":"2024-04-25T15:50:36.223834+0000","last_clean":"2024-04-25T15:50:36.223834+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:50:36.223834+0000","last_undegraded":"2024-04-25T15:50:36.223834+0000","last_fullsized":"2024-04-25T15:50:36.223834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750686+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:50:32.750686+0000","last_peered":"2024-04-25T15:50:32.750686+0000","last_clean":"2024-04-25T15:50:32.750686+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:50:32.750686+0000","last_undegraded":"2024-04-25T15:50:32.750686+0000","last_fullsized":"2024-04-25T15:50:32.750686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.846975+0000","last_change":"2024-04-25T15:50:30.192595+0000","last_active":"2024-04-25T15:50:32.846975+0000","last_peered":"2024-04-25T15:50:32.846975+0000","last_clean":"2024-04-25T15:50:32.846975+0000","last_became_active":"2024-04-25T15:50:30.192044+0000","last_became_peered":"2024-04-25T15:50:30.192044+0000","last_unstale":"2024-04-25T15:50:32.846975+0000","last_undegraded":"2024-04-25T15:50:32.846975+0000","last_fullsized":"2024-04-25T15:50:32.846975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:08.837774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153693+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:50:32.153693+0000","last_peered":"2024-04-25T15:50:32.153693+0000","last_clean":"2024-04-25T15:50:32.153693+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:50:32.153693+0000","last_undegraded":"2024-04-25T15:50:32.153693+0000","last_fullsized":"2024-04-25T15:50:32.153693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749938+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:50:32.749938+0000","last_peered":"2024-04-25T15:50:32.749938+0000","last_clean":"2024-04-25T15:50:32.749938+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:50:32.749938+0000","last_undegraded":"2024-04-25T15:50:32.749938+0000","last_fullsized":"2024-04-25T15:50:32.749938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748511+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:50:32.748511+0000","last_peered":"2024-04-25T15:50:32.748511+0000","last_clean":"2024-04-25T15:50:32.748511+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:50:32.748511+0000","last_undegraded":"2024-04-25T15:50:32.748511+0000","last_fullsized":"2024-04-25T15:50:32.748511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.747640+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:50:32.747640+0000","last_peered":"2024-04-25T15:50:32.747640+0000","last_clean":"2024-04-25T15:50:32.747640+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:50:32.747640+0000","last_undegraded":"2024-04-25T15:50:32.747640+0000","last_fullsized":"2024-04-25T15:50:32.747640+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.256357+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:50:42.256357+0000","last_peered":"2024-04-25T15:50:42.256357+0000","last_clean":"2024-04-25T15:50:42.256357+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:50:42.256357+0000","last_undegraded":"2024-04-25T15:50:42.256357+0000","last_fullsized":"2024-04-25T15:50:42.256357+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247247+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:50:32.247247+0000","last_peered":"2024-04-25T15:50:32.247247+0000","last_clean":"2024-04-25T15:50:32.247247+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:50:32.247247+0000","last_undegraded":"2024-04-25T15:50:32.247247+0000","last_fullsized":"2024-04-25T15:50:32.247247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.833132+0000","last_change":"2024-04-25T15:50:30.197359+0000","last_active":"2024-04-25T15:50:32.833132+0000","last_peered":"2024-04-25T15:50:32.833132+0000","last_clean":"2024-04-25T15:50:32.833132+0000","last_became_active":"2024-04-25T15:50:30.196640+0000","last_became_peered":"2024-04-25T15:50:30.196640+0000","last_unstale":"2024-04-25T15:50:32.833132+0000","last_undegraded":"2024-04-25T15:50:32.833132+0000","last_fullsized":"2024-04-25T15:50:32.833132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.928809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.750609+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:50:32.750609+0000","last_peered":"2024-04-25T15:50:32.750609+0000","last_clean":"2024-04-25T15:50:32.750609+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:50:32.750609+0000","last_undegraded":"2024-04-25T15:50:32.750609+0000","last_fullsized":"2024-04-25T15:50:32.750609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.845515+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:50:32.845515+0000","last_peered":"2024-04-25T15:50:32.845515+0000","last_clean":"2024-04-25T15:50:32.845515+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:50:32.845515+0000","last_undegraded":"2024-04-25T15:50:32.845515+0000","last_fullsized":"2024-04-25T15:50:32.845515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148771+0000","last_change":"2024-04-25T15:50:31.171339+0000","last_active":"2024-04-25T15:50:32.148771+0000","last_peered":"2024-04-25T15:50:32.148771+0000","last_clean":"2024-04-25T15:50:32.148771+0000","last_became_active":"2024-04-25T15:50:31.170349+0000","last_became_peered":"2024-04-25T15:50:31.170349+0000","last_unstale":"2024-04-25T15:50:32.148771+0000","last_undegraded":"2024-04-25T15:50:32.148771+0000","last_fullsized":"2024-04-25T15:50:32.148771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:24.647984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153062+0000","last_change":"2024-04-25T15:50:30.195563+0000","last_active":"2024-04-25T15:50:32.153062+0000","last_peered":"2024-04-25T15:50:32.153062+0000","last_clean":"2024-04-25T15:50:32.153062+0000","last_became_active":"2024-04-25T15:50:30.194468+0000","last_became_peered":"2024-04-25T15:50:30.194468+0000","last_unstale":"2024-04-25T15:50:32.153062+0000","last_undegraded":"2024-04-25T15:50:32.153062+0000","last_fullsized":"2024-04-25T15:50:32.153062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:04.850096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151035+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:50:32.151035+0000","last_peered":"2024-04-25T15:50:32.151035+0000","last_clean":"2024-04-25T15:50:32.151035+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:50:32.151035+0000","last_undegraded":"2024-04-25T15:50:32.151035+0000","last_fullsized":"2024-04-25T15:50:32.151035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149161+0000","last_change":"2024-04-25T15:50:30.175727+0000","last_active":"2024-04-25T15:50:32.149161+0000","last_peered":"2024-04-25T15:50:32.149161+0000","last_clean":"2024-04-25T15:50:32.149161+0000","last_became_active":"2024-04-25T15:50:30.175136+0000","last_became_peered":"2024-04-25T15:50:30.175136+0000","last_unstale":"2024-04-25T15:50:32.149161+0000","last_undegraded":"2024-04-25T15:50:32.149161+0000","last_fullsized":"2024-04-25T15:50:32.149161+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:22.434474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153102+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:50:32.153102+0000","last_peered":"2024-04-25T15:50:32.153102+0000","last_clean":"2024-04-25T15:50:32.153102+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:50:32.153102+0000","last_undegraded":"2024-04-25T15:50:32.153102+0000","last_fullsized":"2024-04-25T15:50:32.153102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.833424+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:50:32.833424+0000","last_peered":"2024-04-25T15:50:32.833424+0000","last_clean":"2024-04-25T15:50:32.833424+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:50:32.833424+0000","last_undegraded":"2024-04-25T15:50:32.833424+0000","last_fullsized":"2024-04-25T15:50:32.833424+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.149163+0000","last_change":"2024-04-25T15:50:31.177424+0000","last_active":"2024-04-25T15:50:32.149163+0000","last_peered":"2024-04-25T15:50:32.149163+0000","last_clean":"2024-04-25T15:50:32.149163+0000","last_became_active":"2024-04-25T15:50:31.176553+0000","last_became_peered":"2024-04-25T15:50:31.176553+0000","last_unstale":"2024-04-25T15:50:32.149163+0000","last_undegraded":"2024-04-25T15:50:32.149163+0000","last_fullsized":"2024-04-25T15:50:32.149163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:29.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.847017+0000","last_change":"2024-04-25T15:50:30.191244+0000","last_active":"2024-04-25T15:50:32.847017+0000","last_peered":"2024-04-25T15:50:32.847017+0000","last_clean":"2024-04-25T15:50:32.847017+0000","last_became_active":"2024-04-25T15:50:30.190437+0000","last_became_peered":"2024-04-25T15:50:30.190437+0000","last_unstale":"2024-04-25T15:50:32.847017+0000","last_undegraded":"2024-04-25T15:50:32.847017+0000","last_fullsized":"2024-04-25T15:50:32.847017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:55.506915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247702+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:50:32.247702+0000","last_peered":"2024-04-25T15:50:32.247702+0000","last_clean":"2024-04-25T15:50:32.247702+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:50:32.247702+0000","last_undegraded":"2024-04-25T15:50:32.247702+0000","last_fullsized":"2024-04-25T15:50:32.247702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.254863+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:50:42.254863+0000","last_peered":"2024-04-25T15:50:42.254863+0000","last_clean":"2024-04-25T15:50:42.254863+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:50:42.254863+0000","last_undegraded":"2024-04-25T15:50:42.254863+0000","last_fullsized":"2024-04-25T15:50:42.254863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749239+0000","last_change":"2024-04-25T15:50:31.182607+0000","last_active":"2024-04-25T15:50:32.749239+0000","last_peered":"2024-04-25T15:50:32.749239+0000","last_clean":"2024-04-25T15:50:32.749239+0000","last_became_active":"2024-04-25T15:50:31.181641+0000","last_became_peered":"2024-04-25T15:50:31.181641+0000","last_unstale":"2024-04-25T15:50:32.749239+0000","last_undegraded":"2024-04-25T15:50:32.749239+0000","last_fullsized":"2024-04-25T15:50:32.749239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:49.636675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.250651+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:50:42.250651+0000","last_peered":"2024-04-25T15:50:42.250651+0000","last_clean":"2024-04-25T15:50:42.250651+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:50:42.250651+0000","last_undegraded":"2024-04-25T15:50:42.250651+0000","last_fullsized":"2024-04-25T15:50:42.250651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.151509+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:50:32.151509+0000","last_peered":"2024-04-25T15:50:32.151509+0000","last_clean":"2024-04-25T15:50:32.151509+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:50:32.151509+0000","last_undegraded":"2024-04-25T15:50:32.151509+0000","last_fullsized":"2024-04-25T15:50:32.151509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.258014+0000","last_change":"2024-04-25T15:50:30.178078+0000","last_active":"2024-04-25T15:50:42.258014+0000","last_peered":"2024-04-25T15:50:42.258014+0000","last_clean":"2024-04-25T15:50:42.258014+0000","last_became_active":"2024-04-25T15:50:30.177718+0000","last_became_peered":"2024-04-25T15:50:30.177718+0000","last_unstale":"2024-04-25T15:50:42.258014+0000","last_undegraded":"2024-04-25T15:50:42.258014+0000","last_fullsized":"2024-04-25T15:50:42.258014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.246699+0000","last_change":"2024-04-25T15:50:31.192152+0000","last_active":"2024-04-25T15:50:32.246699+0000","last_peered":"2024-04-25T15:50:32.246699+0000","last_clean":"2024-04-25T15:50:32.246699+0000","last_became_active":"2024-04-25T15:50:31.179200+0000","last_became_peered":"2024-04-25T15:50:31.179200+0000","last_unstale":"2024-04-25T15:50:32.246699+0000","last_undegraded":"2024-04-25T15:50:32.246699+0000","last_fullsized":"2024-04-25T15:50:32.246699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:03.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.148915+0000","last_change":"2024-04-25T15:50:30.184989+0000","last_active":"2024-04-25T15:50:32.148915+0000","last_peered":"2024-04-25T15:50:32.148915+0000","last_clean":"2024-04-25T15:50:32.148915+0000","last_became_active":"2024-04-25T15:50:30.184132+0000","last_became_peered":"2024-04-25T15:50:30.184132+0000","last_unstale":"2024-04-25T15:50:32.148915+0000","last_undegraded":"2024-04-25T15:50:32.148915+0000","last_fullsized":"2024-04-25T15:50:32.148915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:03.833303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247766+0000","last_change":"2024-04-25T15:50:31.190275+0000","last_active":"2024-04-25T15:50:32.247766+0000","last_peered":"2024-04-25T15:50:32.247766+0000","last_clean":"2024-04-25T15:50:32.247766+0000","last_became_active":"2024-04-25T15:50:31.189302+0000","last_became_peered":"2024-04-25T15:50:31.189302+0000","last_unstale":"2024-04-25T15:50:32.247766+0000","last_undegraded":"2024-04-25T15:50:32.247766+0000","last_fullsized":"2024-04-25T15:50:32.247766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:39.479706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:42.248840+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:50:42.248840+0000","last_peered":"2024-04-25T15:50:42.248840+0000","last_clean":"2024-04-25T15:50:42.248840+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:50:42.248840+0000","last_undegraded":"2024-04-25T15:50:42.248840+0000","last_fullsized":"2024-04-25T15:50:42.248840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.748648+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:50:32.748648+0000","last_peered":"2024-04-25T15:50:32.748648+0000","last_clean":"2024-04-25T15:50:32.748648+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:50:32.748648+0000","last_undegraded":"2024-04-25T15:50:32.748648+0000","last_fullsized":"2024-04-25T15:50:32.748648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153111+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:50:32.153111+0000","last_peered":"2024-04-25T15:50:32.153111+0000","last_clean":"2024-04-25T15:50:32.153111+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:50:32.153111+0000","last_undegraded":"2024-04-25T15:50:32.153111+0000","last_fullsized":"2024-04-25T15:50:32.153111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.214957+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:50:36.214957+0000","last_peered":"2024-04-25T15:50:36.214957+0000","last_clean":"2024-04-25T15:50:36.214957+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:50:36.214957+0000","last_undegraded":"2024-04-25T15:50:36.214957+0000","last_fullsized":"2024-04-25T15:50:36.214957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.153450+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:50:32.153450+0000","last_peered":"2024-04-25T15:50:32.153450+0000","last_clean":"2024-04-25T15:50:32.153450+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:50:32.153450+0000","last_undegraded":"2024-04-25T15:50:32.153450+0000","last_fullsized":"2024-04-25T15:50:32.153450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.220304+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:50:36.220304+0000","last_peered":"2024-04-25T15:50:36.220304+0000","last_clean":"2024-04-25T15:50:36.220304+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:50:36.220304+0000","last_undegraded":"2024-04-25T15:50:36.220304+0000","last_fullsized":"2024-04-25T15:50:36.220304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.749087+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:50:32.749087+0000","last_peered":"2024-04-25T15:50:32.749087+0000","last_clean":"2024-04-25T15:50:32.749087+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:50:32.749087+0000","last_undegraded":"2024-04-25T15:50:32.749087+0000","last_fullsized":"2024-04-25T15:50:32.749087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.848640+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:50:32.848640+0000","last_peered":"2024-04-25T15:50:32.848640+0000","last_clean":"2024-04-25T15:50:32.848640+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:50:32.848640+0000","last_undegraded":"2024-04-25T15:50:32.848640+0000","last_fullsized":"2024-04-25T15:50:32.848640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:32.247337+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:50:32.247337+0000","last_peered":"2024-04-25T15:50:32.247337+0000","last_clean":"2024-04-25T15:50:32.247337+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:50:32.247337+0000","last_undegraded":"2024-04-25T15:50:32.247337+0000","last_fullsized":"2024-04-25T15:50:32.247337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-25T15:50:36.226700+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:50:36.226700+0000","last_peered":"2024-04-25T15:50:36.226700+0000","last_clean":"2024-04-25T15:50:36.226700+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:50:36.226700+0000","last_undegraded":"2024-04-25T15:50:36.226700+0000","last_fullsized":"2024-04-25T15:50:36.226700+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3726,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":7452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":662528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":72636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":73168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":661962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:50:44.030 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T15:50:44.030 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T15:50:44.031 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T15:50:44.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:50:44.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5be53640 1 Processor -- start 2024-04-25T15:50:44.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5be53640 1 -- start start 2024-04-25T15:50:44.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5be53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 0x7f9a5415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5be53640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9a540ca540 0x7f9a540ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5be53640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a5415de10 con 0x7f9a54071bc0 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5be53640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9a5415ebb0 con 0x7f9a540ca540 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5be53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a5415e720 con 0x7f9a5415a720 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a59bef640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9a54071bc0 0x7f9a540cbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49864/0 (socket says 172.21.15.67:49864) 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a59bef640 1 -- 172.21.15.67:0/2072256405 learned_addr learned my addr 172.21.15.67:0/2072256405 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a593ee640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 0x7f9a5415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2072256405 >> v2:172.21.15.176:3300/0 conn(0x7f9a540ca540 0x7f9a540ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2183367533 0 0) 0x7f9a5415de10 con 0x7f9a54071bc0 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a40003530 con 0x7f9a54071bc0 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2097481097 0 0) 0x7f9a40003530 con 0x7f9a54071bc0 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 msgr2=0x7f9a5415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 0x7f9a5415ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 >> v2:172.21.15.176:3300/0 conn(0x7f9a540ca540 msgr2=0x7f9a540ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 --2- 172.21.15.67:0/2072256405 >> v2:172.21.15.176:3300/0 conn(0x7f9a540ca540 0x7f9a540ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a54072390 con 0x7f9a54071bc0 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f9a50001680 con 0x7f9a54071bc0 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 >> v1:172.21.15.67:6789/0 conn(0x7f9a54071bc0 legacy=0x7f9a540cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a400039c0 0x7f9a40003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 0x7f9a40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40004d50 0x7f9a40005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5415ebb0 con 0x7f9a40004d50 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5415e720 con 0x7f9a400039c0 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a4000ba00 con 0x7f9a5415a720 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40004d50 0x7f9a40005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.221+0000 7f9a59bef640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a400039c0 0x7f9a40003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a593ee640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 0x7f9a40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 msgr2=0x7f9a40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:44.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a5415a720 0x7f9a40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a400039c0 msgr2=0x7f9a40003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:44.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a400039c0 0x7f9a40003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2072256405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a40003530 con 0x7f9a40004d50 2024-04-25T15:50:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40004d50 0x7f9a40005160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009f90 tx=0x7f9a4c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d897821a237e8d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0b2070 con 0x7f9a40004d50 2024-04-25T15:50:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a4c0a5c90 con 0x7f9a40004d50 2024-04-25T15:50:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a58bed640 1 -- 172.21.15.67:0/2072256405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c009c10 con 0x7f9a40004d50 2024-04-25T15:50:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5be53640 1 -- 172.21.15.67:0/2072256405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40004d50 msgr2=0x7f9a40005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5be53640 1 --2- 172.21.15.67:0/2072256405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40004d50 0x7f9a40005160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009f90 tx=0x7f9a4c01cc80 comp rx=0 tx=0).stop 2024-04-25T15:50:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2072256405 reap_dead start 2024-04-25T15:50:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5be53640 1 -- 172.21.15.67:0/2072256405 shutdown_connections 2024-04-25T15:50:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5be53640 1 -- 172.21.15.67:0/2072256405 >> 172.21.15.67:0/2072256405 conn(0x7f9a540700b0 msgr2=0x7f9a540704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5be53640 1 -- 172.21.15.67:0/2072256405 shutdown_connections 2024-04-25T15:50:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.225+0000 7f9a5be53640 1 -- 172.21.15.67:0/2072256405 wait complete. 2024-04-25T15:50:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 Processor -- start 2024-04-25T15:50:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 -- start start 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a540b57c0 0x7f9a54068760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a541f68e0 0x7f9a541f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54152e60 0x7f9a54153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9a540c93b0 con 0x7f9a54152e60 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5415e3b0 con 0x7f9a541f68e0 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a540cc2d0 con 0x7f9a540b57c0 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54152e60 0x7f9a54153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54152e60 0x7f9a54153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36928/0 (socket says 172.21.15.67:36928) 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2275335431 learned_addr learned my addr 172.21.15.67:0/2275335431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a593ee640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a541f68e0 0x7f9a541f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a59bef640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a540b57c0 0x7f9a54068760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a540b57c0 msgr2=0x7f9a54068760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a540b57c0 0x7f9a54068760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a541f68e0 msgr2=0x7f9a541f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a541f68e0 0x7f9a541f6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 -- 172.21.15.67:0/2275335431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5415a280 con 0x7f9a54152e60 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a593ee640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a541f68e0 0x7f9a541f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a59bef640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a540b57c0 0x7f9a54068760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5a3f0640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54152e60 0x7f9a54153270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0b0040 tx=0x7f9a4c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a4a7fc640 1 -- 172.21.15.67:0/2275335431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0b2040 con 0x7f9a54152e60 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a4a7fc640 1 -- 172.21.15.67:0/2275335431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a4c0a5a90 con 0x7f9a54152e60 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a54155510 con 0x7f9a54152e60 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a4a7fc640 1 -- 172.21.15.67:0/2275335431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c009c10 con 0x7f9a54152e60 2024-04-25T15:50:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.229+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a54155ad0 con 0x7f9a54152e60 2024-04-25T15:50:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.233+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5406a480 con 0x7f9a54152e60 2024-04-25T15:50:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.233+0000 7f9a4a7fc640 1 -- 172.21.15.67:0/2275335431 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9a4c0be020 con 0x7f9a54152e60 2024-04-25T15:50:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.233+0000 7f9a4a7fc640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3806f400 0x7f9a38071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.233+0000 7f9a4a7fc640 1 -- 172.21.15.67:0/2275335431 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9a4c129520 con 0x7f9a54152e60 2024-04-25T15:50:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.233+0000 7f9a59bef640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3806f400 0x7f9a38071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.233+0000 7f9a59bef640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3806f400 0x7f9a38071910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a50004850 tx=0x7f9a50001c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.237+0000 7f9a4a7fc640 1 -- 172.21.15.67:0/2275335431 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9a4c016990 con 0x7f9a54152e60 2024-04-25T15:50:44.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9a541f7230 con 0x7f9a54152e60 2024-04-25T15:50:44.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a4a7fc640 1 -- 172.21.15.67:0/2275335431 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f9a4c0f6530 con 0x7f9a54152e60 2024-04-25T15:50:44.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:44.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T15:50:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3806f400 msgr2=0x7f9a38071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3806f400 0x7f9a38071910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a50004850 tx=0x7f9a50001c90 comp rx=0 tx=0).stop 2024-04-25T15:50:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54152e60 msgr2=0x7f9a54153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54152e60 0x7f9a54153270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0b0040 tx=0x7f9a4c009750 comp rx=0 tx=0).stop 2024-04-25T15:50:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 shutdown_connections 2024-04-25T15:50:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3806f400 0x7f9a38071910 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a54152e60 0x7f9a54153270 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a541f68e0 0x7f9a541f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 --2- 172.21.15.67:0/2275335431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a540b57c0 0x7f9a54068760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 >> 172.21.15.67:0/2275335431 conn(0x7f9a540700b0 msgr2=0x7f9a540d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 shutdown_connections 2024-04-25T15:50:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:44.445+0000 7f9a5be53640 1 -- 172.21.15.67:0/2275335431 wait complete. 2024-04-25T15:50:45.639 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-04-25T15:50:45.632+0000 7f240c636640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:50:47.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:50:47.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f126df8f640 1 Processor -- start 2024-04-25T15:50:47.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f126df8f640 1 -- start start 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f126df8f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f12680cdfb0 0x7f126806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f126df8f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12680ce3c0 0x7f126806cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f126df8f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1268158d20 con 0x7f1268152b00 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f126df8f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f12680722c0 con 0x7f12680ce3c0 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f126df8f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1268158fe0 con 0x7f12680cdfb0 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1267fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12680ce3c0 0x7f126806cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1267fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f12680ce3c0 0x7f126806cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36940/0 (socket says 172.21.15.67:36940) 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1267fff640 1 -- 172.21.15.67:0/1193203138 learned_addr learned my addr 172.21.15.67:0/1193203138 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1266ffd640 1 --2- 172.21.15.67:0/1193203138 >> v2:172.21.15.176:3300/0 conn(0x7f12680cdfb0 0x7f126806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1267fff640 1 -- 172.21.15.67:0/1193203138 >> v2:172.21.15.176:3300/0 conn(0x7f12680cdfb0 msgr2=0x7f126806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1267fff640 1 --2- 172.21.15.67:0/1193203138 >> v2:172.21.15.176:3300/0 conn(0x7f12680cdfb0 0x7f126806c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1267fff640 1 -- 172.21.15.67:0/1193203138 >> v1:172.21.15.176:6789/0 conn(0x7f1268152b00 legacy=0x7f12680c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.633+0000 7f1267fff640 1 -- 172.21.15.67:0/1193203138 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12681592c0 con 0x7f12680ce3c0 2024-04-25T15:50:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f1267fff640 1 --2- 172.21.15.67:0/1193203138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12680ce3c0 0x7f126806cb80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12680c5be0 tx=0x7f126001ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e35b695b7f5ea00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f12667fc640 1 -- 172.21.15.67:0/1193203138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126000b2d0 con 0x7f12680ce3c0 2024-04-25T15:50:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f12667fc640 1 -- 172.21.15.67:0/1193203138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126000b470 con 0x7f12680ce3c0 2024-04-25T15:50:47.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f12667fc640 1 -- 172.21.15.67:0/1193203138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12600a7ae0 con 0x7f12680ce3c0 2024-04-25T15:50:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 -- 172.21.15.67:0/1193203138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12680ce3c0 msgr2=0x7f126806cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 --2- 172.21.15.67:0/1193203138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12680ce3c0 0x7f126806cb80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12680c5be0 tx=0x7f126001ebf0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 -- 172.21.15.67:0/1193203138 shutdown_connections 2024-04-25T15:50:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 --2- 172.21.15.67:0/1193203138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12680ce3c0 0x7f126806cb80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 --2- 172.21.15.67:0/1193203138 >> v2:172.21.15.176:3300/0 conn(0x7f12680cdfb0 0x7f126806c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 -- 172.21.15.67:0/1193203138 >> 172.21.15.67:0/1193203138 conn(0x7f126806fe70 msgr2=0x7f12680cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 -- 172.21.15.67:0/1193203138 shutdown_connections 2024-04-25T15:50:47.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 -- 172.21.15.67:0/1193203138 wait complete. 2024-04-25T15:50:47.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 Processor -- start 2024-04-25T15:50:47.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.637+0000 7f126df8f640 1 -- start start 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12680cdfb0 0x7f1268150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12680ce3c0 0x7f12681f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1268152b00 0x7f1268073fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f126815d480 con 0x7f1268152b00 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1268152f10 con 0x7f12680ce3c0 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f126815d600 con 0x7f12680cdfb0 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1268152b00 0x7f1268073fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1268152b00 0x7f1268073fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36950/0 (socket says 172.21.15.67:36950) 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 -- 172.21.15.67:0/1668614923 learned_addr learned my addr 172.21.15.67:0/1668614923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1266ffd640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12680ce3c0 0x7f12681f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f12677fe640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12680cdfb0 0x7f1268150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 -- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12680cdfb0 msgr2=0x7f1268150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12680cdfb0 0x7f1268150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 -- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12680ce3c0 msgr2=0x7f12681f5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12680ce3c0 0x7f12681f5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 -- 172.21.15.67:0/1668614923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12681f6070 con 0x7f1268152b00 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f12677fe640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12680cdfb0 0x7f1268150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f1267fff640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1268152b00 0x7f1268073fb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12600b1040 tx=0x7f126001b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f124ffff640 1 -- 172.21.15.67:0/1668614923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126000b720 con 0x7f1268152b00 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f124ffff640 1 -- 172.21.15.67:0/1668614923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12600c0070 con 0x7f1268152b00 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12681f2ad0 con 0x7f1268152b00 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f124ffff640 1 -- 172.21.15.67:0/1668614923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260009a40 con 0x7f1268152b00 2024-04-25T15:50:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12681f1d10 con 0x7f1268152b00 2024-04-25T15:50:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.641+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12681515f0 con 0x7f1268152b00 2024-04-25T15:50:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.645+0000 7f124ffff640 1 -- 172.21.15.67:0/1668614923 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f126000b2e0 con 0x7f1268152b00 2024-04-25T15:50:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.645+0000 7f124ffff640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f125406f400 0x7f1254071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.645+0000 7f124ffff640 1 -- 172.21.15.67:0/1668614923 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f126012c3d0 con 0x7f1268152b00 2024-04-25T15:50:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.645+0000 7f12677fe640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f125406f400 0x7f1254071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.645+0000 7f12677fe640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f125406f400 0x7f1254071910 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1258001670 tx=0x7f1258009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.649+0000 7f124ffff640 1 -- 172.21.15.67:0/1668614923 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f12600187e0 con 0x7f1268152b00 2024-04-25T15:50:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.845+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f126806a480 con 0x7f1268152b00 2024-04-25T15:50:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.845+0000 7f124ffff640 1 -- 172.21.15.67:0/1668614923 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f12600f92c0 con 0x7f1268152b00 2024-04-25T15:50:47.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:47.849 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T15:50:47.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f125406f400 msgr2=0x7f1254071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:47.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f125406f400 0x7f1254071910 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1258001670 tx=0x7f1258009040 comp rx=0 tx=0).stop 2024-04-25T15:50:47.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1268152b00 msgr2=0x7f1268073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1268152b00 0x7f1268073fb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12600b1040 tx=0x7f126001b990 comp rx=0 tx=0).stop 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 shutdown_connections 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f125406f400 0x7f1254071910 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1268152b00 0x7f1268073fb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12680ce3c0 0x7f12681f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 --2- 172.21.15.67:0/1668614923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12680cdfb0 0x7f1268150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 >> 172.21.15.67:0/1668614923 conn(0x7f126806fe70 msgr2=0x7f126814d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 shutdown_connections 2024-04-25T15:50:47.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:47.849+0000 7f126df8f640 1 -- 172.21.15.67:0/1668614923 wait complete. 2024-04-25T15:50:50.639 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-04-25T15:50:50.636+0000 7f240c636640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T15:50:50.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:50:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f31371e3640 1 Processor -- start 2024-04-25T15:50:51.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f31371e3640 1 -- start start 2024-04-25T15:50:51.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f31371e3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3130073bf0 0x7f313006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:51.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f31371e3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f313006dc90 0x7f3130163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f31371e3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31300c77a0 con 0x7f31300caf60 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f31371e3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f31300cab60 con 0x7f3130073bf0 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f31371e3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3130159620 con 0x7f313006dc90 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f312ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3130073bf0 0x7f313006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f3134f7f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f31300caf60 0x7f313015d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49880/0 (socket says 172.21.15.67:49880) 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f312ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3130073bf0 0x7f313006d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36952/0 (socket says 172.21.15.67:36952) 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f3134f7f640 1 -- 172.21.15.67:0/1579843882 learned_addr learned my addr 172.21.15.67:0/1579843882 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f3135780640 1 --2- 172.21.15.67:0/1579843882 >> v2:172.21.15.176:3300/0 conn(0x7f313006dc90 0x7f3130163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f312ffff640 1 -- 172.21.15.67:0/1579843882 >> v2:172.21.15.176:3300/0 conn(0x7f313006dc90 msgr2=0x7f3130163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f312ffff640 1 --2- 172.21.15.67:0/1579843882 >> v2:172.21.15.176:3300/0 conn(0x7f313006dc90 0x7f3130163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f312ffff640 1 -- 172.21.15.67:0/1579843882 >> v1:172.21.15.67:6789/0 conn(0x7f31300caf60 legacy=0x7f313015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f312ffff640 1 -- 172.21.15.67:0/1579843882 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31300c7a60 con 0x7f3130073bf0 2024-04-25T15:50:51.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.033+0000 7f312f7fe640 1 -- 172.21.15.67:0/1579843882 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041013824 0 0) 0x7f31300c77a0 con 0x7f31300caf60 2024-04-25T15:50:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f312ffff640 1 --2- 172.21.15.67:0/1579843882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3130073bf0 0x7f313006d750 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f31300c61d0 tx=0x7f312401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea934a2d2d8f0cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f312f7fe640 1 -- 172.21.15.67:0/1579843882 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31240b0070 con 0x7f3130073bf0 2024-04-25T15:50:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f312f7fe640 1 -- 172.21.15.67:0/1579843882 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31240ab600 con 0x7f3130073bf0 2024-04-25T15:50:51.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f312f7fe640 1 -- 172.21.15.67:0/1579843882 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31240ab9d0 con 0x7f3130073bf0 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 -- 172.21.15.67:0/1579843882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3130073bf0 msgr2=0x7f313006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 --2- 172.21.15.67:0/1579843882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3130073bf0 0x7f313006d750 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f31300c61d0 tx=0x7f312401c480 comp rx=0 tx=0).stop 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 -- 172.21.15.67:0/1579843882 shutdown_connections 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 --2- 172.21.15.67:0/1579843882 >> v2:172.21.15.176:3300/0 conn(0x7f313006dc90 0x7f3130163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 --2- 172.21.15.67:0/1579843882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3130073bf0 0x7f313006d750 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 -- 172.21.15.67:0/1579843882 >> 172.21.15.67:0/1579843882 conn(0x7f3130070db0 msgr2=0x7f31300ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 -- 172.21.15.67:0/1579843882 shutdown_connections 2024-04-25T15:50:51.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.037+0000 7f31371e3640 1 -- 172.21.15.67:0/1579843882 wait complete. 2024-04-25T15:50:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 Processor -- start 2024-04-25T15:50:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 -- start start 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f313006dc90 0x7f31301f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3130073bf0 0x7f313014d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31300caf60 0x7f31300cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f313015a570 con 0x7f31300caf60 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f313015a3f0 con 0x7f313006dc90 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3130166c20 con 0x7f3130073bf0 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31300caf60 0x7f31300cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31300caf60 0x7f31300cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36954/0 (socket says 172.21.15.67:36954) 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 -- 172.21.15.67:0/450211551 learned_addr learned my addr 172.21.15.67:0/450211551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f312ffff640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3130073bf0 0x7f313014d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:51.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3134f7f640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f313006dc90 0x7f31301f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 -- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3130073bf0 msgr2=0x7f313014d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3130073bf0 0x7f313014d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 -- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f313006dc90 msgr2=0x7f31301f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f313006dc90 0x7f31301f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 -- 172.21.15.67:0/450211551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31301f76c0 con 0x7f31300caf60 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f3135780640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31300caf60 0x7f31300cd780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3128009cb0 tx=0x7f312800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f312d7fa640 1 -- 172.21.15.67:0/450211551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128018020 con 0x7f31300caf60 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f312d7fa640 1 -- 172.21.15.67:0/450211551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3128014480 con 0x7f31300caf60 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31300c9fc0 con 0x7f31300caf60 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f312d7fa640 1 -- 172.21.15.67:0/450211551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128014870 con 0x7f31300caf60 2024-04-25T15:50:51.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f313014fea0 con 0x7f31300caf60 2024-04-25T15:50:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.041+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f313006a480 con 0x7f31300caf60 2024-04-25T15:50:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.045+0000 7f312d7fa640 1 -- 172.21.15.67:0/450211551 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f312801b020 con 0x7f31300caf60 2024-04-25T15:50:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.045+0000 7f312d7fa640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f311c06f430 0x7f311c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.045+0000 7f312d7fa640 1 -- 172.21.15.67:0/450211551 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f31280902a0 con 0x7f31300caf60 2024-04-25T15:50:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.045+0000 7f3134f7f640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f311c06f430 0x7f311c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:51.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.045+0000 7f3134f7f640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f311c06f430 0x7f311c071940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3120000f70 tx=0x7f3120007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:51.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.049+0000 7f312d7fa640 1 -- 172.21.15.67:0/450211551 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3128022030 con 0x7f31300caf60 2024-04-25T15:50:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f31300cc020 con 0x7f31300caf60 2024-04-25T15:50:51.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f312d7fa640 1 -- 172.21.15.67:0/450211551 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f3128010040 con 0x7f31300caf60 2024-04-25T15:50:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:51.247 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T15:50:51.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f311c06f430 msgr2=0x7f311c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:51.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f31371e3640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f311c06f430 0x7f311c071940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3120000f70 tx=0x7f3120007df0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31300caf60 msgr2=0x7f31300cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f31371e3640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31300caf60 0x7f31300cd780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3128009cb0 tx=0x7f312800c490 comp rx=0 tx=0).stop 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 shutdown_connections 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.245+0000 7f31371e3640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f311c06f430 0x7f311c071940 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.249+0000 7f31371e3640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31300caf60 0x7f31300cd780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.249+0000 7f31371e3640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3130073bf0 0x7f313014d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.249+0000 7f31371e3640 1 --2- 172.21.15.67:0/450211551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f313006dc90 0x7f31301f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.249+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 >> 172.21.15.67:0/450211551 conn(0x7f3130070db0 msgr2=0x7f31300c8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.249+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 shutdown_connections 2024-04-25T15:50:51.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:51.249+0000 7f31371e3640 1 -- 172.21.15.67:0/450211551 wait complete. 2024-04-25T15:50:54.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:50:54.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f6d5f640 1 Processor -- start 2024-04-25T15:50:54.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f6d5f640 1 -- start start 2024-04-25T15:50:54.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f6d5f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 0x7fd4f0165e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f6d5f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f0072dc0 con 0x7fd4f00723a0 2024-04-25T15:50:54.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f6d5f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4f015b430 con 0x7fd4f0074790 2024-04-25T15:50:54.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f6d5f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f015a670 con 0x7fd4f00727b0 2024-04-25T15:50:54.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd4f00723a0 0x7fd4f00c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33186/0 (socket says 172.21.15.67:33186) 2024-04-25T15:50:54.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 -- 172.21.15.67:0/490724733 learned_addr learned my addr 172.21.15.67:0/490724733 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f52fc640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 0x7fd4f0165e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3008102602 0 0) 0x7fd4f0072dc0 con 0x7fd4f00723a0 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4d8003530 con 0x7fd4f00723a0 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3190532934 0 0) 0x7fd4d8003530 con 0x7fd4f00723a0 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 msgr2=0x7fd4f0165e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 0x7fd4f0165e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 >> v1:172.21.15.176:6789/0 conn(0x7fd4f0074790 legacy=0x7fd4f015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f015a950 con 0x7fd4f00723a0 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fd4e4001680 con 0x7fd4f00723a0 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 >> v1:172.21.15.67:6789/0 conn(0x7fd4f00723a0 legacy=0x7fd4f00c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4d80039f0 0x7fd4d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d8004810 0x7fd4d8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 0x7fd4d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f015a670 con 0x7fd4d8004810 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d800ba80 con 0x7fd4d80039f0 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d800bc80 con 0x7fd4f00727b0 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d8004810 0x7fd4d8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f4afb640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4d80039f0 0x7fd4d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4f52fc640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 0x7fd4d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 -- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 msgr2=0x7fd4d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4f00727b0 0x7fd4d8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 -- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4d80039f0 msgr2=0x7fd4d8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4d80039f0 0x7fd4d8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.429+0000 7fd4effff640 1 -- 172.21.15.67:0/490724733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d8003530 con 0x7fd4d8004810 2024-04-25T15:50:54.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4effff640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d8004810 0x7fd4d8004c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400c190 tx=0x7fd4e401f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5cd1cb283294b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4007cd0 con 0x7fd4d8004810 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4e4005ae0 con 0x7fd4d8004810 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4ef7fe640 1 -- 172.21.15.67:0/490724733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4007070 con 0x7fd4d8004810 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/490724733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d8004810 msgr2=0x7fd4d8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4f6d5f640 1 --2- 172.21.15.67:0/490724733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d8004810 0x7fd4d8004c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd4e400c190 tx=0x7fd4e401f370 comp rx=0 tx=0).stop 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4f52fc640 1 -- 172.21.15.67:0/490724733 reap_dead start 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/490724733 shutdown_connections 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/490724733 >> 172.21.15.67:0/490724733 conn(0x7fd4f006fe70 msgr2=0x7fd4f00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/490724733 shutdown_connections 2024-04-25T15:50:54.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.433+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/490724733 wait complete. 2024-04-25T15:50:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 Processor -- start 2024-04-25T15:50:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 -- start start 2024-04-25T15:50:54.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4d8004810 0x7fd4f015cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f014aa20 0x7fd4f014ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4f01f3d60 0x7fd4f01f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f015e1f0 con 0x7fd4f014aa20 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f015af10 con 0x7fd4f01f3d60 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f0073080 con 0x7fd4d8004810 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f014aa20 0x7fd4f014ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f014aa20 0x7fd4f014ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33666/0 (socket says 172.21.15.67:33666) 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 -- 172.21.15.67:0/2501104339 learned_addr learned my addr 172.21.15.67:0/2501104339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f52fc640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4f01f3d60 0x7fd4f01f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f4afb640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4d8004810 0x7fd4f015cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 -- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4d8004810 msgr2=0x7fd4f015cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4d8004810 0x7fd4f015cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 -- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4f01f3d60 msgr2=0x7fd4f01f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4f01f3d60 0x7fd4f01f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 -- 172.21.15.67:0/2501104339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f014b370 con 0x7fd4f014aa20 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4effff640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f014aa20 0x7fd4f014ae30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd4e401f1a0 tx=0x7fd4d8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4ed7fa640 1 -- 172.21.15.67:0/2501104339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40047f0 con 0x7fd4f014aa20 2024-04-25T15:50:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4ed7fa640 1 -- 172.21.15.67:0/2501104339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4e40b6040 con 0x7fd4f014aa20 2024-04-25T15:50:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f015d050 con 0x7fd4f014aa20 2024-04-25T15:50:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4ed7fa640 1 -- 172.21.15.67:0/2501104339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4004bd0 con 0x7fd4f014aa20 2024-04-25T15:50:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f01532d0 con 0x7fd4f014aa20 2024-04-25T15:50:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.437+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f006a480 con 0x7fd4f014aa20 2024-04-25T15:50:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.441+0000 7fd4ed7fa640 1 -- 172.21.15.67:0/2501104339 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd4e4007070 con 0x7fd4f014aa20 2024-04-25T15:50:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.441+0000 7fd4ed7fa640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd4dc06f400 0x7fd4dc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.441+0000 7fd4ed7fa640 1 -- 172.21.15.67:0/2501104339 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd4e412c7b0 con 0x7fd4f014aa20 2024-04-25T15:50:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.441+0000 7fd4f4afb640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd4dc06f400 0x7fd4dc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.441+0000 7fd4f4afb640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd4dc06f400 0x7fd4dc071910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0007fd0 tx=0x7fd4e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:54.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.445+0000 7fd4ed7fa640 1 -- 172.21.15.67:0/2501104339 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd4e40c1060 con 0x7fd4f014aa20 2024-04-25T15:50:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.645+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd4f014bac0 con 0x7fd4f014aa20 2024-04-25T15:50:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.645+0000 7fd4ed7fa640 1 -- 172.21.15.67:0/2501104339 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7fd4f014b370 con 0x7fd4f014aa20 2024-04-25T15:50:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:54.649 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T15:50:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd4dc06f400 msgr2=0x7fd4dc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd4dc06f400 0x7fd4dc071910 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0007fd0 tx=0x7fd4e0009040 comp rx=0 tx=0).stop 2024-04-25T15:50:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f014aa20 msgr2=0x7fd4f014ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f014aa20 0x7fd4f014ae30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd4e401f1a0 tx=0x7fd4d8005840 comp rx=0 tx=0).stop 2024-04-25T15:50:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 shutdown_connections 2024-04-25T15:50:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd4dc06f400 0x7fd4dc071910 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4f01f3d60 0x7fd4f01f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f014aa20 0x7fd4f014ae30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 --2- 172.21.15.67:0/2501104339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4d8004810 0x7fd4f015cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 >> 172.21.15.67:0/2501104339 conn(0x7fd4f006fe70 msgr2=0x7fd4f014e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 shutdown_connections 2024-04-25T15:50:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:54.649+0000 7fd4f6d5f640 1 -- 172.21.15.67:0/2501104339 wait complete. 2024-04-25T15:50:55.639 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-04-25T15:50:55.636+0000 7f240c636640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T15:50:57.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:50:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.833+0000 7f0be45af640 1 Processor -- start 2024-04-25T15:50:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.833+0000 7f0be45af640 1 -- start start 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be45af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 0x7f0bdc06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be45af640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bdc06d780 con 0x7f0bdc0b5d70 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be45af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bdc160920 con 0x7f0bdc06ca60 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be45af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc15d510 con 0x7f0bdc160510 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1b4a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0bdc0b5d70 0x7f0bdc15fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33196/0 (socket says 172.21.15.67:33196) 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1b4a640 1 -- 172.21.15.67:0/911114152 learned_addr learned my addr 172.21.15.67:0/911114152 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be2b4c640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 0x7f0bdc06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234956285 0 0) 0x7f0bdc06d780 con 0x7f0bdc0b5d70 2024-04-25T15:50:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bc8003530 con 0x7f0bdc0b5d70 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2559968287 0 0) 0x7f0bc8003530 con 0x7f0bdc0b5d70 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 msgr2=0x7f0bdc06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 0x7f0bdc06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 >> v1:172.21.15.176:6789/0 conn(0x7f0bdc06ca60 legacy=0x7f0bdc0b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc160be0 con 0x7f0bdc0b5d70 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f0bd00016a0 con 0x7f0bdc0b5d70 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 >> v1:172.21.15.67:6789/0 conn(0x7f0bdc0b5d70 legacy=0x7f0bdc15fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 0x7f0bc80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bc8003f70 0x7f0bc8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bc8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc15d510 con 0x7f0bc8004d50 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc800bbf0 con 0x7f0bc8003f70 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc800bdf0 con 0x7f0bdc160510 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be2b4c640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bc8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be234b640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 0x7f0bc80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be1b4a640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bc8003f70 0x7f0bc8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be2b4c640 1 -- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 msgr2=0x7f0bc80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be2b4c640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc160510 0x7f0bc80039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be2b4c640 1 -- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bc8003f70 msgr2=0x7f0bc8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be2b4c640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bc8003f70 0x7f0bc8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.837+0000 7f0be2b4c640 1 -- 172.21.15.67:0/911114152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc8003530 con 0x7f0bc8004d50 2024-04-25T15:50:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be2b4c640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bc8005160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009fc0 tx=0x7f0bd401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2791188eb246b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd40b2070 con 0x7f0bc8004d50 2024-04-25T15:50:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd40a5cb0 con 0x7f0bc8004d50 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be1349640 1 -- 172.21.15.67:0/911114152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4009c10 con 0x7f0bc8004d50 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 -- 172.21.15.67:0/911114152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 msgr2=0x7f0bc8005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 --2- 172.21.15.67:0/911114152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bc8005160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009fc0 tx=0x7f0bd401cca0 comp rx=0 tx=0).stop 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be2b4c640 1 -- 172.21.15.67:0/911114152 reap_dead start 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 -- 172.21.15.67:0/911114152 shutdown_connections 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 -- 172.21.15.67:0/911114152 >> 172.21.15.67:0/911114152 conn(0x7f0bdc06e420 msgr2=0x7f0bdc06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 -- 172.21.15.67:0/911114152 shutdown_connections 2024-04-25T15:50:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 -- 172.21.15.67:0/911114152 wait complete. 2024-04-25T15:50:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 Processor -- start 2024-04-25T15:50:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.841+0000 7f0be45af640 1 -- start start 2024-04-25T15:50:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bdc1f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bdc0b4650 0x7f0bdc0d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc0d15f0 0x7f0bdc14fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc15d1a0 con 0x7f0bc8004d50 2024-04-25T15:50:57.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc15cec0 con 0x7f0bdc0b4650 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc15c890 con 0x7f0bdc0d15f0 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bdc1f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bdc1f4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33680/0 (socket says 172.21.15.67:33680) 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 -- 172.21.15.67:0/2290840924 learned_addr learned my addr 172.21.15.67:0/2290840924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be1b4a640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bdc0b4650 0x7f0bdc0d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be2b4c640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc0d15f0 0x7f0bdc14fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 -- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc0d15f0 msgr2=0x7f0bdc14fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc0d15f0 0x7f0bdc14fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 -- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bdc0b4650 msgr2=0x7f0bdc0d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bdc0b4650 0x7f0bdc0d10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 -- 172.21.15.67:0/2290840924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc1e7010 con 0x7f0bc8004d50 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be2b4c640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc0d15f0 0x7f0bdc14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be234b640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bdc1f4460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800f5f0 tx=0x7f0bd800f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0bceffd640 1 -- 172.21.15.67:0/2290840924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd80052e0 con 0x7f0bc8004d50 2024-04-25T15:50:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0bceffd640 1 -- 172.21.15.67:0/2290840924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd8018070 con 0x7f0bc8004d50 2024-04-25T15:50:57.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc1e72f0 con 0x7f0bc8004d50 2024-04-25T15:50:57.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0bceffd640 1 -- 172.21.15.67:0/2290840924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8005480 con 0x7f0bc8004d50 2024-04-25T15:50:57.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc1e78b0 con 0x7f0bc8004d50 2024-04-25T15:50:57.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc06a480 con 0x7f0bc8004d50 2024-04-25T15:50:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.845+0000 7f0bceffd640 1 -- 172.21.15.67:0/2290840924 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0bd8003840 con 0x7f0bc8004d50 2024-04-25T15:50:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.849+0000 7f0bceffd640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0bc006f400 0x7f0bc0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.849+0000 7f0bceffd640 1 -- 172.21.15.67:0/2290840924 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0bd808efb0 con 0x7f0bc8004d50 2024-04-25T15:50:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.849+0000 7f0be1b4a640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0bc006f400 0x7f0bc0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:57.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.849+0000 7f0be1b4a640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0bc006f400 0x7f0bc0071910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc15be80 tx=0x7f0bd0007900 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:57.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:57.853+0000 7f0bceffd640 1 -- 172.21.15.67:0/2290840924 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0bd805bf30 con 0x7f0bc8004d50 2024-04-25T15:50:58.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.057+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0bdc15bfd0 con 0x7f0bc8004d50 2024-04-25T15:50:58.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.057+0000 7f0bceffd640 1 -- 172.21.15.67:0/2290840924 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f0bd805b830 con 0x7f0bc8004d50 2024-04-25T15:50:58.061 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:50:58.061 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-25T15:50:58.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0bc006f400 msgr2=0x7f0bc0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:58.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0bc006f400 0x7f0bc0071910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc15be80 tx=0x7f0bd0007900 comp rx=0 tx=0).stop 2024-04-25T15:50:58.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 msgr2=0x7f0bdc1f4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:58.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bdc1f4460 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800f5f0 tx=0x7f0bd800f9f0 comp rx=0 tx=0).stop 2024-04-25T15:50:58.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 shutdown_connections 2024-04-25T15:50:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0bc006f400 0x7f0bc0071910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0bdc0d15f0 0x7f0bdc14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0bdc0b4650 0x7f0bdc0d10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 --2- 172.21.15.67:0/2290840924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bc8004d50 0x7f0bdc1f4460 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 >> 172.21.15.67:0/2290840924 conn(0x7f0bdc06e420 msgr2=0x7f0bdc0cca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 shutdown_connections 2024-04-25T15:50:58.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:50:58.061+0000 7f0be45af640 1 -- 172.21.15.67:0/2290840924 wait complete. 2024-04-25T15:50:58.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:50:58.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:51:00.640 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-04-25T15:51:00.636+0000 7f240c636640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T15:51:01.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T15:51:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.245+0000 7f512f5ec640 1 Processor -- start 2024-04-25T15:51:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.245+0000 7f512f5ec640 1 -- start start 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512f5ec640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f512815dd00 0x7f5128068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512f5ec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5128156db0 0x7f5128167190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512f5ec640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5128160990 con 0x7f51281569a0 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512f5ec640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f512815e110 con 0x7f5128156db0 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512f5ec640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5128160c70 con 0x7f512815dd00 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512cb87640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f51281569a0 0x7f512806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33198/0 (socket says 172.21.15.67:33198) 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512cb87640 1 -- 172.21.15.67:0/3272875887 learned_addr learned my addr 172.21.15.67:0/3272875887 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512db89640 1 --2- 172.21.15.67:0/3272875887 >> v2:172.21.15.67:3300/0 conn(0x7f5128156db0 0x7f5128167190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512d388640 1 --2- 172.21.15.67:0/3272875887 >> v2:172.21.15.176:3300/0 conn(0x7f512815dd00 0x7f5128068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f511ffff640 1 -- 172.21.15.67:0/3272875887 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678417070 0 0) 0x7f5128160990 con 0x7f51281569a0 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f511ffff640 1 -- 172.21.15.67:0/3272875887 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5114003530 con 0x7f51281569a0 2024-04-25T15:51:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512db89640 1 -- 172.21.15.67:0/3272875887 >> v2:172.21.15.176:3300/0 conn(0x7f512815dd00 msgr2=0x7f5128068830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512db89640 1 --2- 172.21.15.67:0/3272875887 >> v2:172.21.15.176:3300/0 conn(0x7f512815dd00 0x7f5128068830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512db89640 1 -- 172.21.15.67:0/3272875887 >> v1:172.21.15.67:6789/0 conn(0x7f51281569a0 legacy=0x7f512806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.249+0000 7f512db89640 1 -- 172.21.15.67:0/3272875887 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51280c3bf0 con 0x7f5128156db0 2024-04-25T15:51:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512db89640 1 --2- 172.21.15.67:0/3272875887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5128156db0 0x7f5128167190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5128071a40 tx=0x7f512001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471e2560fbdb512 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f511ffff640 1 -- 172.21.15.67:0/3272875887 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200b2070 con 0x7f5128156db0 2024-04-25T15:51:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f511ffff640 1 -- 172.21.15.67:0/3272875887 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51200a57f0 con 0x7f5128156db0 2024-04-25T15:51:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f511ffff640 1 -- 172.21.15.67:0/3272875887 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200a5bc0 con 0x7f5128156db0 2024-04-25T15:51:01.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 -- 172.21.15.67:0/3272875887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5128156db0 msgr2=0x7f5128167190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 --2- 172.21.15.67:0/3272875887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5128156db0 0x7f5128167190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5128071a40 tx=0x7f512001c930 comp rx=0 tx=0).stop 2024-04-25T15:51:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 -- 172.21.15.67:0/3272875887 shutdown_connections 2024-04-25T15:51:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 --2- 172.21.15.67:0/3272875887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5128156db0 0x7f5128167190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 --2- 172.21.15.67:0/3272875887 >> v2:172.21.15.176:3300/0 conn(0x7f512815dd00 0x7f5128068830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 -- 172.21.15.67:0/3272875887 >> 172.21.15.67:0/3272875887 conn(0x7f51280ce1c0 msgr2=0x7f51280ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 -- 172.21.15.67:0/3272875887 shutdown_connections 2024-04-25T15:51:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 -- 172.21.15.67:0/3272875887 wait complete. 2024-04-25T15:51:01.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 Processor -- start 2024-04-25T15:51:01.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.253+0000 7f512f5ec640 1 -- start start 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51281569a0 0x7f51281f9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5128156db0 0x7f512815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f512815dd00 0x7f5128147220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f512806d7e0 con 0x7f51281569a0 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f512806d660 con 0x7f5128156db0 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f512806cd50 con 0x7f512815dd00 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51281569a0 0x7f51281f9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51281569a0 0x7f51281f9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33694/0 (socket says 172.21.15.67:33694) 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 -- 172.21.15.67:0/1266541869 learned_addr learned my addr 172.21.15.67:0/1266541869 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512cb87640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5128156db0 0x7f512815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512db89640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f512815dd00 0x7f5128147220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 -- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f512815dd00 msgr2=0x7f5128147220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f512815dd00 0x7f5128147220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 -- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5128156db0 msgr2=0x7f512815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5128156db0 0x7f512815cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 -- 172.21.15.67:0/1266541869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51281f9a60 con 0x7f51281569a0 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512db89640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f512815dd00 0x7f5128147220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512cb87640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5128156db0 0x7f512815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512d388640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51281569a0 0x7f51281f9520 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f511800e870 tx=0x7f511800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f511dffb640 1 -- 172.21.15.67:0/1266541869 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118017070 con 0x7f51281569a0 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f511dffb640 1 -- 172.21.15.67:0/1266541869 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511800c940 con 0x7f51281569a0 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5128158a60 con 0x7f51281569a0 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f511dffb640 1 -- 172.21.15.67:0/1266541869 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511800ccf0 con 0x7f51281569a0 2024-04-25T15:51:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5128159020 con 0x7f51281569a0 2024-04-25T15:51:01.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f512806a480 con 0x7f51281569a0 2024-04-25T15:51:01.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.257+0000 7f511dffb640 1 -- 172.21.15.67:0/1266541869 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f511801a070 con 0x7f51281569a0 2024-04-25T15:51:01.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.261+0000 7f511dffb640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510c06f430 0x7f510c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.261+0000 7f511dffb640 1 -- 172.21.15.67:0/1266541869 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f511808f140 con 0x7f51281569a0 2024-04-25T15:51:01.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.261+0000 7f512cb87640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510c06f430 0x7f510c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.261+0000 7f512cb87640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510c06f430 0x7f510c071940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5128069f70 tx=0x7f5124009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.265+0000 7f511dffb640 1 -- 172.21.15.67:0/1266541869 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f511805c0b0 con 0x7f51281569a0 2024-04-25T15:51:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.465+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5128152ce0 con 0x7f51281569a0 2024-04-25T15:51:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.465+0000 7f511dffb640 1 -- 172.21.15.67:0/1266541869 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f511805b9b0 con 0x7f51281569a0 2024-04-25T15:51:01.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:01.467 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T15:51:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.465+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510c06f430 msgr2=0x7f510c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.465+0000 7f512f5ec640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510c06f430 0x7f510c071940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5128069f70 tx=0x7f5124009040 comp rx=0 tx=0).stop 2024-04-25T15:51:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.465+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51281569a0 msgr2=0x7f51281f9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51281569a0 0x7f51281f9520 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f511800e870 tx=0x7f511800ec70 comp rx=0 tx=0).stop 2024-04-25T15:51:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 shutdown_connections 2024-04-25T15:51:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510c06f430 0x7f510c071940 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f512815dd00 0x7f5128147220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5128156db0 0x7f512815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 --2- 172.21.15.67:0/1266541869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51281569a0 0x7f51281f9520 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 >> 172.21.15.67:0/1266541869 conn(0x7f51280ce1c0 msgr2=0x7f51281518c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 shutdown_connections 2024-04-25T15:51:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.469+0000 7f512f5ec640 1 -- 172.21.15.67:0/1266541869 wait complete. 2024-04-25T15:51:01.488 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T15:51:01.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:51:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb24858640 1 Processor -- start 2024-04-25T15:51:01.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb24858640 1 -- start start 2024-04-25T15:51:01.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb24858640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 0x7fdb2006ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb24858640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb200c8fa0 con 0x7fdb2015c750 2024-04-25T15:51:01.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb24858640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb200c9600 con 0x7fdb20073a90 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb24858640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdb2015cb60 con 0x7fdb2006c640 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdb2015c750 0x7fdb201568a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33210/0 (socket says 172.21.15.67:33210) 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1e59c640 1 -- 172.21.15.67:0/2490067481 learned_addr learned my addr 172.21.15.67:0/2490067481 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 0x7fdb2006ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134178498 0 0) 0x7fdb200c8fa0 con 0x7fdb2015c750 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb08003530 con 0x7fdb2015c750 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2439275325 0 0) 0x7fdb08003530 con 0x7fdb2015c750 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 msgr2=0x7fdb2006ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 0x7fdb2006ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 >> v1:172.21.15.176:6789/0 conn(0x7fdb20073a90 legacy=0x7fdb20068590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb200c98e0 con 0x7fdb2015c750 2024-04-25T15:51:01.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fdb14001680 con 0x7fdb2015c750 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 >> v1:172.21.15.67:6789/0 conn(0x7fdb2015c750 legacy=0x7fdb201568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb080039f0 0x7fdb08003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 0x7fdb08004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb08004d50 0x7fdb08005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2015cb60 con 0x7fdb08004d50 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0800bb00 con 0x7fdb080039f0 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0800bd00 con 0x7fdb2006c640 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb08004d50 0x7fdb08005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1e59c640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb080039f0 0x7fdb08003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1dd9b640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 0x7fdb08004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.629+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 msgr2=0x7fdb08004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb2006c640 0x7fdb08004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb080039f0 msgr2=0x7fdb08003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb080039f0 0x7fdb08003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/2490067481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb08003530 con 0x7fdb08004d50 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb08004d50 0x7fdb08005160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009f90 tx=0x7fdb1801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d319fc2a51900cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb180b2070 con 0x7fdb08004d50 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb180a5c70 con 0x7fdb08004d50 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1d59a640 1 -- 172.21.15.67:0/2490067481 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18009c10 con 0x7fdb08004d50 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb24858640 1 -- 172.21.15.67:0/2490067481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb08004d50 msgr2=0x7fdb08005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb24858640 1 --2- 172.21.15.67:0/2490067481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb08004d50 0x7fdb08005160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdb18009f90 tx=0x7fdb1801cc60 comp rx=0 tx=0).stop 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/2490067481 reap_dead start 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb24858640 1 -- 172.21.15.67:0/2490067481 shutdown_connections 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb24858640 1 -- 172.21.15.67:0/2490067481 >> 172.21.15.67:0/2490067481 conn(0x7fdb20069220 msgr2=0x7fdb200cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb24858640 1 -- 172.21.15.67:0/2490067481 shutdown_connections 2024-04-25T15:51:01.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.633+0000 7fdb24858640 1 -- 172.21.15.67:0/2490067481 wait complete. 2024-04-25T15:51:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 Processor -- start 2024-04-25T15:51:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 -- start start 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb08004d50 0x7fdb201e5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb201495b0 0x7fdb201e7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb201499c0 0x7fdb201e7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb20072490 con 0x7fdb201499c0 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdb20073ea0 con 0x7fdb08004d50 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdb200c92b0 con 0x7fdb201495b0 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb201499c0 0x7fdb201e7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb201499c0 0x7fdb201e7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33704/0 (socket says 172.21.15.67:33704) 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/1625032436 learned_addr learned my addr 172.21.15.67:0/1625032436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1dd9b640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb201495b0 0x7fdb201e7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1e59c640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb08004d50 0x7fdb201e5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb201495b0 msgr2=0x7fdb201e7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb201495b0 0x7fdb201e7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb08004d50 msgr2=0x7fdb201e5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb08004d50 0x7fdb201e5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 -- 172.21.15.67:0/1625032436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb201e82d0 con 0x7fdb201499c0 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb1ed9d640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb201499c0 0x7fdb201e7d90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb180b0040 tx=0x7fdb18009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb0effd640 1 -- 172.21.15.67:0/1625032436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb180b2040 con 0x7fdb201499c0 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb0effd640 1 -- 172.21.15.67:0/1625032436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb180a5a70 con 0x7fdb201499c0 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb20154d00 con 0x7fdb201499c0 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb0effd640 1 -- 172.21.15.67:0/1625032436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18009c10 con 0x7fdb201499c0 2024-04-25T15:51:01.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.637+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2015b8e0 con 0x7fdb201499c0 2024-04-25T15:51:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.641+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2006a480 con 0x7fdb201499c0 2024-04-25T15:51:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.641+0000 7fdb0effd640 1 -- 172.21.15.67:0/1625032436 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdb180be020 con 0x7fdb201499c0 2024-04-25T15:51:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.641+0000 7fdb0effd640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb0006f400 0x7fdb00071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.641+0000 7fdb0effd640 1 -- 172.21.15.67:0/1625032436 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdb18129800 con 0x7fdb201499c0 2024-04-25T15:51:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.641+0000 7fdb1e59c640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb0006f400 0x7fdb00071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.641+0000 7fdb1e59c640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb0006f400 0x7fdb00071910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb140059a0 tx=0x7fdb14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.645+0000 7fdb0effd640 1 -- 172.21.15.67:0/1625032436 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdb201e82d0 con 0x7fdb201499c0 2024-04-25T15:51:01.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.801+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb2015a170 con 0x7fdb201499c0 2024-04-25T15:51:01.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb0effd640 1 -- 172.21.15.67:0/1625032436 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7fdb180a5e20 con 0x7fdb201499c0 2024-04-25T15:51:01.806 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:01.806 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:32.143126+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:51:01.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb0006f400 msgr2=0x7fdb00071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb0006f400 0x7fdb00071910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb140059a0 tx=0x7fdb14009040 comp rx=0 tx=0).stop 2024-04-25T15:51:01.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb201499c0 msgr2=0x7fdb201e7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb201499c0 0x7fdb201e7d90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb180b0040 tx=0x7fdb18009750 comp rx=0 tx=0).stop 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 shutdown_connections 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb0006f400 0x7fdb00071910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb201499c0 0x7fdb201e7d90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdb201495b0 0x7fdb201e7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 --2- 172.21.15.67:0/1625032436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdb08004d50 0x7fdb201e5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 >> 172.21.15.67:0/1625032436 conn(0x7fdb20069220 msgr2=0x7fdb201568a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 shutdown_connections 2024-04-25T15:51:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.805+0000 7fdb24858640 1 -- 172.21.15.67:0/1625032436 wait complete. 2024-04-25T15:51:01.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:51:01.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:51:01.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd540089640 1 Processor -- start 2024-04-25T15:51:01.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd540089640 1 -- start start 2024-04-25T15:51:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd540089640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd538068840 0x7fd5380c7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd540089640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5380c7bd0 0x7fd53815a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd540089640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5380cb320 con 0x7fd53815aa50 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd540089640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd53815ae60 con 0x7fd5380c7bd0 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd540089640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd5380732d0 con 0x7fd538068840 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53d624640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5380c7bd0 0x7fd53815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53d624640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5380c7bd0 0x7fd53815a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33706/0 (socket says 172.21.15.67:33706) 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53d624640 1 -- 172.21.15.67:0/2201962476 learned_addr learned my addr 172.21.15.67:0/2201962476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53de25640 1 --2- 172.21.15.67:0/2201962476 >> v2:172.21.15.176:3300/0 conn(0x7fd538068840 0x7fd5380c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53d624640 1 -- 172.21.15.67:0/2201962476 >> v2:172.21.15.176:3300/0 conn(0x7fd538068840 msgr2=0x7fd5380c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53d624640 1 --2- 172.21.15.67:0/2201962476 >> v2:172.21.15.176:3300/0 conn(0x7fd538068840 0x7fd5380c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53d624640 1 -- 172.21.15.67:0/2201962476 >> v1:172.21.15.176:6789/0 conn(0x7fd53815aa50 legacy=0x7fd53806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.969+0000 7fd53d624640 1 -- 172.21.15.67:0/2201962476 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5380cb5c0 con 0x7fd5380c7bd0 2024-04-25T15:51:01.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd53d624640 1 --2- 172.21.15.67:0/2201962476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380c7bd0 0x7fd53815a510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd5380cefb0 tx=0x7fd52801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416357adbc966a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd53ce23640 1 -- 172.21.15.67:0/2201962476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5280b0070 con 0x7fd5380c7bd0 2024-04-25T15:51:01.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd53ce23640 1 -- 172.21.15.67:0/2201962476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5280ab600 con 0x7fd5380c7bd0 2024-04-25T15:51:01.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd53ce23640 1 -- 172.21.15.67:0/2201962476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5280ab9d0 con 0x7fd5380c7bd0 2024-04-25T15:51:01.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 -- 172.21.15.67:0/2201962476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380c7bd0 msgr2=0x7fd53815a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 --2- 172.21.15.67:0/2201962476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380c7bd0 0x7fd53815a510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd5380cefb0 tx=0x7fd52801c520 comp rx=0 tx=0).stop 2024-04-25T15:51:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 -- 172.21.15.67:0/2201962476 shutdown_connections 2024-04-25T15:51:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 --2- 172.21.15.67:0/2201962476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5380c7bd0 0x7fd53815a510 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 --2- 172.21.15.67:0/2201962476 >> v2:172.21.15.176:3300/0 conn(0x7fd538068840 0x7fd5380c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 -- 172.21.15.67:0/2201962476 >> 172.21.15.67:0/2201962476 conn(0x7fd538069220 msgr2=0x7fd5380cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 -- 172.21.15.67:0/2201962476 shutdown_connections 2024-04-25T15:51:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 -- 172.21.15.67:0/2201962476 wait complete. 2024-04-25T15:51:01.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.973+0000 7fd540089640 1 Processor -- start 2024-04-25T15:51:01.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 -- start start 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538068840 0x7fd538147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5380c7bd0 0x7fd53814fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd53815aa50 0x7fd538141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5380ca120 con 0x7fd538068840 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd5380ca9e0 con 0x7fd5380c7bd0 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd5380725d0 con 0x7fd53815aa50 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538068840 0x7fd538147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538068840 0x7fd538147e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33712/0 (socket says 172.21.15.67:33712) 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 -- 172.21.15.67:0/3973093579 learned_addr learned my addr 172.21.15.67:0/3973093579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53d624640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5380c7bd0 0x7fd53814fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53e626640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd53815aa50 0x7fd538141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 -- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd53815aa50 msgr2=0x7fd538141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd53815aa50 0x7fd538141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 -- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5380c7bd0 msgr2=0x7fd53814fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5380c7bd0 0x7fd53814fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 -- 172.21.15.67:0/3973093579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd538145aa0 con 0x7fd538068840 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd53de25640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538068840 0x7fd538147e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd53400d600 tx=0x7fd53400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd52e7fc640 1 -- 172.21.15.67:0/3973093579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534004240 con 0x7fd538068840 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd52e7fc640 1 -- 172.21.15.67:0/3973093579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5340043e0 con 0x7fd538068840 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd52e7fc640 1 -- 172.21.15.67:0/3973093579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534003840 con 0x7fd538068840 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5381f73b0 con 0x7fd538068840 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5381f7970 con 0x7fd538068840 2024-04-25T15:51:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd53806a480 con 0x7fd538068840 2024-04-25T15:51:01.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.977+0000 7fd52e7fc640 1 -- 172.21.15.67:0/3973093579 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd5340039e0 con 0x7fd538068840 2024-04-25T15:51:01.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.981+0000 7fd52e7fc640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd51c06f400 0x7fd51c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:01.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.981+0000 7fd52e7fc640 1 -- 172.21.15.67:0/3973093579 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd53408f410 con 0x7fd538068840 2024-04-25T15:51:01.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.981+0000 7fd53d624640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd51c06f400 0x7fd51c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:01.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.981+0000 7fd53d624640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd51c06f400 0x7fd51c071910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd5280ad000 tx=0x7fd528008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:01.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:01.985+0000 7fd52e7fc640 1 -- 172.21.15.67:0/3973093579 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd53405c420 con 0x7fd538068840 2024-04-25T15:51:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.145+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd538072e30 con 0x7fd538068840 2024-04-25T15:51:02.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.145+0000 7fd52e7fc640 1 -- 172.21.15.67:0/3973093579 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fd53405bd20 con 0x7fd538068840 2024-04-25T15:51:02.148 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:51:02.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd51c06f400 msgr2=0x7fd51c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd51c06f400 0x7fd51c071910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd5280ad000 tx=0x7fd528008040 comp rx=0 tx=0).stop 2024-04-25T15:51:02.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538068840 msgr2=0x7fd538147e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538068840 0x7fd538147e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd53400d600 tx=0x7fd53400dac0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 shutdown_connections 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd51c06f400 0x7fd51c071910 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd53815aa50 0x7fd538141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5380c7bd0 0x7fd53814fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 --2- 172.21.15.67:0/3973093579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd538068840 0x7fd538147e60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 >> 172.21.15.67:0/3973093579 conn(0x7fd538069220 msgr2=0x7fd53806c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 shutdown_connections 2024-04-25T15:51:02.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.149+0000 7fd540089640 1 -- 172.21.15.67:0/3973093579 wait complete. 2024-04-25T15:51:02.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:51:02.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd76bc640 1 Processor -- start 2024-04-25T15:51:02.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd76bc640 1 -- start start 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd76bc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bd01663f0 0x7f6bd0168810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd76bc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd0160fa0 con 0x7f6bd0156f10 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd76bc640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd00c6d10 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd76bc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6bd0157320 con 0x7f6bd01663f0 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd5c59640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bd01663f0 0x7f6bd0168810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd5458640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6bd0156f10 0x7f6bd015d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33218/0 (socket says 172.21.15.67:33218) 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd5458640 1 -- 172.21.15.67:0/2017661049 learned_addr learned my addr 172.21.15.67:0/2017661049 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd5c59640 1 -- 172.21.15.67:0/2017661049 >> v1:172.21.15.176:6789/0 conn(0x7f6bd00cbfc0 legacy=0x7f6bd00cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd5c59640 1 -- 172.21.15.67:0/2017661049 >> v1:172.21.15.67:6789/0 conn(0x7f6bd0156f10 legacy=0x7f6bd015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.321+0000 7f6bd5c59640 1 -- 172.21.15.67:0/2017661049 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd0160420 con 0x7f6bd01663f0 2024-04-25T15:51:02.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd5c59640 1 --2- 172.21.15.67:0/2017661049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd01663f0 0x7f6bd0168810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009d90 tx=0x7f6bc801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e07b9a90263284 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bc7fff640 1 -- 172.21.15.67:0/2017661049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc80b2070 con 0x7f6bd01663f0 2024-04-25T15:51:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bc7fff640 1 -- 172.21.15.67:0/2017661049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bc80a5830 con 0x7f6bd01663f0 2024-04-25T15:51:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bc7fff640 1 -- 172.21.15.67:0/2017661049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc80a5c00 con 0x7f6bd01663f0 2024-04-25T15:51:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 -- 172.21.15.67:0/2017661049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd01663f0 msgr2=0x7f6bd0168810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/2017661049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd01663f0 0x7f6bd0168810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009d90 tx=0x7f6bc801c970 comp rx=0 tx=0).stop 2024-04-25T15:51:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 -- 172.21.15.67:0/2017661049 shutdown_connections 2024-04-25T15:51:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/2017661049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd01663f0 0x7f6bd0168810 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 -- 172.21.15.67:0/2017661049 >> 172.21.15.67:0/2017661049 conn(0x7f6bd0068c70 msgr2=0x7f6bd0071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 -- 172.21.15.67:0/2017661049 shutdown_connections 2024-04-25T15:51:02.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 -- 172.21.15.67:0/2017661049 wait complete. 2024-04-25T15:51:02.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 Processor -- start 2024-04-25T15:51:02.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.325+0000 7f6bd76bc640 1 -- start start 2024-04-25T15:51:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd00cbfc0 0x7f6bd0167340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bd0156f10 0x7f6bd01475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bd01663f0 0x7f6bd014ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd01600c0 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd015ccf0 con 0x7f6bd01663f0 2024-04-25T15:51:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd015d920 con 0x7f6bd0156f10 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd00cbfc0 0x7f6bd0167340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd00cbfc0 0x7f6bd0167340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33728/0 (socket says 172.21.15.67:33728) 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 -- 172.21.15.67:0/926723414 learned_addr learned my addr 172.21.15.67:0/926723414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5c59640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bd01663f0 0x7f6bd014ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd4c57640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bd0156f10 0x7f6bd01475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 -- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bd0156f10 msgr2=0x7f6bd01475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bd0156f10 0x7f6bd01475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 -- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bd01663f0 msgr2=0x7f6bd014ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bd01663f0 0x7f6bd014ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 -- 172.21.15.67:0/926723414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd01668a0 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd5458640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd00cbfc0 0x7f6bd0167340 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c400 tx=0x7f6bcc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bc5ffb640 1 -- 172.21.15.67:0/926723414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc017070 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bc5ffb640 1 -- 172.21.15.67:0/926723414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bcc005cf0 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd00cf3f0 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bc5ffb640 1 -- 172.21.15.67:0/926723414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc0072d0 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd00cf6d0 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd006a480 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bc5ffb640 1 -- 172.21.15.67:0/926723414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6bcc01a070 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bc5ffb640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bb406f430 0x7f6bb4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bc5ffb640 1 -- 172.21.15.67:0/926723414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6bcc08ecd0 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.329+0000 7f6bd4c57640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bb406f430 0x7f6bb4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.333+0000 7f6bd4c57640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bb406f430 0x7f6bb4071940 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0072660 tx=0x7f6bc0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.333+0000 7f6bc5ffb640 1 -- 172.21.15.67:0/926723414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6bcc05b650 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.497+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6bd0069b40 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.497+0000 7f6bc5ffb640 1 -- 172.21.15.67:0/926723414 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f6bcc05f950 con 0x7f6bd00cbfc0 2024-04-25T15:51:02.500 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:51:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bb406f430 msgr2=0x7f6bb4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bb406f430 0x7f6bb4071940 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0072660 tx=0x7f6bc0009040 comp rx=0 tx=0).stop 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd00cbfc0 msgr2=0x7f6bd0167340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd00cbfc0 0x7f6bd0167340 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00c400 tx=0x7f6bcc00c8c0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 shutdown_connections 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bb406f430 0x7f6bb4071940 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bd01663f0 0x7f6bd014ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bd0156f10 0x7f6bd01475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 --2- 172.21.15.67:0/926723414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bd00cbfc0 0x7f6bd0167340 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 >> 172.21.15.67:0/926723414 conn(0x7f6bd0068c70 msgr2=0x7f6bd0075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 shutdown_connections 2024-04-25T15:51:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.501+0000 7f6bd76bc640 1 -- 172.21.15.67:0/926723414 wait complete. 2024-04-25T15:51:02.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:51:02.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 Processor -- start 2024-04-25T15:51:02.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 -- start start 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f910006ca60 0x7f910006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f91000b5660 0x7f91000b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f910015fe00 0x7f9100160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f910015d510 con 0x7f910006ca60 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f910006d780 con 0x7f91000b5660 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f9106c22640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9100160be0 con 0x7f910015fe00 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f910006ca60 0x7f910006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f910006ca60 0x7f910006ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46984/0 (socket says 172.21.15.67:46984) 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 -- 172.21.15.67:0/4153210473 learned_addr learned my addr 172.21.15.67:0/4153210473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f90fffff640 1 --2- 172.21.15.67:0/4153210473 >> v2:172.21.15.176:3300/0 conn(0x7f91000b5660 0x7f91000b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91051bf640 1 --2- 172.21.15.67:0/4153210473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f910015fe00 0x7f9100160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 -- 172.21.15.67:0/4153210473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f910015fe00 msgr2=0x7f9100160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 --2- 172.21.15.67:0/4153210473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f910015fe00 0x7f9100160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 -- 172.21.15.67:0/4153210473 >> v2:172.21.15.176:3300/0 conn(0x7f91000b5660 msgr2=0x7f91000b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 --2- 172.21.15.67:0/4153210473 >> v2:172.21.15.176:3300/0 conn(0x7f91000b5660 0x7f91000b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91049be640 1 -- 172.21.15.67:0/4153210473 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910015cbe0 con 0x7f910006ca60 2024-04-25T15:51:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.673+0000 7f91051bf640 1 --2- 172.21.15.67:0/4153210473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f910015fe00 0x7f9100160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:02.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f91049be640 1 --2- 172.21.15.67:0/4153210473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f910006ce70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009870 tx=0x7f90f001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becfe5258a5d7fe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f90ff7fe640 1 -- 172.21.15.67:0/4153210473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00b1070 con 0x7f910006ca60 2024-04-25T15:51:02.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f90ff7fe640 1 -- 172.21.15.67:0/4153210473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f00ac630 con 0x7f910006ca60 2024-04-25T15:51:02.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f90ff7fe640 1 -- 172.21.15.67:0/4153210473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00a8050 con 0x7f910006ca60 2024-04-25T15:51:02.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 -- 172.21.15.67:0/4153210473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 msgr2=0x7f910006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 --2- 172.21.15.67:0/4153210473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f910006ce70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009870 tx=0x7f90f001c520 comp rx=0 tx=0).stop 2024-04-25T15:51:02.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 -- 172.21.15.67:0/4153210473 shutdown_connections 2024-04-25T15:51:02.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 --2- 172.21.15.67:0/4153210473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f910015fe00 0x7f9100160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 --2- 172.21.15.67:0/4153210473 >> v2:172.21.15.176:3300/0 conn(0x7f91000b5660 0x7f91000b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 --2- 172.21.15.67:0/4153210473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f910006ce70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 -- 172.21.15.67:0/4153210473 >> 172.21.15.67:0/4153210473 conn(0x7f910006e420 msgr2=0x7f910006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 -- 172.21.15.67:0/4153210473 shutdown_connections 2024-04-25T15:51:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.677+0000 7f9106c22640 1 -- 172.21.15.67:0/4153210473 wait complete. 2024-04-25T15:51:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 Processor -- start 2024-04-25T15:51:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 -- start start 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f9100151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91000b5660 0x7f91001550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f910015fe00 0x7f910014b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91000b6e90 con 0x7f910006ca60 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f910015c890 con 0x7f910015fe00 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f91000b7010 con 0x7f91000b5660 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f9100151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f9100151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46996/0 (socket says 172.21.15.67:46996) 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 -- 172.21.15.67:0/3376982601 learned_addr learned my addr 172.21.15.67:0/3376982601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91051bf640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f910015fe00 0x7f910014b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f90fffff640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91000b5660 0x7f91001550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 -- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91000b5660 msgr2=0x7f91001550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91000b5660 0x7f91001550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 -- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f910015fe00 msgr2=0x7f910014b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f910015fe00 0x7f910014b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 -- 172.21.15.67:0/3376982601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9100151c50 con 0x7f910006ca60 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f91049be640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f9100151710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f90f00af040 tx=0x7f90f00192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f90fd7fa640 1 -- 172.21.15.67:0/3376982601 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00bc070 con 0x7f910006ca60 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f90fd7fa640 1 -- 172.21.15.67:0/3376982601 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f00ab5e0 con 0x7f910006ca60 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f90fd7fa640 1 -- 172.21.15.67:0/3376982601 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00b1070 con 0x7f910006ca60 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91001f9690 con 0x7f910006ca60 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9100155ce0 con 0x7f910006ca60 2024-04-25T15:51:02.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.681+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910006a480 con 0x7f910006ca60 2024-04-25T15:51:02.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.685+0000 7f90fd7fa640 1 -- 172.21.15.67:0/3376982601 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f90f00bf070 con 0x7f910006ca60 2024-04-25T15:51:02.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.685+0000 7f90fd7fa640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f90ec06f400 0x7f90ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.685+0000 7f90fd7fa640 1 -- 172.21.15.67:0/3376982601 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f90f0128ab0 con 0x7f910006ca60 2024-04-25T15:51:02.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.685+0000 7f90fffff640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f90ec06f400 0x7f90ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.685+0000 7f90fffff640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f90ec06f400 0x7f90ec071910 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91001f3340 tx=0x7f90f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.689+0000 7f90fd7fa640 1 -- 172.21.15.67:0/3376982601 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f90f0016110 con 0x7f910006ca60 2024-04-25T15:51:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f91000cd380 con 0x7f910006ca60 2024-04-25T15:51:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f90fd7fa640 1 -- 172.21.15.67:0/3376982601 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f90f00f59a0 con 0x7f910006ca60 2024-04-25T15:51:02.851 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:51:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f90ec06f400 msgr2=0x7f90ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f90ec06f400 0x7f90ec071910 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91001f3340 tx=0x7f90f4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 msgr2=0x7f9100151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f9100151710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f90f00af040 tx=0x7f90f00192c0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 shutdown_connections 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f90ec06f400 0x7f90ec071910 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f910015fe00 0x7f910014b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91000b5660 0x7f91001550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 --2- 172.21.15.67:0/3376982601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f910006ca60 0x7f9100151710 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 >> 172.21.15.67:0/3376982601 conn(0x7f910006e420 msgr2=0x7f910006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 shutdown_connections 2024-04-25T15:51:02.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:02.849+0000 7f9106c22640 1 -- 172.21.15.67:0/3376982601 wait complete. 2024-04-25T15:51:02.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:51:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.005+0000 7f20a5949640 1 Processor -- start 2024-04-25T15:51:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.005+0000 7f20a5949640 1 -- start start 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f20a5949640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20a0074370 0x7f20a0074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f20a5949640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20a00c7fc0 0x7f20a00c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f20a5949640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20a00c92d0 con 0x7f20a006c640 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f20a5949640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f20a006e210 con 0x7f20a0074370 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f20a5949640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f20a006e4d0 con 0x7f20a00c7fc0 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209e7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20a0074370 0x7f20a0074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209e7fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20a0074370 0x7f20a0074780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47010/0 (socket says 172.21.15.67:47010) 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209e7fc640 1 -- 172.21.15.67:0/1091017900 learned_addr learned my addr 172.21.15.67:0/1091017900 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209f7fe640 1 --2- 172.21.15.67:0/1091017900 >> v2:172.21.15.176:3300/0 conn(0x7f20a00c7fc0 0x7f20a00c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209e7fc640 1 -- 172.21.15.67:0/1091017900 >> v2:172.21.15.176:3300/0 conn(0x7f20a00c7fc0 msgr2=0x7f20a00c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209e7fc640 1 --2- 172.21.15.67:0/1091017900 >> v2:172.21.15.176:3300/0 conn(0x7f20a00c7fc0 0x7f20a00c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209e7fc640 1 -- 172.21.15.67:0/1091017900 >> v1:172.21.15.176:6789/0 conn(0x7f20a006c640 legacy=0x7f20a015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.009+0000 7f209e7fc640 1 -- 172.21.15.67:0/1091017900 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a006e7b0 con 0x7f20a0074370 2024-04-25T15:51:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f209e7fc640 1 --2- 172.21.15.67:0/1091017900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0074370 0x7f20a0074780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f20a00694c0 tx=0x7f209401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e7ba45c94eb6e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f209dffb640 1 -- 172.21.15.67:0/1091017900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20940b5070 con 0x7f20a0074370 2024-04-25T15:51:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f209dffb640 1 -- 172.21.15.67:0/1091017900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20940a8a10 con 0x7f20a0074370 2024-04-25T15:51:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f209dffb640 1 -- 172.21.15.67:0/1091017900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20940a8de0 con 0x7f20a0074370 2024-04-25T15:51:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 -- 172.21.15.67:0/1091017900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0074370 msgr2=0x7f20a0074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 --2- 172.21.15.67:0/1091017900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0074370 0x7f20a0074780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f20a00694c0 tx=0x7f209401ed10 comp rx=0 tx=0).stop 2024-04-25T15:51:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 -- 172.21.15.67:0/1091017900 shutdown_connections 2024-04-25T15:51:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 --2- 172.21.15.67:0/1091017900 >> v2:172.21.15.176:3300/0 conn(0x7f20a00c7fc0 0x7f20a00c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 --2- 172.21.15.67:0/1091017900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a0074370 0x7f20a0074780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 -- 172.21.15.67:0/1091017900 >> 172.21.15.67:0/1091017900 conn(0x7f20a0070db0 msgr2=0x7f20a00ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 -- 172.21.15.67:0/1091017900 shutdown_connections 2024-04-25T15:51:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 -- 172.21.15.67:0/1091017900 wait complete. 2024-04-25T15:51:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 Processor -- start 2024-04-25T15:51:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.013+0000 7f20a5949640 1 -- start start 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a006c640 0x7f20a01e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20a0074370 0x7f20a0150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00c7fc0 0x7f20a01450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f20a0073e50 con 0x7f20a006c640 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f20a0073b70 con 0x7f20a00c7fc0 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f20a0073420 con 0x7f20a0074370 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a006c640 0x7f20a01e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a006c640 0x7f20a01e3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47018/0 (socket says 172.21.15.67:47018) 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 -- 172.21.15.67:0/679499495 learned_addr learned my addr 172.21.15.67:0/679499495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209f7fe640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00c7fc0 0x7f20a01450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209e7fc640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20a0074370 0x7f20a0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 -- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20a0074370 msgr2=0x7f20a0150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20a0074370 0x7f20a0150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 -- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00c7fc0 msgr2=0x7f20a01450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00c7fc0 0x7f20a01450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 -- 172.21.15.67:0/679499495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a01e5260 con 0x7f20a006c640 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209f7fe640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00c7fc0 0x7f20a01450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209effd640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a006c640 0x7f20a01e3c70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f209000c760 tx=0x7f209000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f2087fff640 1 -- 172.21.15.67:0/679499495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090016020 con 0x7f20a006c640 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f2087fff640 1 -- 172.21.15.67:0/679499495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2090006d80 con 0x7f20a006c640 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a01e67f0 con 0x7f20a006c640 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f2087fff640 1 -- 172.21.15.67:0/679499495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2090003bc0 con 0x7f20a006c640 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a01e5500 con 0x7f20a006c640 2024-04-25T15:51:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a006a480 con 0x7f20a006c640 2024-04-25T15:51:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f2087fff640 1 -- 172.21.15.67:0/679499495 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f209000a7f0 con 0x7f20a006c640 2024-04-25T15:51:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f2087fff640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f208c06f400 0x7f208c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f2087fff640 1 -- 172.21.15.67:0/679499495 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f209008ee90 con 0x7f20a006c640 2024-04-25T15:51:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.017+0000 7f209e7fc640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f208c06f400 0x7f208c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.021+0000 7f209e7fc640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f208c06f400 0x7f208c071910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20940b2000 tx=0x7f2094008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.021+0000 7f2087fff640 1 -- 172.21.15.67:0/679499495 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2090020020 con 0x7f20a006c640 2024-04-25T15:51:03.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.217+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20a01522c0 con 0x7f20a006c640 2024-04-25T15:51:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.217+0000 7f2087fff640 1 -- 172.21.15.67:0/679499495 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6907 (secure 0 0 0) 0x7f209005bea0 con 0x7f20a006c640 2024-04-25T15:51:03.220 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:03.220 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/920422475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":920422475},{"type":"v1","addr":"172.21.15.67:6835","nonce":920422475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:50:30.570677+0000","modified":"2024-04-25T15:50:58.911813+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4294,"mds_1":4288,"mds_2":4276,"mds_3":4272,"mds_4":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"a","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.67:6834/144842397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":144842397},{"type":"v1","addr":"172.21.15.67:6834","nonce":144842397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4272":{"gid":4272,"name":"d","rank":3,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.176:6838/2023672746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2023672746},{"type":"v1","addr":"172.21.15.176:6838","nonce":2023672746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4276":{"gid":4276,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":9,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6835/2385091301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2385091301},{"type":"v1","addr":"172.21.15.176:6835","nonce":2385091301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.176:6839/1886671623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6837","nonce":1886671623},{"type":"v1","addr":"172.21.15.176:6839","nonce":1886671623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294,4288,4276,4272]},"id":1}]} 2024-04-25T15:51:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 16 2024-04-25T15:51:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f208c06f400 msgr2=0x7f208c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f208c06f400 0x7f208c071910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20940b2000 tx=0x7f2094008040 comp rx=0 tx=0).stop 2024-04-25T15:51:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a006c640 msgr2=0x7f20a01e3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a006c640 0x7f20a01e3c70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f209000c760 tx=0x7f209000cc20 comp rx=0 tx=0).stop 2024-04-25T15:51:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 shutdown_connections 2024-04-25T15:51:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f208c06f400 0x7f208c071910 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00c7fc0 0x7f20a01450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f20a0074370 0x7f20a0150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 --2- 172.21.15.67:0/679499495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20a006c640 0x7f20a01e3c70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 >> 172.21.15.67:0/679499495 conn(0x7f20a0070db0 msgr2=0x7f20a00cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 shutdown_connections 2024-04-25T15:51:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.221+0000 7f20a5949640 1 -- 172.21.15.67:0/679499495 wait complete. 2024-04-25T15:51:03.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:51:03.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac64a0f640 1 Processor -- start 2024-04-25T15:51:03.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac64a0f640 1 -- start start 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac64a0f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac6006ca60 0x7fac6006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac64a0f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fac6015fe00 0x7fac60160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac64a0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac6006d780 con 0x7fac600b5660 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac64a0f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fac6015d510 con 0x7fac6006ca60 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac64a0f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fac60160be0 con 0x7fac6015fe00 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac6006ca60 0x7fac6006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5e59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac6006ca60 0x7fac6006ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47030/0 (socket says 172.21.15.67:47030) 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5e59c640 1 -- 172.21.15.67:0/1066766340 learned_addr learned my addr 172.21.15.67:0/1066766340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5ed9d640 1 --2- 172.21.15.67:0/1066766340 >> v2:172.21.15.176:3300/0 conn(0x7fac6015fe00 0x7fac60160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5e59c640 1 -- 172.21.15.67:0/1066766340 >> v2:172.21.15.176:3300/0 conn(0x7fac6015fe00 msgr2=0x7fac60160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:03.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5e59c640 1 --2- 172.21.15.67:0/1066766340 >> v2:172.21.15.176:3300/0 conn(0x7fac6015fe00 0x7fac60160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5e59c640 1 -- 172.21.15.67:0/1066766340 >> v1:172.21.15.176:6789/0 conn(0x7fac600b5660 legacy=0x7fac600b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5e59c640 1 -- 172.21.15.67:0/1066766340 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6015cbe0 con 0x7fac6006ca60 2024-04-25T15:51:03.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.385+0000 7fac5ed9d640 1 --2- 172.21.15.67:0/1066766340 >> v2:172.21.15.176:3300/0 conn(0x7fac6015fe00 0x7fac60160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:03.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac5e59c640 1 --2- 172.21.15.67:0/1066766340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac6006ca60 0x7fac6006ce70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fac50009e50 tx=0x7fac5001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8f06d97bdb1068 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac5d59a640 1 -- 172.21.15.67:0/1066766340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac500a57a0 con 0x7fac6006ca60 2024-04-25T15:51:03.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac5d59a640 1 -- 172.21.15.67:0/1066766340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac500a5940 con 0x7fac6006ca60 2024-04-25T15:51:03.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac5d59a640 1 -- 172.21.15.67:0/1066766340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac500a5d10 con 0x7fac6006ca60 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 -- 172.21.15.67:0/1066766340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac6006ca60 msgr2=0x7fac6006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1066766340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac6006ca60 0x7fac6006ce70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fac50009e50 tx=0x7fac5001c9f0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 -- 172.21.15.67:0/1066766340 shutdown_connections 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1066766340 >> v2:172.21.15.176:3300/0 conn(0x7fac6015fe00 0x7fac60160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1066766340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac6006ca60 0x7fac6006ce70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 -- 172.21.15.67:0/1066766340 >> 172.21.15.67:0/1066766340 conn(0x7fac6006e420 msgr2=0x7fac6006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 -- 172.21.15.67:0/1066766340 shutdown_connections 2024-04-25T15:51:03.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.389+0000 7fac64a0f640 1 -- 172.21.15.67:0/1066766340 wait complete. 2024-04-25T15:51:03.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 Processor -- start 2024-04-25T15:51:03.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 -- start start 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fac6006ca60 0x7fac601f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac600b5660 0x7fac6015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac6015fe00 0x7fac601536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac600b7010 con 0x7fac600b5660 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fac600b6e90 con 0x7fac6015fe00 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fac60160750 con 0x7fac6006ca60 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac600b5660 0x7fac6015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac600b5660 0x7fac6015ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47036/0 (socket says 172.21.15.67:47036) 2024-04-25T15:51:03.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 -- 172.21.15.67:0/1617654577 learned_addr learned my addr 172.21.15.67:0/1617654577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5ed9d640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac6015fe00 0x7fac601536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5e59c640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fac6006ca60 0x7fac601f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 -- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fac6006ca60 msgr2=0x7fac601f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fac6006ca60 0x7fac601f65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 -- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac6015fe00 msgr2=0x7fac601536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac6015fe00 0x7fac601536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 -- 172.21.15.67:0/1617654577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac601f6b00 con 0x7fac600b5660 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac5dd9b640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac600b5660 0x7fac6015ef50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c3f0 tx=0x7fac4800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac56ffd640 1 -- 172.21.15.67:0/1617654577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48017070 con 0x7fac600b5660 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac56ffd640 1 -- 172.21.15.67:0/1617654577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac480052f0 con 0x7fac600b5660 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac600c99b0 con 0x7fac600b5660 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac56ffd640 1 -- 172.21.15.67:0/1617654577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48003860 con 0x7fac600b5660 2024-04-25T15:51:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac600c93f0 con 0x7fac600b5660 2024-04-25T15:51:03.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.393+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac600c8e60 con 0x7fac600b5660 2024-04-25T15:51:03.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.397+0000 7fac56ffd640 1 -- 172.21.15.67:0/1617654577 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fac4801a070 con 0x7fac600b5660 2024-04-25T15:51:03.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.397+0000 7fac56ffd640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac4006f400 0x7fac40071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.397+0000 7fac56ffd640 1 -- 172.21.15.67:0/1617654577 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fac4808e560 con 0x7fac600b5660 2024-04-25T15:51:03.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.397+0000 7fac5e59c640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac4006f400 0x7fac40071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.397+0000 7fac5e59c640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac4006f400 0x7fac40071910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fac500ae000 tx=0x7fac500a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.401+0000 7fac56ffd640 1 -- 172.21.15.67:0/1617654577 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fac4805b4e0 con 0x7fac600b5660 2024-04-25T15:51:03.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.557+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac6006a480 con 0x7fac600b5660 2024-04-25T15:51:03.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.557+0000 7fac56ffd640 1 -- 172.21.15.67:0/1617654577 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7fac4805ade0 con 0x7fac600b5660 2024-04-25T15:51:03.561 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:03.561 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:32.143126+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:51:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac4006f400 msgr2=0x7fac40071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac4006f400 0x7fac40071910 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fac500ae000 tx=0x7fac500a7070 comp rx=0 tx=0).stop 2024-04-25T15:51:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac600b5660 msgr2=0x7fac6015ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac600b5660 0x7fac6015ef50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c3f0 tx=0x7fac4800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 shutdown_connections 2024-04-25T15:51:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac4006f400 0x7fac40071910 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac6015fe00 0x7fac601536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac600b5660 0x7fac6015ef50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 --2- 172.21.15.67:0/1617654577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fac6006ca60 0x7fac601f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 >> 172.21.15.67:0/1617654577 conn(0x7fac6006e420 msgr2=0x7fac6006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 shutdown_connections 2024-04-25T15:51:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.561+0000 7fac64a0f640 1 -- 172.21.15.67:0/1617654577 wait complete. 2024-04-25T15:51:03.582 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:51:03.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:51:03.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f160292f640 1 Processor -- start 2024-04-25T15:51:03.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f160292f640 1 -- start start 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f160292f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f15fc074790 0x7f15fc15dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f160292f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 0x7f15fc0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f160292f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15fc072dc0 con 0x7f15fc15e020 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f160292f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f15fc15b430 con 0x7f15fc074790 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f160292f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15fc15a670 con 0x7f15fc0c6fb0 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15f3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f15fc15e020 0x7f15fc0723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36046/0 (socket says 172.21.15.67:36046) 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15f3fff640 1 -- 172.21.15.67:0/1982347541 learned_addr learned my addr 172.21.15.67:0/1982347541 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f1600ecc640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 0x7f15fc0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fbfff640 1 --2- 172.21.15.67:0/1982347541 >> v2:172.21.15.176:3300/0 conn(0x7f15fc074790 0x7f15fc15dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148817244 0 0) 0x7f15fc072dc0 con 0x7f15fc15e020 2024-04-25T15:51:03.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15e8003530 con 0x7f15fc15e020 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4127079069 0 0) 0x7f15e8003530 con 0x7f15fc15e020 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 msgr2=0x7f15fc0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 0x7f15fc0c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 >> v2:172.21.15.176:3300/0 conn(0x7f15fc074790 msgr2=0x7f15fc15dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 --2- 172.21.15.67:0/1982347541 >> v2:172.21.15.176:3300/0 conn(0x7f15fc074790 0x7f15fc15dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc15a950 con 0x7f15fc15e020 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f15e4001680 con 0x7f15fc15e020 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 >> v1:172.21.15.67:6789/0 conn(0x7f15fc15e020 legacy=0x7f15fc0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e80039c0 0x7f15e8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 0x7f15e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8004d50 0x7f15e8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc15b430 con 0x7f15e8004d50 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc15a670 con 0x7f15e80039c0 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15e800ba50 con 0x7f15fc0c6fb0 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fbfff640 1 --2- 172.21.15.67:0/1982347541 >> v2:172.21.15.176:3300/0 conn(0x7f15fc074790 0x7f15fc15dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f1600ecc640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8004d50 0x7f15e8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15f3fff640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 0x7f15e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f15fbfff640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e80039c0 0x7f15e8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f1600ecc640 1 -- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 msgr2=0x7f15e8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f1600ecc640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc0c6fb0 0x7f15e8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f1600ecc640 1 -- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e80039c0 msgr2=0x7f15e8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f1600ecc640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e80039c0 0x7f15e8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.733+0000 7f1600ecc640 1 -- 172.21.15.67:0/1982347541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e8003530 con 0x7f15e8004d50 2024-04-25T15:51:03.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f1600ecc640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8004d50 0x7f15e8005160 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0cb750 tx=0x7f15f401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceb86c0220a36ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f40b2070 con 0x7f15e8004d50 2024-04-25T15:51:03.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f40a5c50 con 0x7f15e8004d50 2024-04-25T15:51:03.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f15fb7fe640 1 -- 172.21.15.67:0/1982347541 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f4009c10 con 0x7f15e8004d50 2024-04-25T15:51:03.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f160292f640 1 -- 172.21.15.67:0/1982347541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8004d50 msgr2=0x7f15e8005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f160292f640 1 --2- 172.21.15.67:0/1982347541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e8004d50 0x7f15e8005160 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0cb750 tx=0x7f15f401cc40 comp rx=0 tx=0).stop 2024-04-25T15:51:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f1600ecc640 1 -- 172.21.15.67:0/1982347541 reap_dead start 2024-04-25T15:51:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f160292f640 1 -- 172.21.15.67:0/1982347541 shutdown_connections 2024-04-25T15:51:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f160292f640 1 -- 172.21.15.67:0/1982347541 >> 172.21.15.67:0/1982347541 conn(0x7f15fc06fe70 msgr2=0x7f15fc0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f160292f640 1 -- 172.21.15.67:0/1982347541 shutdown_connections 2024-04-25T15:51:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.737+0000 7f160292f640 1 -- 172.21.15.67:0/1982347541 wait complete. 2024-04-25T15:51:03.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 Processor -- start 2024-04-25T15:51:03.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 -- start start 2024-04-25T15:51:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15fc1523a0 0x7f15fc14e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15fc0cbfb0 0x7f15fc0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc149020 0x7f15fc149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc073080 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc15af10 con 0x7f15fc1523a0 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f160292f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15fc072ab0 con 0x7f15fc149020 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15fc0cbfb0 0x7f15fc0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15fc0cbfb0 0x7f15fc0cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47052/0 (socket says 172.21.15.67:47052) 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 -- 172.21.15.67:0/1951259907 learned_addr learned my addr 172.21.15.67:0/1951259907 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15fbfff640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15fc1523a0 0x7f15fc14e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f1600ecc640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc149020 0x7f15fc149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 -- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc149020 msgr2=0x7f15fc149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc149020 0x7f15fc149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 -- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15fc1523a0 msgr2=0x7f15fc14e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15fc1523a0 0x7f15fc14e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 -- 172.21.15.67:0/1951259907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc1596f0 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f1600ecc640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc149020 0x7f15fc149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.741+0000 7f15f3fff640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15fc0cbfb0 0x7f15fc0cc3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e8f0 tx=0x7f15e400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15f97fa640 1 -- 172.21.15.67:0/1951259907 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e400c870 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15f97fa640 1 -- 172.21.15.67:0/1951259907 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e4018070 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15f97fa640 1 -- 172.21.15.67:0/1951259907 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e400cc50 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15fc0c5440 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15fc1f6970 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15fc06a480 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15f97fa640 1 -- 172.21.15.67:0/1951259907 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f15e401d460 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15f97fa640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f15dc06f400 0x7f15dc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15f97fa640 1 -- 172.21.15.67:0/1951259907 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f15e408fc60 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15fbfff640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f15dc06f400 0x7f15dc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.745+0000 7f15fbfff640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f15dc06f400 0x7f15dc071910 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0093f0 tx=0x7f15ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.749+0000 7f15f97fa640 1 -- 172.21.15.67:0/1951259907 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f15e405ccc0 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15fc15a380 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f15f97fa640 1 -- 172.21.15.67:0/1951259907 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f15e405c5c0 con 0x7f15fc0cbfb0 2024-04-25T15:51:03.914 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f15dc06f400 msgr2=0x7f15dc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f15dc06f400 0x7f15dc071910 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0093f0 tx=0x7f15ec008040 comp rx=0 tx=0).stop 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15fc0cbfb0 msgr2=0x7f15fc0cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15fc0cbfb0 0x7f15fc0cc3c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e8f0 tx=0x7f15e400edb0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 shutdown_connections 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f15dc06f400 0x7f15dc071910 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15fc149020 0x7f15fc149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15fc0cbfb0 0x7f15fc0cc3c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 --2- 172.21.15.67:0/1951259907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15fc1523a0 0x7f15fc14e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 >> 172.21.15.67:0/1951259907 conn(0x7f15fc06fe70 msgr2=0x7f15fc06c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 shutdown_connections 2024-04-25T15:51:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:03.913+0000 7f160292f640 1 -- 172.21.15.67:0/1951259907 wait complete. 2024-04-25T15:51:03.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:51:04.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2cf04d9640 1 Processor -- start 2024-04-25T15:51:04.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2cf04d9640 1 -- start start 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2cf04d9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2ce8154ac0 0x7f2ce80cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2cf04d9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce80ca8a0 con 0x7f2ce815de00 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2cf04d9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce80cab80 con 0x7f2ce815e210 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2cf04d9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2ce815a920 con 0x7f2ce8154ac0 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2ceda74640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2ce815de00 0x7f2ce80cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36056/0 (socket says 172.21.15.67:36056) 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2ceda74640 1 -- 172.21.15.67:0/111238614 learned_addr learned my addr 172.21.15.67:0/111238614 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2cee275640 1 --2- 172.21.15.67:0/111238614 >> v2:172.21.15.176:3300/0 conn(0x7f2ce8154ac0 0x7f2ce80cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171217560 0 0) 0x7f2ce80ca8a0 con 0x7f2ce815de00 2024-04-25T15:51:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cd4003530 con 0x7f2ce815de00 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.089+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 249563938 0 0) 0x7f2cd4003530 con 0x7f2ce815de00 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 >> v2:172.21.15.176:3300/0 conn(0x7f2ce8154ac0 msgr2=0x7f2ce80cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 --2- 172.21.15.67:0/111238614 >> v2:172.21.15.176:3300/0 conn(0x7f2ce8154ac0 0x7f2ce80cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 >> v1:172.21.15.176:6789/0 conn(0x7f2ce815e210 legacy=0x7f2ce8165190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce815ac00 con 0x7f2ce815de00 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f2cd8001680 con 0x7f2ce815de00 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 >> v1:172.21.15.67:6789/0 conn(0x7f2ce815de00 legacy=0x7f2ce80cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd40039c0 0x7f2cd4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4004810 0x7f2cd4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cd4005260 0x7f2cd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce815a920 con 0x7f2cd4005260 2024-04-25T15:51:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd400c1f0 con 0x7f2cd40039c0 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2cd400c3f0 con 0x7f2cd4004810 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cd4005260 0x7f2cd400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cee275640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd40039c0 0x7f2cd4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceda74640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4004810 0x7f2cd4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 -- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4004810 msgr2=0x7f2cd4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4004810 0x7f2cd4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 -- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd40039c0 msgr2=0x7f2cd4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd40039c0 0x7f2cd4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 -- 172.21.15.67:0/111238614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd4003530 con 0x7f2cd4005260 2024-04-25T15:51:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 -- 172.21.15.67:0/111238614 reap_dead start 2024-04-25T15:51:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ceea76640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cd4005260 0x7f2cd400b9d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0001e70 tx=0x7f2ce001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d343db50cafd1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0001a90 con 0x7f2cd4005260 2024-04-25T15:51:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce0003ba0 con 0x7f2cd4005260 2024-04-25T15:51:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2ced273640 1 -- 172.21.15.67:0/111238614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce00050d0 con 0x7f2cd4005260 2024-04-25T15:51:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cf04d9640 1 -- 172.21.15.67:0/111238614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cd4005260 msgr2=0x7f2cd400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cd4005260 0x7f2cd400b9d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0001e70 tx=0x7f2ce001c780 comp rx=0 tx=0).stop 2024-04-25T15:51:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cf04d9640 1 -- 172.21.15.67:0/111238614 shutdown_connections 2024-04-25T15:51:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/111238614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2cd4005260 0x7f2cd400b9d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cf04d9640 1 -- 172.21.15.67:0/111238614 >> 172.21.15.67:0/111238614 conn(0x7f2ce8068c70 msgr2=0x7f2ce8071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cf04d9640 1 -- 172.21.15.67:0/111238614 shutdown_connections 2024-04-25T15:51:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.093+0000 7f2cf04d9640 1 -- 172.21.15.67:0/111238614 wait complete. 2024-04-25T15:51:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 Processor -- start 2024-04-25T15:51:04.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 -- start start 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4005260 0x7f2ce815cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce814d7f0 0x7f2ce8159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ce80ceee0 0x7f2ce80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce815b1c0 con 0x7f2ce814d7f0 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce80ca700 con 0x7f2ce80ceee0 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce8154ed0 con 0x7f2cd4005260 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce814d7f0 0x7f2ce8159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce814d7f0 0x7f2ce8159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47070/0 (socket says 172.21.15.67:47070) 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 -- 172.21.15.67:0/35205019 learned_addr learned my addr 172.21.15.67:0/35205019 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cee275640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4005260 0x7f2ce815cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 -- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4005260 msgr2=0x7f2ce815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4005260 0x7f2ce815cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 -- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ce80ceee0 msgr2=0x7f2ce80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceea76640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ce80ceee0 0x7f2ce80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ce80ceee0 0x7f2ce80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 -- 172.21.15.67:0/35205019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce81f0a00 con 0x7f2ce814d7f0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cee275640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4005260 0x7f2ce815cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2ceda74640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce814d7f0 0x7f2ce8159bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8011820 tx=0x7f2cd8011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cdeffd640 1 -- 172.21.15.67:0/35205019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8010040 con 0x7f2ce814d7f0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cdeffd640 1 -- 172.21.15.67:0/35205019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd801a070 con 0x7f2ce814d7f0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce815d190 con 0x7f2ce814d7f0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cdeffd640 1 -- 172.21.15.67:0/35205019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd800fd60 con 0x7f2ce814d7f0 2024-04-25T15:51:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.097+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce814f240 con 0x7f2ce814d7f0 2024-04-25T15:51:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.101+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce806a480 con 0x7f2ce814d7f0 2024-04-25T15:51:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.101+0000 7f2cdeffd640 1 -- 172.21.15.67:0/35205019 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2cd801f460 con 0x7f2ce814d7f0 2024-04-25T15:51:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.101+0000 7f2cdeffd640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ccc06f400 0x7f2ccc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.101+0000 7f2cdeffd640 1 -- 172.21.15.67:0/35205019 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2cd80923f0 con 0x7f2ce814d7f0 2024-04-25T15:51:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.101+0000 7f2cee275640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ccc06f400 0x7f2ccc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.101+0000 7f2cee275640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ccc06f400 0x7f2ccc071910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009900 tx=0x7f2ce4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.105+0000 7f2cdeffd640 1 -- 172.21.15.67:0/35205019 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2cd805f400 con 0x7f2ce814d7f0 2024-04-25T15:51:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.261+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2ce80cb260 con 0x7f2ce814d7f0 2024-04-25T15:51:04.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.261+0000 7f2cdeffd640 1 -- 172.21.15.67:0/35205019 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f2cd805ed00 con 0x7f2ce814d7f0 2024-04-25T15:51:04.265 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ccc06f400 msgr2=0x7f2ccc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ccc06f400 0x7f2ccc071910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009900 tx=0x7f2ce4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce814d7f0 msgr2=0x7f2ce8159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce814d7f0 0x7f2ce8159bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8011820 tx=0x7f2cd8011ce0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 shutdown_connections 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ccc06f400 0x7f2ccc071910 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ce80ceee0 0x7f2ce80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce814d7f0 0x7f2ce8159bd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 --2- 172.21.15.67:0/35205019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd4005260 0x7f2ce815cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 >> 172.21.15.67:0/35205019 conn(0x7f2ce8068c70 msgr2=0x7f2ce8150350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 shutdown_connections 2024-04-25T15:51:04.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.265+0000 7f2cf04d9640 1 -- 172.21.15.67:0/35205019 wait complete. 2024-04-25T15:51:04.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:51:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43d014640 1 Processor -- start 2024-04-25T15:51:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43d014640 1 -- start start 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43d014640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4380c7a20 0x7fa438164d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43d014640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4380c6890 con 0x7fa438071b20 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43d014640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4380c6b70 con 0x7fa4380c7610 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43d014640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa43815cd40 con 0x7fa4380c7a20 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4380c7a20 0x7fa438164d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa436d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa438071b20 0x7fa43815d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36068/0 (socket says 172.21.15.67:36068) 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4380c7a20 0x7fa438164d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47080/0 (socket says 172.21.15.67:47080) 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa436d9d640 1 -- 172.21.15.67:0/4223673299 learned_addr learned my addr 172.21.15.67:0/4223673299 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43759e640 1 -- 172.21.15.67:0/4223673299 >> v1:172.21.15.176:6789/0 conn(0x7fa4380c7610 legacy=0x7fa4380c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43759e640 1 -- 172.21.15.67:0/4223673299 >> v1:172.21.15.67:6789/0 conn(0x7fa438071b20 legacy=0x7fa43815d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa43759e640 1 -- 172.21.15.67:0/4223673299 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4380c6e50 con 0x7fa4380c7a20 2024-04-25T15:51:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.429+0000 7fa435d9b640 1 -- 172.21.15.67:0/4223673299 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118708728 0 0) 0x7fa4380c6890 con 0x7fa438071b20 2024-04-25T15:51:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43759e640 1 --2- 172.21.15.67:0/4223673299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7a20 0x7fa438164d70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa430009d90 tx=0x7fa43001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be125566d30334ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa435d9b640 1 -- 172.21.15.67:0/4223673299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4300b2070 con 0x7fa4380c7a20 2024-04-25T15:51:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa435d9b640 1 -- 172.21.15.67:0/4223673299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4300a5940 con 0x7fa4380c7a20 2024-04-25T15:51:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa435d9b640 1 -- 172.21.15.67:0/4223673299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4300a5d10 con 0x7fa4380c7a20 2024-04-25T15:51:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 -- 172.21.15.67:0/4223673299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7a20 msgr2=0x7fa438164d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 --2- 172.21.15.67:0/4223673299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7a20 0x7fa438164d70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa430009d90 tx=0x7fa43001c970 comp rx=0 tx=0).stop 2024-04-25T15:51:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 -- 172.21.15.67:0/4223673299 shutdown_connections 2024-04-25T15:51:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 --2- 172.21.15.67:0/4223673299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7a20 0x7fa438164d70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 -- 172.21.15.67:0/4223673299 >> 172.21.15.67:0/4223673299 conn(0x7fa43806fe70 msgr2=0x7fa4380cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 -- 172.21.15.67:0/4223673299 shutdown_connections 2024-04-25T15:51:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 -- 172.21.15.67:0/4223673299 wait complete. 2024-04-25T15:51:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 Processor -- start 2024-04-25T15:51:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.433+0000 7fa43d014640 1 -- start start 2024-04-25T15:51:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa438071b20 0x7fa4380ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7610 0x7fa43814ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4380c7a20 0x7fa43814c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4380c62c0 con 0x7fa4380c7610 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4380734f0 con 0x7fa4380c7a20 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4380c6440 con 0x7fa438071b20 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7610 0x7fa43814ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7610 0x7fa43814ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47094/0 (socket says 172.21.15.67:47094) 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 -- 172.21.15.67:0/3965470616 learned_addr learned my addr 172.21.15.67:0/3965470616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43759e640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4380c7a20 0x7fa43814c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa436d9d640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa438071b20 0x7fa4380ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 -- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa438071b20 msgr2=0x7fa4380ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa438071b20 0x7fa4380ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 -- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4380c7a20 msgr2=0x7fa43814c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4380c7a20 0x7fa43814c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 -- 172.21.15.67:0/3965470616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43814f680 con 0x7fa4380c7610 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43759e640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4380c7a20 0x7fa43814c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa436d9d640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa438071b20 0x7fa4380ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43659c640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7610 0x7fa43814ee20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c390 tx=0x7fa42c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa41f7fe640 1 -- 172.21.15.67:0/3965470616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c017070 con 0x7fa4380c7610 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa41f7fe640 1 -- 172.21.15.67:0/3965470616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42c0052d0 con 0x7fa4380c7610 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa43814e340 con 0x7fa4380c7610 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa41f7fe640 1 -- 172.21.15.67:0/3965470616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c003820 con 0x7fa4380c7610 2024-04-25T15:51:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa43814e900 con 0x7fa4380c7610 2024-04-25T15:51:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa43806a480 con 0x7fa4380c7610 2024-04-25T15:51:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa41f7fe640 1 -- 172.21.15.67:0/3965470616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa42c01a070 con 0x7fa4380c7610 2024-04-25T15:51:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa41f7fe640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa42406f430 0x7fa424071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.437+0000 7fa41f7fe640 1 -- 172.21.15.67:0/3965470616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa42c08e4f0 con 0x7fa4380c7610 2024-04-25T15:51:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.441+0000 7fa436d9d640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa42406f430 0x7fa424071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.441+0000 7fa436d9d640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa42406f430 0x7fa424071940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa428000f70 tx=0x7fa428009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.445+0000 7fa41f7fe640 1 -- 172.21.15.67:0/3965470616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa42c005480 con 0x7fa4380c7610 2024-04-25T15:51:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.605+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa438074ba0 con 0x7fa4380c7610 2024-04-25T15:51:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.605+0000 7fa41f7fe640 1 -- 172.21.15.67:0/3965470616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fa42c05b500 con 0x7fa4380c7610 2024-04-25T15:51:04.609 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa42406f430 msgr2=0x7fa424071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa42406f430 0x7fa424071940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa428000f70 tx=0x7fa428009040 comp rx=0 tx=0).stop 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7610 msgr2=0x7fa43814ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7610 0x7fa43814ee20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00c390 tx=0x7fa42c00c850 comp rx=0 tx=0).stop 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 shutdown_connections 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa42406f430 0x7fa424071940 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4380c7a20 0x7fa43814c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4380c7610 0x7fa43814ee20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 --2- 172.21.15.67:0/3965470616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa438071b20 0x7fa4380ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 >> 172.21.15.67:0/3965470616 conn(0x7fa43806fe70 msgr2=0x7fa43815d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 shutdown_connections 2024-04-25T15:51:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.609+0000 7fa43d014640 1 -- 172.21.15.67:0/3965470616 wait complete. 2024-04-25T15:51:04.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:51:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e9a052640 1 Processor -- start 2024-04-25T15:51:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e9a052640 1 -- start start 2024-04-25T15:51:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e9a052640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e94071bc0 0x7f6e940cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e9a052640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6e940ca540 0x7f6e940ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e9a052640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e9415de10 con 0x7f6e940cc100 2024-04-25T15:51:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e9a052640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6e9415ebb0 con 0x7f6e94071bc0 2024-04-25T15:51:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e9a052640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6e9415e720 con 0x7f6e940ca540 2024-04-25T15:51:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e937fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e94071bc0 0x7f6e940cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e937fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e94071bc0 0x7f6e940cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47098/0 (socket says 172.21.15.67:47098) 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e937fe640 1 -- 172.21.15.67:0/4214312616 learned_addr learned my addr 172.21.15.67:0/4214312616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e93fff640 1 --2- 172.21.15.67:0/4214312616 >> v2:172.21.15.176:3300/0 conn(0x7f6e940ca540 0x7f6e940ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e937fe640 1 -- 172.21.15.67:0/4214312616 >> v2:172.21.15.176:3300/0 conn(0x7f6e940ca540 msgr2=0x7f6e940ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e937fe640 1 --2- 172.21.15.67:0/4214312616 >> v2:172.21.15.176:3300/0 conn(0x7f6e940ca540 0x7f6e940ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e937fe640 1 -- 172.21.15.67:0/4214312616 >> v1:172.21.15.176:6789/0 conn(0x7f6e940cc100 legacy=0x7f6e9415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e937fe640 1 -- 172.21.15.67:0/4214312616 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e94072390 con 0x7f6e94071bc0 2024-04-25T15:51:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.777+0000 7f6e93fff640 1 --2- 172.21.15.67:0/4214312616 >> v2:172.21.15.176:3300/0 conn(0x7f6e940ca540 0x7f6e940ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e937fe640 1 --2- 172.21.15.67:0/4214312616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e940cbbc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6e94074880 tx=0x7f6e8401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea8db3844feaf87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e927fc640 1 -- 172.21.15.67:0/4214312616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840ab5f0 con 0x7f6e94071bc0 2024-04-25T15:51:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e927fc640 1 -- 172.21.15.67:0/4214312616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e840b0070 con 0x7f6e94071bc0 2024-04-25T15:51:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e927fc640 1 -- 172.21.15.67:0/4214312616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840ab9b0 con 0x7f6e94071bc0 2024-04-25T15:51:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 -- 172.21.15.67:0/4214312616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 msgr2=0x7f6e940cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 --2- 172.21.15.67:0/4214312616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e940cbbc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6e94074880 tx=0x7f6e8401c4e0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 -- 172.21.15.67:0/4214312616 shutdown_connections 2024-04-25T15:51:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 --2- 172.21.15.67:0/4214312616 >> v2:172.21.15.176:3300/0 conn(0x7f6e940ca540 0x7f6e940ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 --2- 172.21.15.67:0/4214312616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e940cbbc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 -- 172.21.15.67:0/4214312616 >> 172.21.15.67:0/4214312616 conn(0x7f6e940700b0 msgr2=0x7f6e940704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 -- 172.21.15.67:0/4214312616 shutdown_connections 2024-04-25T15:51:04.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.781+0000 7f6e9a052640 1 -- 172.21.15.67:0/4214312616 wait complete. 2024-04-25T15:51:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 Processor -- start 2024-04-25T15:51:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 -- start start 2024-04-25T15:51:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e9414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e940ca540 0x7f6e94152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e940cc100 0x7f6e940cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6e94072150 con 0x7f6e94071bc0 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6e94071fd0 con 0x7f6e940ca540 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6e9415ae30 con 0x7f6e940cc100 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e9414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e9414fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47114/0 (socket says 172.21.15.67:47114) 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 -- 172.21.15.67:0/1619280171 learned_addr learned my addr 172.21.15.67:0/1619280171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e93fff640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e940cc100 0x7f6e940cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e92ffd640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e940ca540 0x7f6e94152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 -- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e940cc100 msgr2=0x7f6e940cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e940cc100 0x7f6e940cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 -- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e940ca540 msgr2=0x7f6e94152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e940ca540 0x7f6e94152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 -- 172.21.15.67:0/1619280171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9406f200 con 0x7f6e94071bc0 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e93fff640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e940cc100 0x7f6e940cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e937fe640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e9414fbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6e8401a110 tx=0x7f6e8401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e7bfff640 1 -- 172.21.15.67:0/1619280171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840be5f0 con 0x7f6e94071bc0 2024-04-25T15:51:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e7bfff640 1 -- 172.21.15.67:0/1619280171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e840be790 con 0x7f6e94071bc0 2024-04-25T15:51:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e9406f470 con 0x7f6e94071bc0 2024-04-25T15:51:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e7bfff640 1 -- 172.21.15.67:0/1619280171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840beb60 con 0x7f6e94071bc0 2024-04-25T15:51:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e941f5e00 con 0x7f6e94071bc0 2024-04-25T15:51:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.785+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e9406a480 con 0x7f6e94071bc0 2024-04-25T15:51:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.789+0000 7f6e7bfff640 1 -- 172.21.15.67:0/1619280171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6e840b43f0 con 0x7f6e94071bc0 2024-04-25T15:51:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.789+0000 7f6e7bfff640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e8006f400 0x7f6e80071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.789+0000 7f6e7bfff640 1 -- 172.21.15.67:0/1619280171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6e84129760 con 0x7f6e94071bc0 2024-04-25T15:51:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.789+0000 7f6e92ffd640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e8006f400 0x7f6e80071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.789+0000 7f6e92ffd640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e8006f400 0x7f6e80071910 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e9415b6b0 tx=0x7f6e88008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.793+0000 7f6e7bfff640 1 -- 172.21.15.67:0/1619280171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6e840c4060 con 0x7f6e94071bc0 2024-04-25T15:51:04.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.989+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e940c8a50 con 0x7f6e94071bc0 2024-04-25T15:51:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.989+0000 7f6e7bfff640 1 -- 172.21.15.67:0/1619280171 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6911 (secure 0 0 0) 0x7f6e840160d0 con 0x7f6e94071bc0 2024-04-25T15:51:04.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:04.991 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/920422475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":920422475},{"type":"v1","addr":"172.21.15.67:6835","nonce":920422475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:50:30.570677+0000","modified":"2024-04-25T15:51:04.945545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4294,"mds_1":4288,"mds_2":4276,"mds_3":4272,"mds_4":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"a","rank":4,"incarnation":16,"state":"up:active","state_seq":12,"addr":"172.21.15.67:6834/144842397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":144842397},{"type":"v1","addr":"172.21.15.67:6834","nonce":144842397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4272":{"gid":4272,"name":"d","rank":3,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.176:6838/2023672746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2023672746},{"type":"v1","addr":"172.21.15.176:6838","nonce":2023672746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4276":{"gid":4276,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":9,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6835/2385091301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2385091301},{"type":"v1","addr":"172.21.15.176:6835","nonce":2385091301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.176:6839/1886671623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6837","nonce":1886671623},{"type":"v1","addr":"172.21.15.176:6839","nonce":1886671623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4270,4272,4276,4288,4294]},"id":1}]} 2024-04-25T15:51:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e8006f400 msgr2=0x7f6e80071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e8006f400 0x7f6e80071910 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e9415b6b0 tx=0x7f6e88008040 comp rx=0 tx=0).stop 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 msgr2=0x7f6e9414fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e9414fbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6e8401a110 tx=0x7f6e8401ad20 comp rx=0 tx=0).stop 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 shutdown_connections 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e8006f400 0x7f6e80071910 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e940cc100 0x7f6e940cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e940ca540 0x7f6e94152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 --2- 172.21.15.67:0/1619280171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e94071bc0 0x7f6e9414fbc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 >> 172.21.15.67:0/1619280171 conn(0x7f6e940700b0 msgr2=0x7f6e940704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 shutdown_connections 2024-04-25T15:51:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:04.993+0000 7f6e9a052640 1 -- 172.21.15.67:0/1619280171 wait complete. 2024-04-25T15:51:05.012 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:51:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f71293e7640 1 Processor -- start 2024-04-25T15:51:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f71293e7640 1 -- start start 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f71293e7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f712406c640 0x7f712415b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f71293e7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7124074370 0x7f7124074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f71293e7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71240c92d0 con 0x7f71240c7fc0 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f71293e7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f712415b560 con 0x7f7124074370 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f71293e7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f712406e210 con 0x7f712406c640 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f711bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7124074370 0x7f7124074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f711bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7124074370 0x7f7124074780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47122/0 (socket says 172.21.15.67:47122) 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f711bfff640 1 -- 172.21.15.67:0/646607164 learned_addr learned my addr 172.21.15.67:0/646607164 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f7122ffd640 1 --2- 172.21.15.67:0/646607164 >> v2:172.21.15.176:3300/0 conn(0x7f712406c640 0x7f712415b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f711bfff640 1 -- 172.21.15.67:0/646607164 >> v2:172.21.15.176:3300/0 conn(0x7f712406c640 msgr2=0x7f712415b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f711bfff640 1 --2- 172.21.15.67:0/646607164 >> v2:172.21.15.176:3300/0 conn(0x7f712406c640 0x7f712415b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f711bfff640 1 -- 172.21.15.67:0/646607164 >> v1:172.21.15.176:6789/0 conn(0x7f71240c7fc0 legacy=0x7f712415dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f711bfff640 1 -- 172.21.15.67:0/646607164 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712406e4d0 con 0x7f7124074370 2024-04-25T15:51:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.165+0000 7f7122ffd640 1 --2- 172.21.15.67:0/646607164 >> v2:172.21.15.176:3300/0 conn(0x7f712406c640 0x7f712415b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f711bfff640 1 --2- 172.21.15.67:0/646607164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7124074370 0x7f7124074780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f71240c8c00 tx=0x7f710c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cea2ce350ccf5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71227fc640 1 -- 172.21.15.67:0/646607164 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c0b5070 con 0x7f7124074370 2024-04-25T15:51:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71227fc640 1 -- 172.21.15.67:0/646607164 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f710c0a89d0 con 0x7f7124074370 2024-04-25T15:51:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71227fc640 1 -- 172.21.15.67:0/646607164 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c0a8da0 con 0x7f7124074370 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 -- 172.21.15.67:0/646607164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7124074370 msgr2=0x7f7124074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 --2- 172.21.15.67:0/646607164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7124074370 0x7f7124074780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f71240c8c00 tx=0x7f710c01ecd0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 -- 172.21.15.67:0/646607164 shutdown_connections 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 --2- 172.21.15.67:0/646607164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7124074370 0x7f7124074780 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 --2- 172.21.15.67:0/646607164 >> v2:172.21.15.176:3300/0 conn(0x7f712406c640 0x7f712415b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 -- 172.21.15.67:0/646607164 >> 172.21.15.67:0/646607164 conn(0x7f7124070db0 msgr2=0x7f71240ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 -- 172.21.15.67:0/646607164 shutdown_connections 2024-04-25T15:51:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.169+0000 7f71293e7640 1 -- 172.21.15.67:0/646607164 wait complete. 2024-04-25T15:51:05.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 Processor -- start 2024-04-25T15:51:05.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 -- start start 2024-04-25T15:51:05.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712406c640 0x7f7124149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7124074370 0x7f712415e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71240c7fc0 0x7f712415c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7124073b70 con 0x7f712406c640 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7124073890 con 0x7f71240c7fc0 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7124073e50 con 0x7f7124074370 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712406c640 0x7f7124149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f711bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7124074370 0x7f712415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71237fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71240c7fc0 0x7f712415c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712406c640 0x7f7124149550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47136/0 (socket says 172.21.15.67:47136) 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 -- 172.21.15.67:0/330190337 learned_addr learned my addr 172.21.15.67:0/330190337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f711bfff640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7124074370 0x7f712415e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:42786/0 (socket says 172.21.15.67:42786) 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 -- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7124074370 msgr2=0x7f712415e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7124074370 0x7f712415e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 -- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71240c7fc0 msgr2=0x7f712415c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71240c7fc0 0x7f712415c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 -- 172.21.15.67:0/330190337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712414c360 con 0x7f712406c640 2024-04-25T15:51:05.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f7122ffd640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712406c640 0x7f7124149550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f711400d800 tx=0x7f711400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f711b7fe640 1 -- 172.21.15.67:0/330190337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114018020 con 0x7f712406c640 2024-04-25T15:51:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f711b7fe640 1 -- 172.21.15.67:0/330190337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71140096a0 con 0x7f712406c640 2024-04-25T15:51:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71241f1c60 con 0x7f712406c640 2024-04-25T15:51:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f711b7fe640 1 -- 172.21.15.67:0/330190337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114009a90 con 0x7f712406c640 2024-04-25T15:51:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71241f1f40 con 0x7f712406c640 2024-04-25T15:51:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.173+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f712406a480 con 0x7f712406c640 2024-04-25T15:51:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.177+0000 7f711b7fe640 1 -- 172.21.15.67:0/330190337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f711401b020 con 0x7f712406c640 2024-04-25T15:51:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.177+0000 7f711b7fe640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f710406f400 0x7f7104071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.177+0000 7f711b7fe640 1 -- 172.21.15.67:0/330190337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f711408f640 con 0x7f712406c640 2024-04-25T15:51:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.177+0000 7f711bfff640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f710406f400 0x7f7104071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.177+0000 7f711bfff640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f710406f400 0x7f7104071910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f710c0b2000 tx=0x7f710c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.181+0000 7f711b7fe640 1 -- 172.21.15.67:0/330190337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7114022020 con 0x7f712406c640 2024-04-25T15:51:05.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.349+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7124149b20 con 0x7f712406c640 2024-04-25T15:51:05.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.349+0000 7f711b7fe640 1 -- 172.21.15.67:0/330190337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16569 (secure 0 0 0) 0x7f711405c000 con 0x7f712406c640 2024-04-25T15:51:05.353 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:05.353 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:50:32.143126+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:50:07.834016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:51:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f710406f400 msgr2=0x7f7104071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f710406f400 0x7f7104071910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f710c0b2000 tx=0x7f710c009040 comp rx=0 tx=0).stop 2024-04-25T15:51:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712406c640 msgr2=0x7f7124149550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712406c640 0x7f7124149550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f711400d800 tx=0x7f711400dcc0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 shutdown_connections 2024-04-25T15:51:05.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f710406f400 0x7f7104071910 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71240c7fc0 0x7f712415c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7124074370 0x7f712415e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 --2- 172.21.15.67:0/330190337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f712406c640 0x7f7124149550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 >> 172.21.15.67:0/330190337 conn(0x7f7124070db0 msgr2=0x7f71240cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 shutdown_connections 2024-04-25T15:51:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.353+0000 7f71293e7640 1 -- 172.21.15.67:0/330190337 wait complete. 2024-04-25T15:51:05.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:51:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3106adb640 1 Processor -- start 2024-04-25T15:51:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3106adb640 1 -- start start 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3106adb640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3100071b20 0x7f310015d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3106adb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31000c5bb0 0x7f31000c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3106adb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31000c6890 con 0x7f310015d710 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3106adb640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f310015cd40 con 0x7f3100071b20 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3106adb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f31000c6b70 con 0x7f31000c5bb0 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3104877640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3100071b20 0x7f310015d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3104877640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3100071b20 0x7f310015d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:33278/0 (socket says 172.21.15.67:33278) 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3105078640 1 --2- 172.21.15.67:0/1365222773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31000c5bb0 0x7f31000c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3104877640 1 -- 172.21.15.67:0/1365222773 learned_addr learned my addr 172.21.15.67:0/1365222773 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3105078640 1 -- 172.21.15.67:0/1365222773 >> v2:172.21.15.176:3300/0 conn(0x7f3100071b20 msgr2=0x7f310015d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3105078640 1 --2- 172.21.15.67:0/1365222773 >> v2:172.21.15.176:3300/0 conn(0x7f3100071b20 0x7f310015d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3105078640 1 -- 172.21.15.67:0/1365222773 >> v1:172.21.15.176:6789/0 conn(0x7f310015d710 legacy=0x7f31000c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3105078640 1 -- 172.21.15.67:0/1365222773 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31000c6e50 con 0x7f31000c5bb0 2024-04-25T15:51:05.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.533+0000 7f3104877640 1 --2- 172.21.15.67:0/1365222773 >> v2:172.21.15.176:3300/0 conn(0x7f3100071b20 0x7f310015d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:51:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3105078640 1 --2- 172.21.15.67:0/1365222773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31000c5bb0 0x7f31000c5fc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009d90 tx=0x7f30f801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca760772ebf16068 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f30f77fe640 1 -- 172.21.15.67:0/1365222773 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f80b2070 con 0x7f31000c5bb0 2024-04-25T15:51:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f30f77fe640 1 -- 172.21.15.67:0/1365222773 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f80a5840 con 0x7f31000c5bb0 2024-04-25T15:51:05.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f30f77fe640 1 -- 172.21.15.67:0/1365222773 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f80a5c10 con 0x7f31000c5bb0 2024-04-25T15:51:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 -- 172.21.15.67:0/1365222773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31000c5bb0 msgr2=0x7f31000c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 --2- 172.21.15.67:0/1365222773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31000c5bb0 0x7f31000c5fc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009d90 tx=0x7f30f801c970 comp rx=0 tx=0).stop 2024-04-25T15:51:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 -- 172.21.15.67:0/1365222773 shutdown_connections 2024-04-25T15:51:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 --2- 172.21.15.67:0/1365222773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31000c5bb0 0x7f31000c5fc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 --2- 172.21.15.67:0/1365222773 >> v2:172.21.15.176:3300/0 conn(0x7f3100071b20 0x7f310015d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 -- 172.21.15.67:0/1365222773 >> 172.21.15.67:0/1365222773 conn(0x7f310006fe70 msgr2=0x7f31000cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 -- 172.21.15.67:0/1365222773 shutdown_connections 2024-04-25T15:51:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.537+0000 7f3106adb640 1 -- 172.21.15.67:0/1365222773 wait complete. 2024-04-25T15:51:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 Processor -- start 2024-04-25T15:51:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 -- start start 2024-04-25T15:51:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3100071b20 0x7f3100152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31000c5bb0 0x7f3100148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310015d710 0x7f31001595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f310006ce50 con 0x7f31000c5bb0 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f31000c7d20 con 0x7f3100071b20 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f310006cfd0 con 0x7f310015d710 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31000c5bb0 0x7f3100148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31000c5bb0 0x7f3100148780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47148/0 (socket says 172.21.15.67:47148) 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 -- 172.21.15.67:0/3708768213 learned_addr learned my addr 172.21.15.67:0/3708768213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3104877640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3100071b20 0x7f3100152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3105078640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310015d710 0x7f31001595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 -- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310015d710 msgr2=0x7f31001595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310015d710 0x7f31001595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 -- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3100071b20 msgr2=0x7f3100152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3100071b20 0x7f3100152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 -- 172.21.15.67:0/3708768213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31000c5200 con 0x7f31000c5bb0 2024-04-25T15:51:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3105078640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310015d710 0x7f31001595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f7fff640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31000c5bb0 0x7f3100148780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c370 tx=0x7f30f000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f57fa640 1 -- 172.21.15.67:0/3708768213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f00052d0 con 0x7f31000c5bb0 2024-04-25T15:51:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f57fa640 1 -- 172.21.15.67:0/3708768213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30f0005470 con 0x7f31000c5bb0 2024-04-25T15:51:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f30f57fa640 1 -- 172.21.15.67:0/3708768213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0007260 con 0x7f31000c5bb0 2024-04-25T15:51:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31001534e0 con 0x7f31000c5bb0 2024-04-25T15:51:05.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.541+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31001f9310 con 0x7f31000c5bb0 2024-04-25T15:51:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.545+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f310006a480 con 0x7f31000c5bb0 2024-04-25T15:51:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.545+0000 7f30f57fa640 1 -- 172.21.15.67:0/3708768213 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f30f0007400 con 0x7f31000c5bb0 2024-04-25T15:51:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.545+0000 7f30f57fa640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f30ec06f400 0x7f30ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.545+0000 7f30f57fa640 1 -- 172.21.15.67:0/3708768213 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f30f008e530 con 0x7f31000c5bb0 2024-04-25T15:51:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.545+0000 7f3104877640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f30ec06f400 0x7f30ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:05.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.545+0000 7f3104877640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f30ec06f400 0x7f30ec071910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00b240 tx=0x7f30fc00b450 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:05.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.549+0000 7f30f57fa640 1 -- 172.21.15.67:0/3708768213 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f30f005aef0 con 0x7f31000c5bb0 2024-04-25T15:51:05.640 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-04-25T15:51:05.636+0000 7f240c636640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T15:51:05.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.749+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31000c9460 con 0x7f31000c5bb0 2024-04-25T15:51:05.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f30f57fa640 1 -- 172.21.15.67:0/3708768213 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6911 (secure 0 0 0) 0x7f30f005f1f0 con 0x7f31000c5bb0 2024-04-25T15:51:05.754 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:05.754 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/920422475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":920422475},{"type":"v1","addr":"172.21.15.67:6835","nonce":920422475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:50:30.570677+0000","modified":"2024-04-25T15:51:04.945545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4294,"mds_1":4288,"mds_2":4276,"mds_3":4272,"mds_4":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"a","rank":4,"incarnation":16,"state":"up:active","state_seq":12,"addr":"172.21.15.67:6834/144842397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":144842397},{"type":"v1","addr":"172.21.15.67:6834","nonce":144842397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4272":{"gid":4272,"name":"d","rank":3,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.176:6838/2023672746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2023672746},{"type":"v1","addr":"172.21.15.176:6838","nonce":2023672746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4276":{"gid":4276,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":9,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6835/2385091301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2385091301},{"type":"v1","addr":"172.21.15.176:6835","nonce":2385091301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.176:6839/1886671623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6837","nonce":1886671623},{"type":"v1","addr":"172.21.15.176:6839","nonce":1886671623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4270,4272,4276,4288,4294]},"id":1}]} 2024-04-25T15:51:05.754 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f30ec06f400 msgr2=0x7f30ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f30ec06f400 0x7f30ec071910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00b240 tx=0x7f30fc00b450 comp rx=0 tx=0).stop 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31000c5bb0 msgr2=0x7f3100148780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31000c5bb0 0x7f3100148780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f30f000c370 tx=0x7f30f000c830 comp rx=0 tx=0).stop 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 shutdown_connections 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f30ec06f400 0x7f30ec071910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310015d710 0x7f31001595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31000c5bb0 0x7f3100148780 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 --2- 172.21.15.67:0/3708768213 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3100071b20 0x7f3100152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 >> 172.21.15.67:0/3708768213 conn(0x7f310006fe70 msgr2=0x7f31000ce110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 shutdown_connections 2024-04-25T15:51:05.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:05.753+0000 7f3106adb640 1 -- 172.21.15.67:0/3708768213 wait complete. 2024-04-25T15:51:06.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T15:50:30.570677+0000', 'modified': '2024-04-25T15:51:04.945545+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4294, 'mds_1': 4288, 'mds_2': 4276, 'mds_3': 4272, 'mds_4': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'a', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.67:6834/144842397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 144842397}, {'type': 'v1', 'addr': '172.21.15.67:6834', 'nonce': 144842397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4272': {'gid': 4272, 'name': 'd', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.176:6838/2023672746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6836', 'nonce': 2023672746}, {'type': 'v1', 'addr': '172.21.15.176:6838', 'nonce': 2023672746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4276': {'gid': 4276, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.67:6837/2429063224', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6836', 'nonce': 2429063224}, {'type': 'v1', 'addr': '172.21.15.67:6837', 'nonce': 2429063224}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4288': {'gid': 4288, 'name': 'f', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.176:6835/2385091301', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6834', 'nonce': 2385091301}, {'type': 'v1', 'addr': '172.21.15.176:6835', 'nonce': 2385091301}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4294': {'gid': 4294, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.176:6839/1886671623', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6837', 'nonce': 1886671623}, {'type': 'v1', 'addr': '172.21.15.176:6839', 'nonce': 1886671623}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4294, 'qdb_cluster': [4270, 4272, 4276, 4288, 4294]} 2024-04-25T15:51:06.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T15:51:06.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T15:51:06.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f0150ea9640 1 Processor -- start 2024-04-25T15:51:06.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f0150ea9640 1 -- start start 2024-04-25T15:51:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f0150ea9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c152b00 0x7f014c0c4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f0150ea9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f014c0c8d90 0x7f014c06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f0150ea9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f014c158d20 con 0x7f014c06c640 2024-04-25T15:51:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f0150ea9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f014c0722c0 con 0x7f014c0c8d90 2024-04-25T15:51:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f0150ea9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f014c158fe0 con 0x7f014c152b00 2024-04-25T15:51:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f014ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f014c0c8d90 0x7f014c06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f014ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f014c0c8d90 0x7f014c06ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47158/0 (socket says 172.21.15.67:47158) 2024-04-25T15:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f014ad9d640 1 -- 172.21.15.67:0/3399919562 learned_addr learned my addr 172.21.15.67:0/3399919562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.545+0000 7f014a59c640 1 --2- 172.21.15.67:0/3399919562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c152b00 0x7f014c0c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014ad9d640 1 -- 172.21.15.67:0/3399919562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c152b00 msgr2=0x7f014c0c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014ad9d640 1 --2- 172.21.15.67:0/3399919562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c152b00 0x7f014c0c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014ad9d640 1 -- 172.21.15.67:0/3399919562 >> v1:172.21.15.176:6789/0 conn(0x7f014c06c640 legacy=0x7f014c0c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:06.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014ad9d640 1 -- 172.21.15.67:0/3399919562 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014c1592c0 con 0x7f014c0c8d90 2024-04-25T15:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014ad9d640 1 --2- 172.21.15.67:0/3399919562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c0c8d90 0x7f014c06ca50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f014400af60 tx=0x7f014401db30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196f040aa92927ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:06.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014959a640 1 -- 172.21.15.67:0/3399919562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014400a2d0 con 0x7f014c0c8d90 2024-04-25T15:51:06.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014959a640 1 -- 172.21.15.67:0/3399919562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01440a6770 con 0x7f014c0c8d90 2024-04-25T15:51:06.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f014959a640 1 -- 172.21.15.67:0/3399919562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440a6b40 con 0x7f014c0c8d90 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 -- 172.21.15.67:0/3399919562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c0c8d90 msgr2=0x7f014c06ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 --2- 172.21.15.67:0/3399919562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c0c8d90 0x7f014c06ca50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f014400af60 tx=0x7f014401db30 comp rx=0 tx=0).stop 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 -- 172.21.15.67:0/3399919562 shutdown_connections 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 --2- 172.21.15.67:0/3399919562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c0c8d90 0x7f014c06ca50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 --2- 172.21.15.67:0/3399919562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c152b00 0x7f014c0c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 -- 172.21.15.67:0/3399919562 >> 172.21.15.67:0/3399919562 conn(0x7f014c06fe70 msgr2=0x7f014c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 -- 172.21.15.67:0/3399919562 shutdown_connections 2024-04-25T15:51:06.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.549+0000 7f0150ea9640 1 -- 172.21.15.67:0/3399919562 wait complete. 2024-04-25T15:51:06.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 Processor -- start 2024-04-25T15:51:06.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 -- start start 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f014c06c640 0x7f014c146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c0c8d90 0x7f014c147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c152b00 0x7f014c157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f014c0c4960 con 0x7f014c152b00 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f014c152f10 con 0x7f014c06c640 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f014c0c4ae0 con 0x7f014c0c8d90 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c152b00 0x7f014c157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c152b00 0x7f014c157d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47170/0 (socket says 172.21.15.67:47170) 2024-04-25T15:51:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 -- 172.21.15.67:0/1872671622 learned_addr learned my addr 172.21.15.67:0/1872671622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014a59c640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f014c06c640 0x7f014c146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0149d9b640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c0c8d90 0x7f014c147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 -- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c0c8d90 msgr2=0x7f014c147980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c0c8d90 0x7f014c147980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 -- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f014c06c640 msgr2=0x7f014c146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f014c06c640 0x7f014c146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 -- 172.21.15.67:0/1872671622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014c14f300 con 0x7f014c152b00 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0149d9b640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c0c8d90 0x7f014c147980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014a59c640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f014c06c640 0x7f014c146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f014ad9d640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c152b00 0x7f014c157d20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f01440b0040 tx=0x7f014401a8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0142ffd640 1 -- 172.21.15.67:0/1872671622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144008500 con 0x7f014c152b00 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0142ffd640 1 -- 172.21.15.67:0/1872671622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01440bd070 con 0x7f014c152b00 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0142ffd640 1 -- 172.21.15.67:0/1872671622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144008940 con 0x7f014c152b00 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014c146c60 con 0x7f014c152b00 2024-04-25T15:51:06.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.553+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014c147ec0 con 0x7f014c152b00 2024-04-25T15:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.557+0000 7f0142ffd640 1 -- 172.21.15.67:0/1872671622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0144008ae0 con 0x7f014c152b00 2024-04-25T15:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.557+0000 7f0142ffd640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f012c06f430 0x7f012c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.557+0000 7f0142ffd640 1 -- 172.21.15.67:0/1872671622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f014412a240 con 0x7f014c152b00 2024-04-25T15:51:06.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.557+0000 7f014a59c640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f012c06f430 0x7f012c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.557+0000 7f0158206000 1 Processor -- start 2024-04-25T15:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.557+0000 7f0158206000 1 -- start start 2024-04-25T15:51:06.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.557+0000 7f014a59c640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f012c06f430 0x7f012c071940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f013400b9f0 tx=0x7f0134008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669e2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55dc6684e160 con 0x55dc6686cea0 2024-04-25T15:51:06.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55dc667e3e20 con 0x55dc66880c20 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55dc6679ca20 con 0x55dc66465280 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669e2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669e2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47176/0 (socket says 172.21.15.67:47176) 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 -- 172.21.15.67:0/3583341308 learned_addr learned my addr 172.21.15.67:0/3583341308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f014ad9d640 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f014a59c640 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 -- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 msgr2=0x55dc669e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 -- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 msgr2=0x55dc669d13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669d13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 -- 172.21.15.67:0/3583341308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc667a1100 con 0x55dc6686cea0 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f014a59c640 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0149d9b640 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669e2530 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f013c001550 tx=0x7f013c001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71eebd1819fcc35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f012bfff640 1 -- 172.21.15.67:0/3583341308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c019070 con 0x55dc6686cea0 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f012bfff640 1 -- 172.21.15.67:0/3583341308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013c0050d0 con 0x55dc6686cea0 2024-04-25T15:51:06.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f012bfff640 1 -- 172.21.15.67:0/3583341308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c0067e0 con 0x55dc6686cea0 2024-04-25T15:51:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- 172.21.15.67:0/3583341308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 msgr2=0x55dc669e2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:06.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669e2530 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f013c001550 tx=0x7f013c001a10 comp rx=0 tx=0).stop 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- 172.21.15.67:0/3583341308 shutdown_connections 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669e2530 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 --2- 172.21.15.67:0/3583341308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- 172.21.15.67:0/3583341308 >> 172.21.15.67:0/3583341308 conn(0x55dc664a4ff0 msgr2=0x55dc669d4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- 172.21.15.67:0/3583341308 shutdown_connections 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- 172.21.15.67:0/3583341308 wait complete. 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 -- start start 2024-04-25T15:51:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.561+0000 7f0158206000 1 Processor -- start 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669d1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669d1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669e0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55dc667f1220 con 0x55dc6686cea0 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55dc66787ea0 con 0x55dc66880c20 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55dc667deb00 con 0x55dc66465280 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 10 client.0 _ll_register_callbacks cb 0x55dc667f0dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669d1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47192/0 (socket says 172.21.15.67:47192) 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 -- 172.21.15.67:0/227485294 learned_addr learned my addr 172.21.15.67:0/227485294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:06.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014a59c640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669d1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014ad9d640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669e0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 -- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 msgr2=0x55dc669d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669d1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 -- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 msgr2=0x55dc669e0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669e0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 -- 172.21.15.67:0/227485294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc66783500 con 0x55dc6686cea0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014a59c640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0149d9b640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669d1940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f013c000fc0 tx=0x7f013c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11e314865f5bf73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f01297fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f01297fa640 1 -- 172.21.15.67:0/227485294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013c019040 con 0x55dc6686cea0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f01297fa640 1 -- 172.21.15.67:0/227485294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f013c0069e0 con 0x55dc6686cea0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f013c0050d0 con 0x55dc6686cea0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f01297fa640 1 -- 172.21.15.67:0/227485294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f013c02b050 con 0x55dc6686cea0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 10 client.4722 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55dc66769ea0 con 0x55dc6686cea0 2024-04-25T15:51:06.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f01297fa640 1 -- 172.21.15.67:0/227485294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013c0050d0 con 0x55dc6686cea0 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f01297fa640 1 -- 172.21.15.67:0/227485294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f013c02e020 con 0x55dc6686cea0 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 10 client.4722 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 10 client.4722 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 20 client.4722 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 0x55dc669e11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 4 client.4722 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 --> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc6677cbe0 con 0x55dc664897d0 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014a59c640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 0x55dc669e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014a59c640 1 -- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 msgr2=0x55dc669e11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014a59c640 1 -- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 msgr2=0x55dc669e11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014a59c640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 0x55dc669e11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:51:06.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.565+0000 7f014a59c640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 0x55dc669e11d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:51:06.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.769+0000 7f014a59c640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 0x55dc669e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.769+0000 7f014a59c640 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 0x55dc669e11d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1a1b2191ee05c6a2 server_cookie=fc62ec4c93878a48 in_seq=0 out_seq=0 2024-04-25T15:51:06.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:06.769+0000 7f01297fa640 10 client.4722 ms_handle_connect on v2:172.21.15.67:6832/144842397 2024-04-25T15:51:09.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f01297fa640 1 -- 172.21.15.67:0/227485294 <== mds.4 v2:172.21.15.67:6832/144842397 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dc6677cbe0 con 0x55dc664897d0 2024-04-25T15:51:09.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f01297fa640 10 client.4722 handle_command_reply: tid=0 2024-04-25T15:51:09.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 client.4722 shutdown 2024-04-25T15:51:09.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 msgr2=0x55dc669d1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:09.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669d1940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f013c000fc0 tx=0x7f013c000bd0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 shutdown_connections 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc664897d0 0x55dc669e11d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc66880c20 0x55dc669e0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc6686cea0 0x55dc669d1940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 --2- 172.21.15.67:0/227485294 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669d1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f01297fa640 0 client.4722 ms_handle_reset on v2:172.21.15.67:6832/144842397 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 >> 172.21.15.67:0/227485294 conn(0x55dc664a4ff0 msgr2=0x55dc669d2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 shutdown_connections 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.948+0000 7f0158206000 1 -- 172.21.15.67:0/227485294 wait complete. 2024-04-25T15:51:09.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.952+0000 7f0158206000 20 client.4722 trim_cache size 0 max 16384 2024-04-25T15:51:09.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 Processor -- start 2024-04-25T15:51:09.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 -- start start 2024-04-25T15:51:09.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66465280 0x55dc669e2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 0x55dc669998e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66880c20 0x55dc66999e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f014c15be90 con 0x55dc66465280 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f014c158820 con 0x55dc6686cea0 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55dc667797d0 con 0x55dc66880c20 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66465280 0x55dc669e2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66465280 0x55dc669e2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47196/0 (socket says 172.21.15.67:47196) 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 -- 172.21.15.67:0/143997283 learned_addr learned my addr 172.21.15.67:0/143997283 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014ad9d640 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66880c20 0x55dc66999e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 -- 172.21.15.67:0/143997283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66880c20 msgr2=0x55dc66999e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66880c20 0x55dc66999e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 -- 172.21.15.67:0/143997283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 msgr2=0x55dc669998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 0x55dc669998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.984+0000 7f014a59c640 1 -- 172.21.15.67:0/143997283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc6679cc70 con 0x55dc66465280 2024-04-25T15:51:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f014a59c640 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66465280 0x55dc669e2810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55dc667eb440 tx=0x7f0134015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdee6f63ea26930 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f012b7fe640 1 -- 172.21.15.67:0/143997283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013401b050 con 0x55dc66465280 2024-04-25T15:51:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f012b7fe640 1 -- 172.21.15.67:0/143997283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0134031070 con 0x55dc66465280 2024-04-25T15:51:09.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f012b7fe640 1 -- 172.21.15.67:0/143997283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134014cc0 con 0x55dc66465280 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 -- 172.21.15.67:0/143997283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66465280 msgr2=0x55dc669e2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66465280 0x55dc669e2810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55dc667eb440 tx=0x7f0134015000 comp rx=0 tx=0).stop 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 -- 172.21.15.67:0/143997283 shutdown_connections 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66880c20 0x55dc66999e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 0x55dc669998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 --2- 172.21.15.67:0/143997283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66465280 0x55dc669e2810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 -- 172.21.15.67:0/143997283 >> 172.21.15.67:0/143997283 conn(0x55dc664a4ff0 msgr2=0x55dc669f6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 -- 172.21.15.67:0/143997283 shutdown_connections 2024-04-25T15:51:09.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.988+0000 7f0158206000 1 -- 172.21.15.67:0/143997283 wait complete. 2024-04-25T15:51:09.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 -- start start 2024-04-25T15:51:09.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 Processor -- start 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669f2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 0x55dc669f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66880c20 0x55dc669f3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55dc667e8450 con 0x55dc66880c20 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55dc667e3130 con 0x55dc6686cea0 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55dc667f00c0 con 0x55dc66465280 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 10 client.0 _ll_register_callbacks cb 0x7f013c017070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66880c20 0x55dc669f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66880c20 0x55dc669f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47206/0 (socket says 172.21.15.67:47206) 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 -- 172.21.15.67:0/1943931703 learned_addr learned my addr 172.21.15.67:0/1943931703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:09.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0149d9b640 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 0x55dc669f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014a59c640 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 -- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 msgr2=0x55dc669f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 -- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 msgr2=0x55dc669f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 0x55dc669f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 -- 172.21.15.67:0/1943931703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc66866180 con 0x55dc66880c20 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f014ad9d640 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66880c20 0x55dc669f3300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f01440f2b40 tx=0x7f01440f6890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103282230731d962 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f01297fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f01297fa640 1 -- 172.21.15.67:0/1943931703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01440f6230 con 0x55dc66880c20 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f01297fa640 1 -- 172.21.15.67:0/1943931703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f01440f6f30 con 0x55dc66880c20 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0134031070 con 0x55dc66880c20 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f01297fa640 1 -- 172.21.15.67:0/1943931703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f01440a6770 con 0x55dc66880c20 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 10 client.4728 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55dc6649d0c0 con 0x55dc66880c20 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.992+0000 7f01297fa640 1 -- 172.21.15.67:0/1943931703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0134031070 con 0x55dc66880c20 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f01297fa640 1 -- 172.21.15.67:0/1943931703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f0144008040 con 0x55dc66880c20 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 10 client.4728 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 10 client.4728 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 20 client.4728 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc66477830 0x55dc669f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 4 client.4728 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 --> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] -- command(tid 0: {"prefix": "status"}) -- 0x55dc667f1a30 con 0x55dc66477830 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f014a59c640 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc66477830 0x55dc669f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f01297fa640 10 client.4728 ms_handle_connect on v2:172.21.15.67:6832/144842397 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f014a59c640 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc66477830 0x55dc669f3c60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9961576925e0a9df server_cookie=444f9b69fb8b774f in_seq=0 out_seq=0 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f01297fa640 1 -- 172.21.15.67:0/1943931703 <== mds.4 v2:172.21.15.67:6832/144842397 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x55dc667f1a30 con 0x55dc66477830 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f01297fa640 10 client.4728 handle_command_reply: tid=0 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 client.4728 shutdown 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66880c20 msgr2=0x55dc669f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66880c20 0x55dc669f3300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f01440f2b40 tx=0x7f01440f6890 comp rx=0 tx=0).stop 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 shutdown_connections 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:6832/144842397,v1:172.21.15.67:6834/144842397] conn(0x55dc66477830 0x55dc669f3c60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dc66880c20 0x55dc669f3300 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55dc6686cea0 0x55dc669f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 --2- 172.21.15.67:0/1943931703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55dc66465280 0x55dc669f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f01297fa640 0 client.4728 ms_handle_reset on v2:172.21.15.67:6832/144842397 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 >> 172.21.15.67:0/1943931703 conn(0x55dc664a4ff0 msgr2=0x55dc669988d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 shutdown_connections 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 1 -- 172.21.15.67:0/1943931703 wait complete. 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:09.996+0000 7f0158206000 20 client.4728 trim_cache size 0 max 16384 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 4, 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4270, 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 11.081241684, 2024-04-25T15:51:10.003 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 17, 2024-04-25T15:51:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 21, 2024-04-25T15:51:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 20, 2024-04-25T15:51:10.004 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 46.390788031 2024-04-25T15:51:10.004 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:10.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f012c06f430 msgr2=0x7f012c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f012c06f430 0x7f012c071940 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f013400b9f0 tx=0x7f0134008040 comp rx=0 tx=0).stop 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c152b00 msgr2=0x7f014c157d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c152b00 0x7f014c157d20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f01440b0040 tx=0x7f014401a8d0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 shutdown_connections 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f012c06f430 0x7f012c071940 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f014c152b00 0x7f014c157d20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f014c0c8d90 0x7f014c147980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 --2- 172.21.15.67:0/1872671622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f014c06c640 0x7f014c146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 >> 172.21.15.67:0/1872671622 conn(0x7f014c06fe70 msgr2=0x7f014c0c6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 shutdown_connections 2024-04-25T15:51:10.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.000+0000 7f0150ea9640 1 -- 172.21.15.67:0/1872671622 wait complete. 2024-04-25T15:51:10.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T15:51:10.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 Processor -- start 2024-04-25T15:51:10.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 -- start start 2024-04-25T15:51:10.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49dc06c640 0x7f49dc154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc15e4a0 0x7f49dc0c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f49dc15e8b0 0x7f49dc0c8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f49dc0749d0 con 0x7f49dc06c640 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f49dc06e510 con 0x7f49dc15e8b0 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49e08f8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f49dc15dca0 con 0x7f49dc15e4a0 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49dc06c640 0x7f49dc154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49dc06c640 0x7f49dc154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47220/0 (socket says 172.21.15.67:47220) 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 -- 172.21.15.67:0/4107234988 learned_addr learned my addr 172.21.15.67:0/4107234988 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/4107234988 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc15e4a0 0x7f49dc0c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49dad9d640 1 --2- 172.21.15.67:0/4107234988 >> v2:172.21.15.176:3300/0 conn(0x7f49dc15e8b0 0x7f49dc0c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 -- 172.21.15.67:0/4107234988 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc15e4a0 msgr2=0x7f49dc0c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 --2- 172.21.15.67:0/4107234988 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc15e4a0 0x7f49dc0c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 -- 172.21.15.67:0/4107234988 >> v2:172.21.15.176:3300/0 conn(0x7f49dc15e8b0 msgr2=0x7f49dc0c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 --2- 172.21.15.67:0/4107234988 >> v2:172.21.15.176:3300/0 conn(0x7f49dc15e8b0 0x7f49dc0c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.172+0000 7f49da59c640 1 -- 172.21.15.67:0/4107234988 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc15df80 con 0x7f49dc06c640 2024-04-25T15:51:10.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49da59c640 1 --2- 172.21.15.67:0/4107234988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc06c640 0x7f49dc154a60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f49dc158280 tx=0x7f49d401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14976b7d6ff3c3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49d959a640 1 -- 172.21.15.67:0/4107234988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d40a5730 con 0x7f49dc06c640 2024-04-25T15:51:10.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49d959a640 1 -- 172.21.15.67:0/4107234988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d40a58d0 con 0x7f49dc06c640 2024-04-25T15:51:10.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49d959a640 1 -- 172.21.15.67:0/4107234988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d40a5ca0 con 0x7f49dc06c640 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 -- 172.21.15.67:0/4107234988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc06c640 msgr2=0x7f49dc154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 --2- 172.21.15.67:0/4107234988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc06c640 0x7f49dc154a60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f49dc158280 tx=0x7f49d401c960 comp rx=0 tx=0).stop 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 -- 172.21.15.67:0/4107234988 shutdown_connections 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 --2- 172.21.15.67:0/4107234988 >> v2:172.21.15.176:3300/0 conn(0x7f49dc15e8b0 0x7f49dc0c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 --2- 172.21.15.67:0/4107234988 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc15e4a0 0x7f49dc0c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 --2- 172.21.15.67:0/4107234988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc06c640 0x7f49dc154a60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 -- 172.21.15.67:0/4107234988 >> 172.21.15.67:0/4107234988 conn(0x7f49dc070db0 msgr2=0x7f49dc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 -- 172.21.15.67:0/4107234988 shutdown_connections 2024-04-25T15:51:10.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.176+0000 7f49e08f8640 1 -- 172.21.15.67:0/4107234988 wait complete. 2024-04-25T15:51:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.180+0000 7f49e08f8640 1 Processor -- start 2024-04-25T15:51:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.180+0000 7f49e08f8640 1 -- start start 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.180+0000 7f49e08f8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc06c640 0x7f49dc14a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49e08f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15e4a0 0x7f49dc1f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49e08f8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49dc15e8b0 0x7f49dc154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49e08f8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc0cad60 con 0x7f49dc15e4a0 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49e08f8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc072bc0 con 0x7f49dc15e8b0 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49e08f8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f49dc147c90 con 0x7f49dc06c640 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15e4a0 0x7f49dc1f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15e4a0 0x7f49dc1f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47236/0 (socket says 172.21.15.67:47236) 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3040905875 learned_addr learned my addr 172.21.15.67:0/3040905875 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49da59c640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc06c640 0x7f49dc14a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49dad9d640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49dc15e8b0 0x7f49dc154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc06c640 msgr2=0x7f49dc14a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc06c640 0x7f49dc14a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49dc15e8b0 msgr2=0x7f49dc154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49dc15e8b0 0x7f49dc154080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3040905875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc15b9f0 con 0x7f49dc15e4a0 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49dad9d640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49dc15e8b0 0x7f49dc154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15e4a0 0x7f49dc1f40f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f49c400d800 tx=0x7f49c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d2ffd640 1 -- 172.21.15.67:0/3040905875 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c400b6c0 con 0x7f49dc15e4a0 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d2ffd640 1 -- 172.21.15.67:0/3040905875 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c4019070 con 0x7f49dc15e4a0 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49dc1477c0 con 0x7f49dc15e4a0 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d2ffd640 1 -- 172.21.15.67:0/3040905875 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c400ba80 con 0x7f49dc15e4a0 2024-04-25T15:51:10.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49dc1f3610 con 0x7f49dc15e4a0 2024-04-25T15:51:10.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d2ffd640 1 -- 172.21.15.67:0/3040905875 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f49c400bc20 con 0x7f49dc15e4a0 2024-04-25T15:51:10.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d2ffd640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f49bc06f430 0x7f49bc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49d2ffd640 1 -- 172.21.15.67:0/3040905875 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f49c408f5e0 con 0x7f49dc15e4a0 2024-04-25T15:51:10.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.184+0000 7f49da59c640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f49bc06f430 0x7f49bc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49da59c640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f49bc06f430 0x7f49bc071940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49d40ae000 tx=0x7f49d4004ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 Processor -- start 2024-04-25T15:51:10.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 -- start start 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a3078765d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a307841550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a30730a280 0x55a307841a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a3076f3160 con 0x55a307711ea0 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55a307688e20 con 0x55a30730a280 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55a307641a20 con 0x55a307725c20 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a307841550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a307841550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47252/0 (socket says 172.21.15.67:47252) 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 -- 172.21.15.67:0/1060693947 learned_addr learned my addr 172.21.15.67:0/1060693947 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49da59c640 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a3078765d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49dad9d640 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a30730a280 0x55a307841a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 -- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 msgr2=0x55a3078765d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a3078765d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 -- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a30730a280 msgr2=0x55a307841a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a30730a280 0x55a307841a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 -- 172.21.15.67:0/1060693947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a307646100 con 0x55a307711ea0 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49dad9d640 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a30730a280 0x55a307841a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a307841550 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f49c40583d0 tx=0x7f49c405b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f543390c4bacec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49bbfff640 1 -- 172.21.15.67:0/1060693947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c405c4e0 con 0x55a307711ea0 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49bbfff640 1 -- 172.21.15.67:0/1060693947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c405bde0 con 0x55a307711ea0 2024-04-25T15:51:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49bbfff640 1 -- 172.21.15.67:0/1060693947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c405b240 con 0x55a307711ea0 2024-04-25T15:51:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 -- 172.21.15.67:0/1060693947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 msgr2=0x55a307841550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.188+0000 7f49e7c55000 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a307841550 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f49c40583d0 tx=0x7f49c405b730 comp rx=0 tx=0).stop 2024-04-25T15:51:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- 172.21.15.67:0/1060693947 shutdown_connections 2024-04-25T15:51:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a30730a280 0x55a307841a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a307841550 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 --2- 172.21.15.67:0/1060693947 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a3078765d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- 172.21.15.67:0/1060693947 >> 172.21.15.67:0/1060693947 conn(0x55a307349ff0 msgr2=0x55a307879030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- 172.21.15.67:0/1060693947 shutdown_connections 2024-04-25T15:51:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- 172.21.15.67:0/1060693947 wait complete. 2024-04-25T15:51:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- start start 2024-04-25T15:51:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 Processor -- start 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078414e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307841a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307725c20 0x55a3078808f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a307696220 con 0x55a307725c20 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55a30762cea0 con 0x55a307711ea0 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55a307683b00 con 0x55a30730a280 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 10 client.0 _ll_register_callbacks cb 0x55a307695dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307725c20 0x55a3078808f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307725c20 0x55a3078808f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47260/0 (socket says 172.21.15.67:47260) 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 -- 172.21.15.67:0/2709260989 learned_addr learned my addr 172.21.15.67:0/2709260989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49da59c640 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307841a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 -- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 msgr2=0x55a3078414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 -- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 msgr2=0x55a307841a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307841a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 -- 172.21.15.67:0/2709260989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a307628500 con 0x55a307725c20 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49dad9d640 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307725c20 0x55a3078808f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0cf6e0 tx=0x7f49cc010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8cc978ffcb6408e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49b97fa640 1 -- 172.21.15.67:0/2709260989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc00f040 con 0x55a307725c20 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49b97fa640 1 -- 172.21.15.67:0/2709260989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f49cc01c600 con 0x55a307725c20 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49c405bde0 con 0x55a307725c20 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49b97fa640 1 -- 172.21.15.67:0/2709260989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f49cc024050 con 0x55a307725c20 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 10 client.4740 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55a30760eea0 con 0x55a307725c20 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.192+0000 7f49b97fa640 1 -- 172.21.15.67:0/2709260989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c405bde0 con 0x55a307725c20 2024-04-25T15:51:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49b97fa640 1 -- 172.21.15.67:0/2709260989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f49cc027020 con 0x55a307725c20 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 10 client.4740 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 10 client.4740 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 20 client.4740 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30732e7d0 0x55a307881630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 4 client.4740 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 --> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a307621be0 con 0x55a30732e7d0 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49da59c640 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30732e7d0 0x55a307881630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49da59c640 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30732e7d0 0x55a307881630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=77e8c5e4c68f2670 server_cookie=70507ea3a8140136 in_seq=0 out_seq=0 2024-04-25T15:51:10.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49b97fa640 10 client.4740 ms_handle_connect on v2:172.21.15.176:6836/2023672746 2024-04-25T15:51:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49b97fa640 1 -- 172.21.15.67:0/2709260989 <== mds.3 v2:172.21.15.176:6836/2023672746 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a307621be0 con 0x55a30732e7d0 2024-04-25T15:51:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49b97fa640 10 client.4740 handle_command_reply: tid=0 2024-04-25T15:51:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 1 client.4740 shutdown 2024-04-25T15:51:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307725c20 msgr2=0x55a3078808f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.196+0000 7f49e7c55000 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307725c20 0x55a3078808f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0cf6e0 tx=0x7f49cc010680 comp rx=0 tx=0).stop 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 shutdown_connections 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307725c20 0x55a3078808f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30732e7d0 0x55a307881630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307841a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 --2- 172.21.15.67:0/2709260989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49b97fa640 0 client.4740 ms_handle_reset on v2:172.21.15.176:6836/2023672746 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 >> 172.21.15.67:0/2709260989 conn(0x55a307349ff0 msgr2=0x55a3078781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 shutdown_connections 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 1 -- 172.21.15.67:0/2709260989 wait complete. 2024-04-25T15:51:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.200+0000 7f49e7c55000 20 client.4740 trim_cache size 0 max 16384 2024-04-25T15:51:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.216+0000 7f49e7c55000 1 Processor -- start 2024-04-25T15:51:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.216+0000 7f49e7c55000 1 -- start start 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a30730a280 0x55a3078960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307896620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a307879f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc2027c0 con 0x55a30730a280 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1495c0 con 0x55a307711ea0 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55a30761e7d0 con 0x55a307725c20 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a30730a280 0x55a3078960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a30730a280 0x55a3078960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47268/0 (socket says 172.21.15.67:47268) 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 -- 172.21.15.67:0/364426991 learned_addr learned my addr 172.21.15.67:0/364426991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49dad9d640 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a307879f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 -- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 msgr2=0x55a307879f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307896620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a307879f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 -- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 msgr2=0x55a307896620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307896620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 -- 172.21.15.67:0/364426991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a307641c70 con 0x55a30730a280 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49dad9d640 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a307879f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307896620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49da59c640 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a30730a280 0x55a3078960e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f49d40b9040 tx=0x7f49d40a7070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61016fc5c6da906 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49bb7fe640 1 -- 172.21.15.67:0/364426991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d40ce020 con 0x55a30730a280 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49bb7fe640 1 -- 172.21.15.67:0/364426991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d40ca3a0 con 0x55a30730a280 2024-04-25T15:51:10.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49bb7fe640 1 -- 172.21.15.67:0/364426991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d40ca750 con 0x55a30730a280 2024-04-25T15:51:10.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- 172.21.15.67:0/364426991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a30730a280 msgr2=0x55a3078960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a30730a280 0x55a3078960e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f49d40b9040 tx=0x7f49d40a7070 comp rx=0 tx=0).stop 2024-04-25T15:51:10.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- 172.21.15.67:0/364426991 shutdown_connections 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a307725c20 0x55a307879f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307711ea0 0x55a307896620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 --2- 172.21.15.67:0/364426991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a30730a280 0x55a3078960e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- 172.21.15.67:0/364426991 >> 172.21.15.67:0/364426991 conn(0x55a307349ff0 msgr2=0x55a3078eafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- 172.21.15.67:0/364426991 shutdown_connections 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- 172.21.15.67:0/364426991 wait complete. 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 -- start start 2024-04-25T15:51:10.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.220+0000 7f49e7c55000 1 Processor -- start 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a3078b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307725c20 0x55a3078b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a30768d450 con 0x55a307711ea0 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55a307688130 con 0x55a307725c20 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55a3076950c0 con 0x55a30730a280 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 10 client.0 _ll_register_callbacks cb 0x7f49cc006ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a3078b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a3078b2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47278/0 (socket says 172.21.15.67:47278) 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3044035620 learned_addr learned my addr 172.21.15.67:0/3044035620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49da59c640 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49dad9d640 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307725c20 0x55a3078b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 msgr2=0x55a3078b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307725c20 msgr2=0x55a3078b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307725c20 0x55a3078b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 -- 172.21.15.67:0/3044035620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a30770b180 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49d9d9b640 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a3078b2fa0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49c405ba90 tx=0x7f49c4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4fe54ea771cea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 1 -- 172.21.15.67:0/3044035620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c405c4e0 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 1 -- 172.21.15.67:0/3044035620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f49c405b250 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d40ca3a0 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 1 -- 172.21.15.67:0/3044035620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f49c4060190 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 10 client.4746 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55a3073420c0 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 1 -- 172.21.15.67:0/3044035620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d40ca3a0 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 1 -- 172.21.15.67:0/3044035620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f49c4075030 con 0x55a307711ea0 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 10 client.4746 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 10 client.4746 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 20 client.4746 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30731c830 0x55a3078b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 4 client.4746 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 --> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] -- command(tid 0: {"prefix": "status"}) -- 0x55a307696a30 con 0x55a30731c830 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49da59c640 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30731c830 0x55a3078b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 10 client.4746 ms_handle_connect on v2:172.21.15.176:6836/2023672746 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49da59c640 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30731c830 0x55a3078b3e70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f74013bd42a60d9 server_cookie=26da6c7180a21876 in_seq=0 out_seq=0 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 1 -- 172.21.15.67:0/3044035620 <== mds.3 v2:172.21.15.176:6836/2023672746 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55a307696a30 con 0x55a30731c830 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 10 client.4746 handle_command_reply: tid=0 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 client.4746 shutdown 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 msgr2=0x55a3078b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a3078b2fa0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f49c405ba90 tx=0x7f49c4000bd0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 shutdown_connections 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:6836/2023672746,v1:172.21.15.176:6838/2023672746] conn(0x55a30731c830 0x55a3078b3e70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a307725c20 0x55a3078b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a307711ea0 0x55a3078b2fa0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49b97fa640 0 client.4746 ms_handle_reset on v2:172.21.15.176:6836/2023672746 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 --2- 172.21.15.67:0/3044035620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a30730a280 0x55a3078b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 >> 172.21.15.67:0/3044035620 conn(0x55a307349ff0 msgr2=0x55a307895b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 shutdown_connections 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.224+0000 7f49e7c55000 1 -- 172.21.15.67:0/3044035620 wait complete. 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e7c55000 20 client.4746 trim_cache size 0 max 16384 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 3, 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4272, 2024-04-25T15:51:10.233 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 17.348946130000002, 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 17, 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 21, 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 20, 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 46.600588625 2024-04-25T15:51:10.234 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f49bc06f430 msgr2=0x7f49bc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f49bc06f430 0x7f49bc071940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f49d40ae000 tx=0x7f49d4004ac0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15e4a0 msgr2=0x7f49dc1f40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15e4a0 0x7f49dc1f40f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f49c400d800 tx=0x7f49c400dcc0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 shutdown_connections 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f49bc06f430 0x7f49bc071940 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f49dc15e8b0 0x7f49dc154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49dc15e4a0 0x7f49dc1f40f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 --2- 172.21.15.67:0/3040905875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49dc06c640 0x7f49dc14a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 >> 172.21.15.67:0/3040905875 conn(0x7f49dc070db0 msgr2=0x7f49dc073800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 shutdown_connections 2024-04-25T15:51:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.228+0000 7f49e08f8640 1 -- 172.21.15.67:0/3040905875 wait complete. 2024-04-25T15:51:10.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T15:51:10.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46fe71f640 1 Processor -- start 2024-04-25T15:51:10.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46fe71f640 1 -- start start 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46fe71f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46f8074790 0x7f46f815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46fe71f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f46f815e020 0x7f46f80723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46fe71f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f8072dc0 con 0x7f46f80728e0 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46fe71f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f46f815b430 con 0x7f46f8074790 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46fe71f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f46f815a670 con 0x7f46f815e020 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.392+0000 7f46f7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46f8074790 0x7f46f815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46f8074790 0x7f46f815dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47282/0 (socket says 172.21.15.67:47282) 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f7fff640 1 -- 172.21.15.67:0/1196032963 learned_addr learned my addr 172.21.15.67:0/1196032963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f77fe640 1 --2- 172.21.15.67:0/1196032963 >> v2:172.21.15.176:3300/0 conn(0x7f46f815e020 0x7f46f80723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f7fff640 1 -- 172.21.15.67:0/1196032963 >> v2:172.21.15.176:3300/0 conn(0x7f46f815e020 msgr2=0x7f46f80723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f7fff640 1 --2- 172.21.15.67:0/1196032963 >> v2:172.21.15.176:3300/0 conn(0x7f46f815e020 0x7f46f80723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f7fff640 1 -- 172.21.15.67:0/1196032963 >> v1:172.21.15.176:6789/0 conn(0x7f46f80728e0 legacy=0x7f46f80c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f7fff640 1 -- 172.21.15.67:0/1196032963 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f815a950 con 0x7f46f8074790 2024-04-25T15:51:10.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f7fff640 1 --2- 172.21.15.67:0/1196032963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f815dae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f46f8070a50 tx=0x7f46e801f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be894eab56f550b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f6ffd640 1 -- 172.21.15.67:0/1196032963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e80b4070 con 0x7f46f8074790 2024-04-25T15:51:10.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f6ffd640 1 -- 172.21.15.67:0/1196032963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e80af400 con 0x7f46f8074790 2024-04-25T15:51:10.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46f6ffd640 1 -- 172.21.15.67:0/1196032963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e80af7d0 con 0x7f46f8074790 2024-04-25T15:51:10.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46fe71f640 1 -- 172.21.15.67:0/1196032963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 msgr2=0x7f46f815dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.396+0000 7f46fe71f640 1 --2- 172.21.15.67:0/1196032963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f815dae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f46f8070a50 tx=0x7f46e801f1a0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- 172.21.15.67:0/1196032963 shutdown_connections 2024-04-25T15:51:10.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 --2- 172.21.15.67:0/1196032963 >> v2:172.21.15.176:3300/0 conn(0x7f46f815e020 0x7f46f80723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 --2- 172.21.15.67:0/1196032963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f815dae0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- 172.21.15.67:0/1196032963 >> 172.21.15.67:0/1196032963 conn(0x7f46f806fe70 msgr2=0x7f46f80cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- 172.21.15.67:0/1196032963 shutdown_connections 2024-04-25T15:51:10.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- 172.21.15.67:0/1196032963 wait complete. 2024-04-25T15:51:10.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 Processor -- start 2024-04-25T15:51:10.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- start start 2024-04-25T15:51:10.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46f80728e0 0x7f46f815b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f814bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46f815e020 0x7f46f814c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f46f8073080 con 0x7f46f8074790 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f46f815af10 con 0x7f46f815e020 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46fe71f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f46f8074ba0 con 0x7f46f80728e0 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f814bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.400+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f814bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47290/0 (socket says 172.21.15.67:47290) 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f77fe640 1 -- 172.21.15.67:0/207617199 learned_addr learned my addr 172.21.15.67:0/207617199 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f7fff640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46f80728e0 0x7f46f815b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46fccbc640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46f815e020 0x7f46f814c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f77fe640 1 -- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46f80728e0 msgr2=0x7f46f815b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f77fe640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46f80728e0 0x7f46f815b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f77fe640 1 -- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46f815e020 msgr2=0x7f46f814c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f77fe640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46f815e020 0x7f46f814c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f77fe640 1 -- 172.21.15.67:0/207617199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f815d920 con 0x7f46f8074790 2024-04-25T15:51:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f77fe640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f814bf40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00d5f0 tx=0x7f46ec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f4ff9640 1 -- 172.21.15.67:0/207617199 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0125f0 con 0x7f46f8074790 2024-04-25T15:51:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f4ff9640 1 -- 172.21.15.67:0/207617199 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec019070 con 0x7f46f8074790 2024-04-25T15:51:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f4ff9640 1 -- 172.21.15.67:0/207617199 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0129b0 con 0x7f46f8074790 2024-04-25T15:51:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f815dc00 con 0x7f46f8074790 2024-04-25T15:51:10.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f815bbf0 con 0x7f46f8074790 2024-04-25T15:51:10.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f4ff9640 1 -- 172.21.15.67:0/207617199 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f46ec012b50 con 0x7f46f8074790 2024-04-25T15:51:10.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f4ff9640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f46e406f400 0x7f46e4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.404+0000 7f46f4ff9640 1 -- 172.21.15.67:0/207617199 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f46ec08f7d0 con 0x7f46f8074790 2024-04-25T15:51:10.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f7fff640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f46e406f400 0x7f46e4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 Processor -- start 2024-04-25T15:51:10.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 -- start start 2024-04-25T15:51:10.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f7fff640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f46e406f400 0x7f46e4071910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f46e80b1000 tx=0x7f46e80aa000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb886b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb887c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb887ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559fb86e8160 con 0x559fb8706ea0 2024-04-25T15:51:10.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x559fb867de20 con 0x559fb871ac20 2024-04-25T15:51:10.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x559fb8636a20 con 0x559fb82ff280 2024-04-25T15:51:10.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb887c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb887c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47292/0 (socket says 172.21.15.67:47292) 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 -- 172.21.15.67:0/2259912893 learned_addr learned my addr 172.21.15.67:0/2259912893 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46fccbc640 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb887ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f7fff640 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb886b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 -- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 msgr2=0x559fb887ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb887ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 -- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 msgr2=0x559fb886b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb886b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 -- 172.21.15.67:0/2259912893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fb863b100 con 0x559fb8706ea0 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46f77fe640 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb887c530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46ec08dbc0 tx=0x7f46ec05b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46b7256c294aa78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46ddffb640 1 -- 172.21.15.67:0/2259912893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec05bf40 con 0x559fb8706ea0 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46ddffb640 1 -- 172.21.15.67:0/2259912893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec0602f0 con 0x559fb8706ea0 2024-04-25T15:51:10.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f46ddffb640 1 -- 172.21.15.67:0/2259912893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec064da0 con 0x559fb8706ea0 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 -- 172.21.15.67:0/2259912893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 msgr2=0x559fb887c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb887c530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f46ec08dbc0 tx=0x7f46ec05b770 comp rx=0 tx=0).stop 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 -- 172.21.15.67:0/2259912893 shutdown_connections 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb887ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb887c530 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2259912893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb886b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.408+0000 7f4705a7c000 1 -- 172.21.15.67:0/2259912893 >> 172.21.15.67:0/2259912893 conn(0x559fb833eff0 msgr2=0x559fb886e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- 172.21.15.67:0/2259912893 shutdown_connections 2024-04-25T15:51:10.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- 172.21.15.67:0/2259912893 wait complete. 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- start start 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 Processor -- start 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb886b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb886b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb887a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559fb868b220 con 0x559fb8706ea0 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x559fb8621ea0 con 0x559fb871ac20 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x559fb8678b00 con 0x559fb82ff280 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 10 client.0 _ll_register_callbacks cb 0x559fb868adc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb886b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb886b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47306/0 (socket says 172.21.15.67:47306) 2024-04-25T15:51:10.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 -- 172.21.15.67:0/424023792 learned_addr learned my addr 172.21.15.67:0/424023792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f7fff640 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb886b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46fccbc640 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb887a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 -- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 msgr2=0x559fb886b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb886b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 -- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 msgr2=0x559fb887a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb887a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 -- 172.21.15.67:0/424023792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fb861d500 con 0x559fb8706ea0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46f77fe640 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb886b940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f46ec08dc50 tx=0x7f46ec000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7a483d42537cea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46d6ffd640 1 -- 172.21.15.67:0/424023792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec07a070 con 0x559fb8706ea0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46d6ffd640 1 -- 172.21.15.67:0/424023792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f46ec05b3a0 con 0x559fb8706ea0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46ec0602f0 con 0x559fb8706ea0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46d6ffd640 1 -- 172.21.15.67:0/424023792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f46ec0578b0 con 0x559fb8706ea0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 10 client.4758 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x559fb8603ea0 con 0x559fb8706ea0 2024-04-25T15:51:10.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46d6ffd640 1 -- 172.21.15.67:0/424023792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec0602f0 con 0x559fb8706ea0 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f46d6ffd640 1 -- 172.21.15.67:0/424023792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f46ec013070 con 0x559fb8706ea0 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 10 client.4758 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 10 client.4758 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.412+0000 7f4705a7c000 20 client.4758 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb83237d0 0x559fb887b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 4 client.4758 mds_command: new command op to 4276 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 --> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fb8616be0 con 0x559fb83237d0 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f46f7fff640 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb83237d0 0x559fb887b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f46f7fff640 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb83237d0 0x559fb887b470 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62ae815f52a6afed server_cookie=568efd61b0ffc3b7 in_seq=0 out_seq=0 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f46d6ffd640 10 client.4758 ms_handle_connect on v2:172.21.15.67:6836/2429063224 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f46d6ffd640 1 -- 172.21.15.67:0/424023792 <== mds.2 v2:172.21.15.67:6836/2429063224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559fb8616be0 con 0x559fb83237d0 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f46d6ffd640 10 client.4758 handle_command_reply: tid=0 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 client.4758 shutdown 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 msgr2=0x559fb886b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb886b940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f46ec08dc50 tx=0x7f46ec000bd0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 shutdown_connections 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb83237d0 0x559fb887b470 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb871ac20 0x559fb887a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f46d6ffd640 0 client.4758 ms_handle_reset on v2:172.21.15.67:6836/2429063224 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb886b940 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 --2- 172.21.15.67:0/424023792 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb82ff280 0x559fb886b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 >> 172.21.15.67:0/424023792 conn(0x559fb833eff0 msgr2=0x559fb886ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 shutdown_connections 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 1 -- 172.21.15.67:0/424023792 wait complete. 2024-04-25T15:51:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.416+0000 7f4705a7c000 20 client.4758 trim_cache size 0 max 16384 2024-04-25T15:51:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 Processor -- start 2024-04-25T15:51:10.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- start start 2024-04-25T15:51:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb8890360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb88328d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb8832e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559fb86137d0 con 0x559fb8706ea0 2024-04-25T15:51:10.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x559fb867d130 con 0x559fb82ff280 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x559fb8682450 con 0x559fb871ac20 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb88328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb88328d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47322/0 (socket says 172.21.15.67:47322) 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 -- 172.21.15.67:0/2613303850 learned_addr learned my addr 172.21.15.67:0/2613303850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46fccbc640 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb8832e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f7fff640 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb8890360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 -- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 msgr2=0x559fb8832e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb8832e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 -- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 msgr2=0x559fb8890360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb8890360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 -- 172.21.15.67:0/2613303850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fb8636c70 con 0x559fb8706ea0 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46fccbc640 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb8832e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f7fff640 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb8890360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46f77fe640 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb88328d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f46ec05c7f0 tx=0x7f46ec08b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513a2ec840c64b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46dd7fa640 1 -- 172.21.15.67:0/2613303850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec07a070 con 0x559fb8706ea0 2024-04-25T15:51:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46dd7fa640 1 -- 172.21.15.67:0/2613303850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec0602f0 con 0x559fb8706ea0 2024-04-25T15:51:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f46dd7fa640 1 -- 172.21.15.67:0/2613303850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec05b3c0 con 0x559fb8706ea0 2024-04-25T15:51:10.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- 172.21.15.67:0/2613303850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 msgr2=0x559fb88328d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb88328d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f46ec05c7f0 tx=0x7f46ec08b000 comp rx=0 tx=0).stop 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- 172.21.15.67:0/2613303850 shutdown_connections 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb8832e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb88328d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 --2- 172.21.15.67:0/2613303850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb8890360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- 172.21.15.67:0/2613303850 >> 172.21.15.67:0/2613303850 conn(0x559fb833eff0 msgr2=0x559fb886dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- 172.21.15.67:0/2613303850 shutdown_connections 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.436+0000 7f4705a7c000 1 -- 172.21.15.67:0/2613303850 wait complete. 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 -- start start 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 Processor -- start 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb888d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb888df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb888e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559fb8628470 con 0x559fb8706ea0 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x559fb868a0c0 con 0x559fb82ff280 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x559fb868b220 con 0x559fb871ac20 2024-04-25T15:51:10.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 10 client.0 _ll_register_callbacks cb 0x7f46ec05b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb888df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb888df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47338/0 (socket says 172.21.15.67:47338) 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 -- 172.21.15.67:0/560734277 learned_addr learned my addr 172.21.15.67:0/560734277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46fccbc640 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb888e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f7fff640 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb888d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 -- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 msgr2=0x559fb888e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb888e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 -- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 msgr2=0x559fb888d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb888d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 -- 172.21.15.67:0/560734277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fb8700180 con 0x559fb8706ea0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f77fe640 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb888df00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0591b0 tx=0x7f46ec0586c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a9431d8848eec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46d6ffd640 1 -- 172.21.15.67:0/560734277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec07a040 con 0x559fb8706ea0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46d6ffd640 1 -- 172.21.15.67:0/560734277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f46ec061480 con 0x559fb8706ea0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46ec0602f0 con 0x559fb8706ea0 2024-04-25T15:51:10.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46d6ffd640 1 -- 172.21.15.67:0/560734277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f46ec061680 con 0x559fb8706ea0 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 10 client.4764 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x559fb83370c0 con 0x559fb8706ea0 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46d6ffd640 1 -- 172.21.15.67:0/560734277 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec0602f0 con 0x559fb8706ea0 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46d6ffd640 1 -- 172.21.15.67:0/560734277 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f46ec046bc0 con 0x559fb8706ea0 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 10 client.4764 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 10 client.4764 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 20 client.4764 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb8311830 0x559fb888ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 4 client.4764 mds_command: new command op to 4276 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 --> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] -- command(tid 0: {"prefix": "status"}) -- 0x559fb868ba30 con 0x559fb8311830 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.440+0000 7f46f7fff640 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb8311830 0x559fb888ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46d6ffd640 10 client.4764 ms_handle_connect on v2:172.21.15.67:6836/2429063224 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46f7fff640 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb8311830 0x559fb888ee00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b956ef010f716497 server_cookie=aa00be35858f1789 in_seq=0 out_seq=0 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46d6ffd640 1 -- 172.21.15.67:0/560734277 <== mds.2 v2:172.21.15.67:6836/2429063224 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x559fb868ba30 con 0x559fb8311830 2024-04-25T15:51:10.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46d6ffd640 10 client.4764 handle_command_reply: tid=0 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 client.4764 shutdown 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 msgr2=0x559fb888df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb888df00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0591b0 tx=0x7f46ec0586c0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 shutdown_connections 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x559fb871ac20 0x559fb888e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559fb8706ea0 0x559fb888df00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.67:6836/2429063224,v1:172.21.15.67:6837/2429063224] conn(0x559fb8311830 0x559fb888ee00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 --2- 172.21.15.67:0/560734277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x559fb82ff280 0x559fb888d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 >> 172.21.15.67:0/560734277 conn(0x559fb833eff0 msgr2=0x559fb8885df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46d6ffd640 0 client.4764 ms_handle_reset on v2:172.21.15.67:6836/2429063224 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 shutdown_connections 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 1 -- 172.21.15.67:0/560734277 wait complete. 2024-04-25T15:51:10.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f4705a7c000 20 client.4764 trim_cache size 0 max 16384 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 2, 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4276, 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 23.125538121000002, 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 17, 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 21, 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 20, 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 46.832708947999997 2024-04-25T15:51:10.451 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:10.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f46e406f400 msgr2=0x7f46e4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f46e406f400 0x7f46e4071910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f46e80b1000 tx=0x7f46e80aa000 comp rx=0 tx=0).stop 2024-04-25T15:51:10.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 msgr2=0x7f46f814bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f814bf40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00d5f0 tx=0x7f46ec00dab0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 shutdown_connections 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f46e406f400 0x7f46e4071910 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46f815e020 0x7f46f814c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f8074790 0x7f46f814bf40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 --2- 172.21.15.67:0/207617199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46f80728e0 0x7f46f815b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 >> 172.21.15.67:0/207617199 conn(0x7f46f806fe70 msgr2=0x7f46f8071d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 shutdown_connections 2024-04-25T15:51:10.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.444+0000 7f46fe71f640 1 -- 172.21.15.67:0/207617199 wait complete. 2024-04-25T15:51:10.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T15:51:10.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1dfde88640 1 Processor -- start 2024-04-25T15:51:10.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1dfde88640 1 -- start start 2024-04-25T15:51:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1dfde88640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1df80736d0 0x7f1df806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1dfde88640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1df806cb80 0x7f1df815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1dfde88640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df80caa60 con 0x7f1df806d460 2024-04-25T15:51:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1dfde88640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1df8068590 con 0x7f1df80736d0 2024-04-25T15:51:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1dfde88640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1df815a5f0 con 0x7f1df806cb80 2024-04-25T15:51:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1df80736d0 0x7f1df806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1df806d460 0x7f1df81547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36078/0 (socket says 172.21.15.67:36078) 2024-04-25T15:51:10.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df77fe640 1 -- 172.21.15.67:0/30385768 learned_addr learned my addr 172.21.15.67:0/30385768 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:10.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df7fff640 1 --2- 172.21.15.67:0/30385768 >> v2:172.21.15.176:3300/0 conn(0x7f1df806cb80 0x7f1df815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df6ffd640 1 -- 172.21.15.67:0/30385768 >> v2:172.21.15.176:3300/0 conn(0x7f1df806cb80 msgr2=0x7f1df815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/30385768 >> v2:172.21.15.176:3300/0 conn(0x7f1df806cb80 0x7f1df815aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df6ffd640 1 -- 172.21.15.67:0/30385768 >> v1:172.21.15.67:6789/0 conn(0x7f1df806d460 legacy=0x7f1df81547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df67fc640 1 -- 172.21.15.67:0/30385768 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302930040 0 0) 0x7f1df80caa60 con 0x7f1df806d460 2024-04-25T15:51:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df6ffd640 1 -- 172.21.15.67:0/30385768 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df80ca100 con 0x7f1df80736d0 2024-04-25T15:51:10.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.612+0000 7f1df7fff640 1 --2- 172.21.15.67:0/30385768 >> v2:172.21.15.176:3300/0 conn(0x7f1df806cb80 0x7f1df815aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/30385768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df80736d0 0x7f1df806c640 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1df80b14e0 tx=0x7f1dec01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5731c083a9e69232 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1df67fc640 1 -- 172.21.15.67:0/30385768 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0b0070 con 0x7f1df80736d0 2024-04-25T15:51:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1df67fc640 1 -- 172.21.15.67:0/30385768 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec0ab600 con 0x7f1df80736d0 2024-04-25T15:51:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1df67fc640 1 -- 172.21.15.67:0/30385768 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0ab9d0 con 0x7f1df80736d0 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 -- 172.21.15.67:0/30385768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df80736d0 msgr2=0x7f1df806c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 --2- 172.21.15.67:0/30385768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df80736d0 0x7f1df806c640 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1df80b14e0 tx=0x7f1dec01c460 comp rx=0 tx=0).stop 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 -- 172.21.15.67:0/30385768 shutdown_connections 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 --2- 172.21.15.67:0/30385768 >> v2:172.21.15.176:3300/0 conn(0x7f1df806cb80 0x7f1df815aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 --2- 172.21.15.67:0/30385768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df80736d0 0x7f1df806c640 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 -- 172.21.15.67:0/30385768 >> 172.21.15.67:0/30385768 conn(0x7f1df80ce1c0 msgr2=0x7f1df80ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 -- 172.21.15.67:0/30385768 shutdown_connections 2024-04-25T15:51:10.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.616+0000 7f1dfde88640 1 -- 172.21.15.67:0/30385768 wait complete. 2024-04-25T15:51:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 Processor -- start 2024-04-25T15:51:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 -- start start 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1df806cb80 0x7f1df81f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806d460 0x7f1df8148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1df80736d0 0x7f1df8150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8167790 con 0x7f1df806d460 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1df815b440 con 0x7f1df806cb80 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1df8167910 con 0x7f1df80736d0 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806d460 0x7f1df8148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806d460 0x7f1df8148c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47348/0 (socket says 172.21.15.67:47348) 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 -- 172.21.15.67:0/2714802035 learned_addr learned my addr 172.21.15.67:0/2714802035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df77fe640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1df806cb80 0x7f1df81f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df7fff640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1df80736d0 0x7f1df8150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 -- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1df80736d0 msgr2=0x7f1df8150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1df80736d0 0x7f1df8150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 -- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1df806cb80 msgr2=0x7f1df81f6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1df806cb80 0x7f1df81f6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 -- 172.21.15.67:0/2714802035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df81f6650 con 0x7f1df806d460 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df77fe640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1df806cb80 0x7f1df81f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806d460 0x7f1df8148c70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1dec01a090 tx=0x7f1dec01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1ddffff640 1 -- 172.21.15.67:0/2714802035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0b0050 con 0x7f1df806d460 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1ddffff640 1 -- 172.21.15.67:0/2714802035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec0b1040 con 0x7f1df806d460 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df814b0b0 con 0x7f1df806d460 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1ddffff640 1 -- 172.21.15.67:0/2714802035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec004250 con 0x7f1df806d460 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.620+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df81533a0 con 0x7f1df806d460 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1ddffff640 1 -- 172.21.15.67:0/2714802035 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1dec0b5610 con 0x7f1df806d460 2024-04-25T15:51:10.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1ddffff640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1de406f430 0x7f1de4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1ddffff640 1 -- 172.21.15.67:0/2714802035 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f1dec128c30 con 0x7f1df806d460 2024-04-25T15:51:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1df77fe640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1de406f430 0x7f1de4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 Processor -- start 2024-04-25T15:51:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 -- start start 2024-04-25T15:51:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1df77fe640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1de406f430 0x7f1de4071940 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1de80012f0 tx=0x7f1de8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32a9bd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32a9bda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f32a86f160 con 0x55f32a88dea0 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55f32a804e20 con 0x55f32a8a1c20 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55f32a7bda20 con 0x55f32a486280 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32a9bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32a9bd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47358/0 (socket says 172.21.15.67:47358) 2024-04-25T15:51:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.624+0000 7f1df6ffd640 1 -- 172.21.15.67:0/1831224178 learned_addr learned my addr 172.21.15.67:0/1831224178 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df7fff640 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32a9bda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df77fe640 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df6ffd640 1 -- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 msgr2=0x55f32a9bda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32a9bda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df6ffd640 1 -- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 msgr2=0x55f32a9f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df6ffd640 1 -- 172.21.15.67:0/1831224178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f32a7c2100 con 0x55f32a88dea0 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32a9bd550 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0f14b0 tx=0x7f1dec019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ce4fbc2273fb20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1ddd7fa640 1 -- 172.21.15.67:0/1831224178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0f5430 con 0x55f32a88dea0 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1ddd7fa640 1 -- 172.21.15.67:0/1831224178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec0f97e0 con 0x55f32a88dea0 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1ddd7fa640 1 -- 172.21.15.67:0/1831224178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0fe200 con 0x55f32a88dea0 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- 172.21.15.67:0/1831224178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 msgr2=0x55f32a9bd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32a9bd550 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0f14b0 tx=0x7f1dec019200 comp rx=0 tx=0).stop 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- 172.21.15.67:0/1831224178 shutdown_connections 2024-04-25T15:51:10.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32a9bda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32a9bd550 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1831224178 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- 172.21.15.67:0/1831224178 >> 172.21.15.67:0/1831224178 conn(0x55f32a4c5ff0 msgr2=0x55f32a9f5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- 172.21.15.67:0/1831224178 shutdown_connections 2024-04-25T15:51:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- 172.21.15.67:0/1831224178 wait complete. 2024-04-25T15:51:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- start start 2024-04-25T15:51:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 Processor -- start 2024-04-25T15:51:10.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a486280 0x55f32a9bd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9bda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f32a86f160 con 0x55f32a486280 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55f32a7a8ea0 con 0x55f32a8a1c20 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55f32a7ffb00 con 0x55f32a88dea0 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1e051e5000 10 client.0 _ll_register_callbacks cb 0x55f32a811dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a486280 0x55f32a9bd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a486280 0x55f32a9bd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47360/0 (socket says 172.21.15.67:47360) 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.628+0000 7f1df77fe640 1 -- 172.21.15.67:0/1636831176 learned_addr learned my addr 172.21.15.67:0/1636831176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9bda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df7fff640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df77fe640 1 -- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 msgr2=0x55f32a9bda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df77fe640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9bda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df77fe640 1 -- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 msgr2=0x55f32a9fc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df77fe640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9fc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df77fe640 1 -- 172.21.15.67:0/1636831176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f32a7a4500 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df7fff640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df77fe640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a486280 0x55f32a9bd4e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1de80013e0 tx=0x7f1de8001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14ea4745ee5982e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1dd67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1dd67fc640 1 -- 172.21.15.67:0/1636831176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de8026060 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1dd67fc640 1 -- 172.21.15.67:0/1636831176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f1de8021b10 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dec0f97e0 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1dd67fc640 1 -- 172.21.15.67:0/1636831176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f1de8021d10 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 10 client.4779 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55f32a78aea0 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1dd67fc640 1 -- 172.21.15.67:0/1636831176 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec0f97e0 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1dd67fc640 1 -- 172.21.15.67:0/1636831176 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f1de80216f0 con 0x55f32a486280 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 10 client.4779 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 10 client.4779 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 20 client.4779 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4aa7d0 0x55f32a9fd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 4 client.4779 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:51:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 --> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f32a79dbe0 con 0x55f32a4aa7d0 2024-04-25T15:51:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4aa7d0 0x55f32a9fd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1dd67fc640 10 client.4779 ms_handle_connect on v2:172.21.15.176:6834/2385091301 2024-04-25T15:51:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.632+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4aa7d0 0x55f32a9fd690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9d2a4e0e942eaa18 server_cookie=58282f098bb0d645 in_seq=0 out_seq=0 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1dd67fc640 1 -- 172.21.15.67:0/1636831176 <== mds.1 v2:172.21.15.176:6834/2385091301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f32a79dbe0 con 0x55f32a4aa7d0 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1dd67fc640 10 client.4779 handle_command_reply: tid=0 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 client.4779 shutdown 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a486280 msgr2=0x55f32a9bd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a486280 0x55f32a9bd4e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1de80013e0 tx=0x7f1de8001260 comp rx=0 tx=0).stop 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 shutdown_connections 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4aa7d0 0x55f32a9fd690 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a8a1c20 0x55f32a9fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9bda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 --2- 172.21.15.67:0/1636831176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a486280 0x55f32a9bd4e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1dd67fc640 0 client.4779 ms_handle_reset on v2:172.21.15.176:6834/2385091301 2024-04-25T15:51:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 >> 172.21.15.67:0/1636831176 conn(0x55f32a4c5ff0 msgr2=0x55f32a9f41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 shutdown_connections 2024-04-25T15:51:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 1 -- 172.21.15.67:0/1636831176 wait complete. 2024-04-25T15:51:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.636+0000 7f1e051e5000 20 client.4779 trim_cache size 0 max 16384 2024-04-25T15:51:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.652+0000 7f1e051e5000 1 Processor -- start 2024-04-25T15:51:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.652+0000 7f1e051e5000 1 -- start start 2024-04-25T15:51:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a486280 0x55f32aa120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a4aa7d0 0x55f32aa12620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8202080 con 0x55f32a4aa7d0 2024-04-25T15:51:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1df8202750 con 0x55f32a486280 2024-04-25T15:51:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1df8153980 con 0x55f32a88dea0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a4aa7d0 0x55f32aa12620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a4aa7d0 0x55f32aa12620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47374/0 (socket says 172.21.15.67:47374) 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 -- 172.21.15.67:0/735144920 learned_addr learned my addr 172.21.15.67:0/735144920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df7fff640 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df77fe640 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a486280 0x55f32aa120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 -- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 msgr2=0x55f32a9f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 -- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a486280 msgr2=0x55f32aa120e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a486280 0x55f32aa120e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 -- 172.21.15.67:0/735144920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f32a9b5120 con 0x55f32a4aa7d0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a4aa7d0 0x55f32aa12620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0f53c0 tx=0x7f1dec128000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f64341b39ba53d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1ddcff9640 1 -- 172.21.15.67:0/735144920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0fe200 con 0x55f32a4aa7d0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1ddcff9640 1 -- 172.21.15.67:0/735144920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec0f5430 con 0x55f32a4aa7d0 2024-04-25T15:51:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1ddcff9640 1 -- 172.21.15.67:0/735144920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec0fdd70 con 0x55f32a4aa7d0 2024-04-25T15:51:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- 172.21.15.67:0/735144920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a4aa7d0 msgr2=0x55f32aa12620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a4aa7d0 0x55f32aa12620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1dec0f53c0 tx=0x7f1dec128000 comp rx=0 tx=0).stop 2024-04-25T15:51:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- 172.21.15.67:0/735144920 shutdown_connections 2024-04-25T15:51:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a88dea0 0x55f32a9f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a4aa7d0 0x55f32aa12620 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 --2- 172.21.15.67:0/735144920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a486280 0x55f32aa120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- 172.21.15.67:0/735144920 >> 172.21.15.67:0/735144920 conn(0x55f32a4c5ff0 msgr2=0x55f32aa66fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- 172.21.15.67:0/735144920 shutdown_connections 2024-04-25T15:51:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- 172.21.15.67:0/735144920 wait complete. 2024-04-25T15:51:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 -- start start 2024-04-25T15:51:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.656+0000 7f1e051e5000 1 Processor -- start 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32aa6eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a4aa7d0 0x55f32aa6f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32aa2ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f32a804130 con 0x55f32a88dea0 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55f32a79a7d0 con 0x55f32a4aa7d0 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55f32a809450 con 0x55f32a486280 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 10 client.0 _ll_register_callbacks cb 0x7f1de8005b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32aa2ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32aa2ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47376/0 (socket says 172.21.15.67:47376) 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 -- 172.21.15.67:0/770120526 learned_addr learned my addr 172.21.15.67:0/770120526 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df77fe640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32aa6eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a4aa7d0 0x55f32aa6f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 -- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 msgr2=0x55f32aa6eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32aa6eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 -- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a4aa7d0 msgr2=0x55f32aa6f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a4aa7d0 0x55f32aa6f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 -- 172.21.15.67:0/770120526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f32a7bdc70 con 0x55f32a88dea0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df77fe640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32aa6eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df6ffd640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a4aa7d0 0x55f32aa6f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df7fff640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32aa2ea60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1df0011bf0 tx=0x7f1df00136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b90c1c26b078f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 1 -- 172.21.15.67:0/770120526 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df001c070 con 0x55f32a88dea0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 1 -- 172.21.15.67:0/770120526 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f1df0016660 con 0x55f32a88dea0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dec0f5430 con 0x55f32a88dea0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 1 -- 172.21.15.67:0/770120526 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f1df0027050 con 0x55f32a88dea0 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 10 client.4785 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55f32a887180 con 0x55f32a88dea0 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 1 -- 172.21.15.67:0/770120526 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec0f5430 con 0x55f32a88dea0 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 1 -- 172.21.15.67:0/770120526 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f1df002a020 con 0x55f32a88dea0 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 10 client.4785 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 10 client.4785 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 20 client.4785 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4c3e80 0x55f32aa77d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 4 client.4785 mds_command: new command op to 4288 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 --> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] -- command(tid 0: {"prefix": "status"}) -- 0x55f32a812a30 con 0x55f32a4c3e80 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df77fe640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4c3e80 0x55f32aa77d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 10 client.4785 ms_handle_connect on v2:172.21.15.176:6834/2385091301 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1df77fe640 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4c3e80 0x55f32aa77d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d269bba6ed9a1b98 server_cookie=ed80be8008b656c1 in_seq=0 out_seq=0 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.660+0000 7f1dd67fc640 1 -- 172.21.15.67:0/770120526 <== mds.1 v2:172.21.15.176:6834/2385091301 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55f32a812a30 con 0x55f32a4c3e80 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dd67fc640 10 client.4785 handle_command_reply: tid=0 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 client.4785 shutdown 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 msgr2=0x55f32aa2ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32aa2ea60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1df0011bf0 tx=0x7f1df00136b0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 shutdown_connections 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:6834/2385091301,v1:172.21.15.176:6835/2385091301] conn(0x55f32a4c3e80 0x55f32aa77d50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f32a88dea0 0x55f32aa2ea60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55f32a4aa7d0 0x55f32aa6f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dd67fc640 0 client.4785 ms_handle_reset on v2:172.21.15.176:6834/2385091301 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 --2- 172.21.15.67:0/770120526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55f32a486280 0x55f32aa6eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 >> 172.21.15.67:0/770120526 conn(0x55f32a4c5ff0 msgr2=0x55f32aa11b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 shutdown_connections 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 1 -- 172.21.15.67:0/770120526 wait complete. 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1e051e5000 20 client.4785 trim_cache size 0 max 16384 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:10.669 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 1, 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4288, 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 33.428438526999997, 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 17, 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 21, 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 20, 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 47.040337821000001 2024-04-25T15:51:10.670 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1de406f430 msgr2=0x7f1de4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1de406f430 0x7f1de4071940 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1de80012f0 tx=0x7f1de8009040 comp rx=0 tx=0).stop 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806d460 msgr2=0x7f1df8148c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806d460 0x7f1df8148c70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1dec01a090 tx=0x7f1dec01aca0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 shutdown_connections 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1de406f430 0x7f1de4071940 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1df80736d0 0x7f1df8150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806d460 0x7f1df8148c70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 --2- 172.21.15.67:0/2714802035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1df806cb80 0x7f1df81f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 >> 172.21.15.67:0/2714802035 conn(0x7f1df80ce1c0 msgr2=0x7f1df8072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 shutdown_connections 2024-04-25T15:51:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.664+0000 7f1dfde88640 1 -- 172.21.15.67:0/2714802035 wait complete. 2024-04-25T15:51:10.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T15:51:10.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.828+0000 7ff78623d640 1 Processor -- start 2024-04-25T15:51:10.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.828+0000 7ff78623d640 1 -- start start 2024-04-25T15:51:10.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff78623d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff780159520 0x7ff7800c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff78623d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7800c4ec0 0x7ff7800c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff78623d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7800c7340 con 0x7ff780073bd0 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff78623d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff7800c5810 con 0x7ff7800c4ec0 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff78623d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff7800c67c0 con 0x7ff780159520 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7800c4ec0 0x7ff7800c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7800c4ec0 0x7ff7800c52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47388/0 (socket says 172.21.15.67:47388) 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 -- 172.21.15.67:0/3129146328 learned_addr learned my addr 172.21.15.67:0/3129146328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/3129146328 >> v2:172.21.15.176:3300/0 conn(0x7ff780159520 0x7ff7800c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 -- 172.21.15.67:0/3129146328 >> v2:172.21.15.176:3300/0 conn(0x7ff780159520 msgr2=0x7ff7800c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 --2- 172.21.15.67:0/3129146328 >> v2:172.21.15.176:3300/0 conn(0x7ff780159520 0x7ff7800c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 -- 172.21.15.67:0/3129146328 >> v1:172.21.15.176:6789/0 conn(0x7ff780073bd0 legacy=0x7ff7800c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 -- 172.21.15.67:0/3129146328 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7800c6aa0 con 0x7ff7800c4ec0 2024-04-25T15:51:10.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77ffff640 1 --2- 172.21.15.67:0/3129146328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff7800c52d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff78006d750 tx=0x7ff7780213a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03e4ae772593174 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77effd640 1 -- 172.21.15.67:0/3129146328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7780b6070 con 0x7ff7800c4ec0 2024-04-25T15:51:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77effd640 1 -- 172.21.15.67:0/3129146328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77800b2d0 con 0x7ff7800c4ec0 2024-04-25T15:51:10.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.832+0000 7ff77effd640 1 -- 172.21.15.67:0/3129146328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77800b480 con 0x7ff7800c4ec0 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- 172.21.15.67:0/3129146328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 msgr2=0x7ff7800c52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 --2- 172.21.15.67:0/3129146328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff7800c52d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff78006d750 tx=0x7ff7780213a0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- 172.21.15.67:0/3129146328 shutdown_connections 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 --2- 172.21.15.67:0/3129146328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff7800c52d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 --2- 172.21.15.67:0/3129146328 >> v2:172.21.15.176:3300/0 conn(0x7ff780159520 0x7ff7800c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- 172.21.15.67:0/3129146328 >> 172.21.15.67:0/3129146328 conn(0x7ff78006fe70 msgr2=0x7ff7800cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- 172.21.15.67:0/3129146328 shutdown_connections 2024-04-25T15:51:10.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- 172.21.15.67:0/3129146328 wait complete. 2024-04-25T15:51:10.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 Processor -- start 2024-04-25T15:51:10.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- start start 2024-04-25T15:51:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff780073bd0 0x7ff78015a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff78014a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff780159520 0x7ff78014e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7800736b0 con 0x7ff7800c4ec0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff7800c7060 con 0x7ff780073bd0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.836+0000 7ff78623d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff7800c6310 con 0x7ff780159520 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff78014a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff78014a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47396/0 (socket says 172.21.15.67:47396) 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 -- 172.21.15.67:0/3212473601 learned_addr learned my addr 172.21.15.67:0/3212473601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff780073bd0 0x7ff78015a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77ffff640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff780159520 0x7ff78014e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 -- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff780159520 msgr2=0x7ff78014e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff780159520 0x7ff78014e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 -- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff780073bd0 msgr2=0x7ff78015a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff780073bd0 0x7ff78015a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 -- 172.21.15.67:0/3212473601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78014abc0 con 0x7ff7800c4ec0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff777fff640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff78014a680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff76800b410 tx=0x7ff76800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77cff9640 1 -- 172.21.15.67:0/3212473601 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768015020 con 0x7ff7800c4ec0 2024-04-25T15:51:10.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77cff9640 1 -- 172.21.15.67:0/3212473601 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768011830 con 0x7ff7800c4ec0 2024-04-25T15:51:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77cff9640 1 -- 172.21.15.67:0/3212473601 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768011c20 con 0x7ff7800c4ec0 2024-04-25T15:51:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78014ebe0 con 0x7ff7800c4ec0 2024-04-25T15:51:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff780149e80 con 0x7ff7800c4ec0 2024-04-25T15:51:10.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77cff9640 1 -- 172.21.15.67:0/3212473601 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff768018020 con 0x7ff7800c4ec0 2024-04-25T15:51:10.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77cff9640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff76006f400 0x7ff760071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.840+0000 7ff77cff9640 1 -- 172.21.15.67:0/3212473601 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7ff76808f110 con 0x7ff7800c4ec0 2024-04-25T15:51:10.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff76006f400 0x7ff760071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 Processor -- start 2024-04-25T15:51:10.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- start start 2024-04-25T15:51:10.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff76006f400 0x7ff760071910 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff78015db50 tx=0x7ff770008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d27d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d248a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56388d0fa160 con 0x56388d118ea0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x56388d08fe20 con 0x56388cd11280 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x56388d048a20 con 0x56388d12cc20 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47410/0 (socket says 172.21.15.67:47410) 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 -- 172.21.15.67:0/3326452930 learned_addr learned my addr 172.21.15.67:0/3326452930 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff77ffff640 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d248a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d27d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 -- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 msgr2=0x56388d27d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d27d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 -- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 msgr2=0x56388d248a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d248a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 -- 172.21.15.67:0/3326452930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56388d04d100 con 0x56388d118ea0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff777fff640 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248550 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff76805b930 tx=0x7ff76805b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c828d9dac4b887 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff7757fa640 1 -- 172.21.15.67:0/3326452930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680640c0 con 0x56388d118ea0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff7757fa640 1 -- 172.21.15.67:0/3326452930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768064770 con 0x56388d118ea0 2024-04-25T15:51:10.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff7757fa640 1 -- 172.21.15.67:0/3326452930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680642e0 con 0x56388d118ea0 2024-04-25T15:51:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- 172.21.15.67:0/3326452930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 msgr2=0x56388d248550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248550 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff76805b930 tx=0x7ff76805b420 comp rx=0 tx=0).stop 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- 172.21.15.67:0/3326452930 shutdown_connections 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d248a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248550 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 --2- 172.21.15.67:0/3326452930 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d27d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- 172.21.15.67:0/3326452930 >> 172.21.15.67:0/3326452930 conn(0x56388cd50ff0 msgr2=0x56388d280030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- 172.21.15.67:0/3326452930 shutdown_connections 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.844+0000 7ff78d59a000 1 -- 172.21.15.67:0/3326452930 wait complete. 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 -- start start 2024-04-25T15:51:10.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 Processor -- start 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d2484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d2878f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56388d09d220 con 0x56388d118ea0 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x56388d033ea0 con 0x56388d12cc20 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x56388d08ab00 con 0x56388cd11280 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 10 client.0 _ll_register_callbacks cb 0x56388d09cdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47418/0 (socket says 172.21.15.67:47418) 2024-04-25T15:51:10.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 -- 172.21.15.67:0/1920751752 learned_addr learned my addr 172.21.15.67:0/1920751752 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d2484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff77ffff640 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d2878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 -- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 msgr2=0x56388d2484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d2484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 -- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 msgr2=0x56388d2878f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d2878f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 -- 172.21.15.67:0/1920751752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56388d02f500 con 0x56388d118ea0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff777fff640 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248a00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff76805b930 tx=0x7ff76800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5906e248c1ca2605 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff75e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff75e7fc640 1 -- 172.21.15.67:0/1920751752 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76808c050 con 0x56388d118ea0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff75e7fc640 1 -- 172.21.15.67:0/1920751752 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7ff7680642e0 con 0x56388d118ea0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff768064770 con 0x56388d118ea0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff75e7fc640 1 -- 172.21.15.67:0/1920751752 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ff768083050 con 0x56388d118ea0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 10 client.4797 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x56388d015ea0 con 0x56388d118ea0 2024-04-25T15:51:10.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff75e7fc640 1 -- 172.21.15.67:0/1920751752 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768064770 con 0x56388d118ea0 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff75e7fc640 1 -- 172.21.15.67:0/1920751752 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ff768086020 con 0x56388d118ea0 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 10 client.4797 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 10 client.4797 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 20 client.4797 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd357d0 0x56388d288630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 4 client.4797 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.848+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 --> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56388d028be0 con 0x56388cd357d0 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd357d0 0x56388d288630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd357d0 0x56388d288630 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58093bdbac4e427f server_cookie=296c0eebc4df805c in_seq=0 out_seq=0 2024-04-25T15:51:10.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff75e7fc640 10 client.4797 ms_handle_connect on v2:172.21.15.176:6837/1886671623 2024-04-25T15:51:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff75e7fc640 1 -- 172.21.15.67:0/1920751752 <== mds.0 v2:172.21.15.176:6837/1886671623 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56388d028be0 con 0x56388cd357d0 2024-04-25T15:51:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff75e7fc640 10 client.4797 handle_command_reply: tid=0 2024-04-25T15:51:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 client.4797 shutdown 2024-04-25T15:51:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 msgr2=0x56388d248a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248a00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff76805b930 tx=0x7ff76800e000 comp rx=0 tx=0).stop 2024-04-25T15:51:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 shutdown_connections 2024-04-25T15:51:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd357d0 0x56388d288630 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d2878f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff75e7fc640 0 client.4797 ms_handle_reset on v2:172.21.15.176:6837/1886671623 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d248a00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 --2- 172.21.15.67:0/1920751752 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d2484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 >> 172.21.15.67:0/1920751752 conn(0x56388cd50ff0 msgr2=0x56388d27f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 shutdown_connections 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 1 -- 172.21.15.67:0/1920751752 wait complete. 2024-04-25T15:51:10.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.852+0000 7ff78d59a000 20 client.4797 trim_cache size 0 max 16384 2024-04-25T15:51:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 Processor -- start 2024-04-25T15:51:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 -- start start 2024-04-25T15:51:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d29d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d29d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d280f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56388d08f130 con 0x56388d118ea0 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x56388d094450 con 0x56388d12cc20 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x56388d09c0c0 con 0x56388cd11280 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d29d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d29d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47424/0 (socket says 172.21.15.67:47424) 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 -- 172.21.15.67:0/2490691472 learned_addr learned my addr 172.21.15.67:0/2490691472 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 -- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 msgr2=0x56388d29d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d29d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff77ffff640 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d280f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d29d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 -- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 msgr2=0x56388d280f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d280f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.872+0000 7ff777fff640 1 -- 172.21.15.67:0/2490691472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56388d240120 con 0x56388d118ea0 2024-04-25T15:51:10.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d29d620 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff76805b6a0 tx=0x7ff768088070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f54ee9244a3b8b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff774ff9640 1 -- 172.21.15.67:0/2490691472 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76805b770 con 0x56388d118ea0 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff774ff9640 1 -- 172.21.15.67:0/2490691472 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768064770 con 0x56388d118ea0 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff774ff9640 1 -- 172.21.15.67:0/2490691472 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768064530 con 0x56388d118ea0 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- 172.21.15.67:0/2490691472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 msgr2=0x56388d29d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d29d620 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff76805b6a0 tx=0x7ff768088070 comp rx=0 tx=0).stop 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- 172.21.15.67:0/2490691472 shutdown_connections 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388d12cc20 0x56388d280f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d29d620 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2490691472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388cd11280 0x56388d29d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- 172.21.15.67:0/2490691472 >> 172.21.15.67:0/2490691472 conn(0x56388cd50ff0 msgr2=0x56388d2f1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- 172.21.15.67:0/2490691472 shutdown_connections 2024-04-25T15:51:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- 172.21.15.67:0/2490691472 wait complete. 2024-04-25T15:51:10.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- start start 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 Processor -- start 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d2f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d2fa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d2b9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56388d09d220 con 0x56388d118ea0 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x56388d06ca80 con 0x56388cd11280 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x56388d033ea0 con 0x56388d12cc20 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff78d59a000 10 client.0 _ll_register_callbacks cb 0x7ff76805b4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d2fa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d2fa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47428/0 (socket says 172.21.15.67:47428) 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 -- 172.21.15.67:0/2360721102 learned_addr learned my addr 172.21.15.67:0/2360721102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff77ffff640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d2b9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d2f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 -- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 msgr2=0x56388d2b9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d2b9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 -- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 msgr2=0x56388d2f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d2f9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff777fff640 1 -- 172.21.15.67:0/2360721102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56388d048c70 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff77ffff640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d2b9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.876+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d2f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff777fff640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d2fa210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff76808d6a0 tx=0x7ff768057910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce4ac333e3a3bce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 1 -- 172.21.15.67:0/2360721102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768064530 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 1 -- 172.21.15.67:0/2360721102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7ff7680642e0 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff768064770 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 1 -- 172.21.15.67:0/2360721102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ff76805ae10 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 10 client.4803 fetch_fsmap learned FSMap version 17 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x56388d112180 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 1 -- 172.21.15.67:0/2360721102 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768064770 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 1 -- 172.21.15.67:0/2360721102 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ff768086040 con 0x56388d118ea0 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 10 client.4803 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T15:51:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 10 client.4803 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 20 client.4803 populate_metadata read hostname 'smithi067' 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd23830 0x56388d2ba3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 4 client.4803 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 --> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] -- command(tid 0: {"prefix": "status"}) -- 0x56388d09da30 con 0x56388cd23830 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd23830 0x56388d2ba3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 10 client.4803 ms_handle_connect on v2:172.21.15.176:6837/1886671623 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff77f7fe640 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd23830 0x56388d2ba3c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e244282348413d58 server_cookie=30fb88155f590c42 in_seq=0 out_seq=0 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 1 -- 172.21.15.67:0/2360721102 <== mds.0 v2:172.21.15.176:6837/1886671623 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x56388d09da30 con 0x56388cd23830 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 10 client.4803 handle_command_reply: tid=0 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 client.4803 shutdown 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 msgr2=0x56388d2fa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d2fa210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff76808d6a0 tx=0x7ff768057910 comp rx=0 tx=0).stop 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 shutdown_connections 2024-04-25T15:51:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:6837/1886671623,v1:172.21.15.176:6839/1886671623] conn(0x56388cd23830 0x56388d2ba3c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x56388d12cc20 0x56388d2b9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56388d118ea0 0x56388d2fa210 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff75e7fc640 0 client.4803 ms_handle_reset on v2:172.21.15.176:6837/1886671623 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 --2- 172.21.15.67:0/2360721102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56388cd11280 0x56388d2f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 >> 172.21.15.67:0/2360721102 conn(0x56388cd50ff0 msgr2=0x56388d29cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 shutdown_connections 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 1 -- 172.21.15.67:0/2360721102 wait complete. 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78d59a000 20 client.4803 trim_cache size 0 max 16384 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4294, 2024-04-25T15:51:10.888 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 39.683049936000003, 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 17, 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 21, 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 20, 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 47.255271714000003 2024-04-25T15:51:10.889 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff76006f400 msgr2=0x7ff760071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff76006f400 0x7ff760071910 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff78015db50 tx=0x7ff770008040 comp rx=0 tx=0).stop 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 msgr2=0x7ff78014a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff78014a680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff76800b410 tx=0x7ff76800b8d0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 shutdown_connections 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff76006f400 0x7ff760071910 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff780159520 0x7ff78014e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7800c4ec0 0x7ff78014a680 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 --2- 172.21.15.67:0/3212473601 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff780073bd0 0x7ff78015a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 >> 172.21.15.67:0/3212473601 conn(0x7ff78006fe70 msgr2=0x7ff7801513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 shutdown_connections 2024-04-25T15:51:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:10.880+0000 7ff78623d640 1 -- 172.21.15.67:0/3212473601 wait complete. 2024-04-25T15:51:10.905 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T15:51:10.917 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T15:51:10.917 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T15:51:10.917 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T15:51:10.917 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-25T15:51:10.972 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-25T15:51:10.973 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-04-25T15:51:10.973 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-25T15:51:10.973 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T15:51:10.973 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:10.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:10.973 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:51:10.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:10.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-25T15:51:11.038 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-25T15:51:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:51:11.039 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-25T15:51:11.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T15:51:11.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32e60f6640 1 Processor -- start 2024-04-25T15:51:11.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32e60f6640 1 -- start start 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32e60f6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32e60f6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32e00b5660 0x7f32e00b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32e60f6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32e006d780 con 0x7f32e015fe00 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32e60f6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f32e015d510 con 0x7f32e00b5660 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32e60f6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f32e0160920 con 0x7f32e006ca60 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32deffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32e00b5660 0x7f32e00b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32df7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32deffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32e00b5660 0x7f32e00b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47436/0 (socket says 172.21.15.67:47436) 2024-04-25T15:51:11.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32deffd640 1 -- 172.21.15.67:0/1081229108 learned_addr learned my addr 172.21.15.67:0/1081229108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:11.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32deffd640 1 -- 172.21.15.67:0/1081229108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 msgr2=0x7f32e006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:11.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32deffd640 1 --2- 172.21.15.67:0/1081229108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e006ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32deffd640 1 -- 172.21.15.67:0/1081229108 >> v1:172.21.15.176:6789/0 conn(0x7f32e015fe00 legacy=0x7f32e0160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:11.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.244+0000 7f32deffd640 1 -- 172.21.15.67:0/1081229108 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e0160be0 con 0x7f32e00b5660 2024-04-25T15:51:11.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32deffd640 1 --2- 172.21.15.67:0/1081229108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e00b5660 0x7f32e00b5a70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009870 tx=0x7f32d001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31804683732a6296 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32de7fc640 1 -- 172.21.15.67:0/1081229108 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d00ab650 con 0x7f32e00b5660 2024-04-25T15:51:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32de7fc640 1 -- 172.21.15.67:0/1081229108 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d00b0070 con 0x7f32e00b5660 2024-04-25T15:51:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32de7fc640 1 -- 172.21.15.67:0/1081229108 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d00aba10 con 0x7f32e00b5660 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 -- 172.21.15.67:0/1081229108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e00b5660 msgr2=0x7f32e00b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 --2- 172.21.15.67:0/1081229108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e00b5660 0x7f32e00b5a70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009870 tx=0x7f32d001c560 comp rx=0 tx=0).stop 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 -- 172.21.15.67:0/1081229108 shutdown_connections 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 --2- 172.21.15.67:0/1081229108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e00b5660 0x7f32e00b5a70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 --2- 172.21.15.67:0/1081229108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e006ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 -- 172.21.15.67:0/1081229108 >> 172.21.15.67:0/1081229108 conn(0x7f32e006e420 msgr2=0x7f32e006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 -- 172.21.15.67:0/1081229108 shutdown_connections 2024-04-25T15:51:11.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.248+0000 7f32e60f6640 1 -- 172.21.15.67:0/1081229108 wait complete. 2024-04-25T15:51:11.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 Processor -- start 2024-04-25T15:51:11.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 -- start start 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e01f9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32e00b5660 0x7f32e01488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e015fe00 0x7f32e015ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f32e015c890 con 0x7f32e015fe00 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f32e00b5fb0 con 0x7f32e00b5660 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32e60f6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f32e00b6e90 con 0x7f32e006ca60 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e015fe00 0x7f32e015ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e015fe00 0x7f32e015ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47448/0 (socket says 172.21.15.67:47448) 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 -- 172.21.15.67:0/2588426122 learned_addr learned my addr 172.21.15.67:0/2588426122 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32df7fe640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e01f9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32deffd640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32e00b5660 0x7f32e01488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 -- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 msgr2=0x7f32e01f9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:11.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e01f9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 -- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32e00b5660 msgr2=0x7f32e01488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32e00b5660 0x7f32e01488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32dffff640 1 -- 172.21.15.67:0/2588426122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e0157d80 con 0x7f32e015fe00 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.252+0000 7f32df7fe640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e01f9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32dffff640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e015fe00 0x7f32e015ed60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c760 tx=0x7f32d800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32c7fff640 1 -- 172.21.15.67:0/2588426122 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8003ba0 con 0x7f32e015fe00 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32c7fff640 1 -- 172.21.15.67:0/2588426122 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d8003d60 con 0x7f32e015fe00 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e01555e0 con 0x7f32e015fe00 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32c7fff640 1 -- 172.21.15.67:0/2588426122 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d800aa90 con 0x7f32e015fe00 2024-04-25T15:51:11.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e0155ba0 con 0x7f32e015fe00 2024-04-25T15:51:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e006a480 con 0x7f32e015fe00 2024-04-25T15:51:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32c7fff640 1 -- 172.21.15.67:0/2588426122 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f32d800ac30 con 0x7f32e015fe00 2024-04-25T15:51:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32c7fff640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32cc06f400 0x7f32cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32c7fff640 1 -- 172.21.15.67:0/2588426122 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f32d808f0e0 con 0x7f32e015fe00 2024-04-25T15:51:11.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32df7fe640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32cc06f400 0x7f32cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:11.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.256+0000 7f32df7fe640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32cc06f400 0x7f32cc071910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f32e00cd350 tx=0x7f32d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:11.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.260+0000 7f32c7fff640 1 -- 172.21.15.67:0/2588426122 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f32d805bfe0 con 0x7f32e015fe00 2024-04-25T15:51:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.460+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f32e00ce8e0 con 0x7f32e015fe00 2024-04-25T15:51:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.460+0000 7f32c7fff640 1 -- 172.21.15.67:0/2588426122 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f32d8060e20 con 0x7f32e015fe00 2024-04-25T15:51:11.465 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T15:51:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32cc06f400 msgr2=0x7f32cc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32cc06f400 0x7f32cc071910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f32e00cd350 tx=0x7f32d4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e015fe00 msgr2=0x7f32e015ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e015fe00 0x7f32e015ed60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f32d800c760 tx=0x7f32d800cc20 comp rx=0 tx=0).stop 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 shutdown_connections 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f32cc06f400 0x7f32cc071910 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e015fe00 0x7f32e015ed60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32e00b5660 0x7f32e01488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 --2- 172.21.15.67:0/2588426122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32e006ca60 0x7f32e01f9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 >> 172.21.15.67:0/2588426122 conn(0x7f32e006e420 msgr2=0x7f32e0153160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 shutdown_connections 2024-04-25T15:51:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11.464+0000 7f32e60f6640 1 -- 172.21.15.67:0/2588426122 wait complete. 2024-04-25T15:51:11.487 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T15:51:11.487 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T15:51:11.487 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-25T15:51:11.488 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T15:51:11.488 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:11.488 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T15:51:11.488 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T15:51:11.488 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T15:51:11.488 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:51:11.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:11.488 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-25T15:51:11.494 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42488sec preferred_lft 42488sec 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:a5:8f:2f:cd brd ff:ff:ff:ff:ff:ff 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-25T15:51:11.495 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-25T15:51:11.496 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T15:51:11.496 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:51:11.496 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:51:11.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-04-25T15:51:11.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-04-25T15:51:11.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-04-25T15:51:11.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T15:51:11.496 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:51:11.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:51:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:51:11.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:11.624 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T15:51:11.633 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-25T15:51:11.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:11.635 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-25T15:51:11.703 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-25T15:51:11.703 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T15:51:11.703 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-25T15:51:11.703 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T15:51:11.703 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-25T15:51:11.703 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-25T15:51:11.703 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T15:51:11.704 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T15:51:11.704 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T15:51:11.704 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T15:51:11.704 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T15:51:11.704 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:51:11.704 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:51:11.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T15:51:11.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T15:51:11.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T15:51:11.704 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:51:11.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:51:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:51:11.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:11.831 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:51:11.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:11.837 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-25T15:51:11.886 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:51:11.886 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:51:11.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:51:11.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T15:51:11.887 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:51:11.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:51:11.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:51:12.000 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T15:51:12.000 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:51:12.000 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:51:12.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T15:51:12.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T15:51:12.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T15:51:12.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T15:51:12.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T15:51:12.001 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:51:12.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:51:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:51:12.152 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T15:51:12.152 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-25T15:51:12.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-25T15:51:12.152 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T15:51:12.152 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-25T15:51:12.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T15:51:12.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:51:12.211 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-25T15:51:12.211 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:51:12.211 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:12.258 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T15:51:12.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T15:51:12.258 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:12.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-25T15:51:12.363 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:51:12.411 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-25T15:51:12.411 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-25T15:51:12.411 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-25T15:51:12.411 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/run/lock 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/fs/nfsd 2024-04-25T15:51:12.412 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/run/rpc_pipefs 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:51:12.413 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:51:12.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:12.414 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-25T15:51:12.462 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T15:51:12.462 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-25T15:51:12.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-25T15:51:12.521 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:51:12.570 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-25T15:51:12.570 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/run/lock 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-25T15:51:12.571 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/fs/nfsd 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/run/rpc_pipefs 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:51:12.572 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:51:12.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:12.573 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-25T15:51:12.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:2024-04-25T15:51:12.620+0000 7f07ab1d8580 -1 init, newargv = 0x558e7e7e84b0 newargc=15 2024-04-25T15:51:12.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[35381]: starting ceph client 2024-04-25T15:51:12.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[35381]: starting fuse 2024-04-25T15:51:13.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-25T15:51:13.743 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T15:51:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-25T15:51:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-25T15:51:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-25T15:51:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-25T15:51:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-25T15:51:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/run/lock 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/fs/nfsd 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-25T15:51:13.792 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/run/rpc_pipefs 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:51:13.793 INFO:teuthology.orchestra.run.smithi067.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:13.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:13.794 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-25T15:51:13.842 INFO:teuthology.orchestra.run.smithi067.stdout:45 2024-04-25T15:51:13.842 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo stdin-killer -- python3 -c ' 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> import glob 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> import re 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> import subprocess 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> def _find_admin_socket(client_name): 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> files = glob.glob(asok_path) 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> # Given a non-glob path, it better be there 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> if "*" not in asok_path: 2024-04-25T15:51:13.843 DEBUG:teuthology.orchestra.run.smithi067:> assert(len(files) == 1) 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> return files[0] 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> for f in files: 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> contents = proc_f.read() 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> if mountpoint in contents: 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> return f 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> print(_find_admin_socket("client.0")) 2024-04-25T15:51:13.844 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-25T15:51:13.971 INFO:teuthology.orchestra.run.smithi067.stdout:/var/run/ceph/ceph-client.0.35381.asok 2024-04-25T15:51:13.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T15:51:13.983 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.35381.asok 2024-04-25T15:51:13.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:13.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.35381.asok status 2024-04-25T15:51:14.162 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": { 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "entity_id": "0", 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": "smithi067", 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "pid": "35381", 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "root": "/" 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_count": 0, 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_pinned_count": 0, 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4815, 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "inst": { 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "name": { 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "client", 2024-04-25T15:51:14.163 INFO:teuthology.orchestra.run.smithi067.stdout: "num": 4815 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2848005286 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "192.168.0.1:0", 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2848005286 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "inst_str": "client.4815 192.168.0.1:0/2848005286", 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr_str": "192.168.0.1:0/2848005286", 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "inode_count": 1, 2024-04-25T15:51:14.164 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_epoch": 17, 2024-04-25T15:51:14.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch": 22, 2024-04-25T15:51:14.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch_barrier": 0, 2024-04-25T15:51:14.165 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklisted": false, 2024-04-25T15:51:14.165 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs" 2024-04-25T15:51:14.165 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:14.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:14.179 DEBUG:teuthology.orchestra.run.smithi067:> stat --file-system '--printf=%T 2024-04-25T15:51:14.179 DEBUG:teuthology.orchestra.run.smithi067:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:14.186 INFO:teuthology.orchestra.run.smithi067.stdout:fuseblk 2024-04-25T15:51:14.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:14.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:51:14.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:14.246 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-25T15:51:14.258 INFO:tasks.thrashosds:config is {'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-25T15:51:14.258 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'but it is still running', 'objects unfound and apparently lost', 'MDS_SLOW_METADATA_IO', 'MDS_TRIM'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T15:51:14.258 INFO:tasks.thrashosds:config is {'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-25T15:51:14.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-25T15:51:14.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.448+0000 7fc17fbcd640 1 Processor -- start 2024-04-25T15:51:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17fbcd640 1 -- start start 2024-04-25T15:51:14.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17fbcd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17806ca60 0x7fc17806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17fbcd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17806d780 con 0x7fc1780b5660 2024-04-25T15:51:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17fbcd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc178160920 con 0x7fc17815fe00 2024-04-25T15:51:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17fbcd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc17815d510 con 0x7fc17806ca60 2024-04-25T15:51:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17d168640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1780b5660 0x7fc1780b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40010/0 (socket says 172.21.15.67:40010) 2024-04-25T15:51:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17d168640 1 -- 172.21.15.67:0/1358844659 learned_addr learned my addr 172.21.15.67:0/1358844659 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17d969640 1 --2- 172.21.15.67:0/1358844659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17806ca60 0x7fc17806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 411585052 0 0) 0x7fc17806d780 con 0x7fc1780b5660 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc164003530 con 0x7fc1780b5660 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953063417 0 0) 0x7fc178160920 con 0x7fc17815fe00 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc17806d780 con 0x7fc17815fe00 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3038490323 0 0) 0x7fc164003530 con 0x7fc1780b5660 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17d969640 1 -- 172.21.15.67:0/1358844659 >> v1:172.21.15.176:6789/0 conn(0x7fc17815fe00 legacy=0x7fc178160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17d969640 1 -- 172.21.15.67:0/1358844659 >> v1:172.21.15.67:6789/0 conn(0x7fc1780b5660 legacy=0x7fc1780b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.452+0000 7fc17d969640 1 -- 172.21.15.67:0/1358844659 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc178160be0 con 0x7fc17806ca60 2024-04-25T15:51:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17d969640 1 --2- 172.21.15.67:0/1358844659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17806ca60 0x7fc17806ce70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc1740097f0 tx=0x7fc17401ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d387033bbbb20ccf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740b5070 con 0x7fc17806ca60 2024-04-25T15:51:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1740a8670 con 0x7fc17806ca60 2024-04-25T15:51:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17c967640 1 -- 172.21.15.67:0/1358844659 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1680016a0 con 0x7fc17806ca60 2024-04-25T15:51:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17fbcd640 1 -- 172.21.15.67:0/1358844659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17806ca60 msgr2=0x7fc17806ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/1358844659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17806ca60 0x7fc17806ce70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc1740097f0 tx=0x7fc17401ebf0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17fbcd640 1 -- 172.21.15.67:0/1358844659 shutdown_connections 2024-04-25T15:51:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/1358844659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17806ca60 0x7fc17806ce70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17fbcd640 1 -- 172.21.15.67:0/1358844659 >> 172.21.15.67:0/1358844659 conn(0x7fc17806e420 msgr2=0x7fc17806e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17fbcd640 1 -- 172.21.15.67:0/1358844659 shutdown_connections 2024-04-25T15:51:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.456+0000 7fc17fbcd640 1 -- 172.21.15.67:0/1358844659 wait complete. 2024-04-25T15:51:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 Processor -- start 2024-04-25T15:51:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 -- start start 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc17806ca60 0x7fc17815f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1780b5660 0x7fc17814d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17815fe00 0x7fc17814d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc17815d1a0 con 0x7fc1780b5660 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc17815cec0 con 0x7fc17806ca60 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc17815c890 con 0x7fc17815fe00 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1780b5660 0x7fc17814d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1780b5660 0x7fc17814d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43536/0 (socket says 172.21.15.67:43536) 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 -- 172.21.15.67:0/235963552 learned_addr learned my addr 172.21.15.67:0/235963552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d969640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc17806ca60 0x7fc17815f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17e16a640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17815fe00 0x7fc17814d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 -- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17815fe00 msgr2=0x7fc17814d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17815fe00 0x7fc17814d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 -- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc17806ca60 msgr2=0x7fc17815f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc17806ca60 0x7fc17815f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 -- 172.21.15.67:0/235963552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc178158050 con 0x7fc1780b5660 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17d168640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1780b5660 0x7fc17814d3a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c730 tx=0x7fc16800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc16e7fc640 1 -- 172.21.15.67:0/235963552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168018070 con 0x7fc1780b5660 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc16e7fc640 1 -- 172.21.15.67:0/235963552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1680072d0 con 0x7fc1780b5660 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc16e7fc640 1 -- 172.21.15.67:0/235963552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168007680 con 0x7fc1780b5660 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1781f6980 con 0x7fc1780b5660 2024-04-25T15:51:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.460+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc178153860 con 0x7fc1780b5660 2024-04-25T15:51:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.464+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc178157dc0 con 0x7fc1780b5660 2024-04-25T15:51:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.464+0000 7fc16e7fc640 1 -- 172.21.15.67:0/235963552 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc16801b070 con 0x7fc1780b5660 2024-04-25T15:51:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.464+0000 7fc16e7fc640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc15c06f400 0x7fc15c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.464+0000 7fc16e7fc640 1 -- 172.21.15.67:0/235963552 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fc16808e8c0 con 0x7fc1780b5660 2024-04-25T15:51:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.464+0000 7fc17d969640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc15c06f400 0x7fc15c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.464+0000 7fc17d969640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc15c06f400 0x7fc15c071910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc1740b2000 tx=0x7fc1740aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.468+0000 7fc16e7fc640 1 -- 172.21.15.67:0/235963552 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc16805b730 con 0x7fc1780b5660 2024-04-25T15:51:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.632+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) -- 0x7fc17806a480 con 0x7fc1780b5660 2024-04-25T15:51:14.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.636+0000 7fc16e7fc640 1 -- 172.21.15.67:0/235963552 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v10)=0 v10) ==== 136+0+0 (secure 0 0 0) 0x7fc16805b030 con 0x7fc1780b5660 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc15c06f400 msgr2=0x7fc15c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc15c06f400 0x7fc15c071910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc1740b2000 tx=0x7fc1740aa070 comp rx=0 tx=0).stop 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1780b5660 msgr2=0x7fc17814d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1780b5660 0x7fc17814d3a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc16800c730 tx=0x7fc16800cbf0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 shutdown_connections 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc15c06f400 0x7fc15c071910 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc17815fe00 0x7fc17814d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1780b5660 0x7fc17814d3a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 --2- 172.21.15.67:0/235963552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc17806ca60 0x7fc17815f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 >> 172.21.15.67:0/235963552 conn(0x7fc17806e420 msgr2=0x7fc17815c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 shutdown_connections 2024-04-25T15:51:14.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.640+0000 7fc17fbcd640 1 -- 172.21.15.67:0/235963552 wait complete. 2024-04-25T15:51:14.663 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-25T15:51:14.663 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T15:51:14.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:51:14.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.808+0000 7feb963ac640 1 Processor -- start 2024-04-25T15:51:14.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.808+0000 7feb963ac640 1 -- start start 2024-04-25T15:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb963ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 0x7feb900c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb963ac640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb900c7340 con 0x7feb900c6140 2024-04-25T15:51:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb963ac640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb900c67c0 con 0x7feb90159520 2024-04-25T15:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb963ac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feb900c6aa0 con 0x7feb900c4760 2024-04-25T15:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feb900c6140 0x7feb900c4ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40020/0 (socket says 172.21.15.67:40020) 2024-04-25T15:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 -- 172.21.15.67:0/3556881907 learned_addr learned my addr 172.21.15.67:0/3556881907 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477880821 0 0) 0x7feb900c7340 con 0x7feb900c6140 2024-04-25T15:51:14.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb78003530 con 0x7feb900c6140 2024-04-25T15:51:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3994611246 0 0) 0x7feb78003530 con 0x7feb900c6140 2024-04-25T15:51:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 msgr2=0x7feb900c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8f7fe640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 0x7feb900c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 0x7feb900c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 >> v1:172.21.15.176:6789/0 conn(0x7feb90159520 legacy=0x7feb90073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb900c6d80 con 0x7feb900c6140 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7feb88001680 con 0x7feb900c6140 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8f7fe640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 0x7feb900c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 >> v1:172.21.15.67:6789/0 conn(0x7feb900c6140 legacy=0x7feb900c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb780039f0 0x7feb78003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 0x7feb78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb78004d50 0x7feb78005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb900c6aa0 con 0x7feb78004d50 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feb7800bbb0 con 0x7feb780039f0 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feb7800bdb0 con 0x7feb900c4760 2024-04-25T15:51:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb78004d50 0x7feb78005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8f7fe640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 0x7feb78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb8ffff640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb780039f0 0x7feb78003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 -- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 msgr2=0x7feb78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb900c4760 0x7feb78004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 -- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb780039f0 msgr2=0x7feb78003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb780039f0 0x7feb78003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.812+0000 7feb94949640 1 -- 172.21.15.67:0/3556881907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb78003530 con 0x7feb78004d50 2024-04-25T15:51:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb94949640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb78004d50 0x7feb78005220 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c210 tx=0x7feb8801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703847173ca70d00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88007d20 con 0x7feb78004d50 2024-04-25T15:51:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb88005ae0 con 0x7feb78004d50 2024-04-25T15:51:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb8effd640 1 -- 172.21.15.67:0/3556881907 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88007070 con 0x7feb78004d50 2024-04-25T15:51:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb963ac640 1 -- 172.21.15.67:0/3556881907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb78004d50 msgr2=0x7feb78005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb963ac640 1 --2- 172.21.15.67:0/3556881907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb78004d50 0x7feb78005220 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feb8800c210 tx=0x7feb8801f390 comp rx=0 tx=0).stop 2024-04-25T15:51:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb94949640 1 -- 172.21.15.67:0/3556881907 reap_dead start 2024-04-25T15:51:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb963ac640 1 -- 172.21.15.67:0/3556881907 shutdown_connections 2024-04-25T15:51:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb963ac640 1 -- 172.21.15.67:0/3556881907 >> 172.21.15.67:0/3556881907 conn(0x7feb9006fe70 msgr2=0x7feb900cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb963ac640 1 -- 172.21.15.67:0/3556881907 shutdown_connections 2024-04-25T15:51:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.816+0000 7feb963ac640 1 -- 172.21.15.67:0/3556881907 wait complete. 2024-04-25T15:51:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 Processor -- start 2024-04-25T15:51:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 -- start start 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb78004d50 0x7feb901ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9014d470 0x7feb901503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb9015bc40 0x7feb901f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb900c55d0 con 0x7feb9014d470 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feb900736b0 con 0x7feb78004d50 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feb900c5750 con 0x7feb9015bc40 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9014d470 0x7feb901503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9014d470 0x7feb901503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43564/0 (socket says 172.21.15.67:43564) 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 -- 172.21.15.67:0/2703110807 learned_addr learned my addr 172.21.15.67:0/2703110807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:14.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb94949640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb9015bc40 0x7feb901f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8ffff640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb78004d50 0x7feb901ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 -- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb9015bc40 msgr2=0x7feb901f18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb9015bc40 0x7feb901f18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 -- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb78004d50 msgr2=0x7feb901ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb78004d50 0x7feb901ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 -- 172.21.15.67:0/2703110807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb90072ab0 con 0x7feb9014d470 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8ffff640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb78004d50 0x7feb901ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8f7fe640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9014d470 0x7feb901503a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feb8400e870 tx=0x7feb8400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8cff9640 1 -- 172.21.15.67:0/2703110807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84019020 con 0x7feb9014d470 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8cff9640 1 -- 172.21.15.67:0/2703110807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb8400ee60 con 0x7feb9014d470 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9014ccb0 con 0x7feb9014d470 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8cff9640 1 -- 172.21.15.67:0/2703110807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8400ab60 con 0x7feb9014d470 2024-04-25T15:51:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb9015cca0 con 0x7feb9014d470 2024-04-25T15:51:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb9006a480 con 0x7feb9014d470 2024-04-25T15:51:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.820+0000 7feb8cff9640 1 -- 172.21.15.67:0/2703110807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7feb8401c020 con 0x7feb9014d470 2024-04-25T15:51:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.824+0000 7feb8cff9640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb7c06f3a0 0x7feb7c0718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.824+0000 7feb8cff9640 1 -- 172.21.15.67:0/2703110807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7feb8408fc20 con 0x7feb9014d470 2024-04-25T15:51:14.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.824+0000 7feb8ffff640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb7c06f3a0 0x7feb7c0718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:14.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.824+0000 7feb8ffff640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb7c06f3a0 0x7feb7c0718b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feb80007fd0 tx=0x7feb80007f00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:14.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.828+0000 7feb8cff9640 1 -- 172.21.15.67:0/2703110807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feb84023020 con 0x7feb9014d470 2024-04-25T15:51:14.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:14.988+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feb9015c750 con 0x7feb7c06f3a0 2024-04-25T15:51:15.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.008+0000 7feb8cff9640 1 -- 172.21.15.67:0/2703110807 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252030 (secure 0 0 0) 0x7feb9015c750 con 0x7feb7c06f3a0 2024-04-25T15:51:15.016 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:15.017 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:51:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.012+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb7c06f3a0 msgr2=0x7feb7c0718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.012+0000 7feb963ac640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb7c06f3a0 0x7feb7c0718b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feb80007fd0 tx=0x7feb80007f00 comp rx=0 tx=0).stop 2024-04-25T15:51:15.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.012+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9014d470 msgr2=0x7feb901503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.012+0000 7feb963ac640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9014d470 0x7feb901503a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feb8400e870 tx=0x7feb8400ed30 comp rx=0 tx=0).stop 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 shutdown_connections 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb7c06f3a0 0x7feb7c0718b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb9015bc40 0x7feb901f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9014d470 0x7feb901503a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 --2- 172.21.15.67:0/2703110807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb78004d50 0x7feb901ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 >> 172.21.15.67:0/2703110807 conn(0x7feb9006fe70 msgr2=0x7feb90157e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 shutdown_connections 2024-04-25T15:51:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.016+0000 7feb963ac640 1 -- 172.21.15.67:0/2703110807 wait complete. 2024-04-25T15:51:15.037 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-25T15:51:13.813979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600166,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":313,"num_write_kb":1697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1862109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3170,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003063"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142877+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:51:11.142877+0000","last_peered":"2024-04-25T15:51:11.142877+0000","last_clean":"2024-04-25T15:51:11.142877+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:51:11.142877+0000","last_undegraded":"2024-04-25T15:51:11.142877+0000","last_fullsized":"2024-04-25T15:51:11.142877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089664+0000","last_change":"2024-04-25T15:50:31.192224+0000","last_active":"2024-04-25T15:51:11.089664+0000","last_peered":"2024-04-25T15:51:11.089664+0000","last_clean":"2024-04-25T15:51:11.089664+0000","last_became_active":"2024-04-25T15:50:31.189311+0000","last_became_peered":"2024-04-25T15:50:31.189311+0000","last_unstale":"2024-04-25T15:51:11.089664+0000","last_undegraded":"2024-04-25T15:51:11.089664+0000","last_fullsized":"2024-04-25T15:51:11.089664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.837762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139537+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:51:11.139537+0000","last_peered":"2024-04-25T15:51:11.139537+0000","last_clean":"2024-04-25T15:51:11.139537+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:51:11.139537+0000","last_undegraded":"2024-04-25T15:51:11.139537+0000","last_fullsized":"2024-04-25T15:51:11.139537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141584+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:51:11.141584+0000","last_peered":"2024-04-25T15:51:11.141584+0000","last_clean":"2024-04-25T15:51:11.141584+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:51:11.141584+0000","last_undegraded":"2024-04-25T15:51:11.141584+0000","last_fullsized":"2024-04-25T15:51:11.141584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141535+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:51:11.141535+0000","last_peered":"2024-04-25T15:51:11.141535+0000","last_clean":"2024-04-25T15:51:11.141535+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:51:11.141535+0000","last_undegraded":"2024-04-25T15:51:11.141535+0000","last_fullsized":"2024-04-25T15:51:11.141535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140406+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:51:11.140406+0000","last_peered":"2024-04-25T15:51:11.140406+0000","last_clean":"2024-04-25T15:51:11.140406+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:51:11.140406+0000","last_undegraded":"2024-04-25T15:51:11.140406+0000","last_fullsized":"2024-04-25T15:51:11.140406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141119+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:51:11.141119+0000","last_peered":"2024-04-25T15:51:11.141119+0000","last_clean":"2024-04-25T15:51:11.141119+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:51:11.141119+0000","last_undegraded":"2024-04-25T15:51:11.141119+0000","last_fullsized":"2024-04-25T15:51:11.141119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.138746+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:51:11.138746+0000","last_peered":"2024-04-25T15:51:11.138746+0000","last_clean":"2024-04-25T15:51:11.138746+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:51:11.138746+0000","last_undegraded":"2024-04-25T15:51:11.138746+0000","last_fullsized":"2024-04-25T15:51:11.138746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.148148+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:51:11.148148+0000","last_peered":"2024-04-25T15:51:11.148148+0000","last_clean":"2024-04-25T15:51:11.148148+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:51:11.148148+0000","last_undegraded":"2024-04-25T15:51:11.148148+0000","last_fullsized":"2024-04-25T15:51:11.148148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052609+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:51:11.052609+0000","last_peered":"2024-04-25T15:51:11.052609+0000","last_clean":"2024-04-25T15:51:11.052609+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:51:11.052609+0000","last_undegraded":"2024-04-25T15:51:11.052609+0000","last_fullsized":"2024-04-25T15:51:11.052609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052265+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:51:11.052265+0000","last_peered":"2024-04-25T15:51:11.052265+0000","last_clean":"2024-04-25T15:51:11.052265+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:51:11.052265+0000","last_undegraded":"2024-04-25T15:51:11.052265+0000","last_fullsized":"2024-04-25T15:51:11.052265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989527+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:51:10.989527+0000","last_peered":"2024-04-25T15:51:10.989527+0000","last_clean":"2024-04-25T15:51:10.989527+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:51:10.989527+0000","last_undegraded":"2024-04-25T15:51:10.989527+0000","last_fullsized":"2024-04-25T15:51:10.989527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140240+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:51:11.140240+0000","last_peered":"2024-04-25T15:51:11.140240+0000","last_clean":"2024-04-25T15:51:11.140240+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:51:11.140240+0000","last_undegraded":"2024-04-25T15:51:11.140240+0000","last_fullsized":"2024-04-25T15:51:11.140240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139104+0000","last_change":"2024-04-25T15:50:31.171300+0000","last_active":"2024-04-25T15:51:11.139104+0000","last_peered":"2024-04-25T15:51:11.139104+0000","last_clean":"2024-04-25T15:51:11.139104+0000","last_became_active":"2024-04-25T15:50:31.170451+0000","last_became_peered":"2024-04-25T15:50:31.170451+0000","last_unstale":"2024-04-25T15:51:11.139104+0000","last_undegraded":"2024-04-25T15:51:11.139104+0000","last_fullsized":"2024-04-25T15:51:11.139104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:52:45.807418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089100+0000","last_change":"2024-04-25T15:50:31.198817+0000","last_active":"2024-04-25T15:51:11.089100+0000","last_peered":"2024-04-25T15:51:11.089100+0000","last_clean":"2024-04-25T15:51:11.089100+0000","last_became_active":"2024-04-25T15:50:31.198165+0000","last_became_peered":"2024-04-25T15:50:31.198165+0000","last_unstale":"2024-04-25T15:51:11.089100+0000","last_undegraded":"2024-04-25T15:51:11.089100+0000","last_fullsized":"2024-04-25T15:51:11.089100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:05.893280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142311+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:51:11.142311+0000","last_peered":"2024-04-25T15:51:11.142311+0000","last_clean":"2024-04-25T15:51:11.142311+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:51:11.142311+0000","last_undegraded":"2024-04-25T15:51:11.142311+0000","last_fullsized":"2024-04-25T15:51:11.142311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142432+0000","last_change":"2024-04-25T15:50:31.198225+0000","last_active":"2024-04-25T15:51:11.142432+0000","last_peered":"2024-04-25T15:51:11.142432+0000","last_clean":"2024-04-25T15:51:11.142432+0000","last_became_active":"2024-04-25T15:50:31.196737+0000","last_became_peered":"2024-04-25T15:50:31.196737+0000","last_unstale":"2024-04-25T15:51:11.142432+0000","last_undegraded":"2024-04-25T15:51:11.142432+0000","last_fullsized":"2024-04-25T15:51:11.142432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:25.652638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988412+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:51:10.988412+0000","last_peered":"2024-04-25T15:51:10.988412+0000","last_clean":"2024-04-25T15:51:10.988412+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:51:10.988412+0000","last_undegraded":"2024-04-25T15:51:10.988412+0000","last_fullsized":"2024-04-25T15:51:10.988412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143495+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:51:11.143495+0000","last_peered":"2024-04-25T15:51:11.143495+0000","last_clean":"2024-04-25T15:51:11.143495+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:51:11.143495+0000","last_undegraded":"2024-04-25T15:51:11.143495+0000","last_fullsized":"2024-04-25T15:51:11.143495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052605+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:51:11.052605+0000","last_peered":"2024-04-25T15:51:11.052605+0000","last_clean":"2024-04-25T15:51:11.052605+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:51:11.052605+0000","last_undegraded":"2024-04-25T15:51:11.052605+0000","last_fullsized":"2024-04-25T15:51:11.052605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139268+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:51:11.139268+0000","last_peered":"2024-04-25T15:51:11.139268+0000","last_clean":"2024-04-25T15:51:11.139268+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:51:11.139268+0000","last_undegraded":"2024-04-25T15:51:11.139268+0000","last_fullsized":"2024-04-25T15:51:11.139268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.148144+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:51:11.148144+0000","last_peered":"2024-04-25T15:51:11.148144+0000","last_clean":"2024-04-25T15:51:11.148144+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:51:11.148144+0000","last_undegraded":"2024-04-25T15:51:11.148144+0000","last_fullsized":"2024-04-25T15:51:11.148144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139249+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:51:11.139249+0000","last_peered":"2024-04-25T15:51:11.139249+0000","last_clean":"2024-04-25T15:51:11.139249+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:51:11.139249+0000","last_undegraded":"2024-04-25T15:51:11.139249+0000","last_fullsized":"2024-04-25T15:51:11.139249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142891+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:51:11.142891+0000","last_peered":"2024-04-25T15:51:11.142891+0000","last_clean":"2024-04-25T15:51:11.142891+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:51:11.142891+0000","last_undegraded":"2024-04-25T15:51:11.142891+0000","last_fullsized":"2024-04-25T15:51:11.142891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139582+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:51:11.139582+0000","last_peered":"2024-04-25T15:51:11.139582+0000","last_clean":"2024-04-25T15:51:11.139582+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:51:11.139582+0000","last_undegraded":"2024-04-25T15:51:11.139582+0000","last_fullsized":"2024-04-25T15:51:11.139582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052647+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:51:11.052647+0000","last_peered":"2024-04-25T15:51:11.052647+0000","last_clean":"2024-04-25T15:51:11.052647+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:51:11.052647+0000","last_undegraded":"2024-04-25T15:51:11.052647+0000","last_fullsized":"2024-04-25T15:51:11.052647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089526+0000","last_change":"2024-04-25T15:50:31.192245+0000","last_active":"2024-04-25T15:51:11.089526+0000","last_peered":"2024-04-25T15:51:11.089526+0000","last_clean":"2024-04-25T15:51:11.089526+0000","last_became_active":"2024-04-25T15:50:31.190989+0000","last_became_peered":"2024-04-25T15:50:31.190989+0000","last_unstale":"2024-04-25T15:51:11.089526+0000","last_undegraded":"2024-04-25T15:51:11.089526+0000","last_fullsized":"2024-04-25T15:51:11.089526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:29:24.813097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140052+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:51:11.140052+0000","last_peered":"2024-04-25T15:51:11.140052+0000","last_clean":"2024-04-25T15:51:11.140052+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:51:11.140052+0000","last_undegraded":"2024-04-25T15:51:11.140052+0000","last_fullsized":"2024-04-25T15:51:11.140052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139411+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:51:11.139411+0000","last_peered":"2024-04-25T15:51:11.139411+0000","last_clean":"2024-04-25T15:51:11.139411+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:51:11.139411+0000","last_undegraded":"2024-04-25T15:51:11.139411+0000","last_fullsized":"2024-04-25T15:51:11.139411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140929+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:51:11.140929+0000","last_peered":"2024-04-25T15:51:11.140929+0000","last_clean":"2024-04-25T15:51:11.140929+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:51:11.140929+0000","last_undegraded":"2024-04-25T15:51:11.140929+0000","last_fullsized":"2024-04-25T15:51:11.140929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.093454+0000","last_change":"2024-04-25T15:51:11.093454+0000","last_active":"2024-04-25T15:51:11.093454+0000","last_peered":"2024-04-25T15:51:11.093454+0000","last_clean":"2024-04-25T15:51:11.093454+0000","last_became_active":"2024-04-25T15:51:11.092850+0000","last_became_peered":"2024-04-25T15:51:11.092850+0000","last_unstale":"2024-04-25T15:51:11.093454+0000","last_undegraded":"2024-04-25T15:51:11.093454+0000","last_fullsized":"2024-04-25T15:51:11.093454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:39:57.419732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.051833+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:51:11.051833+0000","last_peered":"2024-04-25T15:51:11.051833+0000","last_clean":"2024-04-25T15:51:11.051833+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:51:11.051833+0000","last_undegraded":"2024-04-25T15:51:11.051833+0000","last_fullsized":"2024-04-25T15:51:11.051833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989657+0000","last_change":"2024-04-25T15:50:31.174593+0000","last_active":"2024-04-25T15:51:10.989657+0000","last_peered":"2024-04-25T15:51:10.989657+0000","last_clean":"2024-04-25T15:51:10.989657+0000","last_became_active":"2024-04-25T15:50:31.173352+0000","last_became_peered":"2024-04-25T15:50:31.173352+0000","last_unstale":"2024-04-25T15:51:10.989657+0000","last_undegraded":"2024-04-25T15:51:10.989657+0000","last_fullsized":"2024-04-25T15:51:10.989657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:45.575622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140616+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:51:11.140616+0000","last_peered":"2024-04-25T15:51:11.140616+0000","last_clean":"2024-04-25T15:51:11.140616+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:51:11.140616+0000","last_undegraded":"2024-04-25T15:51:11.140616+0000","last_fullsized":"2024-04-25T15:51:11.140616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139561+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:51:11.139561+0000","last_peered":"2024-04-25T15:51:11.139561+0000","last_clean":"2024-04-25T15:51:11.139561+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:51:11.139561+0000","last_undegraded":"2024-04-25T15:51:11.139561+0000","last_fullsized":"2024-04-25T15:51:11.139561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052020+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:51:11.052020+0000","last_peered":"2024-04-25T15:51:11.052020+0000","last_clean":"2024-04-25T15:51:11.052020+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:51:11.052020+0000","last_undegraded":"2024-04-25T15:51:11.052020+0000","last_fullsized":"2024-04-25T15:51:11.052020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139375+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:51:11.139375+0000","last_peered":"2024-04-25T15:51:11.139375+0000","last_clean":"2024-04-25T15:51:11.139375+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:51:11.139375+0000","last_undegraded":"2024-04-25T15:51:11.139375+0000","last_fullsized":"2024-04-25T15:51:11.139375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.138826+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:51:11.138826+0000","last_peered":"2024-04-25T15:51:11.138826+0000","last_clean":"2024-04-25T15:51:11.138826+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:51:11.138826+0000","last_undegraded":"2024-04-25T15:51:11.138826+0000","last_fullsized":"2024-04-25T15:51:11.138826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142610+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:51:11.142610+0000","last_peered":"2024-04-25T15:51:11.142610+0000","last_clean":"2024-04-25T15:51:11.142610+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:51:11.142610+0000","last_undegraded":"2024-04-25T15:51:11.142610+0000","last_fullsized":"2024-04-25T15:51:11.142610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089072+0000","last_change":"2024-04-25T15:50:31.182857+0000","last_active":"2024-04-25T15:51:11.089072+0000","last_peered":"2024-04-25T15:51:11.089072+0000","last_clean":"2024-04-25T15:51:11.089072+0000","last_became_active":"2024-04-25T15:50:31.181994+0000","last_became_peered":"2024-04-25T15:50:31.181994+0000","last_unstale":"2024-04-25T15:51:11.089072+0000","last_undegraded":"2024-04-25T15:51:11.089072+0000","last_fullsized":"2024-04-25T15:51:11.089072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:58.571421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.053010+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:51:11.053010+0000","last_peered":"2024-04-25T15:51:11.053010+0000","last_clean":"2024-04-25T15:51:11.053010+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:51:11.053010+0000","last_undegraded":"2024-04-25T15:51:11.053010+0000","last_fullsized":"2024-04-25T15:51:11.053010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141819+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:51:11.141819+0000","last_peered":"2024-04-25T15:51:11.141819+0000","last_clean":"2024-04-25T15:51:11.141819+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:51:11.141819+0000","last_undegraded":"2024-04-25T15:51:11.141819+0000","last_fullsized":"2024-04-25T15:51:11.141819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142173+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:51:11.142173+0000","last_peered":"2024-04-25T15:51:11.142173+0000","last_clean":"2024-04-25T15:51:11.142173+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:51:11.142173+0000","last_undegraded":"2024-04-25T15:51:11.142173+0000","last_fullsized":"2024-04-25T15:51:11.142173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140304+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:51:11.140304+0000","last_peered":"2024-04-25T15:51:11.140304+0000","last_clean":"2024-04-25T15:51:11.140304+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:51:11.140304+0000","last_undegraded":"2024-04-25T15:51:11.140304+0000","last_fullsized":"2024-04-25T15:51:11.140304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089534+0000","last_change":"2024-04-25T15:50:30.200105+0000","last_active":"2024-04-25T15:51:11.089534+0000","last_peered":"2024-04-25T15:51:11.089534+0000","last_clean":"2024-04-25T15:51:11.089534+0000","last_became_active":"2024-04-25T15:50:30.199328+0000","last_became_peered":"2024-04-25T15:50:30.199328+0000","last_unstale":"2024-04-25T15:51:11.089534+0000","last_undegraded":"2024-04-25T15:51:11.089534+0000","last_fullsized":"2024-04-25T15:51:11.089534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:59.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989842+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:51:10.989842+0000","last_peered":"2024-04-25T15:51:10.989842+0000","last_clean":"2024-04-25T15:51:10.989842+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:51:10.989842+0000","last_undegraded":"2024-04-25T15:51:10.989842+0000","last_fullsized":"2024-04-25T15:51:10.989842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'12","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:12.645236+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:51:12.645236+0000","last_peered":"2024-04-25T15:51:12.645236+0000","last_clean":"2024-04-25T15:51:12.645236+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:51:12.645236+0000","last_undegraded":"2024-04-25T15:51:12.645236+0000","last_fullsized":"2024-04-25T15:51:12.645236+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143075+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:51:11.143075+0000","last_peered":"2024-04-25T15:51:11.143075+0000","last_clean":"2024-04-25T15:51:11.143075+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:51:11.143075+0000","last_undegraded":"2024-04-25T15:51:11.143075+0000","last_fullsized":"2024-04-25T15:51:11.143075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139706+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:51:11.139706+0000","last_peered":"2024-04-25T15:51:11.139706+0000","last_clean":"2024-04-25T15:51:11.139706+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:51:11.139706+0000","last_undegraded":"2024-04-25T15:51:11.139706+0000","last_fullsized":"2024-04-25T15:51:11.139706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139253+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:51:11.139253+0000","last_peered":"2024-04-25T15:51:11.139253+0000","last_clean":"2024-04-25T15:51:11.139253+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:51:11.139253+0000","last_undegraded":"2024-04-25T15:51:11.139253+0000","last_fullsized":"2024-04-25T15:51:11.139253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141027+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:51:11.141027+0000","last_peered":"2024-04-25T15:51:11.141027+0000","last_clean":"2024-04-25T15:51:11.141027+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:51:11.141027+0000","last_undegraded":"2024-04-25T15:51:11.141027+0000","last_fullsized":"2024-04-25T15:51:11.141027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141837+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:51:11.141837+0000","last_peered":"2024-04-25T15:51:11.141837+0000","last_clean":"2024-04-25T15:51:11.141837+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:51:11.141837+0000","last_undegraded":"2024-04-25T15:51:11.141837+0000","last_fullsized":"2024-04-25T15:51:11.141837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052199+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:51:11.052199+0000","last_peered":"2024-04-25T15:51:11.052199+0000","last_clean":"2024-04-25T15:51:11.052199+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:51:11.052199+0000","last_undegraded":"2024-04-25T15:51:11.052199+0000","last_fullsized":"2024-04-25T15:51:11.052199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.051833+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:51:11.051833+0000","last_peered":"2024-04-25T15:51:11.051833+0000","last_clean":"2024-04-25T15:51:11.051833+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:51:11.051833+0000","last_undegraded":"2024-04-25T15:51:11.051833+0000","last_fullsized":"2024-04-25T15:51:11.051833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988473+0000","last_change":"2024-04-25T15:50:30.179053+0000","last_active":"2024-04-25T15:51:10.988473+0000","last_peered":"2024-04-25T15:51:10.988473+0000","last_clean":"2024-04-25T15:51:10.988473+0000","last_became_active":"2024-04-25T15:50:30.178196+0000","last_became_peered":"2024-04-25T15:50:30.178196+0000","last_unstale":"2024-04-25T15:51:10.988473+0000","last_undegraded":"2024-04-25T15:51:10.988473+0000","last_fullsized":"2024-04-25T15:51:10.988473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:26.635717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141420+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:51:11.141420+0000","last_peered":"2024-04-25T15:51:11.141420+0000","last_clean":"2024-04-25T15:51:11.141420+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:51:11.141420+0000","last_undegraded":"2024-04-25T15:51:11.141420+0000","last_fullsized":"2024-04-25T15:51:11.141420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.990191+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:51:10.990191+0000","last_peered":"2024-04-25T15:51:10.990191+0000","last_clean":"2024-04-25T15:51:10.990191+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:51:10.990191+0000","last_undegraded":"2024-04-25T15:51:10.990191+0000","last_fullsized":"2024-04-25T15:51:10.990191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142031+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:51:11.142031+0000","last_peered":"2024-04-25T15:51:11.142031+0000","last_clean":"2024-04-25T15:51:11.142031+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:51:11.142031+0000","last_undegraded":"2024-04-25T15:51:11.142031+0000","last_fullsized":"2024-04-25T15:51:11.142031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988667+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:51:10.988667+0000","last_peered":"2024-04-25T15:51:10.988667+0000","last_clean":"2024-04-25T15:51:10.988667+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:51:10.988667+0000","last_undegraded":"2024-04-25T15:51:10.988667+0000","last_fullsized":"2024-04-25T15:51:10.988667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143360+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:51:11.143360+0000","last_peered":"2024-04-25T15:51:11.143360+0000","last_clean":"2024-04-25T15:51:11.143360+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:51:11.143360+0000","last_undegraded":"2024-04-25T15:51:11.143360+0000","last_fullsized":"2024-04-25T15:51:11.143360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089645+0000","last_change":"2024-04-25T15:50:30.192595+0000","last_active":"2024-04-25T15:51:11.089645+0000","last_peered":"2024-04-25T15:51:11.089645+0000","last_clean":"2024-04-25T15:51:11.089645+0000","last_became_active":"2024-04-25T15:50:30.192044+0000","last_became_peered":"2024-04-25T15:50:30.192044+0000","last_unstale":"2024-04-25T15:51:11.089645+0000","last_undegraded":"2024-04-25T15:51:11.089645+0000","last_fullsized":"2024-04-25T15:51:11.089645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:08.837774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141897+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:51:11.141897+0000","last_peered":"2024-04-25T15:51:11.141897+0000","last_clean":"2024-04-25T15:51:11.141897+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:51:11.141897+0000","last_undegraded":"2024-04-25T15:51:11.141897+0000","last_fullsized":"2024-04-25T15:51:11.141897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988574+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:51:10.988574+0000","last_peered":"2024-04-25T15:51:10.988574+0000","last_clean":"2024-04-25T15:51:10.988574+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:51:10.988574+0000","last_undegraded":"2024-04-25T15:51:10.988574+0000","last_fullsized":"2024-04-25T15:51:10.988574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143231+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:51:11.143231+0000","last_peered":"2024-04-25T15:51:11.143231+0000","last_clean":"2024-04-25T15:51:11.143231+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:51:11.143231+0000","last_undegraded":"2024-04-25T15:51:11.143231+0000","last_fullsized":"2024-04-25T15:51:11.143231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":114,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140588+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:51:11.140588+0000","last_peered":"2024-04-25T15:51:11.140588+0000","last_clean":"2024-04-25T15:51:11.140588+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:51:11.140588+0000","last_undegraded":"2024-04-25T15:51:11.140588+0000","last_fullsized":"2024-04-25T15:51:11.140588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141013+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:51:11.141013+0000","last_peered":"2024-04-25T15:51:11.141013+0000","last_clean":"2024-04-25T15:51:11.141013+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:51:11.141013+0000","last_undegraded":"2024-04-25T15:51:11.141013+0000","last_fullsized":"2024-04-25T15:51:11.141013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139774+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:51:11.139774+0000","last_peered":"2024-04-25T15:51:11.139774+0000","last_clean":"2024-04-25T15:51:11.139774+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:51:11.139774+0000","last_undegraded":"2024-04-25T15:51:11.139774+0000","last_fullsized":"2024-04-25T15:51:11.139774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140515+0000","last_change":"2024-04-25T15:50:30.197359+0000","last_active":"2024-04-25T15:51:11.140515+0000","last_peered":"2024-04-25T15:51:11.140515+0000","last_clean":"2024-04-25T15:51:11.140515+0000","last_became_active":"2024-04-25T15:50:30.196640+0000","last_became_peered":"2024-04-25T15:50:30.196640+0000","last_unstale":"2024-04-25T15:51:11.140515+0000","last_undegraded":"2024-04-25T15:51:11.140515+0000","last_fullsized":"2024-04-25T15:51:11.140515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.928809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139234+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:51:11.139234+0000","last_peered":"2024-04-25T15:51:11.139234+0000","last_clean":"2024-04-25T15:51:11.139234+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:51:11.139234+0000","last_undegraded":"2024-04-25T15:51:11.139234+0000","last_fullsized":"2024-04-25T15:51:11.139234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988756+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:51:10.988756+0000","last_peered":"2024-04-25T15:51:10.988756+0000","last_clean":"2024-04-25T15:51:10.988756+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:51:10.988756+0000","last_undegraded":"2024-04-25T15:51:10.988756+0000","last_fullsized":"2024-04-25T15:51:10.988756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089113+0000","last_change":"2024-04-25T15:50:31.171339+0000","last_active":"2024-04-25T15:51:11.089113+0000","last_peered":"2024-04-25T15:51:11.089113+0000","last_clean":"2024-04-25T15:51:11.089113+0000","last_became_active":"2024-04-25T15:50:31.170349+0000","last_became_peered":"2024-04-25T15:50:31.170349+0000","last_unstale":"2024-04-25T15:51:11.089113+0000","last_undegraded":"2024-04-25T15:51:11.089113+0000","last_fullsized":"2024-04-25T15:51:11.089113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:24.647984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143940+0000","last_change":"2024-04-25T15:50:30.195563+0000","last_active":"2024-04-25T15:51:11.143940+0000","last_peered":"2024-04-25T15:51:11.143940+0000","last_clean":"2024-04-25T15:51:11.143940+0000","last_became_active":"2024-04-25T15:50:30.194468+0000","last_became_peered":"2024-04-25T15:50:30.194468+0000","last_unstale":"2024-04-25T15:51:11.143940+0000","last_undegraded":"2024-04-25T15:51:11.143940+0000","last_fullsized":"2024-04-25T15:51:11.143940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:04.850096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989877+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:51:10.989877+0000","last_peered":"2024-04-25T15:51:10.989877+0000","last_clean":"2024-04-25T15:51:10.989877+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:51:10.989877+0000","last_undegraded":"2024-04-25T15:51:10.989877+0000","last_fullsized":"2024-04-25T15:51:10.989877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089573+0000","last_change":"2024-04-25T15:50:30.175727+0000","last_active":"2024-04-25T15:51:11.089573+0000","last_peered":"2024-04-25T15:51:11.089573+0000","last_clean":"2024-04-25T15:51:11.089573+0000","last_became_active":"2024-04-25T15:50:30.175136+0000","last_became_peered":"2024-04-25T15:50:30.175136+0000","last_unstale":"2024-04-25T15:51:11.089573+0000","last_undegraded":"2024-04-25T15:51:11.089573+0000","last_fullsized":"2024-04-25T15:51:11.089573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:22.434474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142780+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:51:11.142780+0000","last_peered":"2024-04-25T15:51:11.142780+0000","last_clean":"2024-04-25T15:51:11.142780+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:51:11.142780+0000","last_undegraded":"2024-04-25T15:51:11.142780+0000","last_fullsized":"2024-04-25T15:51:11.142780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143643+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:51:11.143643+0000","last_peered":"2024-04-25T15:51:11.143643+0000","last_clean":"2024-04-25T15:51:11.143643+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:51:11.143643+0000","last_undegraded":"2024-04-25T15:51:11.143643+0000","last_fullsized":"2024-04-25T15:51:11.143643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089395+0000","last_change":"2024-04-25T15:50:31.177424+0000","last_active":"2024-04-25T15:51:11.089395+0000","last_peered":"2024-04-25T15:51:11.089395+0000","last_clean":"2024-04-25T15:51:11.089395+0000","last_became_active":"2024-04-25T15:50:31.176553+0000","last_became_peered":"2024-04-25T15:50:31.176553+0000","last_unstale":"2024-04-25T15:51:11.089395+0000","last_undegraded":"2024-04-25T15:51:11.089395+0000","last_fullsized":"2024-04-25T15:51:11.089395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:29.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089842+0000","last_change":"2024-04-25T15:50:30.191244+0000","last_active":"2024-04-25T15:51:11.089842+0000","last_peered":"2024-04-25T15:51:11.089842+0000","last_clean":"2024-04-25T15:51:11.089842+0000","last_became_active":"2024-04-25T15:50:30.190437+0000","last_became_peered":"2024-04-25T15:50:30.190437+0000","last_unstale":"2024-04-25T15:51:11.089842+0000","last_undegraded":"2024-04-25T15:51:11.089842+0000","last_fullsized":"2024-04-25T15:51:11.089842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:55.506915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139747+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:51:11.139747+0000","last_peered":"2024-04-25T15:51:11.139747+0000","last_clean":"2024-04-25T15:51:11.139747+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:51:11.139747+0000","last_undegraded":"2024-04-25T15:51:11.139747+0000","last_fullsized":"2024-04-25T15:51:11.139747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052518+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:51:11.052518+0000","last_peered":"2024-04-25T15:51:11.052518+0000","last_clean":"2024-04-25T15:51:11.052518+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:51:11.052518+0000","last_undegraded":"2024-04-25T15:51:11.052518+0000","last_fullsized":"2024-04-25T15:51:11.052518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139795+0000","last_change":"2024-04-25T15:50:31.182607+0000","last_active":"2024-04-25T15:51:11.139795+0000","last_peered":"2024-04-25T15:51:11.139795+0000","last_clean":"2024-04-25T15:51:11.139795+0000","last_became_active":"2024-04-25T15:50:31.181641+0000","last_became_peered":"2024-04-25T15:50:31.181641+0000","last_unstale":"2024-04-25T15:51:11.139795+0000","last_undegraded":"2024-04-25T15:51:11.139795+0000","last_fullsized":"2024-04-25T15:51:11.139795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:49.636675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140238+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:51:11.140238+0000","last_peered":"2024-04-25T15:51:11.140238+0000","last_clean":"2024-04-25T15:51:11.140238+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:51:11.140238+0000","last_undegraded":"2024-04-25T15:51:11.140238+0000","last_fullsized":"2024-04-25T15:51:11.140238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989245+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:51:10.989245+0000","last_peered":"2024-04-25T15:51:10.989245+0000","last_clean":"2024-04-25T15:51:10.989245+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:51:10.989245+0000","last_undegraded":"2024-04-25T15:51:10.989245+0000","last_fullsized":"2024-04-25T15:51:10.989245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089316+0000","last_change":"2024-04-25T15:50:30.178078+0000","last_active":"2024-04-25T15:51:11.089316+0000","last_peered":"2024-04-25T15:51:11.089316+0000","last_clean":"2024-04-25T15:51:11.089316+0000","last_became_active":"2024-04-25T15:50:30.177718+0000","last_became_peered":"2024-04-25T15:50:30.177718+0000","last_unstale":"2024-04-25T15:51:11.089316+0000","last_undegraded":"2024-04-25T15:51:11.089316+0000","last_fullsized":"2024-04-25T15:51:11.089316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.138966+0000","last_change":"2024-04-25T15:50:31.192152+0000","last_active":"2024-04-25T15:51:11.138966+0000","last_peered":"2024-04-25T15:51:11.138966+0000","last_clean":"2024-04-25T15:51:11.138966+0000","last_became_active":"2024-04-25T15:50:31.179200+0000","last_became_peered":"2024-04-25T15:50:31.179200+0000","last_unstale":"2024-04-25T15:51:11.138966+0000","last_undegraded":"2024-04-25T15:51:11.138966+0000","last_fullsized":"2024-04-25T15:51:11.138966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:03.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089502+0000","last_change":"2024-04-25T15:50:30.184989+0000","last_active":"2024-04-25T15:51:11.089502+0000","last_peered":"2024-04-25T15:51:11.089502+0000","last_clean":"2024-04-25T15:51:11.089502+0000","last_became_active":"2024-04-25T15:50:30.184132+0000","last_became_peered":"2024-04-25T15:50:30.184132+0000","last_unstale":"2024-04-25T15:51:11.089502+0000","last_undegraded":"2024-04-25T15:51:11.089502+0000","last_fullsized":"2024-04-25T15:51:11.089502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:03.833303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.144623+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:51:11.144623+0000","last_peered":"2024-04-25T15:51:11.144623+0000","last_clean":"2024-04-25T15:51:11.144623+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:51:11.144623+0000","last_undegraded":"2024-04-25T15:51:11.144623+0000","last_fullsized":"2024-04-25T15:51:11.144623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142332+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:51:11.142332+0000","last_peered":"2024-04-25T15:51:11.142332+0000","last_clean":"2024-04-25T15:51:11.142332+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:51:11.142332+0000","last_undegraded":"2024-04-25T15:51:11.142332+0000","last_fullsized":"2024-04-25T15:51:11.142332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.051906+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:51:11.051906+0000","last_peered":"2024-04-25T15:51:11.051906+0000","last_clean":"2024-04-25T15:51:11.051906+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:51:11.051906+0000","last_undegraded":"2024-04-25T15:51:11.051906+0000","last_fullsized":"2024-04-25T15:51:11.051906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142601+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:51:11.142601+0000","last_peered":"2024-04-25T15:51:11.142601+0000","last_clean":"2024-04-25T15:51:11.142601+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:51:11.142601+0000","last_undegraded":"2024-04-25T15:51:11.142601+0000","last_fullsized":"2024-04-25T15:51:11.142601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141667+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:51:11.141667+0000","last_peered":"2024-04-25T15:51:11.141667+0000","last_clean":"2024-04-25T15:51:11.141667+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:51:11.141667+0000","last_undegraded":"2024-04-25T15:51:11.141667+0000","last_fullsized":"2024-04-25T15:51:11.141667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141674+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:51:11.141674+0000","last_peered":"2024-04-25T15:51:11.141674+0000","last_clean":"2024-04-25T15:51:11.141674+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:51:11.141674+0000","last_undegraded":"2024-04-25T15:51:11.141674+0000","last_fullsized":"2024-04-25T15:51:11.141674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143392+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:51:11.143392+0000","last_peered":"2024-04-25T15:51:11.143392+0000","last_clean":"2024-04-25T15:51:11.143392+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:51:11.143392+0000","last_undegraded":"2024-04-25T15:51:11.143392+0000","last_fullsized":"2024-04-25T15:51:11.143392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140169+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:51:11.140169+0000","last_peered":"2024-04-25T15:51:11.140169+0000","last_clean":"2024-04-25T15:51:11.140169+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:51:11.140169+0000","last_undegraded":"2024-04-25T15:51:11.140169+0000","last_fullsized":"2024-04-25T15:51:11.140169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139815+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:51:11.139815+0000","last_peered":"2024-04-25T15:51:11.139815+0000","last_clean":"2024-04-25T15:51:11.139815+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:51:11.139815+0000","last_undegraded":"2024-04-25T15:51:11.139815+0000","last_fullsized":"2024-04-25T15:51:11.139815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139684+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:51:11.139684+0000","last_peered":"2024-04-25T15:51:11.139684+0000","last_clean":"2024-04-25T15:51:11.139684+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:51:11.139684+0000","last_undegraded":"2024-04-25T15:51:11.139684+0000","last_fullsized":"2024-04-25T15:51:11.139684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140317+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:51:11.140317+0000","last_peered":"2024-04-25T15:51:11.140317+0000","last_clean":"2024-04-25T15:51:11.140317+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:51:11.140317+0000","last_undegraded":"2024-04-25T15:51:11.140317+0000","last_fullsized":"2024-04-25T15:51:11.140317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9798,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":675576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":85510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":87481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":84438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":85544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":84570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":84640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":674350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:51:15.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:51:15.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2559afc640 1 Processor -- start 2024-04-25T15:51:15.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2559afc640 1 -- start start 2024-04-25T15:51:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2559afc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 0x7f255415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2559afc640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f255415e020 0x7f25540723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2559afc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2554072dc0 con 0x7f25540728e0 2024-04-25T15:51:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2559afc640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f255415b430 con 0x7f255415e020 2024-04-25T15:51:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2559afc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f255415a670 con 0x7f2554074790 2024-04-25T15:51:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2553fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f25540728e0 0x7f25540c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40032/0 (socket says 172.21.15.67:40032) 2024-04-25T15:51:15.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2553fff640 1 -- 172.21.15.67:0/797784857 learned_addr learned my addr 172.21.15.67:0/797784857 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:15.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f2552ffd640 1 --2- 172.21.15.67:0/797784857 >> v2:172.21.15.176:3300/0 conn(0x7f255415e020 0x7f25540723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25537fe640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 0x7f255415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1937492687 0 0) 0x7f2554072dc0 con 0x7f25540728e0 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f253c003530 con 0x7f25540728e0 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2302847537 0 0) 0x7f253c003530 con 0x7f25540728e0 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 msgr2=0x7f255415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 0x7f255415dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 >> v2:172.21.15.176:3300/0 conn(0x7f255415e020 msgr2=0x7f25540723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 --2- 172.21.15.67:0/797784857 >> v2:172.21.15.176:3300/0 conn(0x7f255415e020 0x7f25540723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255415a950 con 0x7f25540728e0 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f254c001a00 con 0x7f25540728e0 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 >> v1:172.21.15.67:6789/0 conn(0x7f25540728e0 legacy=0x7f25540c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f253c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 0x7f253c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253c004d50 0x7f253c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f255415b430 con 0x7f253c0039c0 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f255415a670 con 0x7f253c004d50 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f253c00ba50 con 0x7f2554074790 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.236+0000 7f25537fe640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f253c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2552ffd640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 0x7f253c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2553fff640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253c004d50 0x7f253c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25537fe640 1 -- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 msgr2=0x7f253c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25537fe640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 0x7f253c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25537fe640 1 -- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253c004d50 msgr2=0x7f253c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25537fe640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253c004d50 0x7f253c005160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25537fe640 1 -- 172.21.15.67:0/797784857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c003530 con 0x7f253c0039c0 2024-04-25T15:51:15.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2552ffd640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2554074790 0x7f253c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25537fe640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f253c003e30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2544009ab0 tx=0x7f254401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffe66d14e2845b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25440b2070 con 0x7f253c0039c0 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25440a5660 con 0x7f253c0039c0 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f25527fc640 1 -- 172.21.15.67:0/797784857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25440096b0 con 0x7f253c0039c0 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2559afc640 1 -- 172.21.15.67:0/797784857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 msgr2=0x7f253c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2559afc640 1 --2- 172.21.15.67:0/797784857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f253c003e30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2544009ab0 tx=0x7f254401ca70 comp rx=0 tx=0).stop 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2553fff640 1 -- 172.21.15.67:0/797784857 reap_dead start 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2559afc640 1 -- 172.21.15.67:0/797784857 shutdown_connections 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2559afc640 1 -- 172.21.15.67:0/797784857 >> 172.21.15.67:0/797784857 conn(0x7f255406fe70 msgr2=0x7f25540cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2559afc640 1 -- 172.21.15.67:0/797784857 shutdown_connections 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.240+0000 7f2559afc640 1 -- 172.21.15.67:0/797784857 wait complete. 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 Processor -- start 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 -- start start 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f25541f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25541f1020 0x7f25541524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2554156290 0x7f25541566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2554073080 con 0x7f253c0039c0 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f255415af10 con 0x7f2554156290 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2554074ba0 con 0x7f25541f1020 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f25541f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f25541f4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43580/0 (socket says 172.21.15.67:43580) 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 -- 172.21.15.67:0/3016340216 learned_addr learned my addr 172.21.15.67:0/3016340216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2552ffd640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25541f1020 0x7f25541524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2553fff640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2554156290 0x7f25541566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 -- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25541f1020 msgr2=0x7f25541524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25541f1020 0x7f25541524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 -- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2554156290 msgr2=0x7f25541566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2554156290 0x7f25541566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 -- 172.21.15.67:0/3016340216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25541574f0 con 0x7f253c0039c0 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2552ffd640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25541f1020 0x7f25541524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2553fff640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2554156290 0x7f25541566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f25537fe640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f25541f4920 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2544009ab0 tx=0x7f253c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f253bfff640 1 -- 172.21.15.67:0/3016340216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25440b2040 con 0x7f253c0039c0 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f253bfff640 1 -- 172.21.15.67:0/3016340216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25440a5e30 con 0x7f253c0039c0 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25541577d0 con 0x7f253c0039c0 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f253bfff640 1 -- 172.21.15.67:0/3016340216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544004020 con 0x7f253c0039c0 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2554157d90 con 0x7f253c0039c0 2024-04-25T15:51:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.244+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f255406a480 con 0x7f253c0039c0 2024-04-25T15:51:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.248+0000 7f253bfff640 1 -- 172.21.15.67:0/3016340216 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f25440c0030 con 0x7f253c0039c0 2024-04-25T15:51:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.248+0000 7f253bfff640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f254006f430 0x7f2540071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.248+0000 7f253bfff640 1 -- 172.21.15.67:0/3016340216 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f25441298f0 con 0x7f253c0039c0 2024-04-25T15:51:15.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.248+0000 7f2552ffd640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f254006f430 0x7f2540071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.248+0000 7f2552ffd640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f254006f430 0x7f2540071940 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f255414ddb0 tx=0x7f2548008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.252+0000 7f253bfff640 1 -- 172.21.15.67:0/3016340216 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f25440c7060 con 0x7f253c0039c0 2024-04-25T15:51:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.412+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2554157280 con 0x7f254006f430 2024-04-25T15:51:15.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.432+0000 7f253bfff640 1 -- 172.21.15.67:0/3016340216 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252030 (secure 0 0 0) 0x7f2554157280 con 0x7f254006f430 2024-04-25T15:51:15.437 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:15.439 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f254006f430 msgr2=0x7f2540071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f254006f430 0x7f2540071940 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f255414ddb0 tx=0x7f2548008040 comp rx=0 tx=0).stop 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 msgr2=0x7f25541f4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f25541f4920 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2544009ab0 tx=0x7f253c005840 comp rx=0 tx=0).stop 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 shutdown_connections 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f254006f430 0x7f2540071940 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2554156290 0x7f25541566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25541f1020 0x7f25541524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 --2- 172.21.15.67:0/3016340216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f253c0039c0 0x7f25541f4920 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 >> 172.21.15.67:0/3016340216 conn(0x7f255406fe70 msgr2=0x7f255414e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:15.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 shutdown_connections 2024-04-25T15:51:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.436+0000 7f2559afc640 1 -- 172.21.15.67:0/3016340216 wait complete. 2024-04-25T15:51:15.458 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-04-25T15:51:13.813979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600166,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":313,"num_write_kb":1697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":258,"ondisk_log_size":258,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1862109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3170,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003063"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142877+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:51:11.142877+0000","last_peered":"2024-04-25T15:51:11.142877+0000","last_clean":"2024-04-25T15:51:11.142877+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:51:11.142877+0000","last_undegraded":"2024-04-25T15:51:11.142877+0000","last_fullsized":"2024-04-25T15:51:11.142877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089664+0000","last_change":"2024-04-25T15:50:31.192224+0000","last_active":"2024-04-25T15:51:11.089664+0000","last_peered":"2024-04-25T15:51:11.089664+0000","last_clean":"2024-04-25T15:51:11.089664+0000","last_became_active":"2024-04-25T15:50:31.189311+0000","last_became_peered":"2024-04-25T15:50:31.189311+0000","last_unstale":"2024-04-25T15:51:11.089664+0000","last_undegraded":"2024-04-25T15:51:11.089664+0000","last_fullsized":"2024-04-25T15:51:11.089664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:13.837762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139537+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:51:11.139537+0000","last_peered":"2024-04-25T15:51:11.139537+0000","last_clean":"2024-04-25T15:51:11.139537+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:51:11.139537+0000","last_undegraded":"2024-04-25T15:51:11.139537+0000","last_fullsized":"2024-04-25T15:51:11.139537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141584+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:51:11.141584+0000","last_peered":"2024-04-25T15:51:11.141584+0000","last_clean":"2024-04-25T15:51:11.141584+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:51:11.141584+0000","last_undegraded":"2024-04-25T15:51:11.141584+0000","last_fullsized":"2024-04-25T15:51:11.141584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141535+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:51:11.141535+0000","last_peered":"2024-04-25T15:51:11.141535+0000","last_clean":"2024-04-25T15:51:11.141535+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:51:11.141535+0000","last_undegraded":"2024-04-25T15:51:11.141535+0000","last_fullsized":"2024-04-25T15:51:11.141535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140406+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:51:11.140406+0000","last_peered":"2024-04-25T15:51:11.140406+0000","last_clean":"2024-04-25T15:51:11.140406+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:51:11.140406+0000","last_undegraded":"2024-04-25T15:51:11.140406+0000","last_fullsized":"2024-04-25T15:51:11.140406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141119+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:51:11.141119+0000","last_peered":"2024-04-25T15:51:11.141119+0000","last_clean":"2024-04-25T15:51:11.141119+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:51:11.141119+0000","last_undegraded":"2024-04-25T15:51:11.141119+0000","last_fullsized":"2024-04-25T15:51:11.141119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.138746+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:51:11.138746+0000","last_peered":"2024-04-25T15:51:11.138746+0000","last_clean":"2024-04-25T15:51:11.138746+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:51:11.138746+0000","last_undegraded":"2024-04-25T15:51:11.138746+0000","last_fullsized":"2024-04-25T15:51:11.138746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.148148+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:51:11.148148+0000","last_peered":"2024-04-25T15:51:11.148148+0000","last_clean":"2024-04-25T15:51:11.148148+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:51:11.148148+0000","last_undegraded":"2024-04-25T15:51:11.148148+0000","last_fullsized":"2024-04-25T15:51:11.148148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052609+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:51:11.052609+0000","last_peered":"2024-04-25T15:51:11.052609+0000","last_clean":"2024-04-25T15:51:11.052609+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:51:11.052609+0000","last_undegraded":"2024-04-25T15:51:11.052609+0000","last_fullsized":"2024-04-25T15:51:11.052609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052265+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:51:11.052265+0000","last_peered":"2024-04-25T15:51:11.052265+0000","last_clean":"2024-04-25T15:51:11.052265+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:51:11.052265+0000","last_undegraded":"2024-04-25T15:51:11.052265+0000","last_fullsized":"2024-04-25T15:51:11.052265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989527+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:51:10.989527+0000","last_peered":"2024-04-25T15:51:10.989527+0000","last_clean":"2024-04-25T15:51:10.989527+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:51:10.989527+0000","last_undegraded":"2024-04-25T15:51:10.989527+0000","last_fullsized":"2024-04-25T15:51:10.989527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140240+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:51:11.140240+0000","last_peered":"2024-04-25T15:51:11.140240+0000","last_clean":"2024-04-25T15:51:11.140240+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:51:11.140240+0000","last_undegraded":"2024-04-25T15:51:11.140240+0000","last_fullsized":"2024-04-25T15:51:11.140240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139104+0000","last_change":"2024-04-25T15:50:31.171300+0000","last_active":"2024-04-25T15:51:11.139104+0000","last_peered":"2024-04-25T15:51:11.139104+0000","last_clean":"2024-04-25T15:51:11.139104+0000","last_became_active":"2024-04-25T15:50:31.170451+0000","last_became_peered":"2024-04-25T15:50:31.170451+0000","last_unstale":"2024-04-25T15:51:11.139104+0000","last_undegraded":"2024-04-25T15:51:11.139104+0000","last_fullsized":"2024-04-25T15:51:11.139104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:52:45.807418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089100+0000","last_change":"2024-04-25T15:50:31.198817+0000","last_active":"2024-04-25T15:51:11.089100+0000","last_peered":"2024-04-25T15:51:11.089100+0000","last_clean":"2024-04-25T15:51:11.089100+0000","last_became_active":"2024-04-25T15:50:31.198165+0000","last_became_peered":"2024-04-25T15:50:31.198165+0000","last_unstale":"2024-04-25T15:51:11.089100+0000","last_undegraded":"2024-04-25T15:51:11.089100+0000","last_fullsized":"2024-04-25T15:51:11.089100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:05.893280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142311+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:51:11.142311+0000","last_peered":"2024-04-25T15:51:11.142311+0000","last_clean":"2024-04-25T15:51:11.142311+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:51:11.142311+0000","last_undegraded":"2024-04-25T15:51:11.142311+0000","last_fullsized":"2024-04-25T15:51:11.142311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142432+0000","last_change":"2024-04-25T15:50:31.198225+0000","last_active":"2024-04-25T15:51:11.142432+0000","last_peered":"2024-04-25T15:51:11.142432+0000","last_clean":"2024-04-25T15:51:11.142432+0000","last_became_active":"2024-04-25T15:50:31.196737+0000","last_became_peered":"2024-04-25T15:50:31.196737+0000","last_unstale":"2024-04-25T15:51:11.142432+0000","last_undegraded":"2024-04-25T15:51:11.142432+0000","last_fullsized":"2024-04-25T15:51:11.142432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:25.652638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988412+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:51:10.988412+0000","last_peered":"2024-04-25T15:51:10.988412+0000","last_clean":"2024-04-25T15:51:10.988412+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:51:10.988412+0000","last_undegraded":"2024-04-25T15:51:10.988412+0000","last_fullsized":"2024-04-25T15:51:10.988412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143495+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:51:11.143495+0000","last_peered":"2024-04-25T15:51:11.143495+0000","last_clean":"2024-04-25T15:51:11.143495+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:51:11.143495+0000","last_undegraded":"2024-04-25T15:51:11.143495+0000","last_fullsized":"2024-04-25T15:51:11.143495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052605+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:51:11.052605+0000","last_peered":"2024-04-25T15:51:11.052605+0000","last_clean":"2024-04-25T15:51:11.052605+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:51:11.052605+0000","last_undegraded":"2024-04-25T15:51:11.052605+0000","last_fullsized":"2024-04-25T15:51:11.052605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139268+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:51:11.139268+0000","last_peered":"2024-04-25T15:51:11.139268+0000","last_clean":"2024-04-25T15:51:11.139268+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:51:11.139268+0000","last_undegraded":"2024-04-25T15:51:11.139268+0000","last_fullsized":"2024-04-25T15:51:11.139268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.148144+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:51:11.148144+0000","last_peered":"2024-04-25T15:51:11.148144+0000","last_clean":"2024-04-25T15:51:11.148144+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:51:11.148144+0000","last_undegraded":"2024-04-25T15:51:11.148144+0000","last_fullsized":"2024-04-25T15:51:11.148144+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139249+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:51:11.139249+0000","last_peered":"2024-04-25T15:51:11.139249+0000","last_clean":"2024-04-25T15:51:11.139249+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:51:11.139249+0000","last_undegraded":"2024-04-25T15:51:11.139249+0000","last_fullsized":"2024-04-25T15:51:11.139249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142891+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:51:11.142891+0000","last_peered":"2024-04-25T15:51:11.142891+0000","last_clean":"2024-04-25T15:51:11.142891+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:51:11.142891+0000","last_undegraded":"2024-04-25T15:51:11.142891+0000","last_fullsized":"2024-04-25T15:51:11.142891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139582+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:51:11.139582+0000","last_peered":"2024-04-25T15:51:11.139582+0000","last_clean":"2024-04-25T15:51:11.139582+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:51:11.139582+0000","last_undegraded":"2024-04-25T15:51:11.139582+0000","last_fullsized":"2024-04-25T15:51:11.139582+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052647+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:51:11.052647+0000","last_peered":"2024-04-25T15:51:11.052647+0000","last_clean":"2024-04-25T15:51:11.052647+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:51:11.052647+0000","last_undegraded":"2024-04-25T15:51:11.052647+0000","last_fullsized":"2024-04-25T15:51:11.052647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089526+0000","last_change":"2024-04-25T15:50:31.192245+0000","last_active":"2024-04-25T15:51:11.089526+0000","last_peered":"2024-04-25T15:51:11.089526+0000","last_clean":"2024-04-25T15:51:11.089526+0000","last_became_active":"2024-04-25T15:50:31.190989+0000","last_became_peered":"2024-04-25T15:50:31.190989+0000","last_unstale":"2024-04-25T15:51:11.089526+0000","last_undegraded":"2024-04-25T15:51:11.089526+0000","last_fullsized":"2024-04-25T15:51:11.089526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:29:24.813097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140052+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:51:11.140052+0000","last_peered":"2024-04-25T15:51:11.140052+0000","last_clean":"2024-04-25T15:51:11.140052+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:51:11.140052+0000","last_undegraded":"2024-04-25T15:51:11.140052+0000","last_fullsized":"2024-04-25T15:51:11.140052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139411+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:51:11.139411+0000","last_peered":"2024-04-25T15:51:11.139411+0000","last_clean":"2024-04-25T15:51:11.139411+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:51:11.139411+0000","last_undegraded":"2024-04-25T15:51:11.139411+0000","last_fullsized":"2024-04-25T15:51:11.139411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140929+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:51:11.140929+0000","last_peered":"2024-04-25T15:51:11.140929+0000","last_clean":"2024-04-25T15:51:11.140929+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:51:11.140929+0000","last_undegraded":"2024-04-25T15:51:11.140929+0000","last_fullsized":"2024-04-25T15:51:11.140929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.093454+0000","last_change":"2024-04-25T15:51:11.093454+0000","last_active":"2024-04-25T15:51:11.093454+0000","last_peered":"2024-04-25T15:51:11.093454+0000","last_clean":"2024-04-25T15:51:11.093454+0000","last_became_active":"2024-04-25T15:51:11.092850+0000","last_became_peered":"2024-04-25T15:51:11.092850+0000","last_unstale":"2024-04-25T15:51:11.093454+0000","last_undegraded":"2024-04-25T15:51:11.093454+0000","last_fullsized":"2024-04-25T15:51:11.093454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:39:57.419732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.051833+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:51:11.051833+0000","last_peered":"2024-04-25T15:51:11.051833+0000","last_clean":"2024-04-25T15:51:11.051833+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:51:11.051833+0000","last_undegraded":"2024-04-25T15:51:11.051833+0000","last_fullsized":"2024-04-25T15:51:11.051833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989657+0000","last_change":"2024-04-25T15:50:31.174593+0000","last_active":"2024-04-25T15:51:10.989657+0000","last_peered":"2024-04-25T15:51:10.989657+0000","last_clean":"2024-04-25T15:51:10.989657+0000","last_became_active":"2024-04-25T15:50:31.173352+0000","last_became_peered":"2024-04-25T15:50:31.173352+0000","last_unstale":"2024-04-25T15:51:10.989657+0000","last_undegraded":"2024-04-25T15:51:10.989657+0000","last_fullsized":"2024-04-25T15:51:10.989657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:57:45.575622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140616+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:51:11.140616+0000","last_peered":"2024-04-25T15:51:11.140616+0000","last_clean":"2024-04-25T15:51:11.140616+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:51:11.140616+0000","last_undegraded":"2024-04-25T15:51:11.140616+0000","last_fullsized":"2024-04-25T15:51:11.140616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139561+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:51:11.139561+0000","last_peered":"2024-04-25T15:51:11.139561+0000","last_clean":"2024-04-25T15:51:11.139561+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:51:11.139561+0000","last_undegraded":"2024-04-25T15:51:11.139561+0000","last_fullsized":"2024-04-25T15:51:11.139561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052020+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:51:11.052020+0000","last_peered":"2024-04-25T15:51:11.052020+0000","last_clean":"2024-04-25T15:51:11.052020+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:51:11.052020+0000","last_undegraded":"2024-04-25T15:51:11.052020+0000","last_fullsized":"2024-04-25T15:51:11.052020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139375+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:51:11.139375+0000","last_peered":"2024-04-25T15:51:11.139375+0000","last_clean":"2024-04-25T15:51:11.139375+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:51:11.139375+0000","last_undegraded":"2024-04-25T15:51:11.139375+0000","last_fullsized":"2024-04-25T15:51:11.139375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.138826+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:51:11.138826+0000","last_peered":"2024-04-25T15:51:11.138826+0000","last_clean":"2024-04-25T15:51:11.138826+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:51:11.138826+0000","last_undegraded":"2024-04-25T15:51:11.138826+0000","last_fullsized":"2024-04-25T15:51:11.138826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142610+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:51:11.142610+0000","last_peered":"2024-04-25T15:51:11.142610+0000","last_clean":"2024-04-25T15:51:11.142610+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:51:11.142610+0000","last_undegraded":"2024-04-25T15:51:11.142610+0000","last_fullsized":"2024-04-25T15:51:11.142610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089072+0000","last_change":"2024-04-25T15:50:31.182857+0000","last_active":"2024-04-25T15:51:11.089072+0000","last_peered":"2024-04-25T15:51:11.089072+0000","last_clean":"2024-04-25T15:51:11.089072+0000","last_became_active":"2024-04-25T15:50:31.181994+0000","last_became_peered":"2024-04-25T15:50:31.181994+0000","last_unstale":"2024-04-25T15:51:11.089072+0000","last_undegraded":"2024-04-25T15:51:11.089072+0000","last_fullsized":"2024-04-25T15:51:11.089072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:58.571421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.053010+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:51:11.053010+0000","last_peered":"2024-04-25T15:51:11.053010+0000","last_clean":"2024-04-25T15:51:11.053010+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:51:11.053010+0000","last_undegraded":"2024-04-25T15:51:11.053010+0000","last_fullsized":"2024-04-25T15:51:11.053010+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141819+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:51:11.141819+0000","last_peered":"2024-04-25T15:51:11.141819+0000","last_clean":"2024-04-25T15:51:11.141819+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:51:11.141819+0000","last_undegraded":"2024-04-25T15:51:11.141819+0000","last_fullsized":"2024-04-25T15:51:11.141819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142173+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:51:11.142173+0000","last_peered":"2024-04-25T15:51:11.142173+0000","last_clean":"2024-04-25T15:51:11.142173+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:51:11.142173+0000","last_undegraded":"2024-04-25T15:51:11.142173+0000","last_fullsized":"2024-04-25T15:51:11.142173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140304+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:51:11.140304+0000","last_peered":"2024-04-25T15:51:11.140304+0000","last_clean":"2024-04-25T15:51:11.140304+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:51:11.140304+0000","last_undegraded":"2024-04-25T15:51:11.140304+0000","last_fullsized":"2024-04-25T15:51:11.140304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089534+0000","last_change":"2024-04-25T15:50:30.200105+0000","last_active":"2024-04-25T15:51:11.089534+0000","last_peered":"2024-04-25T15:51:11.089534+0000","last_clean":"2024-04-25T15:51:11.089534+0000","last_became_active":"2024-04-25T15:50:30.199328+0000","last_became_peered":"2024-04-25T15:50:30.199328+0000","last_unstale":"2024-04-25T15:51:11.089534+0000","last_undegraded":"2024-04-25T15:51:11.089534+0000","last_fullsized":"2024-04-25T15:51:11.089534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:59.394090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989842+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:51:10.989842+0000","last_peered":"2024-04-25T15:51:10.989842+0000","last_clean":"2024-04-25T15:51:10.989842+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:51:10.989842+0000","last_undegraded":"2024-04-25T15:51:10.989842+0000","last_fullsized":"2024-04-25T15:51:10.989842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'12","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:12.645236+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:51:12.645236+0000","last_peered":"2024-04-25T15:51:12.645236+0000","last_clean":"2024-04-25T15:51:12.645236+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:51:12.645236+0000","last_undegraded":"2024-04-25T15:51:12.645236+0000","last_fullsized":"2024-04-25T15:51:12.645236+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143075+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:51:11.143075+0000","last_peered":"2024-04-25T15:51:11.143075+0000","last_clean":"2024-04-25T15:51:11.143075+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:51:11.143075+0000","last_undegraded":"2024-04-25T15:51:11.143075+0000","last_fullsized":"2024-04-25T15:51:11.143075+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139706+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:51:11.139706+0000","last_peered":"2024-04-25T15:51:11.139706+0000","last_clean":"2024-04-25T15:51:11.139706+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:51:11.139706+0000","last_undegraded":"2024-04-25T15:51:11.139706+0000","last_fullsized":"2024-04-25T15:51:11.139706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139253+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:51:11.139253+0000","last_peered":"2024-04-25T15:51:11.139253+0000","last_clean":"2024-04-25T15:51:11.139253+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:51:11.139253+0000","last_undegraded":"2024-04-25T15:51:11.139253+0000","last_fullsized":"2024-04-25T15:51:11.139253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141027+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:51:11.141027+0000","last_peered":"2024-04-25T15:51:11.141027+0000","last_clean":"2024-04-25T15:51:11.141027+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:51:11.141027+0000","last_undegraded":"2024-04-25T15:51:11.141027+0000","last_fullsized":"2024-04-25T15:51:11.141027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141837+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:51:11.141837+0000","last_peered":"2024-04-25T15:51:11.141837+0000","last_clean":"2024-04-25T15:51:11.141837+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:51:11.141837+0000","last_undegraded":"2024-04-25T15:51:11.141837+0000","last_fullsized":"2024-04-25T15:51:11.141837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052199+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:51:11.052199+0000","last_peered":"2024-04-25T15:51:11.052199+0000","last_clean":"2024-04-25T15:51:11.052199+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:51:11.052199+0000","last_undegraded":"2024-04-25T15:51:11.052199+0000","last_fullsized":"2024-04-25T15:51:11.052199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.051833+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:51:11.051833+0000","last_peered":"2024-04-25T15:51:11.051833+0000","last_clean":"2024-04-25T15:51:11.051833+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:51:11.051833+0000","last_undegraded":"2024-04-25T15:51:11.051833+0000","last_fullsized":"2024-04-25T15:51:11.051833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":35,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988473+0000","last_change":"2024-04-25T15:50:30.179053+0000","last_active":"2024-04-25T15:51:10.988473+0000","last_peered":"2024-04-25T15:51:10.988473+0000","last_clean":"2024-04-25T15:51:10.988473+0000","last_became_active":"2024-04-25T15:50:30.178196+0000","last_became_peered":"2024-04-25T15:50:30.178196+0000","last_unstale":"2024-04-25T15:51:10.988473+0000","last_undegraded":"2024-04-25T15:51:10.988473+0000","last_fullsized":"2024-04-25T15:51:10.988473+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:26.635717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141420+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:51:11.141420+0000","last_peered":"2024-04-25T15:51:11.141420+0000","last_clean":"2024-04-25T15:51:11.141420+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:51:11.141420+0000","last_undegraded":"2024-04-25T15:51:11.141420+0000","last_fullsized":"2024-04-25T15:51:11.141420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.990191+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:51:10.990191+0000","last_peered":"2024-04-25T15:51:10.990191+0000","last_clean":"2024-04-25T15:51:10.990191+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:51:10.990191+0000","last_undegraded":"2024-04-25T15:51:10.990191+0000","last_fullsized":"2024-04-25T15:51:10.990191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142031+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:51:11.142031+0000","last_peered":"2024-04-25T15:51:11.142031+0000","last_clean":"2024-04-25T15:51:11.142031+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:51:11.142031+0000","last_undegraded":"2024-04-25T15:51:11.142031+0000","last_fullsized":"2024-04-25T15:51:11.142031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988667+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:51:10.988667+0000","last_peered":"2024-04-25T15:51:10.988667+0000","last_clean":"2024-04-25T15:51:10.988667+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:51:10.988667+0000","last_undegraded":"2024-04-25T15:51:10.988667+0000","last_fullsized":"2024-04-25T15:51:10.988667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143360+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:51:11.143360+0000","last_peered":"2024-04-25T15:51:11.143360+0000","last_clean":"2024-04-25T15:51:11.143360+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:51:11.143360+0000","last_undegraded":"2024-04-25T15:51:11.143360+0000","last_fullsized":"2024-04-25T15:51:11.143360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089645+0000","last_change":"2024-04-25T15:50:30.192595+0000","last_active":"2024-04-25T15:51:11.089645+0000","last_peered":"2024-04-25T15:51:11.089645+0000","last_clean":"2024-04-25T15:51:11.089645+0000","last_became_active":"2024-04-25T15:50:30.192044+0000","last_became_peered":"2024-04-25T15:50:30.192044+0000","last_unstale":"2024-04-25T15:51:11.089645+0000","last_undegraded":"2024-04-25T15:51:11.089645+0000","last_fullsized":"2024-04-25T15:51:11.089645+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:08.837774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141897+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:51:11.141897+0000","last_peered":"2024-04-25T15:51:11.141897+0000","last_clean":"2024-04-25T15:51:11.141897+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:51:11.141897+0000","last_undegraded":"2024-04-25T15:51:11.141897+0000","last_fullsized":"2024-04-25T15:51:11.141897+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988574+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:51:10.988574+0000","last_peered":"2024-04-25T15:51:10.988574+0000","last_clean":"2024-04-25T15:51:10.988574+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:51:10.988574+0000","last_undegraded":"2024-04-25T15:51:10.988574+0000","last_fullsized":"2024-04-25T15:51:10.988574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143231+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:51:11.143231+0000","last_peered":"2024-04-25T15:51:11.143231+0000","last_clean":"2024-04-25T15:51:11.143231+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:51:11.143231+0000","last_undegraded":"2024-04-25T15:51:11.143231+0000","last_fullsized":"2024-04-25T15:51:11.143231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":114,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140588+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:51:11.140588+0000","last_peered":"2024-04-25T15:51:11.140588+0000","last_clean":"2024-04-25T15:51:11.140588+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:51:11.140588+0000","last_undegraded":"2024-04-25T15:51:11.140588+0000","last_fullsized":"2024-04-25T15:51:11.140588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141013+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:51:11.141013+0000","last_peered":"2024-04-25T15:51:11.141013+0000","last_clean":"2024-04-25T15:51:11.141013+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:51:11.141013+0000","last_undegraded":"2024-04-25T15:51:11.141013+0000","last_fullsized":"2024-04-25T15:51:11.141013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139774+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:51:11.139774+0000","last_peered":"2024-04-25T15:51:11.139774+0000","last_clean":"2024-04-25T15:51:11.139774+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:51:11.139774+0000","last_undegraded":"2024-04-25T15:51:11.139774+0000","last_fullsized":"2024-04-25T15:51:11.139774+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140515+0000","last_change":"2024-04-25T15:50:30.197359+0000","last_active":"2024-04-25T15:51:11.140515+0000","last_peered":"2024-04-25T15:51:11.140515+0000","last_clean":"2024-04-25T15:51:11.140515+0000","last_became_active":"2024-04-25T15:50:30.196640+0000","last_became_peered":"2024-04-25T15:50:30.196640+0000","last_unstale":"2024-04-25T15:51:11.140515+0000","last_undegraded":"2024-04-25T15:51:11.140515+0000","last_fullsized":"2024-04-25T15:51:11.140515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.928809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139234+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:51:11.139234+0000","last_peered":"2024-04-25T15:51:11.139234+0000","last_clean":"2024-04-25T15:51:11.139234+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:51:11.139234+0000","last_undegraded":"2024-04-25T15:51:11.139234+0000","last_fullsized":"2024-04-25T15:51:11.139234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.988756+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:51:10.988756+0000","last_peered":"2024-04-25T15:51:10.988756+0000","last_clean":"2024-04-25T15:51:10.988756+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:51:10.988756+0000","last_undegraded":"2024-04-25T15:51:10.988756+0000","last_fullsized":"2024-04-25T15:51:10.988756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089113+0000","last_change":"2024-04-25T15:50:31.171339+0000","last_active":"2024-04-25T15:51:11.089113+0000","last_peered":"2024-04-25T15:51:11.089113+0000","last_clean":"2024-04-25T15:51:11.089113+0000","last_became_active":"2024-04-25T15:50:31.170349+0000","last_became_peered":"2024-04-25T15:50:31.170349+0000","last_unstale":"2024-04-25T15:51:11.089113+0000","last_undegraded":"2024-04-25T15:51:11.089113+0000","last_fullsized":"2024-04-25T15:51:11.089113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:24.647984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143940+0000","last_change":"2024-04-25T15:50:30.195563+0000","last_active":"2024-04-25T15:51:11.143940+0000","last_peered":"2024-04-25T15:51:11.143940+0000","last_clean":"2024-04-25T15:51:11.143940+0000","last_became_active":"2024-04-25T15:50:30.194468+0000","last_became_peered":"2024-04-25T15:50:30.194468+0000","last_unstale":"2024-04-25T15:51:11.143940+0000","last_undegraded":"2024-04-25T15:51:11.143940+0000","last_fullsized":"2024-04-25T15:51:11.143940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:04.850096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989877+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:51:10.989877+0000","last_peered":"2024-04-25T15:51:10.989877+0000","last_clean":"2024-04-25T15:51:10.989877+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:51:10.989877+0000","last_undegraded":"2024-04-25T15:51:10.989877+0000","last_fullsized":"2024-04-25T15:51:10.989877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089573+0000","last_change":"2024-04-25T15:50:30.175727+0000","last_active":"2024-04-25T15:51:11.089573+0000","last_peered":"2024-04-25T15:51:11.089573+0000","last_clean":"2024-04-25T15:51:11.089573+0000","last_became_active":"2024-04-25T15:50:30.175136+0000","last_became_peered":"2024-04-25T15:50:30.175136+0000","last_unstale":"2024-04-25T15:51:11.089573+0000","last_undegraded":"2024-04-25T15:51:11.089573+0000","last_fullsized":"2024-04-25T15:51:11.089573+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:22.434474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142780+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:51:11.142780+0000","last_peered":"2024-04-25T15:51:11.142780+0000","last_clean":"2024-04-25T15:51:11.142780+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:51:11.142780+0000","last_undegraded":"2024-04-25T15:51:11.142780+0000","last_fullsized":"2024-04-25T15:51:11.142780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143643+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:51:11.143643+0000","last_peered":"2024-04-25T15:51:11.143643+0000","last_clean":"2024-04-25T15:51:11.143643+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:51:11.143643+0000","last_undegraded":"2024-04-25T15:51:11.143643+0000","last_fullsized":"2024-04-25T15:51:11.143643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089395+0000","last_change":"2024-04-25T15:50:31.177424+0000","last_active":"2024-04-25T15:51:11.089395+0000","last_peered":"2024-04-25T15:51:11.089395+0000","last_clean":"2024-04-25T15:51:11.089395+0000","last_became_active":"2024-04-25T15:50:31.176553+0000","last_became_peered":"2024-04-25T15:50:31.176553+0000","last_unstale":"2024-04-25T15:51:11.089395+0000","last_undegraded":"2024-04-25T15:51:11.089395+0000","last_fullsized":"2024-04-25T15:51:11.089395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:51:29.186783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089842+0000","last_change":"2024-04-25T15:50:30.191244+0000","last_active":"2024-04-25T15:51:11.089842+0000","last_peered":"2024-04-25T15:51:11.089842+0000","last_clean":"2024-04-25T15:51:11.089842+0000","last_became_active":"2024-04-25T15:50:30.190437+0000","last_became_peered":"2024-04-25T15:50:30.190437+0000","last_unstale":"2024-04-25T15:51:11.089842+0000","last_undegraded":"2024-04-25T15:51:11.089842+0000","last_fullsized":"2024-04-25T15:51:11.089842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:55.506915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139747+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:51:11.139747+0000","last_peered":"2024-04-25T15:51:11.139747+0000","last_clean":"2024-04-25T15:51:11.139747+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:51:11.139747+0000","last_undegraded":"2024-04-25T15:51:11.139747+0000","last_fullsized":"2024-04-25T15:51:11.139747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.052518+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:51:11.052518+0000","last_peered":"2024-04-25T15:51:11.052518+0000","last_clean":"2024-04-25T15:51:11.052518+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:51:11.052518+0000","last_undegraded":"2024-04-25T15:51:11.052518+0000","last_fullsized":"2024-04-25T15:51:11.052518+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139795+0000","last_change":"2024-04-25T15:50:31.182607+0000","last_active":"2024-04-25T15:51:11.139795+0000","last_peered":"2024-04-25T15:51:11.139795+0000","last_clean":"2024-04-25T15:51:11.139795+0000","last_became_active":"2024-04-25T15:50:31.181641+0000","last_became_peered":"2024-04-25T15:50:31.181641+0000","last_unstale":"2024-04-25T15:51:11.139795+0000","last_undegraded":"2024-04-25T15:51:11.139795+0000","last_fullsized":"2024-04-25T15:51:11.139795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:49.636675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140238+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:51:11.140238+0000","last_peered":"2024-04-25T15:51:11.140238+0000","last_clean":"2024-04-25T15:51:11.140238+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:51:11.140238+0000","last_undegraded":"2024-04-25T15:51:11.140238+0000","last_fullsized":"2024-04-25T15:51:11.140238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:10.989245+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:51:10.989245+0000","last_peered":"2024-04-25T15:51:10.989245+0000","last_clean":"2024-04-25T15:51:10.989245+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:51:10.989245+0000","last_undegraded":"2024-04-25T15:51:10.989245+0000","last_fullsized":"2024-04-25T15:51:10.989245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089316+0000","last_change":"2024-04-25T15:50:30.178078+0000","last_active":"2024-04-25T15:51:11.089316+0000","last_peered":"2024-04-25T15:51:11.089316+0000","last_clean":"2024-04-25T15:51:11.089316+0000","last_became_active":"2024-04-25T15:50:30.177718+0000","last_became_peered":"2024-04-25T15:50:30.177718+0000","last_unstale":"2024-04-25T15:51:11.089316+0000","last_undegraded":"2024-04-25T15:51:11.089316+0000","last_fullsized":"2024-04-25T15:51:11.089316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.138966+0000","last_change":"2024-04-25T15:50:31.192152+0000","last_active":"2024-04-25T15:51:11.138966+0000","last_peered":"2024-04-25T15:51:11.138966+0000","last_clean":"2024-04-25T15:51:11.138966+0000","last_became_active":"2024-04-25T15:50:31.179200+0000","last_became_peered":"2024-04-25T15:50:31.179200+0000","last_unstale":"2024-04-25T15:51:11.138966+0000","last_undegraded":"2024-04-25T15:51:11.138966+0000","last_fullsized":"2024-04-25T15:51:11.138966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:03.192874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.089502+0000","last_change":"2024-04-25T15:50:30.184989+0000","last_active":"2024-04-25T15:51:11.089502+0000","last_peered":"2024-04-25T15:51:11.089502+0000","last_clean":"2024-04-25T15:51:11.089502+0000","last_became_active":"2024-04-25T15:50:30.184132+0000","last_became_peered":"2024-04-25T15:50:30.184132+0000","last_unstale":"2024-04-25T15:51:11.089502+0000","last_undegraded":"2024-04-25T15:51:11.089502+0000","last_fullsized":"2024-04-25T15:51:11.089502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:03.833303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.144623+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:51:11.144623+0000","last_peered":"2024-04-25T15:51:11.144623+0000","last_clean":"2024-04-25T15:51:11.144623+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:51:11.144623+0000","last_undegraded":"2024-04-25T15:51:11.144623+0000","last_fullsized":"2024-04-25T15:51:11.144623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142332+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:51:11.142332+0000","last_peered":"2024-04-25T15:51:11.142332+0000","last_clean":"2024-04-25T15:51:11.142332+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:51:11.142332+0000","last_undegraded":"2024-04-25T15:51:11.142332+0000","last_fullsized":"2024-04-25T15:51:11.142332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.051906+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:51:11.051906+0000","last_peered":"2024-04-25T15:51:11.051906+0000","last_clean":"2024-04-25T15:51:11.051906+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:51:11.051906+0000","last_undegraded":"2024-04-25T15:51:11.051906+0000","last_fullsized":"2024-04-25T15:51:11.051906+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.142601+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:51:11.142601+0000","last_peered":"2024-04-25T15:51:11.142601+0000","last_clean":"2024-04-25T15:51:11.142601+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:51:11.142601+0000","last_undegraded":"2024-04-25T15:51:11.142601+0000","last_fullsized":"2024-04-25T15:51:11.142601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141667+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:51:11.141667+0000","last_peered":"2024-04-25T15:51:11.141667+0000","last_clean":"2024-04-25T15:51:11.141667+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:51:11.141667+0000","last_undegraded":"2024-04-25T15:51:11.141667+0000","last_fullsized":"2024-04-25T15:51:11.141667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.141674+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:51:11.141674+0000","last_peered":"2024-04-25T15:51:11.141674+0000","last_clean":"2024-04-25T15:51:11.141674+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:51:11.141674+0000","last_undegraded":"2024-04-25T15:51:11.141674+0000","last_fullsized":"2024-04-25T15:51:11.141674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":30,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.143392+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:51:11.143392+0000","last_peered":"2024-04-25T15:51:11.143392+0000","last_clean":"2024-04-25T15:51:11.143392+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:51:11.143392+0000","last_undegraded":"2024-04-25T15:51:11.143392+0000","last_fullsized":"2024-04-25T15:51:11.143392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140169+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:51:11.140169+0000","last_peered":"2024-04-25T15:51:11.140169+0000","last_clean":"2024-04-25T15:51:11.140169+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:51:11.140169+0000","last_undegraded":"2024-04-25T15:51:11.140169+0000","last_fullsized":"2024-04-25T15:51:11.140169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139815+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:51:11.139815+0000","last_peered":"2024-04-25T15:51:11.139815+0000","last_clean":"2024-04-25T15:51:11.139815+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:51:11.139815+0000","last_undegraded":"2024-04-25T15:51:11.139815+0000","last_fullsized":"2024-04-25T15:51:11.139815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.139684+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:51:11.139684+0000","last_peered":"2024-04-25T15:51:11.139684+0000","last_clean":"2024-04-25T15:51:11.139684+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:51:11.139684+0000","last_undegraded":"2024-04-25T15:51:11.139684+0000","last_fullsized":"2024-04-25T15:51:11.139684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":31,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-04-25T15:51:11.140317+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:51:11.140317+0000","last_peered":"2024-04-25T15:51:11.140317+0000","last_clean":"2024-04-25T15:51:11.140317+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:51:11.140317+0000","last_undegraded":"2024-04-25T15:51:11.140317+0000","last_fullsized":"2024-04-25T15:51:11.140317+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9798,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":675576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":85510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":87481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":84438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":85544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":84570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":84640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":851968,"data_stored":674350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:51:15.462 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T15:51:15.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-25T15:51:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.644+0000 7facbf400640 1 Processor -- start 2024-04-25T15:51:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.644+0000 7facbf400640 1 -- start start 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbf400640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7facb80c96e0 0x7facb8168190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbf400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb806d8e0 con 0x7facb8160800 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbf400640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb8073c70 con 0x7facb80c92d0 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbf400640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7facb806dbc0 con 0x7facb80c96e0 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7facb8160800 0x7facb806d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40038/0 (socket says 172.21.15.67:40038) 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 -- 172.21.15.67:0/1360390217 learned_addr learned my addr 172.21.15.67:0/1360390217 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd99d640 1 --2- 172.21.15.67:0/1360390217 >> v2:172.21.15.176:3300/0 conn(0x7facb80c96e0 0x7facb8168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041560458 0 0) 0x7facb806d8e0 con 0x7facb8160800 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faca4003530 con 0x7facb8160800 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2194109601 0 0) 0x7faca4003530 con 0x7facb8160800 2024-04-25T15:51:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 >> v2:172.21.15.176:3300/0 conn(0x7facb80c96e0 msgr2=0x7facb8168190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 --2- 172.21.15.67:0/1360390217 >> v2:172.21.15.176:3300/0 conn(0x7facb80c96e0 0x7facb8168190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 >> v1:172.21.15.176:6789/0 conn(0x7facb80c92d0 legacy=0x7facb8072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb8073f50 con 0x7facb8160800 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7faca8001680 con 0x7facb8160800 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 >> v1:172.21.15.67:6789/0 conn(0x7facb8160800 legacy=0x7facb806d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca40039c0 0x7faca4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca4004810 0x7faca4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca4005260 0x7faca400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7facb806dbc0 con 0x7faca40039c0 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faca400c1f0 con 0x7faca4005260 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faca400c3f0 con 0x7faca4004810 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca40039c0 0x7faca4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbc99b640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca4004810 0x7faca4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd99d640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca4005260 0x7faca400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 -- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca4004810 msgr2=0x7faca4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca4004810 0x7faca4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 -- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca4005260 msgr2=0x7faca400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faca4005260 0x7faca400b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd19c640 1 -- 172.21.15.67:0/1360390217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca4003530 con 0x7faca40039c0 2024-04-25T15:51:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.648+0000 7facbd99d640 1 -- 172.21.15.67:0/1360390217 reap_dead start 2024-04-25T15:51:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbd19c640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca40039c0 0x7faca4003e30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faca800c1c0 tx=0x7faca801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74693a36ff36b88d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8007d00 con 0x7faca40039c0 2024-04-25T15:51:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faca80a9020 con 0x7faca40039c0 2024-04-25T15:51:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facaffff640 1 -- 172.21.15.67:0/1360390217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8005ae0 con 0x7faca40039c0 2024-04-25T15:51:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 -- 172.21.15.67:0/1360390217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca40039c0 msgr2=0x7faca4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca40039c0 0x7faca4003e30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faca800c1c0 tx=0x7faca801cc20 comp rx=0 tx=0).stop 2024-04-25T15:51:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 -- 172.21.15.67:0/1360390217 shutdown_connections 2024-04-25T15:51:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 --2- 172.21.15.67:0/1360390217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca40039c0 0x7faca4003e30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 -- 172.21.15.67:0/1360390217 >> 172.21.15.67:0/1360390217 conn(0x7facb8070db0 msgr2=0x7facb80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 -- 172.21.15.67:0/1360390217 shutdown_connections 2024-04-25T15:51:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 -- 172.21.15.67:0/1360390217 wait complete. 2024-04-25T15:51:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.652+0000 7facbf400640 1 Processor -- start 2024-04-25T15:51:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 -- start start 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca40039c0 0x7facb80c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb81571f0 0x7facb81f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facb81504b0 0x7facb814f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7facb8160d90 con 0x7facb81571f0 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7facb8160c10 con 0x7facb81504b0 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7facb8072dc0 con 0x7faca40039c0 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb81571f0 0x7facb81f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb81571f0 0x7facb81f8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43598/0 (socket says 172.21.15.67:43598) 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 -- 172.21.15.67:0/3064047431 learned_addr learned my addr 172.21.15.67:0/3064047431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbd19c640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca40039c0 0x7facb80c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbd99d640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facb81504b0 0x7facb814f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 -- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca40039c0 msgr2=0x7facb80c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca40039c0 0x7facb80c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 -- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facb81504b0 msgr2=0x7facb814f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facb81504b0 0x7facb814f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 -- 172.21.15.67:0/3064047431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb80c7960 con 0x7facb81571f0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbd19c640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca40039c0 0x7facb80c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbc99b640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb81571f0 0x7facb81f8eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7facb400e4e0 tx=0x7facb400e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facadffb640 1 -- 172.21.15.67:0/3064047431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40052e0 con 0x7facb81571f0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facadffb640 1 -- 172.21.15.67:0/3064047431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb4018070 con 0x7facb81571f0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facadffb640 1 -- 172.21.15.67:0/3064047431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb4003840 con 0x7facb81571f0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb81f9880 con 0x7facb81571f0 2024-04-25T15:51:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb80c8470 con 0x7facb81571f0 2024-04-25T15:51:15.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb806a480 con 0x7facb81571f0 2024-04-25T15:51:15.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.656+0000 7facadffb640 1 -- 172.21.15.67:0/3064047431 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7facb40039e0 con 0x7facb81571f0 2024-04-25T15:51:15.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.660+0000 7facadffb640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac9c06f430 0x7fac9c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.660+0000 7facadffb640 1 -- 172.21.15.67:0/3064047431 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7facb408eef0 con 0x7facb81571f0 2024-04-25T15:51:15.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.660+0000 7facbd19c640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac9c06f430 0x7fac9c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.660+0000 7facbd19c640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac9c06f430 0x7fac9c071940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faca80aa000 tx=0x7facb806c740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.664+0000 7facadffb640 1 -- 172.21.15.67:0/3064047431 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7facb405bdf0 con 0x7facb81571f0 2024-04-25T15:51:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.820+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7facb8073920 con 0x7facb81571f0 2024-04-25T15:51:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facadffb640 1 -- 172.21.15.67:0/3064047431 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v22) ==== 56+0+3341 (secure 0 0 0) 0x7facb4005480 con 0x7facb81571f0 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 22 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:fsid fd5b4894-828e-4cb5-af55-b54a791f04c8 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-25T15:50:05.181255+0000 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-04-25T15:51:10.981720+0000 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:crush_version 4 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:full_ratio 0.95 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:backfillfull_ratio 0.9 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:nearfull_ratio 0.85 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:require_min_compat_client luminous 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:min_compat_client luminous 2024-04-25T15:51:15.826 INFO:teuthology.orchestra.run.smithi067.stdout:require_osd_release squid 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:stretch_mode_enabled false 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 19 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:pool 3 'cephfs_data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.38 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:max_osd 8 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] [v2:172.21.15.67:6802/851806635,v1:172.21.15.67:6803/851806635] exists,up 2ae04396-30a5-4c1d-86e5-63bc4f26f4c0 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] [v2:172.21.15.67:6814/2360705508,v1:172.21.15.67:6817/2360705508] exists,up efacae28-481c-42e9-9d26-5a019cb078b1 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 up in weight 1 up_from 13 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] [v2:172.21.15.67:6816/2903206440,v1:172.21.15.67:6819/2903206440] exists,up 60e87127-223e-4fd7-aefb-f00326f9deca 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] [v2:172.21.15.67:6815/1688455876,v1:172.21.15.67:6818/1688455876] exists,up 85ccf60b-0609-40a0-aae5-1e6a37eebdf2 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] [v2:172.21.15.176:6812/696101260,v1:172.21.15.176:6814/696101260] exists,up ce4919c0-d2fb-4210-b281-e843e84c94c1 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] [v2:172.21.15.176:6826/2964881609,v1:172.21.15.176:6827/2964881609] exists,up 47dcf3c9-5c23-4584-922c-b774345f2175 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] [v2:172.21.15.176:6813/211167612,v1:172.21.15.176:6815/211167612] exists,up 312d2166-f5ad-4cf3-8339-16a49bf48b19 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] [v2:172.21.15.176:6802/604707640,v1:172.21.15.176:6803/604707640] exists,up 922d3949-fd7b-4d1e-8b59-7311998bbc0b 2024-04-25T15:51:15.827 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.b [3,2] 2024-04-25T15:51:15.828 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.21 [5,1] 2024-04-25T15:51:15.828 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.2a [5,2] 2024-04-25T15:51:15.828 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.37 [7,2] 2024-04-25T15:51:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac9c06f430 msgr2=0x7fac9c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac9c06f430 0x7fac9c071940 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faca80aa000 tx=0x7facb806c740 comp rx=0 tx=0).stop 2024-04-25T15:51:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb81571f0 msgr2=0x7facb81f8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb81571f0 0x7facb81f8eb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7facb400e4e0 tx=0x7facb400e9a0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 shutdown_connections 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fac9c06f430 0x7fac9c071940 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7facb81504b0 0x7facb814f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facb81571f0 0x7facb81f8eb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 --2- 172.21.15.67:0/3064047431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faca40039c0 0x7facb80c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 >> 172.21.15.67:0/3064047431 conn(0x7facb8070db0 msgr2=0x7facb815fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 shutdown_connections 2024-04-25T15:51:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:15.824+0000 7facbf400640 1 -- 172.21.15.67:0/3064047431 wait complete. 2024-04-25T15:51:15.845 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] [v2:172.21.15.67:6802/851806635,v1:172.21.15.67:6803/851806635] exists,up 2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'osd.1 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] [v2:172.21.15.67:6814/2360705508,v1:172.21.15.67:6817/2360705508] exists,up efacae28-481c-42e9-9d26-5a019cb078b1', 'osd.2 up in weight 1 up_from 13 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] [v2:172.21.15.67:6816/2903206440,v1:172.21.15.67:6819/2903206440] exists,up 60e87127-223e-4fd7-aefb-f00326f9deca', 'osd.3 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] [v2:172.21.15.67:6815/1688455876,v1:172.21.15.67:6818/1688455876] exists,up 85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'osd.4 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] [v2:172.21.15.176:6812/696101260,v1:172.21.15.176:6814/696101260] exists,up ce4919c0-d2fb-4210-b281-e843e84c94c1', 'osd.5 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] [v2:172.21.15.176:6826/2964881609,v1:172.21.15.176:6827/2964881609] exists,up 47dcf3c9-5c23-4584-922c-b774345f2175', 'osd.6 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] [v2:172.21.15.176:6813/211167612,v1:172.21.15.176:6815/211167612] exists,up 312d2166-f5ad-4cf3-8339-16a49bf48b19', 'osd.7 up in weight 1 up_from 13 up_thru 18 down_at 0 last_clean_interval [0,0) [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] [v2:172.21.15.176:6802/604707640,v1:172.21.15.176:6803/604707640] exists,up 922d3949-fd7b-4d1e-8b59-7311998bbc0b'] 2024-04-25T15:51:15.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-25T15:51:15.987 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:15.987 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mon.a", 2024-04-25T15:51:15.987 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster": "ceph", 2024-04-25T15:51:15.987 INFO:teuthology.orchestra.run.smithi067.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "admin_socket_mode": "", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_client_required": "cephx, none", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_cluster_required": "cephx", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_debug": "true", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_service_required": "cephx", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_supported": "cephx", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio": "true", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_poll_ms": "250", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_reap_max": "16", 2024-04-25T15:51:15.988 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_async_discard_threads": "0", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_block_size": "4096", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio": "false", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_inflight_ios": "false", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_enable_discard": "false", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_flock_retry": "3", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_inject_crash": "0", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring": "false", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring_hipri": "false", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-25T15:51:15.989 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_type": "", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_alloc_size": "1048576", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_allocator": "hybrid", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_buffered_io": "true", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_for_zeros": "false", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_compact_log_sync": "false", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_max_log_runway": "4194304", 2024-04-25T15:51:15.990 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_max_prefetch": "1048576", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_min_flush_size": "524288", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_min_log_runway": "1048576", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_replay_recovery": "false", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_sync_write": "false", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_allocation_from_file": "true", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_allocator": "hybrid", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-25T15:51:15.991 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_create": "true", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_create": "false", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_path": "", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_size": "0", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_path": "", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_preallocate_file": "false", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_size": "107374182400", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_create": "false", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_path": "", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_size": "100663296", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs": "true", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-25T15:51:15.992 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_autotune": "true", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size": "0", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_type": "2q", 2024-04-25T15:51:15.993 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_clone_cow": "true", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_mode": "none", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_csum_type": "crc32c", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-25T15:51:15.994 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_freelist": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_small_allocations": "0", 2024-04-25T15:51:15.995 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_default_buffered_read": "true", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_default_buffered_write": "false", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fail_eio": "false", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-25T15:51:15.996 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mount": "false", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_umount": "false", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_ignore_data_csum": "false", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-25T15:51:15.997 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_op_age": "5.000000", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_alloc_size": "0", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size": "0", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_deferred_txc": "32", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size": "0", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_nid_prealloc": "1024", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-25T15:51:15.998 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_retry_disk_reads": "3", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_cf": "true", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_mem": "512", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-25T15:51:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_tracing": "false", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_zero_block_detection": "false", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-25T15:51:16.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_cluster_require_signatures": "false", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_cluster_require_version": "2", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_require_signatures": "false", 2024-04-25T15:51:16.001 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_require_version": "2", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_service_require_signatures": "false", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_service_require_version": "2", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_sign_messages": "true", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "chdir": "", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_acl_type": "", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_asio_thread_count": "2", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_cache_mid": "0.750000", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_cache_size": "16384", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_caps_release_delay": "5", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_check_pool_perm": "true", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_force_sync_read": "false", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_getattr_caps": "false", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_inject_tick_delay": "0", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_die_on_failed_remount": "false", 2024-04-25T15:51:16.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_dirsize_rbytes": "true", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_force_lazyio": "false", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_fs": "", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_inject_release_failure": "false", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_max_inline_size": "4096", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mds_namespace": "", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_metadata": "", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_gid": "-1", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_timeout": "300", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_uid": "-1", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mountpoint": "/", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_notify_timeout": "10", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc": "true", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_dirty": "104857600", 2024-04-25T15:51:16.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_objects": "1000", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_size": "209715200", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_target_dirty": "8388608", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_permissions": "true", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_quota": "true", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_quota_df": "true", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_max_bytes": "0", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_max_periods": "4", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_min": "131072", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_reconnect_stale": "false", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_shutdown_timeout": "30", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_snapdir": ".snap", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_tick_interval": "1", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_trace": "", 2024-04-25T15:51:16.004 INFO:teuthology.orchestra.run.smithi067.stdout: "client_try_dentry_invalidate": "false", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client_use_faked_inos": "false", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "client_use_random_mds": "false", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog": "false", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog_port": "12201", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_monitors": "default=true", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog": "false", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog_level": "info", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "-", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_network": "", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_network_interface": "", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_isal": "false", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_level": "5", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_winsize": "-15", 2024-04-25T15:51:16.005 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zstd_level": "1", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_alien_op_num_threads": "6", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_osd_stat_interval": "0", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_seastar_cpu_cores": "", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location": "", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location_hook": "", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location_hook_timeout": "10", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "daemonize": "false", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "daos_pool": "tank", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-25T15:51:16.006 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_alienstore": "0/5", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_allow_any_pool_priority": "false", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asok": "1/5", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asok_assert_abort": "false", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asserts_on_shutdown": "true", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_auth": "20/20", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bdev": "1/3", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bluefs": "1/5", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bluestore": "1/5", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_buffer": "0/1", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_ceph_dedup": "0/5", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_ceph_exporter": "1/5", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cephfs_mirror": "0/5", 2024-04-25T15:51:16.007 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cephsqlite": "0/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_client": "0/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_compressor": "1/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_context": "0/1", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crimson_interrupt": "0/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crush": "1/1", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crypto": "1/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cyanstore": "0/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_deliberately_leak_memory": "false", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_disable_randomized_ping": "false", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_dpdk": "1/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_eventtrace": "1/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_filer": "0/1", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_filestore": "1/3", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_finisher": "1/1", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_fuse": "1/5", 2024-04-25T15:51:16.008 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_heartbeat_testing_span": "0", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_heartbeatmap": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_javaclient": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_journal": "1/3", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_journaler": "0/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_kstore": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_lockdep": "0/1", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mclock": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_balancer": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_locker": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_log": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_log_expire": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_migrator": "1/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_quiesce": "3/5", 2024-04-25T15:51:16.009 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_memstore": "1/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mgr": "2/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mgrc": "1/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mon": "20/20", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_monc": "0/10", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_ms": "1/1", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_none": "0/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objclass": "0/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objectcacher": "0/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objecter": "0/1", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_optracker": "0/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_osd": "1/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_paxos": "20/20", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_perfcounter": "1/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_prioritycache": "1/5", 2024-04-25T15:51:16.010 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rados": "0/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd": "0/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_mirror": "0/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_pwl": "0/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_replay": "0/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_refs": "0/0", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_reserver": "1/1", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_access": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_datacache": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_dbstore": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_flight": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_notification": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_sync": "1/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rocksdb": "4/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore": "0/5", 2024-04-25T15:51:16.011 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_backref": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_cache": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_cleaner": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_device": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_epm": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_journal": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_lba": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_odata": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_omap": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_onode": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_t": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_tm": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_striper": "0/1", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_test": "0/5", 2024-04-25T15:51:16.012 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_throttle": "1/1", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_timer": "0/1", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_tp": "0/5", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_trace": "1/5", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "device_failure_prediction_mode": "none", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_graylog": "false", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_journald": "false", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_stderr": "true", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_syslog": "false", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event_tracing": "false", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_addr": "0.0.0.0", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_http_port": "9926", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_prio_limit": "5", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-25T15:51:16.013 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_sort_metrics": "true", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_stats_period": "5", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "fake_statfs_for_testing": "0", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "fatal_signal_handlers": "true", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filer_max_purge_ops": "10", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filer_max_truncate_ops": "128", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_apply_finisher_threads": "1", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_blackhole": "false", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_btrfs_clone_range": "true", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_btrfs_snap": "true", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_caller_concurrency": "10", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_collect_device_partition_information": "true", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_commit_timeout": "600.000000", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_inject_read_err": "false", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_omap_check": "false", 2024-04-25T15:51:16.014 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_verify_split": "false", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_dump_file": "", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fadvise": "true", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fail_eio": "true", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fd_cache_shards": "16", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fd_cache_size": "128", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fiemap": "false", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fiemap_threshold": "4096", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_inject_stall": "0", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_parallel": "false", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_trailing": "false", 2024-04-25T15:51:16.015 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_writeahead": "false", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_kill_at": "0", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs": "0", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size": "0", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-25T15:51:16.016 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_merge_threshold": "-10", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_odsync_write": "false", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_backend": "rocksdb", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_backend_path": "", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_thread_timeout": "60", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_threads": "2", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_punch_hole": "false", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-25T15:51:16.017 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_ops": "50", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_seek_data_hole": "false", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_sloppy_crc": "false", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_splice": "false", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_split_multiple": "2", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_split_rand_factor": "20", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_update_to": "1000", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-25T15:51:16.018 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_enable": "true", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_xfs_extsize": "false", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_zfs_snap": "false", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "fio_dir": "/tmp/fio", 2024-04-25T15:51:16.019 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_allow_other": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_atomic_o_trunc": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_big_writes": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_debug": "false", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_default_permissions": "false", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_disable_pagecache": "false", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_max_write": "0", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_multithreaded": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_require_active_mds": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_set_user_groups": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_move": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_read": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_write": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-25T15:51:16.020 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_use_invalidate_cb": "true", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "gss_target_name": "ceph", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_file": "", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_inject_failure": "0", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_interval": "5", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "host": "smithi067", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-25T15:51:16.021 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "inject_early_sigterm": "false", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "jaeger_agent_port": "6799", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "jaeger_tracing_enable": "false", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_aio": "true", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_align_min_size": "65536", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_block_align": "true", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_block_size": "4096", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_dio": "true", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_discard": "false", 2024-04-25T15:51:16.022 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_force_aio": "false", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_ignore_corruption": "false", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_max_write_bytes": "10485760", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_max_write_entries": "100", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_replay_from": "0", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_write_header_frequency": "0", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_zero_on_create": "false", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_prefetch_periods": "10", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_prezero_periods": "5", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_write_head_interval": "15", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "", 2024-04-25T15:51:16.023 INFO:teuthology.orchestra.run.smithi067.stdout: "keyfile": "", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_backend": "rocksdb", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_default_stripe_size": "65536", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_fsck_on_mount": "false", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_max_bytes": "67108864", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_max_ops": "512", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_nid_prealloc": "1024", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_onode_map_size": "1024", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_sync_submit_transaction": "false", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_sync_transaction": "false", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "librados_thread_count": "2", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "lockdep": "false", 2024-04-25T15:51:16.024 INFO:teuthology.orchestra.run.smithi067.stdout: "lockdep_force_backtrace": "false", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_coarse_timestamps": "true", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_flush_on_exit": "false", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_graylog_host": "127.0.0.1", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_graylog_port": "12201", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_max_new": "1000", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_max_recent": "10000", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_stderr_prefix": "", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_stop_at_utilization": "0.970000", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": "true", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_graylog": "false", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_journald": "false", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_stderr": "false", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_syslog": "false", 2024-04-25T15:51:16.025 INFO:teuthology.orchestra.run.smithi067.stdout: "max_rotating_auth_attempts": "10", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_action_on_write_error": "1", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_alternate_name_max": "8192", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_asio_thread_count": "2", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_export_pin": "true", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_dirs": "true", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_interval": "5", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_interval": "10", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_max": "-1", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_max_until": "-1", 2024-04-25T15:51:16.026 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_merge_size": "50", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_midchunk": "0.300000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_min_start": "0.200000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_minchunk": "0.001000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_mode": "0", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_need_max": "1.200000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_need_min": "0.800000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_overload_epochs": "2", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_bits": "3", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_size": "10000", 2024-04-25T15:51:16.027 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_target_decay": "10.000000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_grace": "15.000000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_interval": "4.000000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_mid": "0.700000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_quiesce_delay": "0", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_release_free_interval": "10", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_reservation": "0.050000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-25T15:51:16.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_interval": "1", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_threshold": "262144", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_prealloc_inos": "1000", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_connect_bootstrapping": "false", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_damage_table_max_entries": "10000", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_auth_pins": "false", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_frag": "false", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_scatterstat": "false", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_subtrees": "false", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_decay_halflife": "5.000000", 2024-04-25T15:51:16.029 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_default_dir_hash": "2", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_defer_session_stale": "true", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_deny_all_reconnect": "false", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_keys_per_op": "16384", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_max_commit_size": "10", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_max_entries": "0", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_prefetch": "true", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_on_map": "false", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_early_reply": "true", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_enable_op_tracker": "true", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_enforce_unique_name": "true", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-25T15:51:16.030 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_random": "true", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_health_summarize_threshold": "10", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_health_dummy": "false", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_migrator_session_race": "false", 2024-04-25T15:51:16.031 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_join_fs": "", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_journal_format": "1", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_create_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_export_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_import_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_expire_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_replay_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_link_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_mdstable_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_openc_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_rename_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_shutdown_at": "0", 2024-04-25T15:51:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_event_large_threshold": "524288", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_events_per_segment": "1024", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_max_events": "-1", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_max_segments": "128", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_pause": "false", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_segment_size": "0", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_trim_threshold": "128", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_warn_factor": "2.000000", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_caps_per_client": "1048576", 2024-04-25T15:51:16.033 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_completed_flushes": "100000", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_completed_requests": "100000", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_export_size": "20971520", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_file_recover": "32", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_files": "64", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_ops": "8192", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_snaps_per_dir": "100", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_metrics_update_interval": "2", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_min_caps_per_client": "100", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_min_caps_working_set": "10000", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_numa_node": "-1", 2024-04-25T15:51:16.034 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_complaint_time": "30.000000", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_duration": "600", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_size": "20", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_slow_op_size": "20", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_log_threshold": "5", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_ping_grace": "15", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_ping_interval": "5", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_caps": "30000", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_warning_threshold": "262144", 2024-04-25T15:51:16.035 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_replay_interval": "1.000000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_root_ino_gid": "0", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_root_ino_uid": "0", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-25T15:51:16.036 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_shutdown_check": "0", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_skip_ino": "0", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_max_uid": "4294967294", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_min_uid": "0", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_rstat": "false", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_standby_replay_damaged": "false", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_symlink_recovery": "true", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_thrash_exports": "0", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_thrash_fragments": "0", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_tick_interval": "5.000000", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_valgrind_exit": "false", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_verify_backtrace": "1", 2024-04-25T15:51:16.037 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_verify_scatter": "false", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_wipe_sessions": "false", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool_debug": "false", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_device_bytes": "1073741824", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_page_set": "false", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_page_size": "65536", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_bytes": "134217728", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_messages": "512", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_disabled_modules": "", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_enable_op_tracker": "true", 2024-04-25T15:51:16.038 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_max_pg_creating": "1024", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_max_pg_num_change": "128", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mds_bytes": "134217728", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mds_messages": "128", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mon_bytes": "134217728", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mon_messages": "128", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_num_op_tracker_shard": "32", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_op_complaint_time": "30.000000", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_op_history_duration": "600", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_op_history_size": "20", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_op_history_slow_op_size": "20", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_op_log_threshold": "5", 2024-04-25T15:51:16.039 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_osd_bytes": "536870912", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_osd_messages": "8192", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_pool": "true", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_standby_modules": "true", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_stats_period": "5", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_stats_threshold": "5", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_tick_period": "2", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_allow_pool_delete": "true", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_allow_pool_size_one": "true", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_auth_validate_all_caps": "true", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_bytes": "104857600", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_directed_command_retry": "2", 2024-04-25T15:51:16.040 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_parallel": "3", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_log_interval": "1.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_ping_interval": "10.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_target_rank": "-1", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_level": "debug", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_file": "true", 2024-04-25T15:51:16.041 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_journald": "false", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_bootstrap": "false", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_start": "false", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_trim": "true", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cpu_threads": "4", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_crush_min_required_version": "hammer", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_daemon_bytes": "419430400", 2024-04-25T15:51:16.042 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_avail_crit": "5", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_avail_warn": "5", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_size_warn": "16106127360", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_json": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_transactions": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_extra_checks": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_reef": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_squid": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-25T15:51:16.043 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_down_mkfs_grace": "300", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_down_uptime_grace": "60", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_election_default_strategy": "1", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_election_timeout": "5.000000", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ping_divisor": "2", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_enable_op_tracker": "true", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_fake_pool_delete": "false", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_force_quorum_join": "false", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_globalid_prealloc": "10000", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_detail_to_clog": "false", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_log_update_period": "5", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_max_detail": "50", 2024-04-25T15:51:16.044 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog": "true", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog_interval": "600", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_host": "172.21.15.67,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_host_override": "", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_initial_members": "", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease": "5.000000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_full_interval": "50", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_max": "10000", 2024-04-25T15:51:16.045 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_max_summary": "50", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_log_epochs": "500", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_osd": "10000", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_pg_per_osd": "10000", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_pool_pg_num": "65536", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_force_trim_to": "0", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_skip_sanity": "false", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_memory_autotune": "true", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_memory_target": "2147483648", 2024-04-25T15:51:16.046 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_beacon_grace": "30", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_digest_period": "5", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_inactive_grace": "60", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_min_osdmap_epochs": "500", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_complaint_time": "120", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_duration": "600", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_size": "20", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_slow_op_size": "20", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_log_threshold": "5", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_in": "false", 2024-04-25T15:51:16.047 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_cache_size": "500", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_down_out_interval": "600", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_force_trim_to": "0", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-25T15:51:16.048 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_down_reporters": "2", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_report_timeout": "900", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-25T15:51:16.049 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_stuck_threshold": "60", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_probe_timeout": "2.000000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_max_change": "0.050000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_max_osds": "4", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-25T15:51:16.050 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_interval": "86400", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_max_keys": "100", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_timeout": "300", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_session_timeout": "300", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_smart_report_timeout": "5", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stat_smooth_intervals": "6", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_pool_min_size": "2", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_pool_size": "4", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_debug": "false", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_provider_kill_at": "0", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_requester_kill_at": "0", 2024-04-25T15:51:16.051 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_timeout": "60.000000", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_target_pg_per_osd": "100", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_tick_interval": "5", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_timecheck_interval": "300.000000", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_use_min_delay_socket": "false", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_older_version_delay": "604800", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_misplaced": "false", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_older_version": "true", 2024-04-25T15:51:16.052 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "monmap": "", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_tracing_enabled": "false", 2024-04-25T15:51:16.053 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_op_threads": "3", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_cm": "false", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_device_name": "", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_dscp": "96", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_local_gid": "", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_port_num": "1", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-25T15:51:16.054 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_sl": "3", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_support_srq": "true", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_type": "ib", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_reap_threshold": "5", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_before_connect": "false", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_ipv4": "true", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_ipv6": "false", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_msgr1": "true", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_msgr2": "true", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_port_max": "7568", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_port_min": "6800", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_retry_count": "3", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_retry_delay": "5", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_client": "false", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mds": "false", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mgr": "false", 2024-04-25T15:51:16.055 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mon": "false", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_osd": "false", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_client_mode": "crc secure", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_cluster_mode": "crc secure", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_cluster_type": "", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_compress_secure": "false", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_connection_idle_timeout": "900", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_connection_ready_timeout": "10", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_crc_data": "true", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_crc_header": "true", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_bad_msg": "false", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_bug": "true", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_old_message": "true", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_skipped_message": "false", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_coremask": "0xF", 2024-04-25T15:51:16.056 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_enable_tso": "true", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hugepages": "", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_lro": "true", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_memory_channel": "4", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_pmd": "", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_port_id": "0", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dump_on_send": "false", 2024-04-25T15:51:16.057 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_initial_backoff": "0.200000", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_max": "1.000000", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_type": "", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_network_congestion": "0", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_socket_failures": "0", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_learn_addr_from_peer": "true", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_max_accept_failures": "4", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_max_backoff": "15.000000", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_client_mode": "secure crc", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_service_mode": "secure crc", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compress_min_size": "1024", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compress_mode": "none", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-25T15:51:16.058 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_pq_min_cost": "65536", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_public_type": "", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_service_mode": "crc secure", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_listen_backlog": "512", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_nodelay": "true", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_rcvbuf": "0", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_type": "async+posix", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "no_config_file": "false", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_completion_locks_per_session": "32", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inflight_ops": "1024", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-25T15:51:16.059 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_tick_interval": "5.000000", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_timeout": "10.000000", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "objectstore_blackhole": "false", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "openssl_engine_opts": "", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_delay_time": "5.000000", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_hist_halflife": "1000", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_max_low_ops": "2", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_max_ops": "4", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_slop": "0.020000", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_asio_thread_count": "2", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_async_recovery_min_cost": "100", 2024-04-25T15:51:16.060 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_scan_max": "512", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_scan_min": "64", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_degraded": "false", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_peering": "false", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_unfound": "true", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_beacon_report_interval": "300", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_duration": "30", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_max_block_size": "67108864", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_blkin_trace_all": "false", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-25T15:51:16.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_check_for_log_corruption": "false", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_update_on_start": "true", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_message_cap": "256", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_message_size_cap": "524288000", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_op_priority": "63", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_watch_timeout": "30", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_command_max_records": "256", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_compact_on_start": "false", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_update_on_start": "true", 2024-04-25T15:51:16.062 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_update_weight_set": "true", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_feed_pullee": "-1", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_misdirected_ops": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_no_acting_change": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_no_purge_strays": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_op_order": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-25T15:51:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_shutdown": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_trim_objects": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_snaps": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_keys": "1024", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-25T15:51:16.064 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_stride": "524288", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_default_notify_timeout": "30", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep": "0.000000", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_discard_disconnected_ops": "true", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_enable_op_tracker": "true", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_info": "true", 2024-04-25T15:51:16.065 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown": "true", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_function_tracing": "false", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_grace": "20", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_interval": "6", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_peers": "10", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_size": "2000", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_stale": "600", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_max_size": "100000", 2024-04-25T15:51:16.066 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_min_size": "1000", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal_size": "5120", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_kill_backfill_at": "0", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_cache_size": "50", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_dedup": "true", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_message_max": "40", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_share_max_epochs": "40", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_attr_name_len": "100", 2024-04-25T15:51:16.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_attr_size": "0", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_backfills": "1", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_markdown_count": "5", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_markdown_period": "600", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_name_len": "2048", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_namespace_len": "256", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_size": "134217728", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_blocked_by": "16", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_log_entries": "10000", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pgls": "1024", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_push_cost": "8388608", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_push_objects": "10", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_scrubs": "3", 2024-04-25T15:51:16.068 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_trimming_pgs": "2", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_write_op_reply_len": "64", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_write_size": "90", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_profile": "balanced", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-25T15:51:16.069 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_base": "805306368", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_cache_min": "134217728", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target": "4294967296", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target_autotune": "false", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_min_pg_log_entries": "250", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_min_recovery_priority": "0", 2024-04-25T15:51:16.070 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_report_interval": "5", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_num_cache_shards": "32", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_num_op_tracker_shard": "32", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_auto_affinity": "true", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_node": "-1", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_prefer_iface": "true", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objecter_finishers": "1", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore": "bluestore", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore_fuse": "false", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore_tracing": "false", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_complaint_time": "30.000000", 2024-04-25T15:51:16.071 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_duration": "600", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_size": "20", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_slow_op_size": "20", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_log_threshold": "5", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards": "0", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards_hdd": "5", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards_ssd": "8", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_pq_min_cost": "65536", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-25T15:51:16.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_queue_cut_off": "high", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_thread_timeout": "15", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_open_classes_on_start": "true", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_os_flags": "0", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_peering_op_priority": "255", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_delete_cost": "1048576", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_delete_priority": "5", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_trim_max": "10000", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_trim_min": "100", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-25T15:51:16.073 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_crimson": "false", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flags": "0", 2024-04-25T15:51:16.074 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_min_size": "0", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pg_num": "32", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pgp_num": "0", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_read_ratio": "70", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_size": "2", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_type": "replicated", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_push_per_object_cost": "1000", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recover_clone_overlap": "true", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_cost": "20971520", 2024-04-25T15:51:16.075 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active": "0", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_single_start": "1", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_op_priority": "3", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_priority": "5", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep": "0.000000", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_repair_during_recovery": "false", 2024-04-25T15:51:16.076 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_requested_scrub_priority": "120", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_auto_repair": "false", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_begin_hour": "0", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_begin_week_day": "0", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_chunk_max": "25", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_chunk_min": "5", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_cost": "52428800", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_during_recovery": "false", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_end_hour": "0", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_end_week_day": "0", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_event_cost": "4096", 2024-04-25T15:51:16.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_invalid_stats": "true", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_max_preemptions": "5", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_priority": "5", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_sleep": "0.000000", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_skip_data_digest": "false", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_smart_report_timeout": "5", 2024-04-25T15:51:16.078 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_cost": "1048576", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_priority": "5", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_target_transaction_size": "30", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-25T15:51:16.079 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tracing": "false", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_use_stale_snap": "false", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "osdc_blkin_trace_all": "false", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_kill_at": "0", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_max_join_drift": "10", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_min": "500", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_min_wait": "0.050000", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_propose_interval": "1.000000", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_max": "500", 2024-04-25T15:51:16.080 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_min": "250", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_stash_full_interval": "25", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_trim_max": "500", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_trim_min": "250", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "perf": "true", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "-", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrv": "", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "public_bind_addr": "-", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "public_network": "", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "public_network_interface": "", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "qat_compressor_enabled": "false", 2024-04-25T15:51:16.081 INFO:teuthology.orchestra.run.smithi067.stdout: "qat_compressor_session_max_number": "256", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_mon_op_timeout": "0", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_osd_op_timeout": "0", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_tracing": "false", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_atime_update_interval": "60", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_balance_parent_reads": "false", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_balance_snap_reads": "false", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blkin_trace_all": "false", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache": "true", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-25T15:51:16.082 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_policy": "writearound", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_size": "33554432", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_clone_copy_on_read": "false", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_compression_hint": "none", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_concurrent_management_ops": "10", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_clone_format": "auto", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_data_pool": "", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_features": "61", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_format": "2", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_map_options": "", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_order": "22", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_pool": "rbd", 2024-04-25T15:51:16.083 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_stripe_count": "0", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_stripe_unit": "0", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_enable_alloc_hint": "true", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_io_scheduler": "simple", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-25T15:51:16.084 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_order": "24", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_pool": "", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_splay_width": "4", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_localize_parent_reads": "false", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_localize_snap_reads": "false", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-25T15:51:16.085 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-25T15:51:16.086 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mtime_update_interval": "60", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_non_blocking_aio": "true", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_op_thread_timeout": "60", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_op_threads": "1", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_parent_cache_enabled": "false", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_plugins": "", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_burst": "0", 2024-04-25T15:51:16.087 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_limit": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_exclude_ops": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_burst": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_limit": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-25T15:51:16.088 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_read_from_replica_policy": "default", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_skip_partial_discard": "true", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_tracing": "false", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_validate_names": "true", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_validate_pool": "true", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "restapi_base_url": "", 2024-04-25T15:51:16.089 INFO:teuthology.orchestra.run.smithi067.stdout: "restapi_log_level": "", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_account_default_quota_max_size": "-1", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_acl_grants_max_num": "100", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_admin_entry": "admin", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_backend_store": "rados", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_barbican_url": "", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_beast_enable_async": "true", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_counters_cache": "false", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-25T15:51:16.090 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_enabled": "true", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_expiry_interval": "900", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_lru_size": "10000", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_config_store": "rados", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_content_length_compat": "false", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_copy_obj_progress": "true", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cors_rules_max_num": "100", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cross_domain_policy": "", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_addr": "", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-25T15:51:16.091 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_password": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_username": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_require_ssl": "true", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-25T15:51:16.092 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_addr": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_auth": "token", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_namespace": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_prefix": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_token_file": "", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_buffersize": "524288", 2024-04-25T15:51:16.093 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_low_speed_time": "300", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-25T15:51:16.094 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_port": "6379", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_changes_size": "1000", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_num_shards": "128", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_window": "30", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-25T15:51:16.095 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-25T15:51:16.096 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dns_name": "", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dns_s3website_name": "", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dynamic_resharding": "true", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_gc_threads": "true", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_lc_threads": "true", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_ops_log": "false", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_quota_threads": "true", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_static_website": "false", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_usage_log": "false", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enforce_swift_acls": "true", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_exit_timeout_secs": "120", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_expose_bucket": "false", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_extended_http_attrs": "", 2024-04-25T15:51:16.097 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_filter": "none", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_frontends": "beast port=7480", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_deferred": "50", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_objs": "32", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_processor_period": "3600", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-25T15:51:16.098 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_init_timeout": "300", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_kafka_connection_idle": "300", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_domain": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_password": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_password_path": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_project": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_tenant": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_token": "", 2024-04-25T15:51:16.099 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_token_path": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_user": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_api_version": "2", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_domain": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_password": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_project": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_user": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_url": "", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_debug_interval": "-1", 2024-04-25T15:51:16.100 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_lock_max_time": "90", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_objs": "32", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_rules": "1000", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_worker": "3", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_thread_delay": "0", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_dnattr": "uid", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_searchfilter": "", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_uri": "", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-25T15:51:16.101 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_http_headers": "", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_object_name_utc": "false", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attr_name_len": "0", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attr_size": "0", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_chunk_size": "4194304", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_listing_results": "1000", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_notify_retries": "10", 2024-04-25T15:51:16.102 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_put_param_size": "1048576", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_put_size": "5368709120", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_slo_entries": "1000", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_md_log_max_shards": "64", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_mp_lock_max_time": "600", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-25T15:51:16.103 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_max_gc": "300", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_num_async_rados_threads": "32", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_num_control_oids": "8", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_numa_node": "-1", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_chunk_size": "100", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_gc_interval": "600", 2024-04-25T15:51:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_thread_timeout": "600", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_tracing": "false", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_token": "", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_url": "", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_verify_ssl": "true", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_rados": "false", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_socket_path": "", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_push_interval": "2.000000", 2024-04-25T15:51:16.105 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_lanes": "3", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_partitions": "3", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_print_continue": "true", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-25T15:51:16.106 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_tracing": "false", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm": "", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm_id": "", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_region": "", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_request_uri": "", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_batch_size": "64", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_max_aio": "128", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_num_logs": "16", 2024-04-25T15:51:16.107 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_thread_interval": "600", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_resolve_cname": "false", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_run_sync_thread": "true", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_scheduler_type": "throttler", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_script_uri": "", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_service_provider_name": "", 2024-04-25T15:51:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_client_id": "", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_client_secret": "", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_entry": "sts", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_key": "sts", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_min_session_duration": "900", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_token_introspection_url": "", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_account_in_url": "false", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_auth_entry": "auth", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_auth_url": "", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_custom_header": "", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_need_stats": "true", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_tenant_name": "", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_token_expiration": "86400", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_url": "", 2024-04-25T15:51:16.109 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_url_prefix": "swift", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_lease_period": "120", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_thread_pool_size": "512", 2024-04-25T15:51:16.110 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_comment": "", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_createby": "", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_encoding": "", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_flag": "false", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_origin": "", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_tracker": "", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_trust_forwarded_https": "false", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_max_shards": "32", 2024-04-25T15:51:16.111 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_max_user_shards": "1", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_use_opa_authz": "false", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_counters_cache": "false", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_max_buckets": "1000", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_policies_max_num": "100", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_unique_email": "true", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_verify_ssl": "true", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-25T15:51:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone": "", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone_id": "", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup": "", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup_id": "", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rgwlc_auto_session_clear": "true", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_block_size": "4096", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_size": "536870912", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-25T15:51:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_index_type": "binary_search", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_partition_filters": "false", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_perf": "false", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "run_dir": "/var/run/ceph", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_block_create": "true", 2024-04-25T15:51:16.114 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_cache_lru_size": "67108864", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_default_max_object_size": "16777216", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_device_size": "53687091200", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_capacity": "16", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_main_device_type": "SSD", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-25T15:51:16.115 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_segment_size": "67108864", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "set_keepcaps": "false", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "setgroup": "", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "setuser": "", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "setuser_match_path": "", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "thp": "false", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "threadpool_default_timeout": "60", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout: "throttler_perf_counter": "true" 2024-04-25T15:51:16.116 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:16.118 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-25T15:51:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 Processor -- start 2024-04-25T15:51:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 -- start start 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc06ca60 0x7f71cc06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f71cc0b5660 0x7f71cc0b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71cc15fe00 0x7f71cc160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc15d510 con 0x7f71cc15fe00 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc06d780 con 0x7f71cc0b5660 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d2db9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc160be0 con 0x7f71cc06ca60 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71cc15fe00 0x7f71cc160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71cc15fe00 0x7f71cc160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43608/0 (socket says 172.21.15.67:43608) 2024-04-25T15:51:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 -- 172.21.15.67:0/297921148 learned_addr learned my addr 172.21.15.67:0/297921148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d0b55640 1 --2- 172.21.15.67:0/297921148 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc06ca60 0x7f71cc06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71cbfff640 1 --2- 172.21.15.67:0/297921148 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0b5660 0x7f71cc0b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 -- 172.21.15.67:0/297921148 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc06ca60 msgr2=0x7f71cc06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 --2- 172.21.15.67:0/297921148 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc06ca60 0x7f71cc06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 -- 172.21.15.67:0/297921148 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0b5660 msgr2=0x7f71cc0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 --2- 172.21.15.67:0/297921148 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0b5660 0x7f71cc0b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.272+0000 7f71d1356640 1 -- 172.21.15.67:0/297921148 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc15cbe0 con 0x7f71cc15fe00 2024-04-25T15:51:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d1356640 1 --2- 172.21.15.67:0/297921148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc15fe00 0x7f71cc160210 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f71cc069570 tx=0x7f71c401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b48e9fb32d1c007 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71cb7fe640 1 -- 172.21.15.67:0/297921148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40b1070 con 0x7f71cc15fe00 2024-04-25T15:51:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71cb7fe640 1 -- 172.21.15.67:0/297921148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c40ac630 con 0x7f71cc15fe00 2024-04-25T15:51:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71cb7fe640 1 -- 172.21.15.67:0/297921148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40a8050 con 0x7f71cc15fe00 2024-04-25T15:51:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 -- 172.21.15.67:0/297921148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc15fe00 msgr2=0x7f71cc160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 --2- 172.21.15.67:0/297921148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc15fe00 0x7f71cc160210 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f71cc069570 tx=0x7f71c401c520 comp rx=0 tx=0).stop 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 -- 172.21.15.67:0/297921148 shutdown_connections 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 --2- 172.21.15.67:0/297921148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc15fe00 0x7f71cc160210 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 --2- 172.21.15.67:0/297921148 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0b5660 0x7f71cc0b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 --2- 172.21.15.67:0/297921148 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc06ca60 0x7f71cc06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 -- 172.21.15.67:0/297921148 >> 172.21.15.67:0/297921148 conn(0x7f71cc06e420 msgr2=0x7f71cc06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 -- 172.21.15.67:0/297921148 shutdown_connections 2024-04-25T15:51:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.276+0000 7f71d2db9640 1 -- 172.21.15.67:0/297921148 wait complete. 2024-04-25T15:51:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 Processor -- start 2024-04-25T15:51:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 -- start start 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc06ca60 0x7f71cc1f90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc0b5660 0x7f71cc1496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc15fe00 0x7f71cc150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc0b6e90 con 0x7f71cc06ca60 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc15c890 con 0x7f71cc15fe00 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc0b7010 con 0x7f71cc0b5660 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc06ca60 0x7f71cc1f90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc06ca60 0x7f71cc1f90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43614/0 (socket says 172.21.15.67:43614) 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 -- 172.21.15.67:0/3102640721 learned_addr learned my addr 172.21.15.67:0/3102640721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc15fe00 0x7f71cc150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71cbfff640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc0b5660 0x7f71cc1496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc0b5660 msgr2=0x7f71cc1496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc0b5660 0x7f71cc1496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc15fe00 msgr2=0x7f71cc150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc15fe00 0x7f71cc150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc1f9620 con 0x7f71cc06ca60 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d0b55640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc06ca60 0x7f71cc1f90e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f71bc007e60 tx=0x7f71bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc01a070 con 0x7f71cc06ca60 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71bc003ba0 con 0x7f71cc06ca60 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71cc0d1a70 con 0x7f71cc06ca60 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc004030 con 0x7f71cc06ca60 2024-04-25T15:51:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.280+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71cc155f00 con 0x7f71cc06ca60 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f71cc06a480 con 0x7f71cc06ca60 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f71bc01d070 con 0x7f71cc06ca60 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f71ac06f430 0x7f71ac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f71bc08fec0 con 0x7f71cc06ca60 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+1285 (secure 0 0 0) 0x7f71bc090430 con 0x7f71cc06ca60 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71cbfff640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f71ac06f430 0x7f71ac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc1f97f0 0x7f71cc1508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc0d2830 con 0x7f71cc1f97f0 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc1f97f0 0x7f71cc1508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f71cc0cd380 con 0x7f71cc1f97f0 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc1f97f0 0x7f71cc1508d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0cf690 tx=0x7f71c401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71cbfff640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f71ac06f430 0x7f71ac071940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f71cc15dbd0 tx=0x7f71b4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40b1040 con 0x7f71cc1f97f0 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71c4009490 con 0x7f71cc1f97f0 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc1f97f0 msgr2=0x7f71cc1508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc1f97f0 0x7f71cc1508d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0cf690 tx=0x7f71c401ad60 comp rx=0 tx=0).stop 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc1f97f0 msgr2=0x7f71cc1508d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:51:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.284+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc1f97f0 0x7f71cc1508d0 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0cf690 tx=0x7f71c401ad60 comp rx=0 tx=0).stop 2024-04-25T15:51:16.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc156140 0x7f71cc14f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc154f80 con 0x7f71cc156140 2024-04-25T15:51:16.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc156140 0x7f71cc14f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f71cc0cfdc0 con 0x7f71cc156140 2024-04-25T15:51:16.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc156140 0x7f71cc14f2c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71cc06f2f0 tx=0x7f71c40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40c2070 con 0x7f71cc156140 2024-04-25T15:51:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f71c4003c20 con 0x7f71cc156140 2024-04-25T15:51:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc156140 msgr2=0x7f71cc14f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc156140 0x7f71cc14f2c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f71cc06f2f0 tx=0x7f71c40192c0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc156140 msgr2=0x7f71cc14f2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:51:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc156140 0x7f71cc14f2c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.298 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a: {} 2024-04-25T15:51:16.298 INFO:teuthology.orchestra.run.smithi067.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T15:51:16.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc14f800 0x7f71cc14fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.292+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc150150 con 0x7f71cc14f800 2024-04-25T15:51:16.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc14f800 0x7f71cc14fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f71cc0cfa00 con 0x7f71cc14f800 2024-04-25T15:51:16.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc14f800 0x7f71cc14fc10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71c401cdc0 tx=0x7f71c4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40b1040 con 0x7f71cc14f800 2024-04-25T15:51:16.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71c40bee10 con 0x7f71cc14f800 2024-04-25T15:51:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc14f800 msgr2=0x7f71cc14fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc14f800 0x7f71cc14fc10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71c401cdc0 tx=0x7f71c4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc14f800 msgr2=0x7f71cc14fc10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:51:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 reap_dead start 2024-04-25T15:51:16.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.296+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc14f800 0x7f71cc14fc10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.304+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc150300 0x7f71cc1496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.304+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc150710 con 0x7f71cc150300 2024-04-25T15:51:16.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.304+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc150300 0x7f71cc1496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.304+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f71cc0cc0d0 con 0x7f71cc150300 2024-04-25T15:51:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.304+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc150300 0x7f71cc1496f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f71c401abd0 tx=0x7f71c40acf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.304+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40b1040 con 0x7f71cc150300 2024-04-25T15:51:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f71c40d2970 con 0x7f71cc150300 2024-04-25T15:51:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc150300 msgr2=0x7f71cc1496f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc150300 0x7f71cc1496f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f71c401abd0 tx=0x7f71c40acf90 comp rx=0 tx=0).stop 2024-04-25T15:51:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc150300 msgr2=0x7f71cc1496f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:51:16.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc150300 0x7f71cc1496f0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f71c401abd0 tx=0x7f71c40acf90 comp rx=0 tx=0).stop 2024-04-25T15:51:16.312 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b: {} 2024-04-25T15:51:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T15:51:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15a6b0 0x7f71cc15aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc149c30 con 0x7f71cc15a6b0 2024-04-25T15:51:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15a6b0 0x7f71cc15aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.308+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f71cc156c20 con 0x7f71cc15a6b0 2024-04-25T15:51:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.312+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15a6b0 0x7f71cc15aac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71c40045a0 tx=0x7f71c40d1000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.312+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40a8040 con 0x7f71cc15a6b0 2024-04-25T15:51:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.312+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.2 v2:172.21.15.176:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71c40d2770 con 0x7f71cc15a6b0 2024-04-25T15:51:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.312+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15a6b0 msgr2=0x7f71cc15aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.312+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15a6b0 0x7f71cc15aac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71c40045a0 tx=0x7f71c40d1000 comp rx=0 tx=0).stop 2024-04-25T15:51:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.312+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15a6b0 msgr2=0x7f71cc15aac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:51:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.312+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15a6b0 0x7f71cc15aac0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71c40045a0 tx=0x7f71c40d1000 comp rx=0 tx=0).stop 2024-04-25T15:51:16.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.316+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15b000 0x7f71cc02c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.316+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc06c740 con 0x7f71cc15b000 2024-04-25T15:51:16.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.320+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15b000 0x7f71cc02c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.320+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7f71cc1573b0 con 0x7f71cc15b000 2024-04-25T15:51:16.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.320+0000 7f71d1356640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15b000 0x7f71cc02c9e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71c401abd0 tx=0x7f71c4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.320+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40f40a0 con 0x7f71cc15b000 2024-04-25T15:51:16.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 <== mon.2 v2:172.21.15.176:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f71c40eb940 con 0x7f71cc15b000 2024-04-25T15:51:16.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15b000 msgr2=0x7f71cc02c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15b000 0x7f71cc02c9e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71c401abd0 tx=0x7f71c4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:16.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71c97fa640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15b000 msgr2=0x7f71cc02c9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:51:16.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71c97fa640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71cc15b000 0x7f71cc02c9e0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71c401abd0 tx=0x7f71c4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:16.328 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c: {} 2024-04-25T15:51:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T15:51:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f71ac06f430 msgr2=0x7f71ac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f71ac06f430 0x7f71ac071940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f71cc15dbd0 tx=0x7f71b4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc06ca60 msgr2=0x7f71cc1f90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 --2- 172.21.15.67:0/3102640721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71cc06ca60 0x7f71cc1f90e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f71bc007e60 tx=0x7f71bc00e4b0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d1356640 1 -- 172.21.15.67:0/3102640721 reap_dead start 2024-04-25T15:51:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 shutdown_connections 2024-04-25T15:51:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 >> 172.21.15.67:0/3102640721 conn(0x7f71cc06e420 msgr2=0x7f71cc160210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 shutdown_connections 2024-04-25T15:51:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.324+0000 7f71d2db9640 1 -- 172.21.15.67:0/3102640721 wait complete. 2024-04-25T15:51:16.346 DEBUG:teuthology.orchestra.run.smithi067:> type ceph-objectstore-tool 2024-04-25T15:51:16.350 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-25T15:51:16.350 DEBUG:teuthology.orchestra.run.smithi176:> type ceph-objectstore-tool 2024-04-25T15:51:16.356 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-25T15:51:16.356 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T15:51:16.368 INFO:tasks.workunit:Pulling workunits from ref 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:51:16.368 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T15:51:16.369 INFO:tasks.workunit:timeout=3h 2024-04-25T15:51:16.369 INFO:tasks.workunit:cleanup=True 2024-04-25T15:51:16.369 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:16.369 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-25T15:51:16.369 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:16.369 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:16.370 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-25T15:51:16.370 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:51:16.370 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:51:16.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-25T15:51:16.370 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-25T15:51:16.371 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-25T15:51:16.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:16.371 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-25T15:51:16.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:16.371 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-25T15:51:16.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:51:16.394 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:16.392+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:16.400 INFO:teuthology.orchestra.run.smithi067.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:16.401 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-25T15:51:16.401 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 2dh/45d Inode: 1 Links: 2 2024-04-25T15:51:16.401 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:51:16.401 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-25T15:51:16.401 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-25 15:50:36.203243192 +0000 2024-04-25T15:51:16.401 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-25 15:51:14.242137936 +0000 2024-04-25T15:51:16.401 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-04-25T15:51:16.401 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T15:51:16.401 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T15:51:16.472 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:51:16.473 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:16.468+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:16.519 INFO:tasks.workunit.client.0.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T15:51:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a226e640 1 Processor -- start 2024-04-25T15:51:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a226e640 1 -- start start 2024-04-25T15:51:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a226e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb49c156330 0x7fb49c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a226e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 0x7fb49c163630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a226e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49c06fa10 con 0x7fb49c06cb80 2024-04-25T15:51:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a226e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb49c0ca6c0 con 0x7fb49c156330 2024-04-25T15:51:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a226e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb49c06cf90 con 0x7fb49c0c5100 2024-04-25T15:51:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb49c06cb80 0x7fb49c0c49f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40054/0 (socket says 172.21.15.67:40054) 2024-04-25T15:51:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49b7fe640 1 -- 172.21.15.67:0/1411036543 learned_addr learned my addr 172.21.15.67:0/1411036543 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49bfff640 1 --2- 172.21.15.67:0/1411036543 >> v2:172.21.15.176:3300/0 conn(0x7fb49c156330 0x7fb49c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a080b640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 0x7fb49c163630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 581988498 0 0) 0x7fb49c06fa10 con 0x7fb49c06cb80 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb484004640 con 0x7fb49c06cb80 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4100723360 0 0) 0x7fb484004640 con 0x7fb49c06cb80 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 msgr2=0x7fb49c163630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 0x7fb49c163630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 >> v2:172.21.15.176:3300/0 conn(0x7fb49c156330 msgr2=0x7fb49c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 --2- 172.21.15.67:0/1411036543 >> v2:172.21.15.176:3300/0 conn(0x7fb49c156330 0x7fb49c06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c0ca8f0 con 0x7fb49c06cb80 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fb490001800 con 0x7fb49c06cb80 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 >> v1:172.21.15.67:6789/0 conn(0x7fb49c06cb80 legacy=0x7fb49c0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb484004ad0 0x7fb484004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb484005920 0x7fb484005d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 0x7fb484006270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb484003450 con 0x7fb484004ad0 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb484003600 con 0x7fb484005920 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb484003800 con 0x7fb49c0c5100 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49bfff640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb484004ad0 0x7fb484004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb4a080b640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 0x7fb484006270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb484005920 0x7fb484005d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49bfff640 1 -- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 msgr2=0x7fb484006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49bfff640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb49c0c5100 0x7fb484006270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49bfff640 1 -- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb484005920 msgr2=0x7fb484005d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49bfff640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb484005920 0x7fb484005d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.532+0000 7fb49bfff640 1 -- 172.21.15.67:0/1411036543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb484004640 con 0x7fb484004ad0 2024-04-25T15:51:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb49bfff640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb484004ad0 0x7fb484004f40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb48c009ac0 tx=0x7fb48c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3c9ba02b289000 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c0b2070 con 0x7fb484004ad0 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb48c0a5650 con 0x7fb484004ad0 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb49affd640 1 -- 172.21.15.67:0/1411036543 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c009690 con 0x7fb484004ad0 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb4a226e640 1 -- 172.21.15.67:0/1411036543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb484004ad0 msgr2=0x7fb484004f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb4a226e640 1 --2- 172.21.15.67:0/1411036543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb484004ad0 0x7fb484004f40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb48c009ac0 tx=0x7fb48c01ca90 comp rx=0 tx=0).stop 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb4a080b640 1 -- 172.21.15.67:0/1411036543 reap_dead start 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb4a226e640 1 -- 172.21.15.67:0/1411036543 shutdown_connections 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb4a226e640 1 -- 172.21.15.67:0/1411036543 >> 172.21.15.67:0/1411036543 conn(0x7fb49c0ce1c0 msgr2=0x7fb49c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb4a226e640 1 -- 172.21.15.67:0/1411036543 shutdown_connections 2024-04-25T15:51:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.536+0000 7fb4a226e640 1 -- 172.21.15.67:0/1411036543 wait complete. 2024-04-25T15:51:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 Processor -- start 2024-04-25T15:51:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 -- start start 2024-04-25T15:51:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7f4f80475640 1 Processor -- start 2024-04-25T15:51:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7f4f80475640 1 -- start start 2024-04-25T15:51:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb484004ad0 0x7fb49c1f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb49c1549e0 0x7fb49c144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb49c154df0 0x7fb49c144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c0c3d80 con 0x7fb49c1549e0 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c0c43d0 con 0x7fb49c154df0 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb49c0c4060 con 0x7fb484004ad0 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb49c1549e0 0x7fb49c144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb49c1549e0 0x7fb49c144940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43654/0 (socket says 172.21.15.67:43654) 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 -- 172.21.15.67:0/3399478718 learned_addr learned my addr 172.21.15.67:0/3399478718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb484004ad0 0x7fb49c1f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a080b640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb49c154df0 0x7fb49c144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb484004ad0 msgr2=0x7fb49c1f2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb484004ad0 0x7fb49c1f2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb49c154df0 msgr2=0x7fb49c144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb49c154df0 0x7fb49c144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c1f2570 con 0x7fb49c1549e0 2024-04-25T15:51:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb49c1549e0 0x7fb49c144940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb4900011e0 tx=0x7fb49000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb490018020 con 0x7fb49c1549e0 2024-04-25T15:51:16.546 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:16.546 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T15:51:16.546 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb490014600 con 0x7fb49c1549e0 2024-04-25T15:51:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb4a226e640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb49c14a0c0 con 0x7fb49c1549e0 2024-04-25T15:51:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4900149b0 con 0x7fb49c1549e0 2024-04-25T15:51:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7f4f80475640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f780736d0 0x7f4f7806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7f4f80475640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f780caa60 con 0x7f4f7806cb80 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7f4f80475640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7815a5f0 con 0x7f4f7806d460 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7f4f7ea12640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4f7806cb80 0x7f4f7815aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40058/0 (socket says 172.21.15.67:40058) 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.540+0000 7f4f7ea12640 1 -- 172.21.15.67:0/2567653911 learned_addr learned my addr 172.21.15.67:0/2567653911 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7da10640 1 --2- 172.21.15.67:0/2567653911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f780736d0 0x7f4f7806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f80475640 1 -- 172.21.15.67:0/2567653911 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f78068590 con 0x7f4f780736d0 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7d20f640 1 -- 172.21.15.67:0/2567653911 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138854563 0 0) 0x7f4f7815a5f0 con 0x7f4f7806d460 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7d20f640 1 -- 172.21.15.67:0/2567653911 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f64003530 con 0x7f4f7806d460 2024-04-25T15:51:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb4a226e640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb49c1f1830 con 0x7fb49c1549e0 2024-04-25T15:51:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7d20f640 1 -- 172.21.15.67:0/2567653911 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625311481 0 0) 0x7f4f780caa60 con 0x7f4f7806cb80 2024-04-25T15:51:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7d20f640 1 -- 172.21.15.67:0/2567653911 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f7815a5f0 con 0x7f4f7806cb80 2024-04-25T15:51:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7da10640 1 -- 172.21.15.67:0/2567653911 >> v1:172.21.15.176:6789/0 conn(0x7f4f7806d460 legacy=0x7f4f781547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7da10640 1 -- 172.21.15.67:0/2567653911 >> v1:172.21.15.67:6789/0 conn(0x7f4f7806cb80 legacy=0x7f4f7815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7da10640 1 -- 172.21.15.67:0/2567653911 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f780ca100 con 0x7f4f780736d0 2024-04-25T15:51:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb49001b020 con 0x7fb49c1549e0 2024-04-25T15:51:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb474005280 con 0x7fb49c1549e0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb498ff9640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb48806f430 0x7fb488071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fb49008f4e0 con 0x7fb49c1549e0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb48806f430 0x7fb488071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v22) ==== 54+0+16 (secure 0 0 0) 0x7fb49005de20 con 0x7fb49c1549e0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb4740045f0 0x7fb474004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb474005560 con 0x7fb4740045f0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb4a080b640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb4740045f0 0x7fb474004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb4a080b640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb4740045f0 0x7fb474004a00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb48806f430 0x7fb488071940 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0af000 tx=0x7fb48c0046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb474005560 con 0x7fb4740045f0 2024-04-25T15:51:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.544+0000 7f4f7da10640 1 --2- 172.21.15.67:0/2567653911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f780736d0 0x7f4f7806c640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4f78068710 tx=0x7f4f6801c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c41cd723761c0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f7d20f640 1 -- 172.21.15.67:0/2567653911 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f680b0070 con 0x7f4f780736d0 2024-04-25T15:51:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f7d20f640 1 -- 172.21.15.67:0/2567653911 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f680ab600 con 0x7f4f780736d0 2024-04-25T15:51:16.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f7d20f640 1 -- 172.21.15.67:0/2567653911 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f680ab9d0 con 0x7f4f780736d0 2024-04-25T15:51:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 -- 172.21.15.67:0/2567653911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f780736d0 msgr2=0x7f4f7806c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 --2- 172.21.15.67:0/2567653911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f780736d0 0x7f4f7806c640 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4f78068710 tx=0x7f4f6801c460 comp rx=0 tx=0).stop 2024-04-25T15:51:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 -- 172.21.15.67:0/2567653911 shutdown_connections 2024-04-25T15:51:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 --2- 172.21.15.67:0/2567653911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f780736d0 0x7f4f7806c640 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 -- 172.21.15.67:0/2567653911 >> 172.21.15.67:0/2567653911 conn(0x7f4f780ce1c0 msgr2=0x7f4f780ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 -- 172.21.15.67:0/2567653911 shutdown_connections 2024-04-25T15:51:16.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 -- 172.21.15.67:0/2567653911 wait complete. 2024-04-25T15:51:16.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 Processor -- start 2024-04-25T15:51:16.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.552+0000 7f4f80475640 1 -- start start 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f80475640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7806cb80 0x7f4f78157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f80475640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f7806d460 0x7f4f78158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7806cb80 0x7f4f78157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7806cb80 0x7f4f78157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43664/0 (socket says 172.21.15.67:43664) 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 -- 172.21.15.67:0/3568365343 learned_addr learned my addr 172.21.15.67:0/3568365343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f80475640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f780736d0 0x7f4f781593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f80475640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f78168920 con 0x7f4f7806cb80 2024-04-25T15:51:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f80475640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f781687a0 con 0x7f4f780736d0 2024-04-25T15:51:16.560 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:16.560 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:51:16.560 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f80475640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f78168aa0 con 0x7f4f7806d460 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7da10640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f7806d460 0x7f4f78158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7ea12640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f780736d0 0x7f4f781593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 -- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f7806d460 msgr2=0x7f4f78158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f7806d460 0x7f4f78158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 -- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f780736d0 msgr2=0x7f4f781593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f780736d0 0x7f4f781593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f78150230 con 0x7f4f7806cb80 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7da10640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f7806d460 0x7f4f78158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:16.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f7e211640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7806cb80 0x7f4f78157a00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400c3f0 tx=0x7f4f7400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f6effd640 1 -- 172.21.15.67:0/3568365343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f74017070 con 0x7f4f7806cb80 2024-04-25T15:51:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f6effd640 1 -- 172.21.15.67:0/3568365343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f740052d0 con 0x7f4f7806cb80 2024-04-25T15:51:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f80475640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f781f6cf0 con 0x7f4f7806cb80 2024-04-25T15:51:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.556+0000 7f4f6effd640 1 -- 172.21.15.67:0/3568365343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f74003840 con 0x7f4f7806cb80 2024-04-25T15:51:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7f4f80475640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f78157f40 con 0x7f4f7806cb80 2024-04-25T15:51:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7f4f6effd640 1 -- 172.21.15.67:0/3568365343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4f7401a070 con 0x7f4f7806cb80 2024-04-25T15:51:16.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f540052a0 con 0x7f4f7806cb80 2024-04-25T15:51:16.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7f4f6effd640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4f5c06f430 0x7f4f5c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7f4f6effd640 1 -- 172.21.15.67:0/3568365343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f4f74093060 con 0x7f4f7806cb80 2024-04-25T15:51:16.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7f4f7da10640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4f5c06f430 0x7f4f5c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb474005090 con 0x7fb4740045f0 2024-04-25T15:51:16.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.560+0000 7f4f7da10640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4f5c06f430 0x7f4f5c071940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4f680ad000 tx=0x7f4f680a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.564+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb474005090 con 0x7fb4740045f0 2024-04-25T15:51:16.567 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:16.567 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:16.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.564+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb474001c40 0x7fb4740029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.564+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb474002340 con 0x7fb474001c40 2024-04-25T15:51:16.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.564+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb474001c40 0x7fb4740029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.564+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb474001c40 0x7fb4740029a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.564+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb474002340 con 0x7fb474001c40 2024-04-25T15:51:16.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.564+0000 7f4f6effd640 1 -- 172.21.15.67:0/3568365343 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4f74005480 con 0x7f4f7806cb80 2024-04-25T15:51:16.574 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:16.572+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:16.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.580+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb4740031b0 con 0x7fb474001c40 2024-04-25T15:51:16.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:16.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.580+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb4740031b0 con 0x7fb474001c40 2024-04-25T15:51:16.584 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:16.585 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:16.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.580+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb474009c90 0x7fb47400c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.580+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb47400ee80 con 0x7fb474009c90 2024-04-25T15:51:16.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.580+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb474009c90 0x7fb47400c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.580+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb474009c90 0x7fb47400c0b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.584+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb47400ee80 con 0x7fb474009c90 2024-04-25T15:51:16.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.596+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb47400c8c0 con 0x7fb474009c90 2024-04-25T15:51:16.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.600+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb47400c8c0 con 0x7fb474009c90 2024-04-25T15:51:16.603 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:16.603 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:16.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.600+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb47400d110 0x7fb47400d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.600+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb4740127d0 con 0x7fb47400d110 2024-04-25T15:51:16.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.600+0000 7fb4a080b640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb47400d110 0x7fb47400d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.600+0000 7fb4a080b640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb47400d110 0x7fb47400d560 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.600+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4740127d0 con 0x7fb47400d110 2024-04-25T15:51:16.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.616+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb47400dfa0 con 0x7fb47400d110 2024-04-25T15:51:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.616+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb47400dfa0 con 0x7fb47400d110 2024-04-25T15:51:16.620 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.616+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb47400e6c0 0x7fb474011040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.616+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb47400ead0 con 0x7fb47400e6c0 2024-04-25T15:51:16.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.616+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb47400e6c0 0x7fb474011040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.616+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb47400e6c0 0x7fb474011040 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.620+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb47400ead0 con 0x7fb47400e6c0 2024-04-25T15:51:16.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.632+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb47400ead0 con 0x7fb47400e6c0 2024-04-25T15:51:16.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.636+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb47400ead0 con 0x7fb47400e6c0 2024-04-25T15:51:16.641 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:16.641 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:16.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.636+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb474011af0 0x7fb474011f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.636+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb4740192c0 con 0x7fb474011af0 2024-04-25T15:51:16.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.640+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb474011af0 0x7fb474011f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.640+0000 7fb49b7fe640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb474011af0 0x7fb474011f00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.640+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4740192c0 con 0x7fb474011af0 2024-04-25T15:51:16.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.656+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb474016a30 con 0x7fb474011af0 2024-04-25T15:51:16.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.660+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb474016a30 con 0x7fb474011af0 2024-04-25T15:51:16.662 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:16.663 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:16.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.660+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb4740170b0 0x7fb4740174e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.660+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb47401c650 con 0x7fb4740170b0 2024-04-25T15:51:16.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.660+0000 7fb4a080b640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb4740170b0 0x7fb4740174e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.660+0000 7fb4a080b640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb4740170b0 0x7fb4740174e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.664+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb47401c650 con 0x7fb4740170b0 2024-04-25T15:51:16.674 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:16.672+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:16.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.676+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb474017b30 con 0x7fb4740170b0 2024-04-25T15:51:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.684+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb474017b30 con 0x7fb4740170b0 2024-04-25T15:51:16.686 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:16.686 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.684+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb474018040 0x7fb474018450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.684+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb474018a10 con 0x7fb474018040 2024-04-25T15:51:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.684+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb474018040 0x7fb474018450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.684+0000 7fb49bfff640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb474018040 0x7fb474018450 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.684+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb474018a10 con 0x7fb474018040 2024-04-25T15:51:16.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.700+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb4740008e0 con 0x7fb474018040 2024-04-25T15:51:16.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb498ff9640 1 -- 172.21.15.67:0/3399478718 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb4740008e0 con 0x7fb474018040 2024-04-25T15:51:16.710 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:16.711 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:16.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb4740045f0 msgr2=0x7fb474004a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb4740045f0 0x7fb474004a00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb474001c40 msgr2=0x7fb4740029a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb474001c40 0x7fb4740029a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb474009c90 msgr2=0x7fb47400c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb474009c90 0x7fb47400c0b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb47400d110 msgr2=0x7fb47400d560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb47400d110 0x7fb47400d560 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb47400e6c0 msgr2=0x7fb474011040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb47400e6c0 0x7fb474011040 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb474011af0 msgr2=0x7fb474011f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb474011af0 0x7fb474011f00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb4740170b0 msgr2=0x7fb4740174e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb4740170b0 0x7fb4740174e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb474018040 msgr2=0x7fb474018450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb474018040 0x7fb474018450 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4a080b640 1 -- 172.21.15.67:0/3399478718 reap_dead start 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4a080b640 1 -- 172.21.15.67:0/3399478718 reap_dead start 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4a080b640 1 -- 172.21.15.67:0/3399478718 reap_dead start 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4a080b640 1 -- 172.21.15.67:0/3399478718 reap_dead start 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4a080b640 1 -- 172.21.15.67:0/3399478718 reap_dead start 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb48806f430 msgr2=0x7fb488071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb48806f430 0x7fb488071940 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0af000 tx=0x7fb48c0046d0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb49c1549e0 msgr2=0x7fb49c144940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb49c1549e0 0x7fb49c144940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb4900011e0 tx=0x7fb49000e530 comp rx=0 tx=0).stop 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 shutdown_connections 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb48806f430 0x7fb488071940 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 --2- 172.21.15.67:0/3399478718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb49c1549e0 0x7fb49c144940 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 >> 172.21.15.67:0/3399478718 conn(0x7fb49c0ce1c0 msgr2=0x7fb49c152f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:16.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 shutdown_connections 2024-04-25T15:51:16.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.708+0000 7fb4827fc640 1 -- 172.21.15.67:0/3399478718 wait complete. 2024-04-25T15:51:16.725 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:16.725 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:16.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:16.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:16.744+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f4f54005860 con 0x7f4f7806cb80 2024-04-25T15:51:16.775 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:16.772+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:16.876 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:16.872+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:16.879 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:16.879 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:16.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:16.978 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:16.976+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:17.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6effd640 1 -- 172.21.15.67:0/3568365343 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v23) ==== 87+0+0 (secure 0 0 0) 0x7f4f7405b2d0 con 0x7f4f7806cb80 2024-04-25T15:51:17.022 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4f5c06f430 msgr2=0x7f4f5c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4f5c06f430 0x7f4f5c071940 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4f680ad000 tx=0x7f4f680a6000 comp rx=0 tx=0).stop 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7806cb80 msgr2=0x7f4f78157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7806cb80 0x7f4f78157a00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400c3f0 tx=0x7f4f7400c8b0 comp rx=0 tx=0).stop 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 shutdown_connections 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4f5c06f430 0x7f4f5c071940 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f780736d0 0x7f4f781593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f7806d460 0x7f4f78158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 --2- 172.21.15.67:0/3568365343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f7806cb80 0x7f4f78157a00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.020+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 >> 172.21.15.67:0/3568365343 conn(0x7f4f780ce1c0 msgr2=0x7f4f78158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.024+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 shutdown_connections 2024-04-25T15:51:17.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:17.024+0000 7f4f6cff9640 1 -- 172.21.15.67:0/3568365343 wait complete. 2024-04-25T15:51:17.079 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:17.072+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:17.082 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:17.083 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:17.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:17.179 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:17.176+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:17.280 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:17.276+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:17.289 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:17.289 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:17.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:17.381 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:17.376+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:17.454 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:17.454 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:17.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:17.482 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:17.476+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:17.583 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:17.576+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:17.670 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:17.670 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:17.684 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:17.680+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:17.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:17.784 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:17.780+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:17.873 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:17.873 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:17.885 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:17.884+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:17.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:17.986 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:17.980+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:18.083 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:18.083 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:18.087 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:18.084+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:18.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:18.188 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:18.184+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:18.288 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:18.288 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:18.289 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:18.284+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:18.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:18.389 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:18.384+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:18.483 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:18.483 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:18.490 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:18.484+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:18.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:18.591 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:18.588+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:18.651 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:18.651 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:18.668 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:18.715 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:18.712+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:18.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:18.792 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:18.788+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:18.825 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:18.825 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:18.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:18.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab6b10640 1 Processor -- start 2024-04-25T15:51:18.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab6b10640 1 -- start start 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab6b10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ab00736d0 0x7f7ab006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab6b10640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7ab006cb80 0x7f7ab015aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab6b10640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ab00caa60 con 0x7f7ab006d460 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab6b10640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7ab0068590 con 0x7f7ab00736d0 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab6b10640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7ab015a5f0 con 0x7f7ab006cb80 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7aaffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ab00736d0 0x7f7ab006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7aaffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ab00736d0 0x7f7ab006c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43670/0 (socket says 172.21.15.67:43670) 2024-04-25T15:51:18.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7aaffff640 1 -- 172.21.15.67:0/3600529534 learned_addr learned my addr 172.21.15.67:0/3600529534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/3600529534 >> v2:172.21.15.176:3300/0 conn(0x7f7ab006cb80 0x7f7ab015aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7aaffff640 1 -- 172.21.15.67:0/3600529534 >> v2:172.21.15.176:3300/0 conn(0x7f7ab006cb80 msgr2=0x7f7ab015aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7aaffff640 1 --2- 172.21.15.67:0/3600529534 >> v2:172.21.15.176:3300/0 conn(0x7f7ab006cb80 0x7f7ab015aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7aaffff640 1 -- 172.21.15.67:0/3600529534 >> v1:172.21.15.176:6789/0 conn(0x7f7ab006d460 legacy=0x7f7ab01547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:18.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.884+0000 7f7aaffff640 1 -- 172.21.15.67:0/3600529534 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab00ca100 con 0x7f7ab00736d0 2024-04-25T15:51:18.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7aaffff640 1 --2- 172.21.15.67:0/3600529534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab006c640 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00b14e0 tx=0x7f7aa401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868c331122aacb78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7aaf7fe640 1 -- 172.21.15.67:0/3600529534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40b0070 con 0x7f7ab00736d0 2024-04-25T15:51:18.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7aaf7fe640 1 -- 172.21.15.67:0/3600529534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa40ab600 con 0x7f7ab00736d0 2024-04-25T15:51:18.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7aaf7fe640 1 -- 172.21.15.67:0/3600529534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40ab9d0 con 0x7f7ab00736d0 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 -- 172.21.15.67:0/3600529534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 msgr2=0x7f7ab006c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/3600529534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab006c640 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00b14e0 tx=0x7f7aa401c520 comp rx=0 tx=0).stop 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 -- 172.21.15.67:0/3600529534 shutdown_connections 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/3600529534 >> v2:172.21.15.176:3300/0 conn(0x7f7ab006cb80 0x7f7ab015aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/3600529534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab006c640 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 -- 172.21.15.67:0/3600529534 >> 172.21.15.67:0/3600529534 conn(0x7f7ab00ce1c0 msgr2=0x7f7ab00ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 -- 172.21.15.67:0/3600529534 shutdown_connections 2024-04-25T15:51:18.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.888+0000 7f7ab6b10640 1 -- 172.21.15.67:0/3600529534 wait complete. 2024-04-25T15:51:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.892+0000 7f7ab6b10640 1 Processor -- start 2024-04-25T15:51:18.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.892+0000 7f7ab6b10640 1 -- start start 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ab006cb80 0x7f7ab01f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ab006d460 0x7f7ab0145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab0145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab0167790 con 0x7f7ab00736d0 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab015b440 con 0x7f7ab006d460 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab0167910 con 0x7f7ab006cb80 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab0145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab0145eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43674/0 (socket says 172.21.15.67:43674) 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 learned_addr learned my addr 172.21.15.67:0/4261832774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ab006d460 0x7f7ab0145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ab006cb80 0x7f7ab01f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ab006cb80 msgr2=0x7f7ab01f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7ab006cb80 0x7f7ab01f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ab006d460 msgr2=0x7f7ab0145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ab006d460 0x7f7ab0145970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab0149330 con 0x7f7ab00736d0 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7ab006d460 0x7f7ab0145970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab0145eb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800e7f0 tx=0x7f7aa800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8017070 con 0x7f7ab00736d0 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa800c940 con 0x7f7ab00736d0 2024-04-25T15:51:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa800eea0 con 0x7f7ab00736d0 2024-04-25T15:51:18.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab01f7360 con 0x7f7ab00736d0 2024-04-25T15:51:18.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab01f7920 con 0x7f7ab00736d0 2024-04-25T15:51:18.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7ab0149080 con 0x7f7ab00736d0 2024-04-25T15:51:18.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7aa801a070 con 0x7f7ab00736d0 2024-04-25T15:51:18.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.896+0000 7f7aad7fa640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7a9c06f430 0x7f7a9c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7aa808f450 con 0x7f7ab00736d0 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v23) ==== 54+0+16 (secure 0 0 0) 0x7f7aa808f8e0 con 0x7f7ab00736d0 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7a9c06f430 0x7f7a9c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7ab0072930 0x7f7ab0151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ab006a480 con 0x7f7ab0072930 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7ab0072930 0x7f7ab0151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7ab0072930 0x7f7ab0151dc0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7a9c06f430 0x7f7a9c071940 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0074190 tx=0x7f7aa0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.900+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab006a480 con 0x7f7ab0072930 2024-04-25T15:51:18.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.916+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab00c85f0 con 0x7f7ab0072930 2024-04-25T15:51:18.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.916+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab00c85f0 con 0x7f7ab0072930 2024-04-25T15:51:18.920 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.916+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7ab0146860 0x7f7ab0146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.916+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7ab00b09b0 con 0x7f7ab0146860 2024-04-25T15:51:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.916+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7ab0146860 0x7f7ab0146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.916+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7ab0146860 0x7f7ab0146c70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.916+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab00b09b0 con 0x7f7ab0146860 2024-04-25T15:51:18.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.932+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab0152810 con 0x7f7ab0146860 2024-04-25T15:51:18.935 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:18.932+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:18.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.932+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab0152810 con 0x7f7ab0146860 2024-04-25T15:51:18.936 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.932+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7ab0063090 0x7f7ab00634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.932+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7ab01476c0 con 0x7f7ab0063090 2024-04-25T15:51:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.932+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7ab0063090 0x7f7ab00634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.932+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7ab0063090 0x7f7ab00634a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.932+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab01476c0 con 0x7f7ab0063090 2024-04-25T15:51:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.948+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab0072eb0 con 0x7f7ab0063090 2024-04-25T15:51:18.952 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.948+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab0072eb0 con 0x7f7ab0063090 2024-04-25T15:51:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.948+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7ab0063da0 0x7f7ab00cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.948+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7ab00cd870 con 0x7f7ab0063da0 2024-04-25T15:51:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.948+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7ab0063da0 0x7f7ab00cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.948+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7ab0063da0 0x7f7ab00cd330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.948+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab00cd870 con 0x7f7ab0063da0 2024-04-25T15:51:18.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.964+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab00cdd50 con 0x7f7ab0063da0 2024-04-25T15:51:18.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.964+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab00cdd50 con 0x7f7ab0063da0 2024-04-25T15:51:18.969 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:18.970 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:18.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.968+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7ab002cef0 0x7f7ab002d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.968+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7ab00cdcd0 con 0x7f7ab002cef0 2024-04-25T15:51:18.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.968+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7ab002cef0 0x7f7ab002d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.968+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7ab002cef0 0x7f7ab002d300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.968+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab00cdcd0 con 0x7f7ab002cef0 2024-04-25T15:51:18.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.980+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab00cdcd0 con 0x7f7ab002cef0 2024-04-25T15:51:18.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.988+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab00cdcd0 con 0x7f7ab002cef0 2024-04-25T15:51:18.990 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:18.990 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:18.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.988+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7ab0208640 0x7f7ab020aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:18.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.988+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7ab002dd50 con 0x7f7ab0208640 2024-04-25T15:51:18.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.988+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7ab0208640 0x7f7ab020aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:18.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.988+0000 7f7aaffff640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7ab0208640 0x7f7ab020aa60 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:18.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:18.988+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab002dd50 con 0x7f7ab0208640 2024-04-25T15:51:18.995 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:18.988+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:19.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.004+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab002dd50 con 0x7f7ab0208640 2024-04-25T15:51:19.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.008+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab002dd50 con 0x7f7ab0208640 2024-04-25T15:51:19.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:19.011 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:19.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.008+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7ab020b4b0 0x7f7ab020d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.008+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7ab002df70 con 0x7f7ab020b4b0 2024-04-25T15:51:19.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.008+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7ab020b4b0 0x7f7ab020d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.008+0000 7f7ab50ad640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7ab020b4b0 0x7f7ab020d8d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:19.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.012+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab002df70 con 0x7f7ab020b4b0 2024-04-25T15:51:19.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.024+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab020de10 con 0x7f7ab020b4b0 2024-04-25T15:51:19.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.028+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab020de10 con 0x7f7ab020b4b0 2024-04-25T15:51:19.033 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:19.033 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:19.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.028+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7ab020e530 0x7f7ab020e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.028+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7ab0213940 con 0x7f7ab020e530 2024-04-25T15:51:19.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.032+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7ab020e530 0x7f7ab020e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.032+0000 7f7ab48ac640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7ab020e530 0x7f7ab020e940 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:19.035 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:19.036 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:19.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.032+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab0213940 con 0x7f7ab020e530 2024-04-25T15:51:19.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.048+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7ab0212740 con 0x7f7ab020e530 2024-04-25T15:51:19.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:51:19.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:19.056 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:19.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7aad7fa640 1 -- 172.21.15.67:0/4261832774 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7ab0212740 con 0x7f7ab020e530 2024-04-25T15:51:19.057 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:19.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7ab0072930 msgr2=0x7f7ab0151dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7ab0072930 0x7f7ab0151dc0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7ab0146860 msgr2=0x7f7ab0146c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7ab0146860 0x7f7ab0146c70 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7ab0063090 msgr2=0x7f7ab00634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7ab0063090 0x7f7ab00634a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7ab0063da0 msgr2=0x7f7ab00cd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7ab0063da0 0x7f7ab00cd330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7ab002cef0 msgr2=0x7f7ab002d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7ab002cef0 0x7f7ab002d300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7ab0208640 msgr2=0x7f7ab020aa60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7ab0208640 0x7f7ab020aa60 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7ab020b4b0 msgr2=0x7f7ab020d8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7ab020b4b0 0x7f7ab020d8d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7ab020e530 msgr2=0x7f7ab020e940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7ab020e530 0x7f7ab020e940 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 reap_dead start 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 reap_dead start 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7a9c06f430 msgr2=0x7f7a9c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7a9c06f430 0x7f7a9c071940 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0074190 tx=0x7f7aa0009040 comp rx=0 tx=0).stop 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 msgr2=0x7f7ab0145eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 reap_dead start 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 --2- 172.21.15.67:0/4261832774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ab00736d0 0x7f7ab0145eb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800e7f0 tx=0x7f7aa800ecb0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 reap_dead start 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab50ad640 1 -- 172.21.15.67:0/4261832774 reap_dead start 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 shutdown_connections 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 >> 172.21.15.67:0/4261832774 conn(0x7f7ab00ce1c0 msgr2=0x7f7ab0150fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 shutdown_connections 2024-04-25T15:51:19.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.052+0000 7f7ab6b10640 1 -- 172.21.15.67:0/4261832774 wait complete. 2024-04-25T15:51:19.118 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:19.116+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:19.198 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:19.196+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8cd008640 1 Processor -- start 2024-04-25T15:51:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8cd008640 1 -- start start 2024-04-25T15:51:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8cd008640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c8154910 0x7fb8c80c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8cd008640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb8c80c6fd0 0x7fb8c806cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8cd008640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c80c6100 con 0x7fb8c806c9d0 2024-04-25T15:51:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8cd008640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb8c80c6430 con 0x7fb8c80c6fd0 2024-04-25T15:51:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8cd008640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c80c9040 con 0x7fb8c8154910 2024-04-25T15:51:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c6d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c8154910 0x7fb8c80c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c6d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c8154910 0x7fb8c80c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:45774/0 (socket says 172.21.15.67:45774) 2024-04-25T15:51:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c6d9d640 1 -- 172.21.15.67:0/2029179139 learned_addr learned my addr 172.21.15.67:0/2029179139 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c759e640 1 --2- 172.21.15.67:0/2029179139 >> v2:172.21.15.176:3300/0 conn(0x7fb8c80c6fd0 0x7fb8c806cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c759e640 1 -- 172.21.15.67:0/2029179139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c8154910 msgr2=0x7fb8c80c8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c759e640 1 --2- 172.21.15.67:0/2029179139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c8154910 0x7fb8c80c8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c759e640 1 -- 172.21.15.67:0/2029179139 >> v1:172.21.15.176:6789/0 conn(0x7fb8c806c9d0 legacy=0x7fb8c80c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.204+0000 7fb8c759e640 1 -- 172.21.15.67:0/2029179139 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c806e290 con 0x7fb8c80c6fd0 2024-04-25T15:51:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8c759e640 1 --2- 172.21.15.67:0/2029179139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c80c6fd0 0x7fb8c806cde0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80c65b0 tx=0x7fb8c001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=427f7b5e74fc0dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8c659c640 1 -- 172.21.15.67:0/2029179139 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00b2070 con 0x7fb8c80c6fd0 2024-04-25T15:51:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8c659c640 1 -- 172.21.15.67:0/2029179139 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c00a5920 con 0x7fb8c80c6fd0 2024-04-25T15:51:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8c659c640 1 -- 172.21.15.67:0/2029179139 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00a5cf0 con 0x7fb8c80c6fd0 2024-04-25T15:51:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 -- 172.21.15.67:0/2029179139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c80c6fd0 msgr2=0x7fb8c806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 --2- 172.21.15.67:0/2029179139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c80c6fd0 0x7fb8c806cde0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80c65b0 tx=0x7fb8c001c950 comp rx=0 tx=0).stop 2024-04-25T15:51:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 -- 172.21.15.67:0/2029179139 shutdown_connections 2024-04-25T15:51:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 --2- 172.21.15.67:0/2029179139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c80c6fd0 0x7fb8c806cde0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 --2- 172.21.15.67:0/2029179139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c8154910 0x7fb8c80c8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 -- 172.21.15.67:0/2029179139 >> 172.21.15.67:0/2029179139 conn(0x7fb8c8070fa0 msgr2=0x7fb8c80ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 -- 172.21.15.67:0/2029179139 shutdown_connections 2024-04-25T15:51:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.208+0000 7fb8cd008640 1 -- 172.21.15.67:0/2029179139 wait complete. 2024-04-25T15:51:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 Processor -- start 2024-04-25T15:51:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 -- start start 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c806c9d0 0x7fb8c8152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c80c6fd0 0x7fb8c814a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8c8154910 0x7fb8c814b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c815b100 con 0x7fb8c8154910 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c8154d20 con 0x7fb8c806c9d0 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8cd008640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c815b280 con 0x7fb8c80c6fd0 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8c8154910 0x7fb8c814b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8c8154910 0x7fb8c814b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43684/0 (socket says 172.21.15.67:43684) 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 -- 172.21.15.67:0/3300812514 learned_addr learned my addr 172.21.15.67:0/3300812514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c6d9d640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c806c9d0 0x7fb8c8152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8bffff640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c80c6fd0 0x7fb8c814a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 -- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c80c6fd0 msgr2=0x7fb8c814a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c80c6fd0 0x7fb8c814a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 -- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c806c9d0 msgr2=0x7fb8c8152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c806c9d0 0x7fb8c8152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.212+0000 7fb8c759e640 1 -- 172.21.15.67:0/3300812514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c8075ed0 con 0x7fb8c8154910 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8c759e640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8c8154910 0x7fb8c814b120 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00b0040 tx=0x7fb8c00196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8bf7fe640 1 -- 172.21.15.67:0/3300812514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00b2050 con 0x7fb8c8154910 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8bf7fe640 1 -- 172.21.15.67:0/3300812514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c00be070 con 0x7fb8c8154910 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c8151f70 con 0x7fb8c8154910 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8bf7fe640 1 -- 172.21.15.67:0/3300812514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c00ada90 con 0x7fb8c8154910 2024-04-25T15:51:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c81575c0 con 0x7fb8c8154910 2024-04-25T15:51:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8c8157310 con 0x7fb8c8154910 2024-04-25T15:51:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8bf7fe640 1 -- 172.21.15.67:0/3300812514 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb8c00a57a0 con 0x7fb8c8154910 2024-04-25T15:51:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8bf7fe640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8a806f430 0x7fb8a8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8bf7fe640 1 -- 172.21.15.67:0/3300812514 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb8c012a270 con 0x7fb8c8154910 2024-04-25T15:51:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8c6d9d640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8a806f430 0x7fb8a8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.216+0000 7fb8c6d9d640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8a806f430 0x7fb8a8071940 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009ab0 tx=0x7fb8b8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.220+0000 7fb8bf7fe640 1 -- 172.21.15.67:0/3300812514 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb8c8075ed0 con 0x7fb8c8154910 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.034229588999999, 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.247 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:19.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:19.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:19.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:19.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.031614791999999, 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.252 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:19.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:19.256 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.031115894999999, 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.030026649, 2024-04-25T15:51:19.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.029406154, 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:19.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:19.267 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.027011928, 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:19.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:19.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:19.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.997891525, 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.993146791000001, 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:19.279 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.98961929, 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.989486808999999, 2024-04-25T15:51:19.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:19.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.368547351, 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:19.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.357112219000001, 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.293 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:19.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:19.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.356714783000001, 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.356605041999998, 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.356104297000002, 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.355728311, 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.317130128000001, 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.317029869000001, 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.314924954, 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.314549378000001, 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.329 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:19.331 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:19.331 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:19.324+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:19.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:19.392+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb8c806a480 con 0x7fb8c8154910 2024-04-25T15:51:19.431 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:19.428+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:19.529 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:19.529 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:19.529 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:19.529 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:19.532 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:19.532+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:19.545 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:19.633 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:19.628+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:19.734 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:19.732+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:19.738 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:19.739 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:19.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:19.835 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:19.832+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651207816000003, 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:19.902 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:19.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651177255, 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.907 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:19.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651169424000003, 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:19.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651160494999999, 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:19.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.921 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651154263000002, 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.926 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651138852999999, 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651126021000003, 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.935 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.651120806999998, 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 27.56551069, 2024-04-25T15:51:19.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:19.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:19.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 22.012980697, 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.944 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 22.011587858999999, 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:19.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.952 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.973567732999999, 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:19.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:19.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.973544407, 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:19.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:19.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.962 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.973503867, 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:19.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:19.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.973495206999999, 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:19.967 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:19.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.973453534000001, 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:19.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:19.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.973444277, 2024-04-25T15:51:19.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:19.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.978 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.969306851000001, 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:19.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.982 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.969252274, 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.986 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.969219066000001, 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:19.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.990 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:19.991 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:19.991 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:19.992 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:19.992 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:19.988+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.052+0000 7fb8bf7fe640 1 -- 172.21.15.67:0/3300812514 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v24) ==== 97+0+0 (secure 0 0 0) 0x7fb8c0016540 con 0x7fb8c8154910 2024-04-25T15:51:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8a806f430 msgr2=0x7fb8a8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8a806f430 0x7fb8a8071940 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009ab0 tx=0x7fb8b8009240 comp rx=0 tx=0).stop 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8c8154910 msgr2=0x7fb8c814b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8c8154910 0x7fb8c814b120 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00b0040 tx=0x7fb8c00196f0 comp rx=0 tx=0).stop 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 shutdown_connections 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8a806f430 0x7fb8a8071940 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8c8154910 0x7fb8c814b120 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8c80c6fd0 0x7fb8c814a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 --2- 172.21.15.67:0/3300812514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8c806c9d0 0x7fb8c8152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 >> 172.21.15.67:0/3300812514 conn(0x7fb8c8070fa0 msgr2=0x7fb8c80cca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 shutdown_connections 2024-04-25T15:51:20.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:20.056+0000 7fb8cd008640 1 -- 172.21.15.67:0/3300812514 wait complete. 2024-04-25T15:51:20.126 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:20.124+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:20.194 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:20.192+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:20.197 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:20.197 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:20.197 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:20.197 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:20.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:20.295 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:20.292+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:20.361 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:20.407 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:20.407 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:20.407 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:20.407 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:20.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:20.409 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:20.408+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:20.509 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:20.504+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:20.561 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.347798847, 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:51:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:51:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.347769583000002, 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:51:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.347760295, 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:51:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.347752718999999, 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:51:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.347744085999999, 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.582 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:51:20.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:51:20.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:20.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:51:20.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.587 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.347694253, 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:20.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:20.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.347686375000002, 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:20.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:51:20.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.311258367999997, 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:51:20.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:20.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:51:20.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.311227703999997, 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:51:20.601 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:51:20.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.606 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.311215173999997, 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:20.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.611 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.311204541999999, 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:20.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:51:20.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:51:20.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.616 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.311184996999998, 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:51:20.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.621 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.311176105000001, 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:20.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:20.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:51:20.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.307613488999998, 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:51:20.630 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:51:20.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:51:20.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:51:20.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:51:20.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.307492877000001, 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.635 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:51:20.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:51:20.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 22.672494857, 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.639 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:51:20.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.631925459000001, 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.644 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:51:20.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.648 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.631919951, 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:51:20.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:20.653 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.629505436999999, 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:20.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:51:20.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:51:20.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.657 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.628543551, 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:20.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:20.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:20.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:20.661 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:20.662 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:20.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:20.664 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:20.660+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:20.763 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:20.760+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:20.860 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:20.860 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:20.860 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:20.860 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:20.864 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:20.860+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:20.876 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:20.965 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:20.960+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:21.019 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:21.019 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:21.019 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:21.019 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:21.019 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:21.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:21.066 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:21.064+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:21.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:21.167 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:21.164+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.123546888999996, 2024-04-25T15:51:21.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:51:21.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:51:21.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.659946073999997, 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01409815, 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.233 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 25 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562066+0000", 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562072+0000", 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.703e-06 2024-04-25T15:51:21.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562156+0000", 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3486000000000007e-05 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562165+0000", 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.568966+0000", 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006801372 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569110+0000", 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014385799999999999 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569344+0000", 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023376500000000001 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.573357+0000", 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040127870000000003 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.575955+0000", 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025982969999999998 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576077+0000", 2024-04-25T15:51:21.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121966 2024-04-25T15:51:21.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576163+0000", 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6058999999999997e-05 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.533140637999999, 2024-04-25T15:51:21.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013368089999999999, 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:51:21.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688872+0000", 2024-04-25T15:51:21.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0470000000000001e-06 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688892+0000", 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9143e-05 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688940+0000", 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8343000000000001e-05 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688961+0000", 2024-04-25T15:51:21.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0925999999999999e-05 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694131+0000", 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051698639999999997 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694283+0000", 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015229800000000001 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694506+0000", 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222995 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.698950+0000", 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044438300000000002 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702065+0000", 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031151719999999998 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702159+0000", 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4089e-05 2024-04-25T15:51:21.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702238+0000", 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9382999999999995e-05 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.533014086000001, 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013712703999999999, 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.248 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 44 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:51:21.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688998+0000", 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689301+0000", 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000302684 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689729+0000", 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000428486 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689746+0000", 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6690000000000001e-05 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.695997+0000", 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062505779999999997 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696164+0000", 2024-04-25T15:51:21.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167045 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696383+0000", 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.701919+0000", 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055357399999999999 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702410+0000", 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00049136500000000001 2024-04-25T15:51:21.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702581+0000", 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017146899999999999 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702710+0000", 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012822199999999999 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.253 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532701744999997, 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013745535999999999, 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 45 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:51:21.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689310+0000", 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.127e-06 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689317+0000", 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.973e-06 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689793+0000", 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000475468 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689804+0000", 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.082e-05 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697359+0000", 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075556 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697536+0000", 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176658 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697753+0000", 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021705499999999999 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702734+0000", 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049807779999999999 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702850+0000", 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116109 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702934+0000", 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4429000000000001e-05 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703055+0000", 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120519 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532689699000002, 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013995720999999999, 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 46 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689322+0000", 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8999999999999996e-07 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689329+0000", 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1899999999999998e-06 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689831+0000", 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050143900000000001 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689839+0000", 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4069999999999993e-06 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699042+0000", 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092026810000000007 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699214+0000", 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017275100000000001 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699442+0000", 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022718200000000001 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702745+0000", 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003303266 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703161+0000", 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000416253 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703250+0000", 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8806000000000003e-05 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703317+0000", 2024-04-25T15:51:21.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7056000000000003e-05 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532678431999997, 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014300548, 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:51:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689491+0000", 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015768600000000001 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689857+0000", 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000366313 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689865+0000", 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1470000000000001e-06 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700215+0000", 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010350177 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700471+0000", 2024-04-25T15:51:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521499999999999 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700879+0000", 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040780599999999998 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703339+0000", 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024602140000000001 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703443+0000", 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104393 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703524+0000", 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0745e-05 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703633+0000", 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109226 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532514153999998, 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019412740000000001, 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 48 2024-04-25T15:51:21.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689498+0000", 2024-04-25T15:51:21.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5999999999999991e-07 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689505+0000", 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1339999999999998e-06 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689885+0000", 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000380553 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689894+0000", 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0660000000000004e-06 2024-04-25T15:51:21.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703735+0000", 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013841509 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703880+0000", 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014450899999999999 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704121+0000", 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024184799999999999 2024-04-25T15:51:21.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708286+0000", 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041643369999999997 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708530+0000", 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024477400000000001 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708710+0000", 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179715 2024-04-25T15:51:21.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708910+0000", 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199335 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532502143999999, 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019597107999999999, 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.275 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 49 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4700000000000001e-07 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689515+0000", 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3770000000000004e-06 2024-04-25T15:51:21.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689911+0000", 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039616199999999998 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689920+0000", 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2939999999999994e-06 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704840+0000", 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014919412999999999 2024-04-25T15:51:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704990+0000", 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150388 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.705238+0000", 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000248392 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708485+0000", 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032464740000000001 2024-04-25T15:51:21.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708936+0000", 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045072599999999998 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709037+0000", 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101482 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709106+0000", 2024-04-25T15:51:21.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8752999999999997e-05 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532492587, 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.281 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:51:21.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532343965000003, 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.286 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:51:21.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:51:21.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.291 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.532332885999999, 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.225494406000003, 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019231747, 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.297 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 69 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996518+0000", 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6840000000000001e-06 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996645+0000", 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012677899999999999 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996696+0000", 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0912999999999999e-05 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996718+0000", 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1931999999999999e-05 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000632+0000", 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003914509 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000756+0000", 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012338099999999999 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000879+0000", 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012301599999999999 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009110+0000", 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082309819999999995 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015495+0000", 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063848969999999996 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015700+0000", 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020508399999999999 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015748+0000", 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8569999999999997e-05 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.302 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.225242364000003, 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019263037, 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 70 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996770+0000", 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7060000000000001e-06 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996786+0000", 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5497000000000001e-05 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996990+0000", 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000203938 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997014+0000", 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4617000000000002e-05 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002337+0000", 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053230450000000002 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002417+0000", 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9548999999999993e-05 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002579+0000", 2024-04-25T15:51:21.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162385 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009114+0000", 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00653442 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015848+0000", 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067339909999999999 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015951+0000", 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102796 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016032+0000", 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1093000000000004e-05 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:51:21.308 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.225216463999999, 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019380720000000001, 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 71 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996796+0000", 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.063e-06 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996810+0000", 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4100000000000001e-05 2024-04-25T15:51:21.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997061+0000", 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000251797 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997076+0000", 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4970000000000001e-05 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003008+0000", 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005931487 2024-04-25T15:51:21.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003148+0000", 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140097 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003371+0000", 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022273400000000001 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009117+0000", 2024-04-25T15:51:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057466929999999998 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016106+0000", 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069880919999999996 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016144+0000", 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8686000000000002e-05 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016175+0000", 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1000999999999997e-05 2024-04-25T15:51:21.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.225193797999999, 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019495743999999999, 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.314 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996818+0000", 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.046e-06 2024-04-25T15:51:21.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996944+0000", 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012572199999999999 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997139+0000", 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019529899999999999 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997151+0000", 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2116000000000001e-05 2024-04-25T15:51:21.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004407+0000", 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072560519999999998 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004474+0000", 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6810000000000006e-05 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004576+0000", 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101431 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009121+0000", 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045451099999999998 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016239+0000", 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071178359999999998 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016280+0000", 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0914000000000002e-05 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016313+0000", 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3408000000000003e-05 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.224814532000003, 2024-04-25T15:51:21.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019262988000000002, 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 73 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997198+0000", 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.347e-06 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997208+0000", 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0427e-05 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997419+0000", 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021052399999999999 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997433+0000", 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4687e-05 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005473+0000", 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080393610000000001 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005619+0000", 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146157 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005848+0000", 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229001 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009124+0000", 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003276288 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016366+0000", 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072413759999999999 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016408+0000", 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2119000000000001e-05 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016459+0000", 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1700999999999998e-05 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.324 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.224797058, 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019348008999999999, 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 74 2024-04-25T15:51:21.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997215+0000", 2024-04-25T15:51:21.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997220+0000", 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7200000000000003e-06 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997480+0000", 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026014000000000001 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997493+0000", 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2306999999999999e-05 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006512+0000", 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090195669999999992 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006591+0000", 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9058000000000001e-05 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006724+0000", 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013295000000000001 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009130+0000", 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024053659999999999 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073654790000000003 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016533+0000", 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7432999999999999e-05 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016562+0000", 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9379000000000001e-05 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.224787251999999, 2024-04-25T15:51:21.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020191183000000001, 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997376+0000", 2024-04-25T15:51:21.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015132899999999999 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997528+0000", 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152275 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997538+0000", 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0137e-05 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007159+0000", 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096209090000000004 2024-04-25T15:51:21.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007248+0000", 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9078000000000003e-05 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007394+0000", 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014618699999999999 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009133+0000", 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001739008 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017311+0000", 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081775159999999993 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017360+0000", 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9202000000000001e-05 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017415+0000", 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4932000000000002e-05 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:21.336 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.224629847000003, 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:51:21.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:51:21.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:51:21.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:51:21.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:51:21.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:21.342 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:21.347 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:21.340+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:21.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.232+0000 7f0e3ea2b640 1 Processor -- start 2024-04-25T15:51:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.232+0000 7f0e3ea2b640 1 -- start start 2024-04-25T15:51:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e3ea2b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e380b5d70 0x7f0e3806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e3ea2b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3815fe00 0x7f0e38160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e3ea2b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e3806d780 con 0x7f0e3806ca60 2024-04-25T15:51:21.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e3ea2b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0e3815d510 con 0x7f0e380b5d70 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e3ea2b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e38160be0 con 0x7f0e3815fe00 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e380b5d70 0x7f0e3806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e380b5d70 0x7f0e3806ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43696/0 (socket says 172.21.15.67:43696) 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 -- 172.21.15.67:0/1880437580 learned_addr learned my addr 172.21.15.67:0/1880437580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/1880437580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3815fe00 0x7f0e38160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 -- 172.21.15.67:0/1880437580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3815fe00 msgr2=0x7f0e38160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 --2- 172.21.15.67:0/1880437580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3815fe00 0x7f0e38160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 -- 172.21.15.67:0/1880437580 >> v1:172.21.15.176:6789/0 conn(0x7f0e3806ca60 legacy=0x7f0e380b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:21.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 -- 172.21.15.67:0/1880437580 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3815cbe0 con 0x7f0e380b5d70 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e377fe640 1 --2- 172.21.15.67:0/1880437580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e3806ce70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009870 tx=0x7f0e2001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea219b736046cc2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e36ffd640 1 -- 172.21.15.67:0/1880437580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e200b5070 con 0x7f0e380b5d70 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e36ffd640 1 -- 172.21.15.67:0/1880437580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e200a89b0 con 0x7f0e380b5d70 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.236+0000 7f0e36ffd640 1 -- 172.21.15.67:0/1880437580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e200a8d80 con 0x7f0e380b5d70 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/1880437580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 msgr2=0x7f0e3806ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/1880437580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e3806ce70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009870 tx=0x7f0e2001ecf0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/1880437580 shutdown_connections 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/1880437580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3815fe00 0x7f0e38160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/1880437580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e3806ce70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/1880437580 >> 172.21.15.67:0/1880437580 conn(0x7f0e3806e420 msgr2=0x7f0e3806e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/1880437580 shutdown_connections 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/1880437580 wait complete. 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 Processor -- start 2024-04-25T15:51:21.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.240+0000 7f0e3ea2b640 1 -- start start 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3806ca60 0x7f0e381f5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e380c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e3815fe00 0x7f0e3814a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3815c890 con 0x7f0e380b5d70 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3815d1a0 con 0x7f0e3815fe00 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e380b6e90 con 0x7f0e3806ca60 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e380c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e380c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43698/0 (socket says 172.21.15.67:43698) 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 -- 172.21.15.67:0/2511415002 learned_addr learned my addr 172.21.15.67:0/2511415002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e3815fe00 0x7f0e3814a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3806ca60 0x7f0e381f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3806ca60 msgr2=0x7f0e381f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e3806ca60 0x7f0e381f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e3815fe00 msgr2=0x7f0e3814a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e3815fe00 0x7f0e3814a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e381f5760 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e3815fe00 0x7f0e3814a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e377fe640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e380c6de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0e200b3040 tx=0x7f0e2001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e200b5070 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e200b0a90 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e381545c0 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e200bb440 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e38154e70 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0e3806a480 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.244+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0e200c2060 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e34ff9640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e1806f400 0x7f0e18071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f0e2012cb50 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v24) ==== 54+0+16 (secure 0 0 0) 0x7f0e200ca020 con 0x7f0e380b5d70 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e1806f400 0x7f0e18071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0e3814a7b0 0x7f0e3814f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e380cd380 con 0x7f0e3814a7b0 2024-04-25T15:51:21.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0e3814a7b0 0x7f0e3814f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0e3814a7b0 0x7f0e3814f960 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e1806f400 0x7f0e18071910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e380cd350 tx=0x7f0e280094d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.248+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e380cd380 con 0x7f0e3814a7b0 2024-04-25T15:51:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.264+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e3815a7a0 con 0x7f0e3814a7b0 2024-04-25T15:51:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.264+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e3815a7a0 con 0x7f0e3814a7b0 2024-04-25T15:51:21.357 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.264+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0e381f6e90 0x7f0e381f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.264+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0e381f77e0 con 0x7f0e381f6e90 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.264+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0e381f6e90 0x7f0e381f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.264+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0e381f6e90 0x7f0e381f72a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.268+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e381f77e0 con 0x7f0e381f6e90 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.280+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e3806d280 con 0x7f0e381f6e90 2024-04-25T15:51:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.284+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e3806d280 con 0x7f0e381f6e90 2024-04-25T15:51:21.359 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.284+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0e380635a0 0x7f0e380639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.284+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0e381548b0 con 0x7f0e380635a0 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.284+0000 7f0e377fe640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0e380635a0 0x7f0e380639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.284+0000 7f0e377fe640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0e380635a0 0x7f0e380639b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.284+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e381548b0 con 0x7f0e380635a0 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.296+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e38063ef0 con 0x7f0e380635a0 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.300+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e38063ef0 con 0x7f0e380635a0 2024-04-25T15:51:21.360 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.300+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0e3815e4f0 0x7f0e3815e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.300+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0e3815ee40 con 0x7f0e3815e4f0 2024-04-25T15:51:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.300+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0e3815e4f0 0x7f0e3815e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.300+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0e3815e4f0 0x7f0e3815e900 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.300+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e3815ee40 con 0x7f0e3815e4f0 2024-04-25T15:51:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.316+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e3815ee40 con 0x7f0e3815e4f0 2024-04-25T15:51:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.316+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e3815ee40 con 0x7f0e3815e4f0 2024-04-25T15:51:21.362 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.316+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0e3802cef0 0x7f0e3802d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.316+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0e380cbc20 con 0x7f0e3802cef0 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.316+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0e3802cef0 0x7f0e3802d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.320+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0e3802cef0 0x7f0e3802d300 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.320+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e380cbc20 con 0x7f0e3802cef0 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.336+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e38064060 con 0x7f0e3802cef0 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.340+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e38064060 con 0x7f0e3802cef0 2024-04-25T15:51:21.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.340+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0e38207d70 0x7f0e3820a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.340+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0e38064060 con 0x7f0e38207d70 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.340+0000 7f0e377fe640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0e38207d70 0x7f0e3820a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.340+0000 7f0e377fe640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0e38207d70 0x7f0e3820a190 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.340+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e38064060 con 0x7f0e38207d70 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.356+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e38064060 con 0x7f0e38207d70 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.360+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e38064060 con 0x7f0e38207d70 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:21.365 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.360+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0e3820abe0 0x7f0e3820d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.360+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0e3802ddd0 con 0x7f0e3820abe0 2024-04-25T15:51:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.360+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0e3820abe0 0x7f0e3820d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.360+0000 7f0e3cfc8640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0e3820abe0 0x7f0e3820d000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.364+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e3802ddd0 con 0x7f0e3820abe0 2024-04-25T15:51:21.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.376+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e380008e0 con 0x7f0e3820abe0 2024-04-25T15:51:21.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.380+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e380008e0 con 0x7f0e3820abe0 2024-04-25T15:51:21.385 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.384+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0e3820da50 0x7f0e3820fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.384+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0e381f4ac0 con 0x7f0e3820da50 2024-04-25T15:51:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.384+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0e3820da50 0x7f0e3820fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.384+0000 7f0e37fff640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0e3820da50 0x7f0e3820fe70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:21.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.384+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e381f4ac0 con 0x7f0e3820da50 2024-04-25T15:51:21.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.400+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0e3802ddd0 con 0x7f0e3820da50 2024-04-25T15:51:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e34ff9640 1 -- 172.21.15.67:0/2511415002 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0e3802ddd0 con 0x7f0e3820da50 2024-04-25T15:51:21.407 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0e3814a7b0 msgr2=0x7f0e3814f960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0e3814a7b0 0x7f0e3814f960 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0e381f6e90 msgr2=0x7f0e381f72a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0e381f6e90 0x7f0e381f72a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0e380635a0 msgr2=0x7f0e380639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0e380635a0 0x7f0e380639b0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0e3815e4f0 msgr2=0x7f0e3815e900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0e3815e4f0 0x7f0e3815e900 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0e3802cef0 msgr2=0x7f0e3802d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0e3802cef0 0x7f0e3802d300 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0e38207d70 msgr2=0x7f0e3820a190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0e38207d70 0x7f0e3820a190 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0e3820abe0 msgr2=0x7f0e3820d000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3cfc8640 1 -- 172.21.15.67:0/2511415002 reap_dead start 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0e3820abe0 0x7f0e3820d000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3cfc8640 1 -- 172.21.15.67:0/2511415002 reap_dead start 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3cfc8640 1 -- 172.21.15.67:0/2511415002 reap_dead start 2024-04-25T15:51:21.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0e3820da50 msgr2=0x7f0e3820fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0e3820da50 0x7f0e3820fe70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e1806f400 msgr2=0x7f0e18071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e1806f400 0x7f0e18071910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e380cd350 tx=0x7f0e280094d0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 msgr2=0x7f0e380c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e380c6de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0e200b3040 tx=0x7f0e2001ba90 comp rx=0 tx=0).stop 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 shutdown_connections 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0e3820abe0 0x7f0e3820d000 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0e3820da50 0x7f0e3820fe70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e1806f400 0x7f0e18071910 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 --2- 172.21.15.67:0/2511415002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e380b5d70 0x7f0e380c6de0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 >> 172.21.15.67:0/2511415002 conn(0x7f0e3806e420 msgr2=0x7f0e3815c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 shutdown_connections 2024-04-25T15:51:21.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:21.404+0000 7f0e3ea2b640 1 -- 172.21.15.67:0/2511415002 wait complete. 2024-04-25T15:51:21.470 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:21.468+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:21.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:21.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:21.567 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:21.567 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-25T15:51:21.567 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:51:21.567 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:51:21.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-04-25T15:51:21.582 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:21.580+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:21.648 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:21.648+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:21.730 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:21.730 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:21.731 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:21.731 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T15:51:21.731 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:21.749 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:21.744+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:21.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:21.851 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:21.844+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:21.945 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:21.945 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:21.951 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:21.948+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:21.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:22.053 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:22.048+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:22.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:51:22.115 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:22.115 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:22.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:22.154 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:22.148+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.232+0000 7f4ea0a16640 1 Processor -- start 2024-04-25T15:51:22.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4ea0a16640 1 -- start start 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4ea0a16640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e9c0c92d0 0x7f4e9c072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4ea0a16640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c0c96e0 0x7f4e9c073300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4ea0a16640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e9c06d8e0 con 0x7f4e9c160800 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4ea0a16640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9c06dbc0 con 0x7f4e9c0c92d0 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4ea0a16640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9c073c70 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e99d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e9c0c92d0 0x7f4e9c072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e99d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e9c0c92d0 0x7f4e9c072dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43704/0 (socket says 172.21.15.67:43704) 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e99d9b640 1 -- 172.21.15.67:0/2713917361 learned_addr learned my addr 172.21.15.67:0/2713917361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e9ad9d640 1 --2- 172.21.15.67:0/2713917361 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c0c96e0 0x7f4e9c073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e99d9b640 1 -- 172.21.15.67:0/2713917361 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c0c96e0 msgr2=0x7f4e9c073300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e99d9b640 1 --2- 172.21.15.67:0/2713917361 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c0c96e0 0x7f4e9c073300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e99d9b640 1 -- 172.21.15.67:0/2713917361 >> v1:172.21.15.176:6789/0 conn(0x7f4e9c160800 legacy=0x7f4e9c06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:22.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.236+0000 7f4e99d9b640 1 -- 172.21.15.67:0/2713917361 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c073f50 con 0x7f4e9c0c92d0 2024-04-25T15:51:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4e99d9b640 1 --2- 172.21.15.67:0/2713917361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c92d0 0x7f4e9c072dc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009870 tx=0x7f4e9001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1fdca93923f8ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4e9959a640 1 -- 172.21.15.67:0/2713917361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e900b0070 con 0x7f4e9c0c92d0 2024-04-25T15:51:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4e9959a640 1 -- 172.21.15.67:0/2713917361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e900ab600 con 0x7f4e9c0c92d0 2024-04-25T15:51:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4e9959a640 1 -- 172.21.15.67:0/2713917361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e900ab9d0 con 0x7f4e9c0c92d0 2024-04-25T15:51:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 -- 172.21.15.67:0/2713917361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c92d0 msgr2=0x7f4e9c072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/2713917361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c92d0 0x7f4e9c072dc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009870 tx=0x7f4e9001c490 comp rx=0 tx=0).stop 2024-04-25T15:51:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 -- 172.21.15.67:0/2713917361 shutdown_connections 2024-04-25T15:51:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/2713917361 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c0c96e0 0x7f4e9c073300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/2713917361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c92d0 0x7f4e9c072dc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 -- 172.21.15.67:0/2713917361 >> 172.21.15.67:0/2713917361 conn(0x7f4e9c070db0 msgr2=0x7f4e9c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 -- 172.21.15.67:0/2713917361 shutdown_connections 2024-04-25T15:51:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 -- 172.21.15.67:0/2713917361 wait complete. 2024-04-25T15:51:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 Processor -- start 2024-04-25T15:51:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.240+0000 7f4ea0a16640 1 -- start start 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e9c0c92d0 0x7f4e9c14c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c96e0 0x7f4e9c14e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c160800 0x7f4e9c147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c160d90 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c160c10 con 0x7f4e9c0c92d0 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9c1699a0 con 0x7f4e9c160800 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c96e0 0x7f4e9c14e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c96e0 0x7f4e9c14e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43710/0 (socket says 172.21.15.67:43710) 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 -- 172.21.15.67:0/3130792060 learned_addr learned my addr 172.21.15.67:0/3130792060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e9ad9d640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c160800 0x7f4e9c147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e9a59c640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e9c0c92d0 0x7f4e9c14c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 -- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c160800 msgr2=0x7f4e9c147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c160800 0x7f4e9c147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 -- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e9c0c92d0 msgr2=0x7f4e9c14c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e9c0c92d0 0x7f4e9c14c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 -- 172.21.15.67:0/3130792060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c1f5f80 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e99d9b640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c96e0 0x7f4e9c14e120 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4e9001cd30 tx=0x7f4e90019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e8affd640 1 -- 172.21.15.67:0/3130792060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e900b9070 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e8affd640 1 -- 172.21.15.67:0/3130792060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e900b4620 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9c15e1d0 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4e8affd640 1 -- 172.21.15.67:0/3130792060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e900b0040 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9c1f6500 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.244+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9c15df40 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.248+0000 7f4e8affd640 1 -- 172.21.15.67:0/3130792060 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4e900bc070 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.248+0000 7f4e8affd640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4e7c06f430 0x7f4e7c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.248+0000 7f4e8affd640 1 -- 172.21.15.67:0/3130792060 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5825+0+0 (secure 0 0 0) 0x7f4e90128740 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.248+0000 7f4e9a59c640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4e7c06f430 0x7f4e7c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.248+0000 7f4e9a59c640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4e7c06f430 0x7f4e7c071940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00b810 tx=0x7f4e8c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:22.254 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:22.248+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.256+0000 7f4e8affd640 1 -- 172.21.15.67:0/3130792060 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4e900c3080 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.340 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:22.340 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:22.355 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:22.352+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:22.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:22.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:22.432+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f4e9c06a480 con 0x7f4e9c0c96e0 2024-04-25T15:51:22.456 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:22.452+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:22.547 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:22.548 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:22.557 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:22.552+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:22.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:22.658 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:22.652+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:22.759 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:22.759 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:22.759 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:22.756+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:22.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:22.859 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:22.856+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:22.923 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:22.924 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:22.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:22.960 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:22.956+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:23.061 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:23.056+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.064+0000 7f4e8affd640 1 -- 172.21.15.67:0/3130792060 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v25) ==== 91+0+0 (secure 0 0 0) 0x7f4e90016110 con 0x7f4e9c0c96e0 2024-04-25T15:51:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4e7c06f430 msgr2=0x7f4e7c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4e7c06f430 0x7f4e7c071940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00b810 tx=0x7f4e8c008040 comp rx=0 tx=0).stop 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c96e0 msgr2=0x7f4e9c14e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c96e0 0x7f4e9c14e120 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4e9001cd30 tx=0x7f4e90019230 comp rx=0 tx=0).stop 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 shutdown_connections 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4e7c06f430 0x7f4e7c071940 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e9c160800 0x7f4e9c147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e9c0c96e0 0x7f4e9c14e120 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 --2- 172.21.15.67:0/3130792060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e9c0c92d0 0x7f4e9c14c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 >> 172.21.15.67:0/3130792060 conn(0x7f4e9c070db0 msgr2=0x7f4e9c06cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:23.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 shutdown_connections 2024-04-25T15:51:23.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.068+0000 7f4ea0a16640 1 -- 172.21.15.67:0/3130792060 wait complete. 2024-04-25T15:51:23.138 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:23.138 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:23.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:23.161 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:23.156+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:23.262 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:23.260+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:23.308 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:23.308 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:23.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:23.370 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:23.368+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:23.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:23.464 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:23.464+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:23.519 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:23.519 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:23.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:23.565 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:23.560+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:23.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1106fb8640 1 Processor -- start 2024-04-25T15:51:23.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1106fb8640 1 -- start start 2024-04-25T15:51:23.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1106fb8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1100156f10 0x7f11000cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1106fb8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f110015d210 0x7f110015d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1106fb8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1100160fa0 con 0x7f11000cc600 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1106fb8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1100157320 con 0x7f110015d210 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1106fb8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f11000c6d10 con 0x7f1100156f10 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f10fffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f110015d210 0x7f110015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f10fffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f110015d210 0x7f110015d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58224/0 (socket says 172.21.15.67:58224) 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f10fffff640 1 -- 172.21.15.67:0/679641133 learned_addr learned my addr 172.21.15.67:0/679641133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f1104d54640 1 --2- 172.21.15.67:0/679641133 >> v2:172.21.15.176:3300/0 conn(0x7f1100156f10 0x7f11000cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f10fffff640 1 -- 172.21.15.67:0/679641133 >> v2:172.21.15.176:3300/0 conn(0x7f1100156f10 msgr2=0x7f11000cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f10fffff640 1 --2- 172.21.15.67:0/679641133 >> v2:172.21.15.176:3300/0 conn(0x7f1100156f10 0x7f11000cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f10fffff640 1 -- 172.21.15.67:0/679641133 >> v1:172.21.15.176:6789/0 conn(0x7f11000cc600 legacy=0x7f1100073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:23.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.568+0000 7f10fffff640 1 -- 172.21.15.67:0/679641133 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1100160420 con 0x7f110015d210 2024-04-25T15:51:23.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f10fffff640 1 --2- 172.21.15.67:0/679641133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110015d210 0x7f110015d620 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1100072060 tx=0x7f10ec01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686c7920b1585840 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f10ff7fe640 1 -- 172.21.15.67:0/679641133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0b0070 con 0x7f110015d210 2024-04-25T15:51:23.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f10ff7fe640 1 -- 172.21.15.67:0/679641133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec0ab600 con 0x7f110015d210 2024-04-25T15:51:23.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f10ff7fe640 1 -- 172.21.15.67:0/679641133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0ab9d0 con 0x7f110015d210 2024-04-25T15:51:23.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 -- 172.21.15.67:0/679641133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110015d210 msgr2=0x7f110015d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 --2- 172.21.15.67:0/679641133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110015d210 0x7f110015d620 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1100072060 tx=0x7f10ec01c4e0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 -- 172.21.15.67:0/679641133 shutdown_connections 2024-04-25T15:51:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 --2- 172.21.15.67:0/679641133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f110015d210 0x7f110015d620 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 --2- 172.21.15.67:0/679641133 >> v2:172.21.15.176:3300/0 conn(0x7f1100156f10 0x7f11000cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 -- 172.21.15.67:0/679641133 >> 172.21.15.67:0/679641133 conn(0x7f1100068c70 msgr2=0x7f1100071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 -- 172.21.15.67:0/679641133 shutdown_connections 2024-04-25T15:51:23.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 -- 172.21.15.67:0/679641133 wait complete. 2024-04-25T15:51:23.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 Processor -- start 2024-04-25T15:51:23.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.572+0000 7f1106fb8640 1 -- start start 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11000cc600 0x7f110015f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1100156f10 0x7f110014d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110015d210 0x7f110014d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f11000cce20 con 0x7f1100156f10 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f110015ccf0 con 0x7f110015d210 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f11001600c0 con 0x7f11000cc600 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1100156f10 0x7f110014d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1100156f10 0x7f110014d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58240/0 (socket says 172.21.15.67:58240) 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 -- 172.21.15.67:0/3268163261 learned_addr learned my addr 172.21.15.67:0/3268163261 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:23.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11000cc600 0x7f110015f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110015d210 0x7f110014d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11000cc600 msgr2=0x7f110015f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11000cc600 0x7f110015f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110015d210 msgr2=0x7f110014d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110015d210 0x7f110014d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110015fa80 con 0x7f1100156f10 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11000cc600 0x7f110015f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110015d210 0x7f110014d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fffff640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1100156f10 0x7f110014d3a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01a110 tx=0x7f10ec019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0ba070 con 0x7f1100156f10 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec0b4620 con 0x7f1100156f10 2024-04-25T15:51:23.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0b0040 con 0x7f1100156f10 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110015e170 con 0x7f1100156f10 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110015e780 con 0x7f1100156f10 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.576+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f110006a480 con 0x7f1100156f10 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f10ec0ab4c0 con 0x7f1100156f10 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f10fd7fa640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f10e406f430 0x7f10e4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5825+0+0 (secure 0 0 0) 0x7f10ec128620 con 0x7f1100156f10 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v25) ==== 54+0+16 (secure 0 0 0) 0x7f10ec0160d0 con 0x7f1100156f10 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f10e406f430 0x7f10e4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1100155ef0 0x7f11000c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f110015ee60 con 0x7f1100155ef0 2024-04-25T15:51:23.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1100155ef0 0x7f11000c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1100155ef0 0x7f11000c8510 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f10e406f430 0x7f10e4071940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.580+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10f80079e0 con 0x7f1100155ef0 2024-04-25T15:51:23.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.612+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f11001f6af0 con 0x7f1100155ef0 2024-04-25T15:51:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.616+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f11001f6af0 con 0x7f1100155ef0 2024-04-25T15:51:23.618 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:23.618 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.616+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11001f9de0 0x7f11001fa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.616+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f11000cba60 con 0x7f11001f9de0 2024-04-25T15:51:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.616+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11001f9de0 0x7f11001fa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.616+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11001f9de0 0x7f11001fa1f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.616+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11000cba60 con 0x7f11001f9de0 2024-04-25T15:51:23.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.632+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f11000cb260 con 0x7f11001f9de0 2024-04-25T15:51:23.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.632+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f11000cb260 con 0x7f11001f9de0 2024-04-25T15:51:23.636 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.632+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1100063090 0x7f11000634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.632+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f11001fac40 con 0x7f1100063090 2024-04-25T15:51:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.632+0000 7f10fffff640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1100063090 0x7f11000634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.632+0000 7f10fffff640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1100063090 0x7f11000634a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.636+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11001fac40 con 0x7f1100063090 2024-04-25T15:51:23.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.648+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f110006d7f0 con 0x7f1100063090 2024-04-25T15:51:23.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.652+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f110006d7f0 con 0x7f1100063090 2024-04-25T15:51:23.654 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.652+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f110006dd10 0x7f110006e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.652+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f11001521e0 con 0x7f110006dd10 2024-04-25T15:51:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.652+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f110006dd10 0x7f110006e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.652+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f110006dd10 0x7f110006e120 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.652+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11001521e0 con 0x7f110006dd10 2024-04-25T15:51:23.665 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:23.660+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.664+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f11001521e0 con 0x7f110006dd10 2024-04-25T15:51:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.668+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f11001521e0 con 0x7f110006dd10 2024-04-25T15:51:23.671 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.668+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f110002c9e0 0x7f110002cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.668+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f11001521e0 con 0x7f110002c9e0 2024-04-25T15:51:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.668+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f110002c9e0 0x7f110002cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.668+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f110002c9e0 0x7f110002cdf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.672+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11001521e0 con 0x7f110002c9e0 2024-04-25T15:51:23.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.684+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f11001521e0 con 0x7f110002c9e0 2024-04-25T15:51:23.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.688+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f11001521e0 con 0x7f110002c9e0 2024-04-25T15:51:23.693 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.688+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f110002d840 0x7f110020d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.692+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f110002dc50 con 0x7f110002d840 2024-04-25T15:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.692+0000 7f10fffff640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f110002d840 0x7f110020d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.692+0000 7f10fffff640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f110002d840 0x7f110020d170 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.692+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f110002dc50 con 0x7f110002d840 2024-04-25T15:51:23.704 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.490206186999998, 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.490166541999997, 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:23.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:23.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.487259520000002, 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:23.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.487106584000003, 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.485590940000002, 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.721 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:23.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:23.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.485448292999997, 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.485345201999998, 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:23.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:23.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:23.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.485192452, 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:23.733 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:23.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.483244225999997, 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.737 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.455216595000003, 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.453193704, 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.744 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:23.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:23.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.453023154999997, 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.748 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:23.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:23.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:23.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.450667658999997, 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.752 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.450632947000003, 2024-04-25T15:51:23.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.450612137999997, 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:23.761 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.373725091000001, 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:23.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:23.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.769 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.373701009000001, 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:23.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.373691311999998, 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.367796375000001, 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:23.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.778357717999999, 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:23.783 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:23.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:23.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:23.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:23.788 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.708+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f110002dc50 con 0x7f110002d840 2024-04-25T15:51:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.712+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f110002dc50 con 0x7f110002d840 2024-04-25T15:51:23.789 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.712+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f110002dc50 0x7f110020fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.712+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1100212c70 con 0x7f110002dc50 2024-04-25T15:51:23.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.712+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f110002dc50 0x7f110020fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.712+0000 7f1105555640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f110002dc50 0x7f110020fbd0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.716+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1100212c70 con 0x7f110002dc50 2024-04-25T15:51:23.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.728+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1100076a30 con 0x7f110002dc50 2024-04-25T15:51:23.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.732+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1100076a30 con 0x7f110002dc50 2024-04-25T15:51:23.792 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.736+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1100211ea0 0x7f11002122d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.736+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1100216090 con 0x7f1100211ea0 2024-04-25T15:51:23.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.736+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1100211ea0 0x7f11002122d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.736+0000 7f1104d54640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1100211ea0 0x7f11002122d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.736+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1100216090 con 0x7f1100211ea0 2024-04-25T15:51:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.752+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1100210250 con 0x7f1100211ea0 2024-04-25T15:51:23.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f10fd7fa640 1 -- 172.21.15.67:0/3268163261 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1100210250 con 0x7f1100211ea0 2024-04-25T15:51:23.794 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1100155ef0 msgr2=0x7f11000c8510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1100155ef0 0x7f11000c8510 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11001f9de0 msgr2=0x7f11001fa1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11001f9de0 0x7f11001fa1f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1100063090 msgr2=0x7f11000634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1100063090 0x7f11000634a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f110006dd10 msgr2=0x7f110006e120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1105555640 1 -- 172.21.15.67:0/3268163261 reap_dead start 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f110006dd10 0x7f110006e120 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f110002c9e0 msgr2=0x7f110002cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f110002c9e0 0x7f110002cdf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f110002d840 msgr2=0x7f110020d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f110002d840 0x7f110020d170 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f110002dc50 msgr2=0x7f110020fbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f110002dc50 0x7f110020fbd0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1100211ea0 msgr2=0x7f11002122d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1100211ea0 0x7f11002122d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1105555640 1 -- 172.21.15.67:0/3268163261 reap_dead start 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f10e406f430 msgr2=0x7f10e4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f10e406f430 0x7f10e4071940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1100156f10 msgr2=0x7f110014d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1100156f10 0x7f110014d3a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f10ec01a110 tx=0x7f10ec019280 comp rx=0 tx=0).stop 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 shutdown_connections 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f10e406f430 0x7f10e4071940 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 --2- 172.21.15.67:0/3268163261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1100156f10 0x7f110014d3a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 >> 172.21.15.67:0/3268163261 conn(0x7f1100068c70 msgr2=0x7f1100154fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 shutdown_connections 2024-04-25T15:51:23.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:23.756+0000 7f1106fb8640 1 -- 172.21.15.67:0/3268163261 wait complete. 2024-04-25T15:51:23.799 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:23.788+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:23.891 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:23.888+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:23.953 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:23.953 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:23.953 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:23.953 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:23.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:24.015 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:24.012+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:24.093 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:24.088+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:24.122 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:24.123 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:24.123 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:24.123 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:24.123 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:24.140 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:24.194 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:24.192+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:24.295 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:24.292+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.124635187000003, 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.337 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:24.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:24.342 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.122020390000003, 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:24.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:24.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.346 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.121521493000003, 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:24.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:24.350 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.120432246999997, 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.119811751999997, 2024-04-25T15:51:24.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:24.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:24.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.117417525999997, 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:24.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.088297122999997, 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.083552388999998, 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:24.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.080024887999997, 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:24.370 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:24.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.079892407000003, 2024-04-25T15:51:24.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:24.378 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.458952949, 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.447517817000001, 2024-04-25T15:51:24.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:24.387 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.447120381000001, 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:24.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:24.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.447010639999998, 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:24.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:24.395 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.446509894999998, 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:24.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:24.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:24.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:24.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.446133909, 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.407535725999999, 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.407435466999999, 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.408 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:24.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:24.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:24.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.405330551999999, 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.412 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:24.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.404954975999999, 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:24.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:24.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:24.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:24.419 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:24.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:24.421 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:24.416+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:24.521 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:24.516+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:24.615 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:24.615 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:24.616 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:24.616 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:24.622 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:24.616+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:24.632 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:24.723 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:24.720+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:24.824 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:24.820+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:24.826 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:24.826 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:24.826 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:24.826 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:24.827 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:24.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:24.924 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:24.920+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:25.025 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:25.024+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:25.042 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:25.042 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:25.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:25.042 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:25.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.042 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791689611999999, 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:25.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:25.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791659051000003, 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.048 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:25.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:25.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.053 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791651219999999, 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:25.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:25.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:25.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:25.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:25.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791642291000002, 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:25.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791636058999998, 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:25.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:25.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791620649000002, 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.069 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:25.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791607816999999, 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:25.074 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.791602603000001, 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.078 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:25.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:25.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.081 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.705992486, 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:25.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 27.153462492999999, 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.091 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 27.152069654999998, 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:25.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:25.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.114049528999999, 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.096 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:25.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:25.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.114026203000002, 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.101 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:25.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.113985663000001, 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.106 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:25.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.113977002999999, 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:25.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.11393533, 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.116 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.113926072999998, 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.109788646999998, 2024-04-25T15:51:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.109734069999998, 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.109700862, 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.137 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:25.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:51:25.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:25.139 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:25.132+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:25.239 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:25.236+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.296+0000 7f861af87640 1 Processor -- start 2024-04-25T15:51:25.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.296+0000 7f861af87640 1 -- start start 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f861af87640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86140c4760 0x7f86140c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f861af87640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f86140c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f861af87640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86140c7340 con 0x7f8614159520 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f861af87640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f86140c67c0 con 0x7f86140c4760 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f861af87640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86140c6aa0 con 0x7f86140c6140 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86140c4760 0x7f86140c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86140c4760 0x7f86140c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58242/0 (socket says 172.21.15.67:58242) 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 -- 172.21.15.67:0/4187061649 learned_addr learned my addr 172.21.15.67:0/4187061649 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8619524640 1 --2- 172.21.15.67:0/4187061649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f86140c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 -- 172.21.15.67:0/4187061649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 msgr2=0x7f86140c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 --2- 172.21.15.67:0/4187061649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f86140c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 -- 172.21.15.67:0/4187061649 >> v1:172.21.15.176:6789/0 conn(0x7f8614159520 legacy=0x7f8614073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 -- 172.21.15.67:0/4187061649 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86140c6d80 con 0x7f86140c4760 2024-04-25T15:51:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f8613fff640 1 --2- 172.21.15.67:0/4187061649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f86140c5c00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8608009870 tx=0x7f860801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24795d9da5fd50bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f86137fe640 1 -- 172.21.15.67:0/4187061649 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86080b5070 con 0x7f86140c4760 2024-04-25T15:51:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f86137fe640 1 -- 172.21.15.67:0/4187061649 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86080a89d0 con 0x7f86140c4760 2024-04-25T15:51:25.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.300+0000 7f86137fe640 1 -- 172.21.15.67:0/4187061649 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86080a8da0 con 0x7f86140c4760 2024-04-25T15:51:25.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- 172.21.15.67:0/4187061649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 msgr2=0x7f86140c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:25.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 --2- 172.21.15.67:0/4187061649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f86140c5c00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8608009870 tx=0x7f860801ed10 comp rx=0 tx=0).stop 2024-04-25T15:51:25.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- 172.21.15.67:0/4187061649 shutdown_connections 2024-04-25T15:51:25.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 --2- 172.21.15.67:0/4187061649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f86140c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 --2- 172.21.15.67:0/4187061649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f86140c5c00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- 172.21.15.67:0/4187061649 >> 172.21.15.67:0/4187061649 conn(0x7f861406fe70 msgr2=0x7f86140cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:25.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- 172.21.15.67:0/4187061649 shutdown_connections 2024-04-25T15:51:25.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- 172.21.15.67:0/4187061649 wait complete. 2024-04-25T15:51:25.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 Processor -- start 2024-04-25T15:51:25.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- start start 2024-04-25T15:51:25.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f8614148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f861414cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8614159520 0x7f861414ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86140c5400 con 0x7f86140c4760 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86140736b0 con 0x7f8614159520 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f861af87640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86140c5580 con 0x7f86140c6140 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f8618d23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f8614148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f8618d23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f8614148c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58250/0 (socket says 172.21.15.67:58250) 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.304+0000 7f8618d23640 1 -- 172.21.15.67:0/19396703 learned_addr learned my addr 172.21.15.67:0/19396703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8619524640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8614159520 0x7f861414ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8613fff640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f861414cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8618d23640 1 -- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 msgr2=0x7f861414cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8618d23640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f861414cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8618d23640 1 -- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8614159520 msgr2=0x7f861414ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8618d23640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8614159520 0x7f861414ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8618d23640 1 -- 172.21.15.67:0/19396703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8614149170 con 0x7f86140c4760 2024-04-25T15:51:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8618d23640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f8614148c30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f860400b3e0 tx=0x7f860400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f86117fa640 1 -- 172.21.15.67:0/19396703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604015020 con 0x7f86140c4760 2024-04-25T15:51:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f86117fa640 1 -- 172.21.15.67:0/19396703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8604011830 con 0x7f86140c4760 2024-04-25T15:51:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861414d510 con 0x7f86140c4760 2024-04-25T15:51:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f86117fa640 1 -- 172.21.15.67:0/19396703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604011c60 con 0x7f86140c4760 2024-04-25T15:51:25.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861414fdb0 con 0x7f86140c4760 2024-04-25T15:51:25.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f861406a480 con 0x7f86140c4760 2024-04-25T15:51:25.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f86117fa640 1 -- 172.21.15.67:0/19396703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8604018020 con 0x7f86140c4760 2024-04-25T15:51:25.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f86117fa640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f860006f430 0x7f8600071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f86117fa640 1 -- 172.21.15.67:0/19396703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5825+0+0 (secure 0 0 0) 0x7f860408ef50 con 0x7f86140c4760 2024-04-25T15:51:25.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.308+0000 7f8613fff640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f860006f430 0x7f8600071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.312+0000 7f8613fff640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f860006f430 0x7f8600071940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f86080b2000 tx=0x7f8608008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:25.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.316+0000 7f86117fa640 1 -- 172.21.15.67:0/19396703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f860401f030 con 0x7f86140c4760 2024-04-25T15:51:25.338 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:25.338 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:25.338 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:25.338 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:25.340 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:25.336+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:25.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:25.441 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:25.436+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:25.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:25.484+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f86140c59e0 con 0x7f86140c4760 2024-04-25T15:51:25.542 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:25.540+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:25.544 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:25.544 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:25.544 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:25.544 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:25.544 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:25.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:25.647 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:25.644+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:25.744 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:25.740+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.535550459, 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:51:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:51:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.535521195000001, 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:51:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:51:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.535511907, 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:51:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:51:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:51:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:51:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:51:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:51:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:51:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:51:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:51:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:51:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:51:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:51:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:51:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:51:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:51:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:51:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:51:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:51:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:51:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:51:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.535504330999999, 2024-04-25T15:51:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:51:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:51:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:51:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:51:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:51:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:51:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:51:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:51:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.535495697999998, 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:51:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:51:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:51:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.535445865, 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:51:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:51:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:51:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:51:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.535437987000002, 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:51:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.499009979999997, 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:51:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.498979316000003, 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:51:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:51:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:51:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.498966785999997, 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:51:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:51:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.498956153999998, 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:51:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.498936608999998, 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:51:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:51:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:51:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.498927717000001, 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:51:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.495365100999997, 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:51:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 43.495244489000001, 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:51:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 27.860246469, 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:51:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.819677071000001, 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:51:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:51:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:51:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.819671563, 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:51:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:51:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:51:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.817257048999998, 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:51:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.816295162999999, 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:51:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:51:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:25.859 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:25.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:25.860 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:25.861 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:25.856+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:25.961 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:25.956+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:26.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc62279640 1 Processor -- start 2024-04-25T15:51:26.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc62279640 1 -- start start 2024-04-25T15:51:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc62279640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc5c0736d0 0x7fbc5c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc62279640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc5c0caa60 con 0x7fbc5c06cb80 2024-04-25T15:51:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc62279640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc5c15a5f0 con 0x7fbc5c06d460 2024-04-25T15:51:26.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc62279640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbc5c068590 con 0x7fbc5c0736d0 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc5b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc5c0736d0 0x7fbc5c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc60816640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbc5c06cb80 0x7fbc5c15aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41694/0 (socket says 172.21.15.67:41694) 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc5b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc5c0736d0 0x7fbc5c06c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58254/0 (socket says 172.21.15.67:58254) 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc60816640 1 -- 172.21.15.67:0/3738531623 learned_addr learned my addr 172.21.15.67:0/3738531623 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc5b7fe640 1 -- 172.21.15.67:0/3738531623 >> v1:172.21.15.176:6789/0 conn(0x7fbc5c06d460 legacy=0x7fbc5c1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc5b7fe640 1 -- 172.21.15.67:0/3738531623 >> v1:172.21.15.67:6789/0 conn(0x7fbc5c06cb80 legacy=0x7fbc5c15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc5affd640 1 -- 172.21.15.67:0/3738531623 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4113560778 0 0) 0x7fbc5c0caa60 con 0x7fbc5c06cb80 2024-04-25T15:51:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.016+0000 7fbc5b7fe640 1 -- 172.21.15.67:0/3738531623 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c0ca100 con 0x7fbc5c0736d0 2024-04-25T15:51:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/3738531623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c0736d0 0x7fbc5c06c640 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097a0 tx=0x7fbc5001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e787c1a3695481 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc5affd640 1 -- 172.21.15.67:0/3738531623 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc500b0070 con 0x7fbc5c0736d0 2024-04-25T15:51:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc5affd640 1 -- 172.21.15.67:0/3738531623 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc500ab600 con 0x7fbc5c0736d0 2024-04-25T15:51:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc5affd640 1 -- 172.21.15.67:0/3738531623 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc500ab9d0 con 0x7fbc5c0736d0 2024-04-25T15:51:26.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc62279640 1 -- 172.21.15.67:0/3738531623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c0736d0 msgr2=0x7fbc5c06c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc62279640 1 --2- 172.21.15.67:0/3738531623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c0736d0 0x7fbc5c06c640 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097a0 tx=0x7fbc5001c480 comp rx=0 tx=0).stop 2024-04-25T15:51:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc62279640 1 -- 172.21.15.67:0/3738531623 shutdown_connections 2024-04-25T15:51:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc62279640 1 --2- 172.21.15.67:0/3738531623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c0736d0 0x7fbc5c06c640 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc62279640 1 -- 172.21.15.67:0/3738531623 >> 172.21.15.67:0/3738531623 conn(0x7fbc5c0ce1c0 msgr2=0x7fbc5c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc62279640 1 -- 172.21.15.67:0/3738531623 shutdown_connections 2024-04-25T15:51:26.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.020+0000 7fbc62279640 1 -- 172.21.15.67:0/3738531623 wait complete. 2024-04-25T15:51:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 Processor -- start 2024-04-25T15:51:26.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 -- start start 2024-04-25T15:51:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c06cb80 0x7fbc5c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c06d460 0x7fbc5c146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc5c0736d0 0x7fbc5c1f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c168920 con 0x7fbc5c06cb80 2024-04-25T15:51:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c1687a0 con 0x7fbc5c06d460 2024-04-25T15:51:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbc5c168aa0 con 0x7fbc5c0736d0 2024-04-25T15:51:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c06cb80 0x7fbc5c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c06cb80 0x7fbc5c0c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58268/0 (socket says 172.21.15.67:58268) 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 -- 172.21.15.67:0/572131586 learned_addr learned my addr 172.21.15.67:0/572131586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc60816640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc5c0736d0 0x7fbc5c1f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c06d460 0x7fbc5c146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc5c0736d0 msgr2=0x7fbc5c1f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbc5c0736d0 0x7fbc5c1f4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c06d460 msgr2=0x7fbc5c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc5c06d460 0x7fbc5c146700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c1f7380 con 0x7fbc5c06cb80 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c06cb80 0x7fbc5c0c5850 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c370 tx=0x7fbc4c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c017070 con 0x7fbc5c06cb80 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4c0052d0 con 0x7fbc5c06cb80 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c003820 con 0x7fbc5c06cb80 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c1f7660 con 0x7fbc5c06cb80 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c1f7c20 con 0x7fbc5c06cb80 2024-04-25T15:51:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.024+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbc5c14c3c0 con 0x7fbc5c06cb80 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbc4c01a070 con 0x7fbc5c06cb80 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc58ff9640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc4806f430 0x7fbc48071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5825+0+0 (secure 0 0 0) 0x7fbc4c08e9e0 con 0x7fbc5c06cb80 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v25) ==== 54+0+16 (secure 0 0 0) 0x7fbc4c021020 con 0x7fbc5c06cb80 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc4806f430 0x7fbc48071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbc5c0c7e00 0x7fbc5c14e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c06a480 con 0x7fbc5c0c7e00 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc60816640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbc5c0c7e00 0x7fbc5c14e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc60816640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbc5c0c7e00 0x7fbc5c14e400 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc4806f430 0x7fbc48071940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc500ad000 tx=0x7fbc50008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.028+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc5c06a480 con 0x7fbc5c0c7e00 2024-04-25T15:51:26.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.044+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c159e90 con 0x7fbc5c0c7e00 2024-04-25T15:51:26.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.044+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc5c159e90 con 0x7fbc5c0c7e00 2024-04-25T15:51:26.050 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:26.050 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:26.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.048+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbc5c0c5d90 0x7fbc5c14ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.048+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c075e00 con 0x7fbc5c0c5d90 2024-04-25T15:51:26.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.048+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbc5c0c5d90 0x7fbc5c14ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.048+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbc5c0c5d90 0x7fbc5c14ef20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.048+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc5c075e00 con 0x7fbc5c0c5d90 2024-04-25T15:51:26.060 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:26.060 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:26.060 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:26.060 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:26.061 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:26.056+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:26.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.060+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c0c7840 con 0x7fbc5c0c5d90 2024-04-25T15:51:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.064+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc5c0c7840 con 0x7fbc5c0c5d90 2024-04-25T15:51:26.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.064+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbc5c063090 0x7fbc5c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.064+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c14fd30 con 0x7fbc5c063090 2024-04-25T15:51:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.064+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbc5c063090 0x7fbc5c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.064+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbc5c063090 0x7fbc5c0634a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.064+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc5c14fd30 con 0x7fbc5c063090 2024-04-25T15:51:26.077 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:26.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.076+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c063cb0 con 0x7fbc5c063090 2024-04-25T15:51:26.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.080+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc5c063cb0 con 0x7fbc5c063090 2024-04-25T15:51:26.083 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:26.084 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:26.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.080+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbc5c0cd150 0x7fbc5c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.080+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c0cdaa0 con 0x7fbc5c0cd150 2024-04-25T15:51:26.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.080+0000 7fbc60816640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbc5c0cd150 0x7fbc5c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.080+0000 7fbc60816640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbc5c0cd150 0x7fbc5c0cd560 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.080+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc5c0cdaa0 con 0x7fbc5c0cd150 2024-04-25T15:51:26.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f86117fa640 1 -- 172.21.15.67:0/19396703 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v26) ==== 101+0+0 (secure 0 0 0) 0x7f860405be50 con 0x7f86140c4760 2024-04-25T15:51:26.094 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:51:26.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f860006f430 msgr2=0x7f8600071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f860006f430 0x7f8600071940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f86080b2000 tx=0x7f8608008040 comp rx=0 tx=0).stop 2024-04-25T15:51:26.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 msgr2=0x7f8614148c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f8614148c30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f860400b3e0 tx=0x7f860400b8a0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 shutdown_connections 2024-04-25T15:51:26.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f860006f430 0x7f8600071940 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8614159520 0x7f861414ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86140c6140 0x7f861414cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 --2- 172.21.15.67:0/19396703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86140c4760 0x7f8614148c30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 >> 172.21.15.67:0/19396703 conn(0x7f861406fe70 msgr2=0x7f8614152770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:26.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 shutdown_connections 2024-04-25T15:51:26.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.092+0000 7f861af87640 1 -- 172.21.15.67:0/19396703 wait complete. 2024-04-25T15:51:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.096+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c02ccb0 con 0x7fbc5c0cd150 2024-04-25T15:51:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.100+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc5c02ccb0 con 0x7fbc5c0cd150 2024-04-25T15:51:26.104 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:26.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.100+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc5c02d3d0 0x7fbc5c02d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.100+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c02cc30 con 0x7fbc5c02d3d0 2024-04-25T15:51:26.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.100+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc5c02d3d0 0x7fbc5c02d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.100+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc5c02d3d0 0x7fbc5c02d7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.104+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc5c02cc30 con 0x7fbc5c02d3d0 2024-04-25T15:51:26.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.116+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c02c9e0 con 0x7fbc5c02d3d0 2024-04-25T15:51:26.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.120+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc4c08a020 con 0x7fbc5c02d3d0 2024-04-25T15:51:26.123 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.120+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbc5c208ab0 0x7fbc5c20aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.120+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c02cd00 con 0x7fbc5c208ab0 2024-04-25T15:51:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.120+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbc5c208ab0 0x7fbc5c20aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.120+0000 7fbc5b7fe640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbc5c208ab0 0x7fbc5c20aed0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.124+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc500b1040 con 0x7fbc5c208ab0 2024-04-25T15:51:26.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.136+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c20b410 con 0x7fbc5c208ab0 2024-04-25T15:51:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.140+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc500b1020 con 0x7fbc5c208ab0 2024-04-25T15:51:26.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.140+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbc5c20bc40 0x7fbc5c20e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.140+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c210e30 con 0x7fbc5c20bc40 2024-04-25T15:51:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.140+0000 7fbc60816640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbc5c20bc40 0x7fbc5c20e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.140+0000 7fbc60816640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbc5c20bc40 0x7fbc5c20e060 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.140+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc5c210e30 con 0x7fbc5c20bc40 2024-04-25T15:51:26.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.156+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c20e5a0 con 0x7fbc5c20bc40 2024-04-25T15:51:26.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.156+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc5c20e5a0 con 0x7fbc5c20bc40 2024-04-25T15:51:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.156+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbc5c20ec20 0x7fbc5c211040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.156+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbc5c213e10 con 0x7fbc5c20ec20 2024-04-25T15:51:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.160+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbc5c20ec20 0x7fbc5c211040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.160+0000 7fbc5bfff640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbc5c20ec20 0x7fbc5c211040 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.160+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc5c213e10 con 0x7fbc5c20ec20 2024-04-25T15:51:26.167 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:26.164+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:26.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.172+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbc5c211580 con 0x7fbc5c20ec20 2024-04-25T15:51:26.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc58ff9640 1 -- 172.21.15.67:0/572131586 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbc5c211580 con 0x7fbc5c20ec20 2024-04-25T15:51:26.180 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbc5c0c7e00 msgr2=0x7fbc5c14e400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbc5c0c7e00 0x7fbc5c14e400 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbc5c0c5d90 msgr2=0x7fbc5c14ef20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbc5c0c5d90 0x7fbc5c14ef20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbc5c063090 msgr2=0x7fbc5c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbc5c063090 0x7fbc5c0634a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbc5c0cd150 msgr2=0x7fbc5c0cd560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbc5c0cd150 0x7fbc5c0cd560 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc5c02d3d0 msgr2=0x7fbc5c02d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc5c02d3d0 0x7fbc5c02d7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbc5c208ab0 msgr2=0x7fbc5c20aed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc60816640 1 -- 172.21.15.67:0/572131586 reap_dead start 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbc5c208ab0 0x7fbc5c20aed0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc60816640 1 -- 172.21.15.67:0/572131586 reap_dead start 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc60816640 1 -- 172.21.15.67:0/572131586 reap_dead start 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbc5c20bc40 msgr2=0x7fbc5c20e060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbc5c20bc40 0x7fbc5c20e060 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbc5c20ec20 msgr2=0x7fbc5c211040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbc5c20ec20 0x7fbc5c211040 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc4806f430 msgr2=0x7fbc48071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc4806f430 0x7fbc48071940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc500ad000 tx=0x7fbc50008040 comp rx=0 tx=0).stop 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c06cb80 msgr2=0x7fbc5c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 --2- 172.21.15.67:0/572131586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc5c06cb80 0x7fbc5c0c5850 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c370 tx=0x7fbc4c00c830 comp rx=0 tx=0).stop 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc60816640 1 -- 172.21.15.67:0/572131586 reap_dead start 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 shutdown_connections 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 >> 172.21.15.67:0/572131586 conn(0x7fbc5c0ce1c0 msgr2=0x7fbc5c072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 shutdown_connections 2024-04-25T15:51:26.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.176+0000 7fbc62279640 1 -- 172.21.15.67:0/572131586 wait complete. 2024-04-25T15:51:26.262 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:26.256+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:26.275 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:26.275 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:26.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:26.379 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:26.376+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:26.464 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:26.460+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:26.484 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:26.484 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:26.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:26.565 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:26.560+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:26.662 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:26.662 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:26.666 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:26.664+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:26.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:26.753 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:26.753 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:26.753 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:26.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-04-25T15:51:26.766 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:26.764+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:26.867 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:26.864+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:26.873 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:26.874 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:26.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.908+0000 7f6e9332e640 1 Processor -- start 2024-04-25T15:51:26.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.908+0000 7f6e9332e640 1 -- start start 2024-04-25T15:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e9332e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e9332e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e8c0c4ec0 0x7f6e8c0c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e9332e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8c0c7340 con 0x7f6e8c073bd0 2024-04-25T15:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e9332e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6e8c0c5810 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e9332e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6e8c0c67c0 con 0x7f6e8c159520 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e8c0c4ec0 0x7f6e8c0c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e8c0c4ec0 0x7f6e8c0c52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58276/0 (socket says 172.21.15.67:58276) 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 -- 172.21.15.67:0/4045938296 learned_addr learned my addr 172.21.15.67:0/4045938296 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e910ca640 1 --2- 172.21.15.67:0/4045938296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 -- 172.21.15.67:0/4045938296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 msgr2=0x7f6e8c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 --2- 172.21.15.67:0/4045938296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 -- 172.21.15.67:0/4045938296 >> v1:172.21.15.176:6789/0 conn(0x7f6e8c073bd0 legacy=0x7f6e8c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 -- 172.21.15.67:0/4045938296 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8c0c6aa0 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e918cb640 1 --2- 172.21.15.67:0/4045938296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c52d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800d150 tx=0x7f6e880213e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a62927bd17656a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e83fff640 1 -- 172.21.15.67:0/4045938296 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b6070 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e83fff640 1 -- 172.21.15.67:0/4045938296 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8800b2d0 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.912+0000 7f6e83fff640 1 -- 172.21.15.67:0/4045938296 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800b480 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- 172.21.15.67:0/4045938296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 msgr2=0x7f6e8c0c52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 --2- 172.21.15.67:0/4045938296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c52d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800d150 tx=0x7f6e880213e0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- 172.21.15.67:0/4045938296 shutdown_connections 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 --2- 172.21.15.67:0/4045938296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c52d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 --2- 172.21.15.67:0/4045938296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- 172.21.15.67:0/4045938296 >> 172.21.15.67:0/4045938296 conn(0x7f6e8c06fe70 msgr2=0x7f6e8c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- 172.21.15.67:0/4045938296 shutdown_connections 2024-04-25T15:51:26.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- 172.21.15.67:0/4045938296 wait complete. 2024-04-25T15:51:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 Processor -- start 2024-04-25T15:51:26.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- start start 2024-04-25T15:51:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e8c073bd0 0x7f6e8c14f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c0736b0 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c0c7060 con 0x7f6e8c073bd0 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e9332e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6e8c0c6310 con 0x7f6e8c159520 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e908c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e908c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58278/0 (socket says 172.21.15.67:58278) 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e908c9640 1 -- 172.21.15.67:0/1423867246 learned_addr learned my addr 172.21.15.67:0/1423867246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e910ca640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e8c073bd0 0x7f6e8c14f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.916+0000 7f6e918cb640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e908c9640 1 -- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 msgr2=0x7f6e8c0cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e908c9640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e908c9640 1 -- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e8c073bd0 msgr2=0x7f6e8c14f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:26.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e908c9640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e8c073bd0 0x7f6e8c14f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e908c9640 1 -- 172.21.15.67:0/1423867246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8c14fcb0 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e908c9640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c91e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00b3e0 tx=0x7f6e7c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e81ffb640 1 -- 172.21.15.67:0/1423867246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c015020 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e81ffb640 1 -- 172.21.15.67:0/1423867246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7c011830 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e8c1f1f80 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e81ffb640 1 -- 172.21.15.67:0/1423867246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c011c60 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e8c1f2520 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e8c06a480 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e81ffb640 1 -- 172.21.15.67:0/1423867246 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6e7c018020 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e81ffb640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e7006f400 0x7f6e70071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:26.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e81ffb640 1 -- 172.21.15.67:0/1423867246 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5825+0+0 (secure 0 0 0) 0x7f6e7c08f500 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.920+0000 7f6e910ca640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e7006f400 0x7f6e70071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:26.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.924+0000 7f6e910ca640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e7006f400 0x7f6e70071910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c15db50 tx=0x7f6e84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:26.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:26.924+0000 7f6e81ffb640 1 -- 172.21.15.67:0/1423867246 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6e7c01f020 con 0x7f6e8c0c4ec0 2024-04-25T15:51:26.968 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:26.964+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:27.055 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:27.055 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:27.070 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:27.068+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:27.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:27.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:27.100+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd out", "ids": ["1"]} v 0) -- 0x7f6e8c0c9bc0 con 0x7f6e8c0c4ec0 2024-04-25T15:51:27.104 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:51:27.100+0000 7fa4fdaf0640 -1 mon.a@0(leader).osd e26 definitely_dead 0 2024-04-25T15:51:27.171 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:27.164+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:27.250 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:27.251 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:27.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:27.271 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:27.268+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:27.372 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:27.368+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:27.416 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:27.416 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:27.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:27.478 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:27.476+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:27.574 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:27.568+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:27.582 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:27.582 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:27.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:27.686 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:27.684+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:27.777 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:27.772+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:27.789 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:27.789 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:27.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:27.898 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:27.896+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:27.978 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:27.976+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:28.008 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:28.009 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:28.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:28.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.100+0000 7f6e81ffb640 1 -- 172.21.15.67:0/1423867246 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["1"]}]=0 marked out osd.1. v27) ==== 87+0+0 (secure 0 0 0) 0x7f6e7c05bf20 con 0x7f6e8c0c4ec0 2024-04-25T15:51:28.106 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.1. 2024-04-25T15:51:28.106 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e7006f400 msgr2=0x7f6e70071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e7006f400 0x7f6e70071910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c15db50 tx=0x7f6e84008040 comp rx=0 tx=0).stop 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 msgr2=0x7f6e8c0c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c91e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00b3e0 tx=0x7f6e7c00b8a0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 shutdown_connections 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6e7006f400 0x7f6e70071910 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e8c159520 0x7f6e8c0cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e8c0c4ec0 0x7f6e8c0c91e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 --2- 172.21.15.67:0/1423867246 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e8c073bd0 0x7f6e8c14f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 >> 172.21.15.67:0/1423867246 conn(0x7f6e8c06fe70 msgr2=0x7f6e8c152910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 shutdown_connections 2024-04-25T15:51:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.104+0000 7f6e9332e640 1 -- 172.21.15.67:0/1423867246 wait complete. 2024-04-25T15:51:28.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:51:28.179 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:28.176+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:28.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:28.235 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:28.235 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:28.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:28.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d6563c640 1 Processor -- start 2024-04-25T15:51:28.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d6563c640 1 -- start start 2024-04-25T15:51:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d6563c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d6006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d6563c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d600c5100 0x7f3d60163630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d6563c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6006fa10 con 0x7f3d6006cb80 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d6563c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d600ca6c0 con 0x7f3d600c5100 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d6563c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d6006cf90 con 0x7f3d60156330 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d600c5100 0x7f3d60163630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d600c5100 0x7f3d60163630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58294/0 (socket says 172.21.15.67:58294) 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5f7fe640 1 -- 172.21.15.67:0/4245299232 learned_addr learned my addr 172.21.15.67:0/4245299232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5effd640 1 --2- 172.21.15.67:0/4245299232 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d6006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5f7fe640 1 -- 172.21.15.67:0/4245299232 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 msgr2=0x7f3d6006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5f7fe640 1 --2- 172.21.15.67:0/4245299232 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d6006c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5f7fe640 1 -- 172.21.15.67:0/4245299232 >> v1:172.21.15.176:6789/0 conn(0x7f3d6006cb80 legacy=0x7f3d600c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5f7fe640 1 -- 172.21.15.67:0/4245299232 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d600ca8f0 con 0x7f3d600c5100 2024-04-25T15:51:28.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.272+0000 7f3d5effd640 1 --2- 172.21.15.67:0/4245299232 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d6006c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d5f7fe640 1 --2- 172.21.15.67:0/4245299232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d600c5100 0x7f3d60163630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800bb10 tx=0x7f3d58021420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b198bd8c7f4904e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d5dffb640 1 -- 172.21.15.67:0/4245299232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d580b6070 con 0x7f3d600c5100 2024-04-25T15:51:28.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d5dffb640 1 -- 172.21.15.67:0/4245299232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d580b1600 con 0x7f3d600c5100 2024-04-25T15:51:28.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d5dffb640 1 -- 172.21.15.67:0/4245299232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d580b19d0 con 0x7f3d600c5100 2024-04-25T15:51:28.280 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:28.276+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 -- 172.21.15.67:0/4245299232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d600c5100 msgr2=0x7f3d60163630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4245299232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d600c5100 0x7f3d60163630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800bb10 tx=0x7f3d58021420 comp rx=0 tx=0).stop 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 -- 172.21.15.67:0/4245299232 shutdown_connections 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4245299232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d600c5100 0x7f3d60163630 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4245299232 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d6006c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 -- 172.21.15.67:0/4245299232 >> 172.21.15.67:0/4245299232 conn(0x7f3d600ce1c0 msgr2=0x7f3d600ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 -- 172.21.15.67:0/4245299232 shutdown_connections 2024-04-25T15:51:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.276+0000 7f3d6563c640 1 -- 172.21.15.67:0/4245299232 wait complete. 2024-04-25T15:51:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.280+0000 7f3d6563c640 1 Processor -- start 2024-04-25T15:51:28.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.280+0000 7f3d6563c640 1 -- start start 2024-04-25T15:51:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d6006cb80 0x7f3d6015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d600c5100 0x7f3d60153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d601499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d600c3d80 con 0x7f3d6006cb80 2024-04-25T15:51:28.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d600c43d0 con 0x7f3d600c5100 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d600c4060 con 0x7f3d60156330 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d6006cb80 0x7f3d6015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d6006cb80 0x7f3d6015b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58296/0 (socket says 172.21.15.67:58296) 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 -- 172.21.15.67:0/4171322175 learned_addr learned my addr 172.21.15.67:0/4171322175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5f7fe640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d601499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5e7fc640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d600c5100 0x7f3d60153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 -- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 msgr2=0x7f3d601499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d601499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 -- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d600c5100 msgr2=0x7f3d60153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d600c5100 0x7f3d60153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 -- 172.21.15.67:0/4171322175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6014f330 con 0x7f3d6006cb80 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5f7fe640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d601499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d5effd640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d6006cb80 0x7f3d6015b320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3d48007f20 tx=0x7f3d4800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d577fe640 1 -- 172.21.15.67:0/4171322175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4801a070 con 0x7f3d6006cb80 2024-04-25T15:51:28.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d577fe640 1 -- 172.21.15.67:0/4171322175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d48003ba0 con 0x7f3d6006cb80 2024-04-25T15:51:28.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6015a0e0 con 0x7f3d6006cb80 2024-04-25T15:51:28.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d577fe640 1 -- 172.21.15.67:0/4171322175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48004030 con 0x7f3d6006cb80 2024-04-25T15:51:28.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6015a6a0 con 0x7f3d6006cb80 2024-04-25T15:51:28.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d6006a480 con 0x7f3d6006cb80 2024-04-25T15:51:28.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.284+0000 7f3d577fe640 1 -- 172.21.15.67:0/4171322175 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3d4801d070 con 0x7f3d6006cb80 2024-04-25T15:51:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.288+0000 7f3d577fe640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d4006f430 0x7f3d40071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.288+0000 7f3d577fe640 1 -- 172.21.15.67:0/4171322175 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5796+0+0 (secure 0 0 0) 0x7f3d4808f420 con 0x7f3d6006cb80 2024-04-25T15:51:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.288+0000 7f3d5e7fc640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d4006f430 0x7f3d40071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.288+0000 7f3d5e7fc640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d4006f430 0x7f3d40071940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3d50007fd0 tx=0x7f3d50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.292+0000 7f3d577fe640 1 -- 172.21.15.67:0/4171322175 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3d48010070 con 0x7f3d6006cb80 2024-04-25T15:51:28.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.344+0000 7f9a05516640 1 Processor -- start 2024-04-25T15:51:28.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.344+0000 7f9a05516640 1 -- start start 2024-04-25T15:51:28.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.344+0000 7f9a05516640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a00160800 0x7f9a0006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.344+0000 7f9a05516640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0006d8e0 con 0x7f9a000c92d0 2024-04-25T15:51:28.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.344+0000 7f9a05516640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a00073c70 con 0x7f9a000c96e0 2024-04-25T15:51:28.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.344+0000 7f9a05516640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9a0006dbc0 con 0x7f9a00160800 2024-04-25T15:51:28.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99fe7fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9a000c92d0 0x7f9a00072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41698/0 (socket says 172.21.15.67:41698) 2024-04-25T15:51:28.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99feffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a00160800 0x7f9a0006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99fe7fc640 1 -- 172.21.15.67:0/1985643004 learned_addr learned my addr 172.21.15.67:0/1985643004 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:28.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99feffd640 1 -- 172.21.15.67:0/1985643004 >> v1:172.21.15.176:6789/0 conn(0x7f9a000c96e0 legacy=0x7f9a00167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99feffd640 1 -- 172.21.15.67:0/1985643004 >> v1:172.21.15.67:6789/0 conn(0x7f9a000c92d0 legacy=0x7f9a00072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99feffd640 1 -- 172.21.15.67:0/1985643004 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a00073f50 con 0x7f9a00160800 2024-04-25T15:51:28.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99fdffb640 1 -- 172.21.15.67:0/1985643004 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861011099 0 0) 0x7f9a0006d8e0 con 0x7f9a000c92d0 2024-04-25T15:51:28.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99feffd640 1 --2- 172.21.15.67:0/1985643004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a00160800 0x7f9a0006d030 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097c0 tx=0x7f99f001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ae1b6ba45c9da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99fdffb640 1 -- 172.21.15.67:0/1985643004 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00b0070 con 0x7f9a00160800 2024-04-25T15:51:28.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99fdffb640 1 -- 172.21.15.67:0/1985643004 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f00ab600 con 0x7f9a00160800 2024-04-25T15:51:28.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.348+0000 7f99fdffb640 1 -- 172.21.15.67:0/1985643004 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00ab9d0 con 0x7f9a00160800 2024-04-25T15:51:28.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- 172.21.15.67:0/1985643004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a00160800 msgr2=0x7f9a0006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 --2- 172.21.15.67:0/1985643004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a00160800 0x7f9a0006d030 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097c0 tx=0x7f99f001c410 comp rx=0 tx=0).stop 2024-04-25T15:51:28.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- 172.21.15.67:0/1985643004 shutdown_connections 2024-04-25T15:51:28.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 --2- 172.21.15.67:0/1985643004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a00160800 0x7f9a0006d030 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- 172.21.15.67:0/1985643004 >> 172.21.15.67:0/1985643004 conn(0x7f9a00070db0 msgr2=0x7f9a000ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:28.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- 172.21.15.67:0/1985643004 shutdown_connections 2024-04-25T15:51:28.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- 172.21.15.67:0/1985643004 wait complete. 2024-04-25T15:51:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 Processor -- start 2024-04-25T15:51:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- start start 2024-04-25T15:51:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a000c92d0 0x7f9a001f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a000c96e0 0x7f9a001f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a00160800 0x7f9a0014c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9a00160c10 con 0x7f9a000c96e0 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0006d570 con 0x7f9a000c92d0 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f9a05516640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a00160d90 con 0x7f9a00160800 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f99fe7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a000c96e0 0x7f9a001f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f99fe7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a000c96e0 0x7f9a001f9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58312/0 (socket says 172.21.15.67:58312) 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f99fe7fc640 1 -- 172.21.15.67:0/3199911936 learned_addr learned my addr 172.21.15.67:0/3199911936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f99ff7fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a00160800 0x7f9a0014c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.352+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a000c92d0 0x7f9a001f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99fe7fc640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a00160800 msgr2=0x7f9a0014c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99fe7fc640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a00160800 0x7f9a0014c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99fe7fc640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a000c92d0 msgr2=0x7f9a001f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99fe7fc640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a000c92d0 0x7f9a001f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99fe7fc640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a00153be0 con 0x7f9a000c96e0 2024-04-25T15:51:28.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99fe7fc640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a000c96e0 0x7f9a001f9ad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f99f400c410 tx=0x7f99f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f40074e0 con 0x7f9a000c96e0 2024-04-25T15:51:28.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f40047f0 con 0x7f9a000c96e0 2024-04-25T15:51:28.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a00153ec0 con 0x7f9a000c96e0 2024-04-25T15:51:28.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4007680 con 0x7f9a000c96e0 2024-04-25T15:51:28.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a0015b850 con 0x7f9a000c96e0 2024-04-25T15:51:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9a0006a480 con 0x7f9a000c96e0 2024-04-25T15:51:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f99f4011070 con 0x7f9a000c96e0 2024-04-25T15:51:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99e77fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f99ec06f400 0x7f99ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5796+0+0 (secure 0 0 0) 0x7f99f408ee80 con 0x7f9a000c96e0 2024-04-25T15:51:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v27) ==== 54+0+16 (secure 0 0 0) 0x7f99f4091500 con 0x7f9a000c96e0 2024-04-25T15:51:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f99ec06f400 0x7f99ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a0015ec40 0x7f9a0015f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.356+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a001fa010 con 0x7f9a0015ec40 2024-04-25T15:51:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.360+0000 7f99ff7fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a0015ec40 0x7f9a0015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.360+0000 7f99ff7fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a0015ec40 0x7f9a0015f050 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.360+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f99ec06f400 0x7f99ec071910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f99f00ad000 tx=0x7f99f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.360+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f99f8001de0 con 0x7f9a0015ec40 2024-04-25T15:51:28.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.376+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a000c8c50 con 0x7f9a0015ec40 2024-04-25T15:51:28.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.376+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a000c8c50 con 0x7f9a0015ec40 2024-04-25T15:51:28.380 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.376+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a001520e0 0x7f9a001524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.376+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9a00152a30 con 0x7f9a001520e0 2024-04-25T15:51:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.376+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a001520e0 0x7f9a001524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.376+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a001520e0 0x7f9a001524f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.381 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:28.376+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:28.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.380+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a00152a30 con 0x7f9a001520e0 2024-04-25T15:51:28.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.392+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a00073920 con 0x7f9a001520e0 2024-04-25T15:51:28.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.396+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a00073920 con 0x7f9a001520e0 2024-04-25T15:51:28.398 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.396+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a000635a0 0x7f9a000639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.396+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9a00153900 con 0x7f9a000635a0 2024-04-25T15:51:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.396+0000 7f99fe7fc640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a000635a0 0x7f9a000639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.396+0000 7f99fe7fc640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a000635a0 0x7f9a000639b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.396+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a00153900 con 0x7f9a000635a0 2024-04-25T15:51:28.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.408+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a0014cb90 con 0x7f9a000635a0 2024-04-25T15:51:28.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.412+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a0014cb90 con 0x7f9a000635a0 2024-04-25T15:51:28.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.412+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a000c69d0 0x7f9a000c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.412+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9a001535c0 con 0x7f9a000c69d0 2024-04-25T15:51:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.412+0000 7f99ff7fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a000c69d0 0x7f9a000c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.412+0000 7f99ff7fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a000c69d0 0x7f9a000c6de0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.412+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a001535c0 con 0x7f9a000c69d0 2024-04-25T15:51:28.424 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:28.424 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.428+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a0002c9e0 con 0x7f9a000c69d0 2024-04-25T15:51:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.432+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a0002c9e0 con 0x7f9a000c69d0 2024-04-25T15:51:28.435 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.432+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a0002d060 0x7f9a0002d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.432+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9a0002d9b0 con 0x7f9a0002d060 2024-04-25T15:51:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.432+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a0002d060 0x7f9a0002d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.432+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a0002d060 0x7f9a0002d470 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.436+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a0002d9b0 con 0x7f9a0002d060 2024-04-25T15:51:28.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:28.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.448+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a0002d9b0 con 0x7f9a0002d060 2024-04-25T15:51:28.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.452+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a0002d9b0 con 0x7f9a0002d060 2024-04-25T15:51:28.457 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.452+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a0020a460 0x7f9a0020c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.452+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9a0020f650 con 0x7f9a0020a460 2024-04-25T15:51:28.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.452+0000 7f99fe7fc640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a0020a460 0x7f9a0020c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.456+0000 7f99fe7fc640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a0020a460 0x7f9a0020c880 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.456+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a0020f650 con 0x7f9a0020a460 2024-04-25T15:51:28.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.472+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a0020cdc0 con 0x7f9a0020a460 2024-04-25T15:51:28.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.476+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f3d601567d0 con 0x7f3d6006cb80 2024-04-25T15:51:28.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.476+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a0020cdc0 con 0x7f9a0020a460 2024-04-25T15:51:28.480 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:28.480 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:28.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.476+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a0020d440 0x7f9a0020f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.476+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9a002127c0 con 0x7f9a0020d440 2024-04-25T15:51:28.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.476+0000 7f99ff7fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a0020d440 0x7f9a0020f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.476+0000 7f99ff7fe640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a0020d440 0x7f9a0020f860 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.482 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:28.476+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:28.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.480+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a002127c0 con 0x7f9a0020d440 2024-04-25T15:51:28.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:51:28.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:51:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.508+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a002114e0 con 0x7f9a0020d440 2024-04-25T15:51:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a002114e0 con 0x7f9a0020d440 2024-04-25T15:51:28.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 0x7f9a00212010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9a00215b90 con 0x7f9a00211c00 2024-04-25T15:51:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 0x7f9a00212010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f99feffd640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 msgr2=0x7f9a00212010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 28 2024-04-25T15:51:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f99feffd640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 msgr2=0x7f9a00212010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:51:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 0x7f9a00212010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:51:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.516+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 0x7f9a00212010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:51:28.583 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:28.580+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:28.605 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:28.605 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:28.606 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:51:28.606 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:51:28.606 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:28.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:28.684 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:28.680+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:28.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.720+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 0x7f9a00212010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.720+0000 7f99feffd640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 0x7f9a00212010 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.720+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a00215b90 con 0x7f9a00211c00 2024-04-25T15:51:28.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.736+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a00214a10 con 0x7f9a00211c00 2024-04-25T15:51:28.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f99e77fe640 1 -- 172.21.15.67:0/3199911936 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a00214a10 con 0x7f9a00211c00 2024-04-25T15:51:28.744 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:28.745 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:28.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a0015ec40 msgr2=0x7f9a0015f050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a0015ec40 0x7f9a0015f050 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a001520e0 msgr2=0x7f9a001524f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a001520e0 0x7f9a001524f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a000635a0 msgr2=0x7f9a000639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a000635a0 0x7f9a000639b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a000c69d0 msgr2=0x7f9a000c6de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a000c69d0 0x7f9a000c6de0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a0002d060 msgr2=0x7f9a0002d470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a0002d060 0x7f9a0002d470 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a0020a460 msgr2=0x7f9a0020c880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f99ff7fe640 1 -- 172.21.15.67:0/3199911936 reap_dead start 2024-04-25T15:51:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a0020a460 0x7f9a0020c880 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f99ff7fe640 1 -- 172.21.15.67:0/3199911936 reap_dead start 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a0020d440 msgr2=0x7f9a0020f860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a0020d440 0x7f9a0020f860 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 msgr2=0x7f9a00212010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a00211c00 0x7f9a00212010 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f99ff7fe640 1 -- 172.21.15.67:0/3199911936 reap_dead start 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f99ec06f400 msgr2=0x7f99ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f99ec06f400 0x7f99ec071910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f99f00ad000 tx=0x7f99f0008040 comp rx=0 tx=0).stop 2024-04-25T15:51:28.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a000c96e0 msgr2=0x7f9a001f9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a000c96e0 0x7f9a001f9ad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f99f400c410 tx=0x7f99f400c8d0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.740+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 shutdown_connections 2024-04-25T15:51:28.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.744+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f99ec06f400 0x7f99ec071910 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.744+0000 7f9a05516640 1 --2- 172.21.15.67:0/3199911936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a000c96e0 0x7f9a001f9ad0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.744+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 >> 172.21.15.67:0/3199911936 conn(0x7f9a00070db0 msgr2=0x7f9a00072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:28.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.744+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 shutdown_connections 2024-04-25T15:51:28.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:28.744+0000 7f9a05516640 1 -- 172.21.15.67:0/3199911936 wait complete. 2024-04-25T15:51:28.785 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:28.780+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:28.820 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.605635268999997, 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.605595624000003, 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.602688602000001, 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.602535666000001, 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.601020022, 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.600877375000003, 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.600774284000003, 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.600621533999998, 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 52.598673308000002, 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.570645677000002, 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.568622785999999, 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.568452237000002, 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.566096741000003, 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.566062029000001, 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:28.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.566041220000002, 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:28.876 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.489154173000003, 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:28.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:28.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:28.884 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.489130091, 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:28.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:28.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.889 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.489120393999997, 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:28.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.483225457000003, 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:28.893 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.893786800000001, 2024-04-25T15:51:28.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:28.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:28.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:28.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:28.902 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:28.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:28.904 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:28.900+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:29.004 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:29.004+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:29.060 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:29.060 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:29.060 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:29.060 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:29.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:29.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.112+0000 7f3d577fe640 1 -- 172.21.15.67:0/4171322175 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v28) ==== 87+0+0 (secure 0 0 0) 0x7f3d4805bd00 con 0x7f3d6006cb80 2024-04-25T15:51:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d4006f430 msgr2=0x7f3d40071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d4006f430 0x7f3d40071940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3d50007fd0 tx=0x7f3d50009040 comp rx=0 tx=0).stop 2024-04-25T15:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d6006cb80 msgr2=0x7f3d6015b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:29.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d6006cb80 0x7f3d6015b320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3d48007f20 tx=0x7f3d4800d560 comp rx=0 tx=0).stop 2024-04-25T15:51:29.123 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:29.120+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 shutdown_connections 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d4006f430 0x7f3d40071940 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d60156330 0x7f3d601499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d600c5100 0x7f3d60153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 --2- 172.21.15.67:0/4171322175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d6006cb80 0x7f3d6015b320 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 >> 172.21.15.67:0/4171322175 conn(0x7f3d600ce1c0 msgr2=0x7f3d60154fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 shutdown_connections 2024-04-25T15:51:29.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:29.120+0000 7f3d6563c640 1 -- 172.21.15.67:0/4171322175 wait complete. 2024-04-25T15:51:29.206 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:29.204+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:29.238 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:29.238 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:29.238 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:29.238 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:29.238 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:29.255 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:29.343 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:29.340+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:29.408 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:29.404+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:29.450 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.238376631999998, 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:29.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:29.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:29.455 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.235761834999998, 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:29.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:29.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.235262937999998, 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:29.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:29.464 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.234173691999999, 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.233553196999999, 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.468 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:29.472 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.231158970999999, 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:29.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.476 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.202038567999999, 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:29.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.197293834, 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:29.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:29.483 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.193766332999999, 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:29.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:29.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:29.487 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.193633851999998, 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:29.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:29.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:29.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.491 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.572694393999999, 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:29.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:29.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.496 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.561259262, 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:29.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.560861826, 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.504 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.560752085000001, 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.560251340000001, 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:29.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:29.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.512 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.559875353999999, 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:29.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.516 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.521277171000001, 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.521176912000001, 2024-04-25T15:51:29.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:29.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.519071997000001, 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:29.524 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:29.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.518696421000001, 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:29.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:29.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:29.531 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.531 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.531 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.531 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:29.531 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:29.531 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:29.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:29.533 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:29.528+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:29.633 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:29.628+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:29.728 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:29.728 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:29.728 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:29.728 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:29.734 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:29.728+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:29.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:29.835 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:29.832+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:29.935 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:29.935 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:29.935 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:29.935 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:29.935 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:29.936 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:29.932+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:29.952 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:30.037 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:30.032+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:30.102 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:30.102 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:30.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:30.102 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:30.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.102 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.102 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851357583000002, 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:30.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:30.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851327022, 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.107 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:30.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851319191000002, 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.112 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:30.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851310261999998, 2024-04-25T15:51:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:30.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:30.122 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851304030000001, 2024-04-25T15:51:30.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:30.123 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.123 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.123 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.123 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:30.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:30.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851288619999998, 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.129 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:30.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851275788000002, 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.134 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:30.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:30.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:30.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:30.138 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.851270573999997, 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:30.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.765660457000003, 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.142 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:30.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.213130464000002, 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.147 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:30.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.211737626000001, 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.152 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:30.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:30.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.173717499999999, 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.156 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:30.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.173694174000001, 2024-04-25T15:51:30.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:30.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:30.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:30.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:30.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.173653634000001, 2024-04-25T15:51:30.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:30.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:30.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.172 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.173644973999998, 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:30.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:30.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.173603301, 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:30.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:30.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:30.182 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.173594044000001, 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:30.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:30.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:30.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.186 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.169456618000002, 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:30.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:30.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:30.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:30.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:30.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:30.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:30.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:30.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.193 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.169402041000001, 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:30.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.169368833, 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.201 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.202 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:30.203 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:30.203 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:30.200+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:30.304 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:30.300+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:30.355 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:30.355 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:30.356 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:30.356 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:30.372 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:30.405 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:30.404+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:30.505 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:30.500+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:30.521 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:30.521 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:30.521 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:30.522 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:30.522 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:30.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:30.606 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:30.604+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:30.707 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:30.704+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:30.728 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:30.728 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:30.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.515030803999998, 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.729 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.730 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.731 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:51:30.732 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.51500154, 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.734 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:30.736 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.737 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.738 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.514992251999999, 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.739 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:51:30.740 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.741 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.742 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:51:30.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:51:30.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.744 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:30.746 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:30.747 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.514984675999997, 2024-04-25T15:51:30.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:51:30.747 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.747 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.747 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.747 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:51:30.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:30.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:51:30.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.514976042999997, 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:51:30.752 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:51:30.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:51:30.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.514926209999999, 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.757 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:51:30.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:30.762 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.514918332000001, 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:51:30.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.478490325000003, 2024-04-25T15:51:30.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:51:30.767 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.767 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.767 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.767 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:30.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:30.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:51:30.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:51:30.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:51:30.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:51:30.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:51:30.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:51:30.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:51:30.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:51:30.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:51:30.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:51:30.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:51:30.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:51:30.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:30.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:51:30.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:51:30.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:30.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:51:30.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:51:30.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:51:30.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:51:30.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:51:30.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:51:30.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:51:30.778 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.778 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.778 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.779 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.779 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:30.779 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.478459661000002, 2024-04-25T15:51:30.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:51:30.779 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.779 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.780 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.780 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:30.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:51:30.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:51:30.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.478447131000003, 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.785 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:30.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:51:30.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.478436498999997, 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.790 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:30.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:51:30.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.478416953999997, 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.478408062, 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:51:30.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:51:30.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:51:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.474845446000003, 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:51:30.808 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.474724834, 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:30.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:51:30.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:51:30.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:51:30.812 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.839726814000002, 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:51:30.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:51:30.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:51:30.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:51:30.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:51:30.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.799157416, 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:30.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:30.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:51:30.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.799151907999999, 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:51:30.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:51:30.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.829 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.796737394000001, 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.833 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.795775507999998, 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:30.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:51:30.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.837 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:30.838 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:30.838 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:30.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:30.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:30.840 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:30.836+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:30.940 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:30.936+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:30.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f838ad56640 1 Processor -- start 2024-04-25T15:51:30.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f838ad56640 1 -- start start 2024-04-25T15:51:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f838ad56640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f838406d460 0x7f83840736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f838ad56640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 0x7f838415af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f838ad56640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83840caa60 con 0x7f83841547e0 2024-04-25T15:51:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f838ad56640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8384068590 con 0x7f838406d460 2024-04-25T15:51:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f838ad56640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f838415a5f0 con 0x7f838415aaf0 2024-04-25T15:51:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f8383fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83841547e0 0x7f838406c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41708/0 (socket says 172.21.15.67:41708) 2024-04-25T15:51:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f8383fff640 1 -- 172.21.15.67:0/2502167270 learned_addr learned my addr 172.21.15.67:0/2502167270 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:31.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f8388af2640 1 --2- 172.21.15.67:0/2502167270 >> v2:172.21.15.176:3300/0 conn(0x7f838406d460 0x7f83840736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:30.996+0000 7f83892f3640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 0x7f838415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779048938 0 0) 0x7f83840caa60 con 0x7f83841547e0 2024-04-25T15:51:31.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f836c003530 con 0x7f83841547e0 2024-04-25T15:51:31.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1359068697 0 0) 0x7f836c003530 con 0x7f83841547e0 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 msgr2=0x7f838415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 0x7f838415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 >> v2:172.21.15.176:3300/0 conn(0x7f838406d460 msgr2=0x7f83840736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 --2- 172.21.15.67:0/2502167270 >> v2:172.21.15.176:3300/0 conn(0x7f838406d460 0x7f83840736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83840ca100 con 0x7f83841547e0 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8378001800 con 0x7f83841547e0 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 >> v1:172.21.15.67:6789/0 conn(0x7f83841547e0 legacy=0x7f838406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f836c0039c0 0x7f836c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 0x7f836c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836c004d50 0x7f836c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8384068590 con 0x7f836c004d50 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f838415a5f0 con 0x7f836c0039c0 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f836c00b9d0 con 0x7f838415aaf0 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83892f3640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836c004d50 0x7f836c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f8388af2640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f836c0039c0 0x7f836c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f8383fff640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 0x7f836c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83892f3640 1 -- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 msgr2=0x7f836c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83892f3640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f838415aaf0 0x7f836c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83892f3640 1 -- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f836c0039c0 msgr2=0x7f836c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83892f3640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f836c0039c0 0x7f836c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83892f3640 1 -- 172.21.15.67:0/2502167270 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836c003530 con 0x7f836c004d50 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83892f3640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836c004d50 0x7f836c005160 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8374009ae0 tx=0x7f837401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e87b234636b13b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83740b2070 con 0x7f836c004d50 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83740a5640 con 0x7f836c004d50 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.000+0000 7f83837fe640 1 -- 172.21.15.67:0/2502167270 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83740096b0 con 0x7f836c004d50 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- 172.21.15.67:0/2502167270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836c004d50 msgr2=0x7f836c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 --2- 172.21.15.67:0/2502167270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f836c004d50 0x7f836c005160 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8374009ae0 tx=0x7f837401ca50 comp rx=0 tx=0).stop 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f83892f3640 1 -- 172.21.15.67:0/2502167270 reap_dead start 2024-04-25T15:51:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- 172.21.15.67:0/2502167270 shutdown_connections 2024-04-25T15:51:31.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- 172.21.15.67:0/2502167270 >> 172.21.15.67:0/2502167270 conn(0x7f83840ce1c0 msgr2=0x7f83840ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:31.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- 172.21.15.67:0/2502167270 shutdown_connections 2024-04-25T15:51:31.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- 172.21.15.67:0/2502167270 wait complete. 2024-04-25T15:51:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 Processor -- start 2024-04-25T15:51:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- start start 2024-04-25T15:51:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f836c004d50 0x7f83840c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8384156430 0x7f83840b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83840c4c90 0x7f83841535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8384068ac0 con 0x7f83840c4c90 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f83840687e0 con 0x7f8384156430 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f838ad56640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8384154bf0 con 0x7f836c004d50 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f83892f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83840c4c90 0x7f83841535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f83892f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83840c4c90 0x7f83841535e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58342/0 (socket says 172.21.15.67:58342) 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f83892f3640 1 -- 172.21.15.67:0/4176811553 learned_addr learned my addr 172.21.15.67:0/4176811553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f836c004d50 0x7f83840c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.004+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8384156430 0x7f83840b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83892f3640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f836c004d50 msgr2=0x7f83840c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83892f3640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f836c004d50 0x7f83840c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83892f3640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8384156430 msgr2=0x7f83840b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83892f3640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8384156430 0x7f83840b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83892f3640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83840c3c70 con 0x7f83840c4c90 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8384156430 0x7f83840b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83892f3640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83840c4c90 0x7f83841535e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8374009ae0 tx=0x7f836c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83740bb070 con 0x7f83840c4c90 2024-04-25T15:51:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8374000e50 con 0x7f83840c4c90 2024-04-25T15:51:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83841f2c20 con 0x7f83840c4c90 2024-04-25T15:51:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83740b2040 con 0x7f83840c4c90 2024-04-25T15:51:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83841f31e0 con 0x7f83840c4c90 2024-04-25T15:51:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f838406a480 con 0x7f83840c4c90 2024-04-25T15:51:31.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f83740be070 con 0x7f83840c4c90 2024-04-25T15:51:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83817fa640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f837006f400 0x7f8370071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5796+0+0 (secure 0 0 0) 0x7f8374128680 con 0x7f83840c4c90 2024-04-25T15:51:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v28) ==== 54+0+16 (secure 0 0 0) 0x7f83740167a0 con 0x7f83840c4c90 2024-04-25T15:51:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f837006f400 0x7f8370071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8384153b20 0x7f8384153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8384158bf0 con 0x7f8384153b20 2024-04-25T15:51:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.008+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8384153b20 0x7f8384153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.012+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8384153b20 0x7f8384153f30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.012+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f837006f400 0x7f8370071910 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f837c003b90 tx=0x7f837c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.012+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8384158bf0 con 0x7f8384153b20 2024-04-25T15:51:31.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.024+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8384158910 con 0x7f8384153b20 2024-04-25T15:51:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.028+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8384158910 con 0x7f8384153b20 2024-04-25T15:51:31.031 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.028+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f83841f6060 0x7f83841f6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.028+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f83841f69b0 con 0x7f83841f6060 2024-04-25T15:51:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.028+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f83841f6060 0x7f83841f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.028+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f83841f6060 0x7f83841f6470 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.028+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f83841f69b0 con 0x7f83841f6060 2024-04-25T15:51:31.037 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:31.037 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:31.037 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:31.037 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:31.041 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:31.036+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.044+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f83841f2940 con 0x7f83841f6060 2024-04-25T15:51:31.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.044+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f83841f2940 con 0x7f83841f6060 2024-04-25T15:51:31.049 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:31.050 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:31.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.044+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f83840635a0 0x7f83840639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.044+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8384063ef0 con 0x7f83840635a0 2024-04-25T15:51:31.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.044+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f83840635a0 0x7f83840639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.048+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f83840635a0 0x7f83840639b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.048+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8384063ef0 con 0x7f83840635a0 2024-04-25T15:51:31.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:31.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.060+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f83840c32a0 con 0x7f83840635a0 2024-04-25T15:51:31.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.064+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f83840c32a0 con 0x7f83840635a0 2024-04-25T15:51:31.067 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:31.067 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:31.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.064+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8384159790 0x7f8384159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.064+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f838415a0e0 con 0x7f8384159790 2024-04-25T15:51:31.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.064+0000 7f83892f3640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8384159790 0x7f8384159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.064+0000 7f83892f3640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8384159790 0x7f8384159ba0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.064+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f838415a0e0 con 0x7f8384159790 2024-04-25T15:51:31.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.080+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838402ccb0 con 0x7f8384159790 2024-04-25T15:51:31.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.080+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f838402ccb0 con 0x7f8384159790 2024-04-25T15:51:31.084 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.080+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f838402d3d0 0x7f838402d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.080+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f838402dd20 con 0x7f838402d3d0 2024-04-25T15:51:31.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.080+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f838402d3d0 0x7f838402d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.080+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f838402d3d0 0x7f838402d7e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.084+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f838402dd20 con 0x7f838402d3d0 2024-04-25T15:51:31.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.096+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838402dd20 con 0x7f838402d3d0 2024-04-25T15:51:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.100+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f838402dd20 con 0x7f838402d3d0 2024-04-25T15:51:31.104 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.100+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8384207690 0x7f8384209ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.100+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f838402dd20 con 0x7f8384207690 2024-04-25T15:51:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.100+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8384207690 0x7f8384209ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.100+0000 7f8383fff640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8384207690 0x7f8384209ab0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.104+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f838402dd20 con 0x7f8384207690 2024-04-25T15:51:31.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.116+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f838402dd20 con 0x7f8384207690 2024-04-25T15:51:31.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.120+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f838402dd20 con 0x7f8384207690 2024-04-25T15:51:31.123 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.120+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f838420a500 0x7f838420c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.120+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f838420f8b0 con 0x7f838420a500 2024-04-25T15:51:31.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.120+0000 7f83892f3640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f838420a500 0x7f838420c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.120+0000 7f83892f3640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f838420a500 0x7f838420c920 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.124+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f838420f8b0 con 0x7f838420a500 2024-04-25T15:51:31.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.136+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8384000960 con 0x7f838420a500 2024-04-25T15:51:31.141 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:31.136+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.140+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8384000960 con 0x7f838420a500 2024-04-25T15:51:31.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.140+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f838420eb00 0x7f838420ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.140+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8384212c80 con 0x7f838420eb00 2024-04-25T15:51:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.140+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f838420eb00 0x7f838420ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.140+0000 7f8388af2640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f838420eb00 0x7f838420ef50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.144+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8384212c80 con 0x7f838420eb00 2024-04-25T15:51:31.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:51:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.156+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8384211b00 con 0x7f838420eb00 2024-04-25T15:51:31.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f83817fa640 1 -- 172.21.15.67:0/4176811553 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8384211b00 con 0x7f838420eb00 2024-04-25T15:51:31.163 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8384153b20 msgr2=0x7f8384153f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8384153b20 0x7f8384153f30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f83841f6060 msgr2=0x7f83841f6470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f83841f6060 0x7f83841f6470 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f83840635a0 msgr2=0x7f83840639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f83840635a0 0x7f83840639b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f83892f3640 1 -- 172.21.15.67:0/4176811553 reap_dead start 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8384159790 msgr2=0x7f8384159ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8384159790 0x7f8384159ba0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f838402d3d0 msgr2=0x7f838402d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f838402d3d0 0x7f838402d7e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8384207690 msgr2=0x7f8384209ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8384207690 0x7f8384209ab0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f838420a500 msgr2=0x7f838420c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f838420a500 0x7f838420c920 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f838420eb00 msgr2=0x7f838420ef50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f838420eb00 0x7f838420ef50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f83892f3640 1 -- 172.21.15.67:0/4176811553 reap_dead start 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f837006f400 msgr2=0x7f8370071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f837006f400 0x7f8370071910 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f837c003b90 tx=0x7f837c00b040 comp rx=0 tx=0).stop 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83840c4c90 msgr2=0x7f83841535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83840c4c90 0x7f83841535e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8374009ae0 tx=0x7f836c005840 comp rx=0 tx=0).stop 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 shutdown_connections 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f837006f400 0x7f8370071910 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 --2- 172.21.15.67:0/4176811553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83840c4c90 0x7f83841535e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 >> 172.21.15.67:0/4176811553 conn(0x7f83840ce1c0 msgr2=0x7f83840cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 shutdown_connections 2024-04-25T15:51:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.160+0000 7f838ad56640 1 -- 172.21.15.67:0/4176811553 wait complete. 2024-04-25T15:51:31.242 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:31.236+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:31.248 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:31.248 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:31.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:31.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.304+0000 7f003ad93640 1 Processor -- start 2024-04-25T15:51:31.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.304+0000 7f003ad93640 1 -- start start 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f003ad93640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f00340ca2c0 0x7f003415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f003ad93640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00340ca6d0 0x7f0034167040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f003ad93640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00340ccef0 con 0x7f003415d3f0 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f003ad93640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f003415ca20 con 0x7f00340ca6d0 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f003ad93640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f00340cafa0 con 0x7f00340ca2c0 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f0039330640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00340ca6d0 0x7f0034167040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f002bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f003415d3f0 0x7f003406e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41718/0 (socket says 172.21.15.67:41718) 2024-04-25T15:51:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f002bfff640 1 -- 172.21.15.67:0/2170824873 learned_addr learned my addr 172.21.15.67:0/2170824873 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f0038b2f640 1 --2- 172.21.15.67:0/2170824873 >> v2:172.21.15.176:3300/0 conn(0x7f00340ca2c0 0x7f003415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f0039330640 1 -- 172.21.15.67:0/2170824873 >> v2:172.21.15.176:3300/0 conn(0x7f00340ca2c0 msgr2=0x7f003415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f0039330640 1 --2- 172.21.15.67:0/2170824873 >> v2:172.21.15.176:3300/0 conn(0x7f00340ca2c0 0x7f003415ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f0039330640 1 -- 172.21.15.67:0/2170824873 >> v1:172.21.15.67:6789/0 conn(0x7f003415d3f0 legacy=0x7f003406e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f0039330640 1 -- 172.21.15.67:0/2170824873 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00340cb280 con 0x7f00340ca6d0 2024-04-25T15:51:31.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f0039330640 1 --2- 172.21.15.67:0/2170824873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340ca6d0 0x7f0034167040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0034072e60 tx=0x7f002c01f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413189b2b1c96e91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.308+0000 7f002b7fe640 1 -- 172.21.15.67:0/2170824873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c0b4070 con 0x7f00340ca6d0 2024-04-25T15:51:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f002b7fe640 1 -- 172.21.15.67:0/2170824873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002c0af600 con 0x7f00340ca6d0 2024-04-25T15:51:31.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f002b7fe640 1 -- 172.21.15.67:0/2170824873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c0af9d0 con 0x7f00340ca6d0 2024-04-25T15:51:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 -- 172.21.15.67:0/2170824873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340ca6d0 msgr2=0x7f0034167040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 --2- 172.21.15.67:0/2170824873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340ca6d0 0x7f0034167040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0034072e60 tx=0x7f002c01f110 comp rx=0 tx=0).stop 2024-04-25T15:51:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 -- 172.21.15.67:0/2170824873 shutdown_connections 2024-04-25T15:51:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 --2- 172.21.15.67:0/2170824873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00340ca6d0 0x7f0034167040 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 --2- 172.21.15.67:0/2170824873 >> v2:172.21.15.176:3300/0 conn(0x7f00340ca2c0 0x7f003415ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 -- 172.21.15.67:0/2170824873 >> 172.21.15.67:0/2170824873 conn(0x7f0034073c50 msgr2=0x7f0034074060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 -- 172.21.15.67:0/2170824873 shutdown_connections 2024-04-25T15:51:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 -- 172.21.15.67:0/2170824873 wait complete. 2024-04-25T15:51:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 Processor -- start 2024-04-25T15:51:31.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.312+0000 7f003ad93640 1 -- start start 2024-04-25T15:51:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00340ca2c0 0x7f003415f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00340ca6d0 0x7f00341f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003415d3f0 0x7f00341e7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f003406e7e0 con 0x7f003415d3f0 2024-04-25T15:51:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f00340cb840 con 0x7f00340ca2c0 2024-04-25T15:51:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f003406e960 con 0x7f00340ca6d0 2024-04-25T15:51:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003415d3f0 0x7f00341e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003415d3f0 0x7f00341e7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58360/0 (socket says 172.21.15.67:58360) 2024-04-25T15:51:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 -- 172.21.15.67:0/173831162 learned_addr learned my addr 172.21.15.67:0/173831162 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f002bfff640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00340ca6d0 0x7f00341f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0038b2f640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00340ca2c0 0x7f003415f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 -- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00340ca6d0 msgr2=0x7f00341f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00340ca6d0 0x7f00341f6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 -- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00340ca2c0 msgr2=0x7f003415f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00340ca2c0 0x7f003415f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 -- 172.21.15.67:0/173831162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00341f6bb0 con 0x7f003415d3f0 2024-04-25T15:51:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f002bfff640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00340ca6d0 0x7f00341f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0038b2f640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00340ca2c0 0x7f003415f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0039330640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003415d3f0 0x7f00341e7740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f002c01bd20 tx=0x7f002c01beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f00297fa640 1 -- 172.21.15.67:0/173831162 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c0b4040 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f00297fa640 1 -- 172.21.15.67:0/173831162 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002c0c1070 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00341e81b0 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f00297fa640 1 -- 172.21.15.67:0/173831162 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c009450 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00341e5b50 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00341e5940 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f00297fa640 1 -- 172.21.15.67:0/173831162 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f002c0ba690 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f00297fa640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f002006f400 0x7f0020071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f00297fa640 1 -- 172.21.15.67:0/173831162 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5796+0+0 (secure 0 0 0) 0x7f002c12dc80 con 0x7f003415d3f0 2024-04-25T15:51:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.316+0000 7f0038b2f640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f002006f400 0x7f0020071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.320+0000 7f0038b2f640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f002006f400 0x7f0020071910 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f003000b9f0 tx=0x7f0030008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.324+0000 7f00297fa640 1 -- 172.21.15.67:0/173831162 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f002c018c70 con 0x7f003415d3f0 2024-04-25T15:51:31.342 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:31.340+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:31.443 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:31.440+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:31.460 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:31.460 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:31.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:31.496+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f00341e84c0 con 0x7f003415d3f0 2024-04-25T15:51:31.544 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:31.540+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:31.631 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:31.632 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:31.645 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:31.640+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:31.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:31.746 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:31.744+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:31.841 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:31.841 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:31.847 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:31.844+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:31.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:31.950 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:31.948+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:32.049 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:32.044+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:32.056 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:32.056 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:32.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:32.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.128+0000 7f00297fa640 1 -- 172.21.15.67:0/173831162 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v29) ==== 97+0+0 (secure 0 0 0) 0x7f002c0fac60 con 0x7f003415d3f0 2024-04-25T15:51:32.132 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:51:32.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f002006f400 msgr2=0x7f0020071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:32.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f002006f400 0x7f0020071910 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f003000b9f0 tx=0x7f0030008040 comp rx=0 tx=0).stop 2024-04-25T15:51:32.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003415d3f0 msgr2=0x7f00341e7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003415d3f0 0x7f00341e7740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f002c01bd20 tx=0x7f002c01beb0 comp rx=0 tx=0).stop 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 shutdown_connections 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f002006f400 0x7f0020071910 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f003415d3f0 0x7f00341e7740 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00340ca6d0 0x7f00341f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 --2- 172.21.15.67:0/173831162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00340ca2c0 0x7f003415f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 >> 172.21.15.67:0/173831162 conn(0x7f0034073c50 msgr2=0x7f0034156650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 shutdown_connections 2024-04-25T15:51:32.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:32.132+0000 7f003ad93640 1 -- 172.21.15.67:0/173831162 wait complete. 2024-04-25T15:51:32.149 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:32.144+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:32.250 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:32.248+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:32.276 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:32.276 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:32.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:32.386 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:32.384+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:32.451 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:32.448+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:32.490 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:32.490 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:32.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:32.554 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:32.552+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:32.653 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:32.648+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:32.702 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:32.702 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:32.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:32.754 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:32.748+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:32.855 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:32.852+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:32.914 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:32.914 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:32.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:32.974 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:32.972+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:33.056 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:33.052+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:33.129 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:33.129 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:33.146 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:33.146 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:33.146 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-25T15:51:33.147 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:51:33.147 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:51:33.147 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-04-25T15:51:33.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:33.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:33.195 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:33.192+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:33.258 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:33.252+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:33.306 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:33.306 INFO:teuthology.orchestra.run.smithi176.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T15:51:33.307 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c99d6b640 1 Processor -- start 2024-04-25T15:51:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c99d6b640 1 -- start start 2024-04-25T15:51:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c99d6b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c94164190 0x7f5c941665b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c99d6b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9415e510 con 0x7f5c94154ac0 2024-04-25T15:51:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c99d6b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c940ca8a0 con 0x7f5c940cbe90 2024-04-25T15:51:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c99d6b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5c940cab80 con 0x7f5c94164190 2024-04-25T15:51:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c93fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c94164190 0x7f5c941665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c937fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c94154ac0 0x7f5c9415de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47000/0 (socket says 172.21.15.67:47000) 2024-04-25T15:51:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c937fe640 1 -- 172.21.15.67:0/2447835919 learned_addr learned my addr 172.21.15.67:0/2447835919 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c93fff640 1 -- 172.21.15.67:0/2447835919 >> v1:172.21.15.176:6789/0 conn(0x7f5c940cbe90 legacy=0x7f5c940cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c93fff640 1 -- 172.21.15.67:0/2447835919 >> v1:172.21.15.67:6789/0 conn(0x7f5c94154ac0 legacy=0x7f5c9415de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c92ffd640 1 -- 172.21.15.67:0/2447835919 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655601383 0 0) 0x7f5c9415e510 con 0x7f5c94154ac0 2024-04-25T15:51:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.340+0000 7f5c93fff640 1 -- 172.21.15.67:0/2447835919 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9415a920 con 0x7f5c94164190 2024-04-25T15:51:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2447835919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94164190 0x7f5c941665b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009d90 tx=0x7f5c8c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75996e3181f3400 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c92ffd640 1 -- 172.21.15.67:0/2447835919 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c0b2070 con 0x7f5c94164190 2024-04-25T15:51:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c92ffd640 1 -- 172.21.15.67:0/2447835919 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c8c0a5970 con 0x7f5c94164190 2024-04-25T15:51:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c92ffd640 1 -- 172.21.15.67:0/2447835919 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c0a5d40 con 0x7f5c94164190 2024-04-25T15:51:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2447835919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94164190 msgr2=0x7f5c941665b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2447835919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94164190 0x7f5c941665b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009d90 tx=0x7f5c8c01c8b0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2447835919 shutdown_connections 2024-04-25T15:51:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2447835919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94164190 0x7f5c941665b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2447835919 >> 172.21.15.67:0/2447835919 conn(0x7f5c94068c70 msgr2=0x7f5c94071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:33.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2447835919 shutdown_connections 2024-04-25T15:51:33.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2447835919 wait complete. 2024-04-25T15:51:33.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 Processor -- start 2024-04-25T15:51:33.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.344+0000 7f5c99d6b640 1 -- start start 2024-04-25T15:51:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c940cbe90 0x7f5c9415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94154ac0 0x7f5c9415f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c94164190 0x7f5c941657c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c940cc2a0 con 0x7f5c94154ac0 2024-04-25T15:51:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94154ed0 con 0x7f5c94164190 2024-04-25T15:51:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c9415b6e0 con 0x7f5c940cbe90 2024-04-25T15:51:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94154ac0 0x7f5c9415f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94154ac0 0x7f5c9415f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47350/0 (socket says 172.21.15.67:47350) 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 -- 172.21.15.67:0/2077331812 learned_addr learned my addr 172.21.15.67:0/2077331812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c94164190 0x7f5c941657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c940cbe90 0x7f5c9415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c940cbe90 msgr2=0x7f5c9415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c940cbe90 0x7f5c9415eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c94164190 msgr2=0x7f5c941657c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c94164190 0x7f5c941657c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9415fe60 con 0x7f5c94154ac0 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c94164190 0x7f5c941657c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c8affd640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94154ac0 0x7f5c9415f920 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00c3c0 tx=0x7f5c7c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c017070 con 0x7f5c94154ac0 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c7c0052d0 con 0x7f5c94154ac0 2024-04-25T15:51:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c94159ae0 con 0x7f5c94154ac0 2024-04-25T15:51:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c003840 con 0x7f5c94154ac0 2024-04-25T15:51:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c9415a0a0 con 0x7f5c94154ac0 2024-04-25T15:51:33.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5c9406a480 con 0x7f5c94154ac0 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.348+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5c7c01a070 con 0x7f5c94154ac0 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c90ff9640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5796+0+0 (secure 0 0 0) 0x7f5c7c08ed50 con 0x7f5c94154ac0 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v29) ==== 54+0+16 (secure 0 0 0) 0x7f5c7c021070 con 0x7f5c94154ac0 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c9415a2a0 0x7f5c9414a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c9415a6f0 con 0x7f5c9415a2a0 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c9415a2a0 0x7f5c9414a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c84000f70 tx=0x7f5c84007d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c9415a2a0 0x7f5c9414a400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.352+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c9415a6f0 con 0x7f5c9415a2a0 2024-04-25T15:51:33.358 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:33.358 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:33.358 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:33.356+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:33.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.368+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c940cb260 con 0x7f5c9415a2a0 2024-04-25T15:51:33.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.368+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c940cb260 con 0x7f5c9415a2a0 2024-04-25T15:51:33.373 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.368+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c94155a10 0x7f5c941581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.368+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5c9414ae50 con 0x7f5c94155a10 2024-04-25T15:51:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.372+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c94155a10 0x7f5c941581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.372+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c94155a10 0x7f5c941581e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.372+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c9414ae50 con 0x7f5c94155a10 2024-04-25T15:51:33.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:33.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.384+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c9414b820 con 0x7f5c94155a10 2024-04-25T15:51:33.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.388+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c9414b820 con 0x7f5c94155a10 2024-04-25T15:51:33.391 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.388+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c94158c30 0x7f5c94063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.388+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5c94159040 con 0x7f5c94158c30 2024-04-25T15:51:33.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.388+0000 7f5c8affd640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c94158c30 0x7f5c94063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.388+0000 7f5c8affd640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c94158c30 0x7f5c94063090 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.388+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c94159040 con 0x7f5c94158c30 2024-04-25T15:51:33.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.400+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c9414b540 con 0x7f5c94158c30 2024-04-25T15:51:33.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.404+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c9414b540 con 0x7f5c94158c30 2024-04-25T15:51:33.408 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:33.408 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:33.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.404+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c94063ae0 0x7f5c940cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.404+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5c9414b260 con 0x7f5c94063ae0 2024-04-25T15:51:33.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.404+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c94063ae0 0x7f5c940cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.404+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c94063ae0 0x7f5c940cd970 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.404+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c9414b260 con 0x7f5c94063ae0 2024-04-25T15:51:33.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.420+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c9414b260 con 0x7f5c94063ae0 2024-04-25T15:51:33.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.420+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c9414b260 con 0x7f5c94063ae0 2024-04-25T15:51:33.424 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:33.425 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:33.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.420+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c940ce3c0 0x7f5c9402c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.420+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5c940ce850 con 0x7f5c940ce3c0 2024-04-25T15:51:33.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.420+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c940ce3c0 0x7f5c9402c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.424+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c940ce3c0 0x7f5c9402c9e0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.424+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c940ce850 con 0x7f5c940ce3c0 2024-04-25T15:51:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.436+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c9402cf20 con 0x7f5c940ce3c0 2024-04-25T15:51:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.444+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c9402cf20 con 0x7f5c940ce3c0 2024-04-25T15:51:33.446 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.444+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c9402d890 0x7f5c9420a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.444+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f5c9402dd70 con 0x7f5c9402d890 2024-04-25T15:51:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.444+0000 7f5c8affd640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c9402d890 0x7f5c9420a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.444+0000 7f5c8affd640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c9402d890 0x7f5c9420a770 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.444+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c9402dd70 con 0x7f5c9402d890 2024-04-25T15:51:33.460 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:33.456+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:33.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.460+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c940008e0 con 0x7f5c9402d890 2024-04-25T15:51:33.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.464+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c940008e0 con 0x7f5c9402d890 2024-04-25T15:51:33.468 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:33.469 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:33.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.464+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c9420b1c0 0x7f5c9420d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.464+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5c942104c0 con 0x7f5c9420b1c0 2024-04-25T15:51:33.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.464+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c9420b1c0 0x7f5c9420d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.468+0000 7f5c93fff640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c9420b1c0 0x7f5c9420d5e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.468+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c942104c0 con 0x7f5c9420b1c0 2024-04-25T15:51:33.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.480+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c9420db20 con 0x7f5c9420b1c0 2024-04-25T15:51:33.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.488+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c9420db20 con 0x7f5c9420b1c0 2024-04-25T15:51:33.490 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.488+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c9420e240 0x7f5c9420e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.488+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5c94213910 con 0x7f5c9420e240 2024-04-25T15:51:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.488+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c9420e240 0x7f5c9420e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:33.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.488+0000 7f5c937fe640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c9420e240 0x7f5c9420e650 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:33.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.488+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c94213910 con 0x7f5c9420e240 2024-04-25T15:51:33.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.504+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c94212710 con 0x7f5c9420e240 2024-04-25T15:51:33.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c90ff9640 1 -- 172.21.15.67:0/2077331812 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c94212710 con 0x7f5c9420e240 2024-04-25T15:51:33.512 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:33.513 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:33.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c9415a2a0 msgr2=0x7f5c9414a400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c9415a2a0 0x7f5c9414a400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c94155a10 msgr2=0x7f5c941581e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c94155a10 0x7f5c941581e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c94158c30 msgr2=0x7f5c94063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c94158c30 0x7f5c94063090 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c93fff640 1 -- 172.21.15.67:0/2077331812 reap_dead start 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c94063ae0 msgr2=0x7f5c940cd970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c94063ae0 0x7f5c940cd970 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c940ce3c0 msgr2=0x7f5c9402c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c940ce3c0 0x7f5c9402c9e0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c9402d890 msgr2=0x7f5c9420a770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c9402d890 0x7f5c9420a770 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c9420b1c0 msgr2=0x7f5c9420d5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c9420b1c0 0x7f5c9420d5e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c9420e240 msgr2=0x7f5c9420e650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c9420e240 0x7f5c9420e650 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c93fff640 1 -- 172.21.15.67:0/2077331812 reap_dead start 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 msgr2=0x7f5c74071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c84000f70 tx=0x7f5c84007d10 comp rx=0 tx=0).stop 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94154ac0 msgr2=0x7f5c9415f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94154ac0 0x7f5c9415f920 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00c3c0 tx=0x7f5c7c00c880 comp rx=0 tx=0).stop 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 shutdown_connections 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 --2- 172.21.15.67:0/2077331812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c94154ac0 0x7f5c9415f920 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 >> 172.21.15.67:0/2077331812 conn(0x7f5c94068c70 msgr2=0x7f5c94071550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 shutdown_connections 2024-04-25T15:51:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:33.508+0000 7f5c99d6b640 1 -- 172.21.15.67:0/2077331812 wait complete. 2024-04-25T15:51:33.532 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:33.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:33.532 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:51:33.533 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:33.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:33.598 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:33.596+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:33.661 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:33.656+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:33.749 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:33.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:33.749 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:51:33.749 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:51:33.749 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:33.762 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:33.756+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:33.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:33.863 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:33.860+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:33.964 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:33.960+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.752686838999999, 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.752647193999998, 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.749740172000003, 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.749587235999996, 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.748071592000002, 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.747928944999998, 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.747825853999998, 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.747673104, 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 57.745724877999997, 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.004 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.717697246999997, 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.715674356000001, 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:34.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.715503806999997, 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:34.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.713148310999998, 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:34.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:34.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.713113599000003, 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:34.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:34.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:34.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.024 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.713092789999997, 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.636205742999998, 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:34.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:34.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.636181661000002, 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:34.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:34.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.636171963999999, 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:34.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:34.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 41.630277026999998, 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.041 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.040838369999999, 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:34.045 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:34.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:34.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:34.050 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:34.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:34.095 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:34.092+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:34.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:51:34.202 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:34.199+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:34.213 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:34.214 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:34.214 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:34.214 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:34.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:34.275 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:34.272+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.319+0000 7f8f43e19640 1 Processor -- start 2024-04-25T15:51:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.319+0000 7f8f43e19640 1 -- start start 2024-04-25T15:51:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.319+0000 7f8f43e19640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c0723a0 0x7f8f3c0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.319+0000 7f8f43e19640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c072dc0 con 0x7f8f3c074790 2024-04-25T15:51:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f43e19640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f3c15b430 con 0x7f8f3c0c6fb0 2024-04-25T15:51:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f43e19640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8f3c15a670 con 0x7f8f3c0723a0 2024-04-25T15:51:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f41bb5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8f3c074790 0x7f8f3c15dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47010/0 (socket says 172.21.15.67:47010) 2024-04-25T15:51:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f41bb5640 1 -- 172.21.15.67:0/1410609939 learned_addr learned my addr 172.21.15.67:0/1410609939 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:34.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f413b4640 1 --2- 172.21.15.67:0/1410609939 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c0723a0 0x7f8f3c0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f413b4640 1 -- 172.21.15.67:0/1410609939 >> v1:172.21.15.176:6789/0 conn(0x7f8f3c0c6fb0 legacy=0x7f8f3c164e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f413b4640 1 -- 172.21.15.67:0/1410609939 >> v1:172.21.15.67:6789/0 conn(0x7f8f3c074790 legacy=0x7f8f3c15dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f40bb3640 1 -- 172.21.15.67:0/1410609939 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356813703 0 0) 0x7f8f3c15b430 con 0x7f8f3c0c6fb0 2024-04-25T15:51:34.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f413b4640 1 -- 172.21.15.67:0/1410609939 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3c15a950 con 0x7f8f3c0723a0 2024-04-25T15:51:34.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f413b4640 1 --2- 172.21.15.67:0/1410609939 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c0723a0 0x7f8f3c0727b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0097f0 tx=0x7f8f2c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67dffe882600de1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:34.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f40bb3640 1 -- 172.21.15.67:0/1410609939 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0b1070 con 0x7f8f3c0723a0 2024-04-25T15:51:34.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f40bb3640 1 -- 172.21.15.67:0/1410609939 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0ac600 con 0x7f8f3c0723a0 2024-04-25T15:51:34.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.323+0000 7f8f40bb3640 1 -- 172.21.15.67:0/1410609939 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0ac9d0 con 0x7f8f3c0723a0 2024-04-25T15:51:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- 172.21.15.67:0/1410609939 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c0723a0 msgr2=0x7f8f3c0727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 --2- 172.21.15.67:0/1410609939 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c0723a0 0x7f8f3c0727b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0097f0 tx=0x7f8f2c01c400 comp rx=0 tx=0).stop 2024-04-25T15:51:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- 172.21.15.67:0/1410609939 shutdown_connections 2024-04-25T15:51:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 --2- 172.21.15.67:0/1410609939 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c0723a0 0x7f8f3c0727b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- 172.21.15.67:0/1410609939 >> 172.21.15.67:0/1410609939 conn(0x7f8f3c06fe70 msgr2=0x7f8f3c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- 172.21.15.67:0/1410609939 shutdown_connections 2024-04-25T15:51:34.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- 172.21.15.67:0/1410609939 wait complete. 2024-04-25T15:51:34.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 Processor -- start 2024-04-25T15:51:34.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- start start 2024-04-25T15:51:34.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0723a0 0x7f8f3c1f1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c074790 0x7f8f3c153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f3c0c6fb0 0x7f8f3c1525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c15e1f0 con 0x7f8f3c0723a0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c15af10 con 0x7f8f3c0c6fb0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f43e19640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8f3c073080 con 0x7f8f3c074790 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f41bb5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0723a0 0x7f8f3c1f1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f41bb5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0723a0 0x7f8f3c1f1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47352/0 (socket says 172.21.15.67:47352) 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f413b4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c074790 0x7f8f3c153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f41bb5640 1 -- 172.21.15.67:0/4018235381 learned_addr learned my addr 172.21.15.67:0/4018235381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.327+0000 7f8f423b6640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f3c0c6fb0 0x7f8f3c1525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f41bb5640 1 -- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c074790 msgr2=0x7f8f3c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f41bb5640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c074790 0x7f8f3c153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f41bb5640 1 -- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f3c0c6fb0 msgr2=0x7f8f3c1525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f41bb5640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f3c0c6fb0 0x7f8f3c1525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f41bb5640 1 -- 172.21.15.67:0/4018235381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3c153f70 con 0x7f8f3c0723a0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f41bb5640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0723a0 0x7f8f3c1f1490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009e20 tx=0x7f8f3400c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f327fc640 1 -- 172.21.15.67:0/4018235381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34019070 con 0x7f8f3c0723a0 2024-04-25T15:51:34.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f327fc640 1 -- 172.21.15.67:0/4018235381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f340078f0 con 0x7f8f3c0723a0 2024-04-25T15:51:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f3c154180 con 0x7f8f3c0723a0 2024-04-25T15:51:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f327fc640 1 -- 172.21.15.67:0/4018235381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34007ce0 con 0x7f8f3c0723a0 2024-04-25T15:51:34.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f3c15cc20 con 0x7f8f3c0723a0 2024-04-25T15:51:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f3c14bb10 con 0x7f8f3c0723a0 2024-04-25T15:51:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f327fc640 1 -- 172.21.15.67:0/4018235381 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8f3401c070 con 0x7f8f3c0723a0 2024-04-25T15:51:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f327fc640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8f2006f430 0x7f8f20071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f327fc640 1 -- 172.21.15.67:0/4018235381 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5796+0+0 (secure 0 0 0) 0x7f8f34090f40 con 0x7f8f3c0723a0 2024-04-25T15:51:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f413b4640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8f2006f430 0x7f8f20071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.331+0000 7f8f413b4640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8f2006f430 0x7f8f20071940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0ae000 tx=0x7f8f2c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:34.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.335+0000 7f8f327fc640 1 -- 172.21.15.67:0/4018235381 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8f3405d780 con 0x7f8f3c0723a0 2024-04-25T15:51:34.368 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:34.363+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:34.424 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:34.424 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:34.424 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:34.424 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:34.425 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:34.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:34.469 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:34.463+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:34.511+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f8f3c06a480 con 0x7f8f3c0723a0 2024-04-25T15:51:34.570 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:34.564+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 58.381788438000001, 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.594 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:34.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 58.379173641000001, 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:34.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:34.602 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 58.378674744000001, 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 58.377585498000002, 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:34.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 58.376965003000002, 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:34.610 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:34.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:34.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 58.374570777000002, 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.614 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:34.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:34.618 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.345450374000002, 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:34.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.340705640000003, 2024-04-25T15:51:34.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.337178139000002, 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:34.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:34.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:34.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.337045658000001, 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.716106199999999, 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.634 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:34.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:34.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:34.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.704671068000003, 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.639 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:34.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:34.644 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:34.645 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.704273632000003, 2024-04-25T15:51:34.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:34.645 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.645 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.645 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.646 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:34.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:34.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:34.650 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.704163891, 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:34.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:34.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.703663145999997, 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.658 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.703287160000002, 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:34.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:34.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 30.664688977000001, 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:34.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:34.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:34.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:34.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:34.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:34.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 30.664588718000001, 2024-04-25T15:51:34.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:34.670 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.670 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.670 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:34.670 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:34.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:34.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:34.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.674 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 30.662483803000001, 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:34.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:34.677 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 30.662108227000001, 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:34.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:34.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:34.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:34.681 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.681 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.681 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.681 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:34.681 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:34.681 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:34.682 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:34.683 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:34.676+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:34.783 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:34.783+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:34.875 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:34.876 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:34.876 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:34.876 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:34.884 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:34.879+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:34.892 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:34.985 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:34.980+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:35.042 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:35.042 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:35.042 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:35.042 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:35.042 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:35.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:35.099 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:35.096+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:35.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.143+0000 7f8f327fc640 1 -- 172.21.15.67:0/4018235381 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v30) ==== 91+0+0 (secure 0 0 0) 0x7f8f34061a80 con 0x7f8f3c0723a0 2024-04-25T15:51:35.149 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:51:35.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8f2006f430 msgr2=0x7f8f20071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8f2006f430 0x7f8f20071940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0ae000 tx=0x7f8f2c0a7000 comp rx=0 tx=0).stop 2024-04-25T15:51:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0723a0 msgr2=0x7f8f3c1f1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0723a0 0x7f8f3c1f1490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009e20 tx=0x7f8f3400c430 comp rx=0 tx=0).stop 2024-04-25T15:51:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 shutdown_connections 2024-04-25T15:51:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8f2006f430 0x7f8f20071940 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f3c0c6fb0 0x7f8f3c1525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f3c074790 0x7f8f3c153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 --2- 172.21.15.67:0/4018235381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0723a0 0x7f8f3c1f1490 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 >> 172.21.15.67:0/4018235381 conn(0x7f8f3c06fe70 msgr2=0x7f8f3c14fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:35.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 shutdown_connections 2024-04-25T15:51:35.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.147+0000 7f8f43e19640 1 -- 172.21.15.67:0/4018235381 wait complete. 2024-04-25T15:51:35.186 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:35.180+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:35.252 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002695064999997, 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.253 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:35.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:35.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:35.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002664504000002, 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.258 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:35.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:35.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002656672999997, 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:35.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002647744000001, 2024-04-25T15:51:35.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:35.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:35.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.273 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002641511999997, 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:35.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:35.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:35.279 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002626102000001, 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:35.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:35.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002613269999998, 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:35.285 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:35.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.002608056, 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:35.289 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:35.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:35.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.916997938999998, 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.293 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:35.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.364467945999998, 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.298 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:35.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:35.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 37.363075107999997, 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:35.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.306 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.325054982000001, 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:35.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.325031656, 2024-04-25T15:51:35.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:35.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:35.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:35.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.324991116, 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:35.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:35.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:35.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:35.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.322 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.324982456000001, 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:35.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:35.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:35.327 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.324940782999999, 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.331 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.324931526, 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.335 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.320794100000001, 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:35.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:35.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.339 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.320739523, 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.320706314999999, 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:35.344 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:35.345 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:35.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:35.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:35.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:35.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:35.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:35.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:35.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.349 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:35.349 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:35.349 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:35.350 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:35.351 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:35.347+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:35.451 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:35.447+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:35.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:35.546 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:35.546 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:35.546 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:35.546 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:35.552 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:35.548+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:35.564 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:35.653 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:35.648+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff105153640 1 Processor -- start 2024-04-25T15:51:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff105153640 1 -- start start 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff105153640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff100154ac0 0x7ff1000cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff105153640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff10015e210 0x7ff100165190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff105153640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1000ca8a0 con 0x7ff10015de00 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff105153640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff1000cab80 con 0x7ff100154ac0 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff105153640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff10015a920 con 0x7ff10015e210 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff0fed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff100154ac0 0x7ff1000cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff0fed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff100154ac0 0x7ff1000cbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47362/0 (socket says 172.21.15.67:47362) 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff0fed9d640 1 -- 172.21.15.67:0/827380055 learned_addr learned my addr 172.21.15.67:0/827380055 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.695+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/827380055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff10015e210 0x7ff100165190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fed9d640 1 -- 172.21.15.67:0/827380055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff10015e210 msgr2=0x7ff100165190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/827380055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff10015e210 0x7ff100165190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fed9d640 1 -- 172.21.15.67:0/827380055 >> v1:172.21.15.176:6789/0 conn(0x7ff10015de00 legacy=0x7ff1000cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fed9d640 1 -- 172.21.15.67:0/827380055 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10015ac00 con 0x7ff100154ac0 2024-04-25T15:51:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/827380055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff100154ac0 0x7ff1000cbe90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009870 tx=0x7ff0f001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d2930ff0230e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fdd9b640 1 -- 172.21.15.67:0/827380055 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f00b1070 con 0x7ff100154ac0 2024-04-25T15:51:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fdd9b640 1 -- 172.21.15.67:0/827380055 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f00ac620 con 0x7ff100154ac0 2024-04-25T15:51:35.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff0fdd9b640 1 -- 172.21.15.67:0/827380055 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f00ac9f0 con 0x7ff100154ac0 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 -- 172.21.15.67:0/827380055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff100154ac0 msgr2=0x7ff1000cbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 --2- 172.21.15.67:0/827380055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff100154ac0 0x7ff1000cbe90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009870 tx=0x7ff0f001c4d0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 -- 172.21.15.67:0/827380055 shutdown_connections 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 --2- 172.21.15.67:0/827380055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff10015e210 0x7ff100165190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 --2- 172.21.15.67:0/827380055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff100154ac0 0x7ff1000cbe90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 -- 172.21.15.67:0/827380055 >> 172.21.15.67:0/827380055 conn(0x7ff100068c70 msgr2=0x7ff100071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 -- 172.21.15.67:0/827380055 shutdown_connections 2024-04-25T15:51:35.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.699+0000 7ff105153640 1 -- 172.21.15.67:0/827380055 wait complete. 2024-04-25T15:51:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 Processor -- start 2024-04-25T15:51:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 -- start start 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff100154ac0 0x7ff1001f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10015de00 0x7ff10014bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff10015e210 0x7ff100149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff1000ccd10 con 0x7ff10015de00 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff10015b6e0 con 0x7ff10015e210 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff1000cce90 con 0x7ff100154ac0 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10015de00 0x7ff10014bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10015de00 0x7ff10014bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47364/0 (socket says 172.21.15.67:47364) 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 -- 172.21.15.67:0/982569989 learned_addr learned my addr 172.21.15.67:0/982569989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff10015e210 0x7ff100149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff100154ac0 0x7ff1001f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff100154ac0 msgr2=0x7ff1001f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff100154ac0 0x7ff1001f7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff10015e210 msgr2=0x7ff100149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff10015e210 0x7ff100149a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10014c080 con 0x7ff10015de00 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0fe59c640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10015de00 0x7ff10014bb40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400c340 tx=0x7ff0f400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40052d0 con 0x7ff10015de00 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f4005470 con 0x7ff10015de00 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10014b0f0 con 0x7ff10015de00 2024-04-25T15:51:35.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4007260 con 0x7ff10015de00 2024-04-25T15:51:35.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.703+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff100153850 con 0x7ff10015de00 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff10006a480 con 0x7ff10015de00 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff0f4007400 con 0x7ff10015de00 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0e77fe640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0ec06f400 0x7ff0ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5796+0+0 (secure 0 0 0) 0x7ff0f408f210 con 0x7ff10015de00 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v30) ==== 54+0+16 (secure 0 0 0) 0x7ff0f408f6a0 con 0x7ff10015de00 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0ec06f400 0x7ff0ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff10014b530 0x7ff1000cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1000cb260 con 0x7ff10014b530 2024-04-25T15:51:35.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff10014b530 0x7ff1000cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff10014b530 0x7ff1000cfb50 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0ec06f400 0x7ff0ec071910 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00ae000 tx=0x7ff0f00a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.707+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1000cb260 con 0x7ff10014b530 2024-04-25T15:51:35.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.723+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1000cc940 con 0x7ff10014b530 2024-04-25T15:51:35.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.727+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1000cc940 con 0x7ff10014b530 2024-04-25T15:51:35.730 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:35.731 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:35.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.727+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff100152d70 0x7ff10015cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.727+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff100153180 con 0x7ff100152d70 2024-04-25T15:51:35.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.727+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff100152d70 0x7ff10015cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.727+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff100152d70 0x7ff10015cc30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.727+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff100153180 con 0x7ff100152d70 2024-04-25T15:51:35.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.739+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff10015d170 con 0x7ff100152d70 2024-04-25T15:51:35.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.743+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff10015d170 con 0x7ff100152d70 2024-04-25T15:51:35.747 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.743+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ff10015d740 0x7ff1001459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.743+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff100145ee0 con 0x7ff10015d740 2024-04-25T15:51:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.743+0000 7ff0fe59c640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ff10015d740 0x7ff1001459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.743+0000 7ff0fe59c640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ff10015d740 0x7ff1001459a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.743+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff100145ee0 con 0x7ff10015d740 2024-04-25T15:51:35.754 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:35.754 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:35.754 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:35.754 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:35.754 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:35.754 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:35.748+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:35.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.759+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff100063090 con 0x7ff10015d740 2024-04-25T15:51:35.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.759+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff100063090 con 0x7ff10015d740 2024-04-25T15:51:35.764 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.759+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff100063b10 0x7ff1000cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.759+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff100063f20 con 0x7ff100063b10 2024-04-25T15:51:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.759+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff100063b10 0x7ff1000cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.759+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff100063b10 0x7ff1000cda70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.763+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff100063f20 con 0x7ff100063b10 2024-04-25T15:51:35.774 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:35.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.775+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1000ce2c0 con 0x7ff100063b10 2024-04-25T15:51:35.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.775+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1000ce2c0 con 0x7ff100063b10 2024-04-25T15:51:35.781 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:35.781 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:35.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.775+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff1000ce750 0x7ff10002c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.775+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff100063f20 con 0x7ff1000ce750 2024-04-25T15:51:35.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.779+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff1000ce750 0x7ff10002c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.779+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff1000ce750 0x7ff10002c9e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.779+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff100063f20 con 0x7ff1000ce750 2024-04-25T15:51:35.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.795+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff100063f20 con 0x7ff1000ce750 2024-04-25T15:51:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.799+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff100063f20 con 0x7ff1000ce750 2024-04-25T15:51:35.803 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.799+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ff10002d430 0x7ff10002d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.799+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff10002deb0 con 0x7ff10002d430 2024-04-25T15:51:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.799+0000 7ff0fe59c640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ff10002d430 0x7ff10002d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.799+0000 7ff0fe59c640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ff10002d430 0x7ff10002d840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.799+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff10002deb0 con 0x7ff10002d430 2024-04-25T15:51:35.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.815+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff100000960 con 0x7ff10002d430 2024-04-25T15:51:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.815+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff100000960 con 0x7ff10002d430 2024-04-25T15:51:35.821 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.815+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ff10020aad0 0x7ff10020aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.819+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff100210060 con 0x7ff10020aad0 2024-04-25T15:51:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.819+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ff10020aad0 0x7ff10020aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.819+0000 7ff0ff59e640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ff10020aad0 0x7ff10020aee0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.819+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff100210060 con 0x7ff10020aad0 2024-04-25T15:51:35.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.835+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff10020b510 con 0x7ff10020aad0 2024-04-25T15:51:35.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.835+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff10020b510 con 0x7ff10020aad0 2024-04-25T15:51:35.840 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.835+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ff10020ba20 0x7ff10020be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.835+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff10020c3f0 con 0x7ff10020ba20 2024-04-25T15:51:35.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.835+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ff10020ba20 0x7ff10020be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.835+0000 7ff0fed9d640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ff10020ba20 0x7ff10020be30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.839+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff10020c3f0 con 0x7ff10020ba20 2024-04-25T15:51:35.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.851+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff10002de30 con 0x7ff10020ba20 2024-04-25T15:51:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff0e77fe640 1 -- 172.21.15.67:0/982569989 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff10002de30 con 0x7ff10020ba20 2024-04-25T15:51:35.859 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff10014b530 msgr2=0x7ff1000cfb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff10014b530 0x7ff1000cfb50 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff100152d70 msgr2=0x7ff10015cc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff100152d70 0x7ff10015cc30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ff10015d740 msgr2=0x7ff1001459a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ff10015d740 0x7ff1001459a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff100063b10 msgr2=0x7ff1000cda70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ff100063b10 0x7ff1000cda70 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff1000ce750 msgr2=0x7ff10002c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ff1000ce750 0x7ff10002c9e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ff10002d430 msgr2=0x7ff10002d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ff10002d430 0x7ff10002d840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ff10020aad0 msgr2=0x7ff10020aee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ff10020aad0 0x7ff10020aee0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff0ff59e640 1 -- 172.21.15.67:0/982569989 reap_dead start 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff0ff59e640 1 -- 172.21.15.67:0/982569989 reap_dead start 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ff10020ba20 msgr2=0x7ff10020be30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ff10020ba20 0x7ff10020be30 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff0ff59e640 1 -- 172.21.15.67:0/982569989 reap_dead start 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff0ff59e640 1 -- 172.21.15.67:0/982569989 reap_dead start 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0ec06f400 msgr2=0x7ff0ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0ec06f400 0x7ff0ec071910 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00ae000 tx=0x7ff0f00a7000 comp rx=0 tx=0).stop 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10015de00 msgr2=0x7ff10014bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10015de00 0x7ff10014bb40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400c340 tx=0x7ff0f400c800 comp rx=0 tx=0).stop 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 shutdown_connections 2024-04-25T15:51:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0ec06f400 0x7ff0ec071910 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 --2- 172.21.15.67:0/982569989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff10015de00 0x7ff10014bb40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 >> 172.21.15.67:0/982569989 conn(0x7ff100068c70 msgr2=0x7ff10014c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 shutdown_connections 2024-04-25T15:51:35.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:35.855+0000 7ff105153640 1 -- 172.21.15.67:0/982569989 wait complete. 2024-04-25T15:51:35.863 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:35.856+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:35.955 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:35.951+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:35.964 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:35.964 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:35.981 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:36.067 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:36.064+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:36.157 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:36.151+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:36.175 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:36.175 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:36.192 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:36.257 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:36.255+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:36.359 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:36.352+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:36.382 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:36.383 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:36.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:36.459 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:36.455+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:36.560 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:36.559+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:36.592 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:36.592 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:36.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:36.661 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:36.656+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:36.762 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:36.756+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:36.762 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:36.762 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:36.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:36.863 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:36.859+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:36.933 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:36.933 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:36.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:36.994 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:36.991+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:37.064 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:37.060+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:37.147 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:37.147 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:37.165 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:37.159+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:37.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:37.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:51:37.265 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:37.260+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:37.359 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:37.359 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:37.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae33a0640 1 Processor -- start 2024-04-25T15:51:37.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae33a0640 1 -- start start 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae33a0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2adc0736d0 0x7f2adc06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae33a0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 0x7f2adc15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae33a0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2adc0caa60 con 0x7f2adc06d460 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae33a0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2adc068590 con 0x7f2adc0736d0 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae33a0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2adc15a5f0 con 0x7f2adc06cb80 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae113c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2adc06d460 0x7f2adc1547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47022/0 (socket says 172.21.15.67:47022) 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae113c640 1 -- 172.21.15.67:0/1524328606 learned_addr learned my addr 172.21.15.67:0/1524328606 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae193d640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 0x7f2adc15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae093b640 1 --2- 172.21.15.67:0/1524328606 >> v2:172.21.15.176:3300/0 conn(0x7f2adc0736d0 0x7f2adc06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491938742 0 0) 0x7f2adc0caa60 con 0x7f2adc06d460 2024-04-25T15:51:37.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac8003530 con 0x7f2adc06d460 2024-04-25T15:51:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 318177043 0 0) 0x7f2ac8003530 con 0x7f2adc06d460 2024-04-25T15:51:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 msgr2=0x7f2adc15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 0x7f2adc15aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 >> v2:172.21.15.176:3300/0 conn(0x7f2adc0736d0 msgr2=0x7f2adc06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 --2- 172.21.15.67:0/1524328606 >> v2:172.21.15.176:3300/0 conn(0x7f2adc0736d0 0x7f2adc06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc0ca100 con 0x7f2adc06d460 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f2ad8001680 con 0x7f2adc06d460 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 >> v1:172.21.15.67:6789/0 conn(0x7f2adc06d460 legacy=0x7f2adc1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 0x7f2ac80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8003f40 0x7f2ac8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8004d50 0x7f2ac8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc068590 con 0x7f2ac8003f40 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc15a5f0 con 0x7f2ac8004d50 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac800ba20 con 0x7f2adc06cb80 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae093b640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8003f40 0x7f2ac8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.359+0000 7f2ae113c640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 0x7f2ac80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae193d640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8004d50 0x7f2ac8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae093b640 1 -- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 msgr2=0x7f2ac80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae093b640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 0x7f2ac80039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae093b640 1 -- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8004d50 msgr2=0x7f2ac8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae093b640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8004d50 0x7f2ac8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae093b640 1 -- 172.21.15.67:0/1524328606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac8003530 con 0x7f2ac8003f40 2024-04-25T15:51:37.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae113c640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc06cb80 0x7f2ac80039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:37.369 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae093b640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8003f40 0x7f2ac8004810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009a90 tx=0x7f2acc01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654cc05b0e9fdb75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc009690 con 0x7f2ac8003f40 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc0059a0 con 0x7f2ac8003f40 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ad3fff640 1 -- 172.21.15.67:0/1524328606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc005d90 con 0x7f2ac8003f40 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae33a0640 1 -- 172.21.15.67:0/1524328606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8003f40 msgr2=0x7f2ac8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae33a0640 1 --2- 172.21.15.67:0/1524328606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ac8003f40 0x7f2ac8004810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009a90 tx=0x7f2acc01ca70 comp rx=0 tx=0).stop 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae193d640 1 -- 172.21.15.67:0/1524328606 reap_dead start 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae33a0640 1 -- 172.21.15.67:0/1524328606 shutdown_connections 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.363+0000 7f2ae33a0640 1 -- 172.21.15.67:0/1524328606 >> 172.21.15.67:0/1524328606 conn(0x7f2adc0ce1c0 msgr2=0x7f2adc0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 -- 172.21.15.67:0/1524328606 shutdown_connections 2024-04-25T15:51:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 -- 172.21.15.67:0/1524328606 wait complete. 2024-04-25T15:51:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 Processor -- start 2024-04-25T15:51:37.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 -- start start 2024-04-25T15:51:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8003f40 0x7f2adc14bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2adc0b3490 0x7f2adc0c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc0cc830 0x7f2adc14ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc068ac0 con 0x7f2adc0b3490 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc0687e0 con 0x7f2ac8003f40 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae33a0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2adc15aaf0 con 0x7f2adc0cc830 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae093b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2adc0b3490 0x7f2adc0c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae093b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2adc0b3490 0x7f2adc0c3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47378/0 (socket says 172.21.15.67:47378) 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae093b640 1 -- 172.21.15.67:0/2434507780 learned_addr learned my addr 172.21.15.67:0/2434507780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae113c640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8003f40 0x7f2adc14bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.367+0000 7f2ae193d640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc0cc830 0x7f2adc14ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae093b640 1 -- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc0cc830 msgr2=0x7f2adc14ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae093b640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc0cc830 0x7f2adc14ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae093b640 1 -- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8003f40 msgr2=0x7f2adc14bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae093b640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8003f40 0x7f2adc14bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae093b640 1 -- 172.21.15.67:0/2434507780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc1f6ea0 con 0x7f2adc0b3490 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae093b640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2adc0b3490 0x7f2adc0c3580 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0af040 tx=0x7f2ac8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ad1ffb640 1 -- 172.21.15.67:0/2434507780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0bc070 con 0x7f2adc0b3490 2024-04-25T15:51:37.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ad1ffb640 1 -- 172.21.15.67:0/2434507780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc0a5c50 con 0x7f2adc0b3490 2024-04-25T15:51:37.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2adc1f2e30 con 0x7f2adc0b3490 2024-04-25T15:51:37.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ad1ffb640 1 -- 172.21.15.67:0/2434507780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc004020 con 0x7f2adc0b3490 2024-04-25T15:51:37.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2adc1f33f0 con 0x7f2adc0b3490 2024-04-25T15:51:37.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2adc06a480 con 0x7f2adc0b3490 2024-04-25T15:51:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ad1ffb640 1 -- 172.21.15.67:0/2434507780 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2acc0041c0 con 0x7f2adc0b3490 2024-04-25T15:51:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ad1ffb640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac006f430 0x7f2ac0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.371+0000 7f2ad1ffb640 1 -- 172.21.15.67:0/2434507780 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5796+0+0 (secure 0 0 0) 0x7f2acc12a2b0 con 0x7f2adc0b3490 2024-04-25T15:51:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.375+0000 7f2ae113c640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac006f430 0x7f2ac0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.375+0000 7f2ae113c640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac006f430 0x7f2ac0071940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8000c10 tx=0x7f2ad8001d20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:37.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.379+0000 7f2ad1ffb640 1 -- 172.21.15.67:0/2434507780 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2acc016780 con 0x7f2adc0b3490 2024-04-25T15:51:37.468 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:37.463+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:37.524 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:37.524 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:37.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:37.551+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f2adc0cb7d0 con 0x7f2adc0b3490 2024-04-25T15:51:37.606 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:37.603+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:37.669 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:37.664+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:37.708 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:37.708 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:37.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:37.771 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:37.768+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:37.872 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:37.872+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:37.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:37.918 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:37.919 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:37.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:37.982 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:37.979+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:38.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.031+0000 7fc654401640 1 Processor -- start 2024-04-25T15:51:38.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.031+0000 7fc654401640 1 -- start start 2024-04-25T15:51:38.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc654401640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc64c06ca60 0x7fc64c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc654401640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc64c06d780 con 0x7fc64c15fe00 2024-04-25T15:51:38.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc654401640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc64c160920 con 0x7fc64c0b5660 2024-04-25T15:51:38.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc654401640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc64c15d510 con 0x7fc64c06ca60 2024-04-25T15:51:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc65219d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc64c06ca60 0x7fc64c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc65299e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc64c15fe00 0x7fc64c160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47036/0 (socket says 172.21.15.67:47036) 2024-04-25T15:51:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc65299e640 1 -- 172.21.15.67:0/2872566548 learned_addr learned my addr 172.21.15.67:0/2872566548 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc65219d640 1 -- 172.21.15.67:0/2872566548 >> v1:172.21.15.176:6789/0 conn(0x7fc64c0b5660 legacy=0x7fc64c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc65219d640 1 -- 172.21.15.67:0/2872566548 >> v1:172.21.15.67:6789/0 conn(0x7fc64c15fe00 legacy=0x7fc64c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc65119b640 1 -- 172.21.15.67:0/2872566548 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2798772794 0 0) 0x7fc64c06d780 con 0x7fc64c15fe00 2024-04-25T15:51:38.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.035+0000 7fc65219d640 1 -- 172.21.15.67:0/2872566548 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc64c160be0 con 0x7fc64c06ca60 2024-04-25T15:51:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc65219d640 1 --2- 172.21.15.67:0/2872566548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c06ca60 0x7fc64c06ce70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6480097f0 tx=0x7fc64801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3359e04dd8fcb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc65119b640 1 -- 172.21.15.67:0/2872566548 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6480b0070 con 0x7fc64c06ca60 2024-04-25T15:51:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc65119b640 1 -- 172.21.15.67:0/2872566548 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc648004230 con 0x7fc64c06ca60 2024-04-25T15:51:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc65119b640 1 -- 172.21.15.67:0/2872566548 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc648003620 con 0x7fc64c06ca60 2024-04-25T15:51:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc654401640 1 -- 172.21.15.67:0/2872566548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c06ca60 msgr2=0x7fc64c06ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc654401640 1 --2- 172.21.15.67:0/2872566548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c06ca60 0x7fc64c06ce70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6480097f0 tx=0x7fc64801c460 comp rx=0 tx=0).stop 2024-04-25T15:51:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc654401640 1 -- 172.21.15.67:0/2872566548 shutdown_connections 2024-04-25T15:51:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc654401640 1 --2- 172.21.15.67:0/2872566548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c06ca60 0x7fc64c06ce70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc654401640 1 -- 172.21.15.67:0/2872566548 >> 172.21.15.67:0/2872566548 conn(0x7fc64c06e420 msgr2=0x7fc64c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc654401640 1 -- 172.21.15.67:0/2872566548 shutdown_connections 2024-04-25T15:51:38.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.039+0000 7fc654401640 1 -- 172.21.15.67:0/2872566548 wait complete. 2024-04-25T15:51:38.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 Processor -- start 2024-04-25T15:51:38.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 -- start start 2024-04-25T15:51:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc64c06ca60 0x7fc64c1f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c0b5660 0x7fc64c14fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc64c15fe00 0x7fc64c0ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc64c0b6e90 con 0x7fc64c0b5660 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc64c15c890 con 0x7fc64c15fe00 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc64c0b7010 con 0x7fc64c06ca60 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c0b5660 0x7fc64c14fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c0b5660 0x7fc64c14fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47394/0 (socket says 172.21.15.67:47394) 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 -- 172.21.15.67:0/3580638102 learned_addr learned my addr 172.21.15.67:0/3580638102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc64c06ca60 0x7fc64c1f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc64c15fe00 0x7fc64c0ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc64c06ca60 msgr2=0x7fc64c1f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc64c06ca60 0x7fc64c1f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc64c15fe00 msgr2=0x7fc64c0ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc64c15fe00 0x7fc64c0ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc64c1f6220 con 0x7fc64c0b5660 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc64c15fe00 0x7fc64c0ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc65199c640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c0b5660 0x7fc64c14fda0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc63c00b3e0 tx=0x7fc63c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c015020 con 0x7fc64c0b5660 2024-04-25T15:51:38.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc63c011830 con 0x7fc64c0b5660 2024-04-25T15:51:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc64c1f9dc0 con 0x7fc64c0b5660 2024-04-25T15:51:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c011c60 con 0x7fc64c0b5660 2024-04-25T15:51:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc64c0cb7d0 con 0x7fc64c0b5660 2024-04-25T15:51:38.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.043+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc64c06a480 con 0x7fc64c0b5660 2024-04-25T15:51:38.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc63c018020 con 0x7fc64c0b5660 2024-04-25T15:51:38.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc642ffd640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc63006f400 0x7fc630071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5796+0+0 (secure 0 0 0) 0x7fc63c08f1f0 con 0x7fc64c0b5660 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v30) ==== 54+0+16 (secure 0 0 0) 0x7fc63c01f020 con 0x7fc64c0b5660 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc63006f400 0x7fc630071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc64c155bc0 0x7fc64c15a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc64c0cd380 con 0x7fc64c155bc0 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc64c155bc0 0x7fc64c15a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc64c155bc0 0x7fc64c15a700 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc63006f400 0x7fc630071910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6480ad000 tx=0x7fc648008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.047+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c0cd380 con 0x7fc64c155bc0 2024-04-25T15:51:38.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.063+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c0d0060 con 0x7fc64c155bc0 2024-04-25T15:51:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.063+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c0d0060 con 0x7fc64c155bc0 2024-04-25T15:51:38.069 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.063+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc64c159290 0x7fc64c1596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.063+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc64c0cbe20 con 0x7fc64c159290 2024-04-25T15:51:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.063+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc64c159290 0x7fc64c1596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.063+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc64c159290 0x7fc64c1596a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.067+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c0cbe20 con 0x7fc64c159290 2024-04-25T15:51:38.075 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:38.068+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:38.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.079+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c15bbb0 con 0x7fc64c159290 2024-04-25T15:51:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.079+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c15bbb0 con 0x7fc64c159290 2024-04-25T15:51:38.086 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.079+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc64c063090 0x7fc64c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.079+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc64c15a0f0 con 0x7fc64c063090 2024-04-25T15:51:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.083+0000 7fc65199c640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc64c063090 0x7fc64c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.083+0000 7fc65199c640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc64c063090 0x7fc64c0634a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.083+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c15a0f0 con 0x7fc64c063090 2024-04-25T15:51:38.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.095+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c0cad10 con 0x7fc64c063090 2024-04-25T15:51:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.099+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c0cad10 con 0x7fc64c063090 2024-04-25T15:51:38.103 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.099+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc64c063da0 0x7fc64c1539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.099+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc64c0cad10 con 0x7fc64c063da0 2024-04-25T15:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.099+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc64c063da0 0x7fc64c1539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.099+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc64c063da0 0x7fc64c1539f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.099+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c0cad10 con 0x7fc64c063da0 2024-04-25T15:51:38.117 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:38.117 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:38.117 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:51:38.117 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:51:38.117 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.115+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c0cad10 con 0x7fc64c063da0 2024-04-25T15:51:38.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.115+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c0cad10 con 0x7fc64c063da0 2024-04-25T15:51:38.120 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:38.121 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:38.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.115+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc64c154180 0x7fc64c02cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.115+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc64c0caf20 con 0x7fc64c154180 2024-04-25T15:51:38.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.115+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc64c154180 0x7fc64c02cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.119+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc64c154180 0x7fc64c02cef0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.119+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c0caf20 con 0x7fc64c154180 2024-04-25T15:51:38.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:38.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.131+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c02d430 con 0x7fc64c154180 2024-04-25T15:51:38.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.139+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c02d430 con 0x7fc64c154180 2024-04-25T15:51:38.142 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:38.142 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:38.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.139+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc64c02dab0 0x7fc64c20c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.139+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc64c02dec0 con 0x7fc64c02dab0 2024-04-25T15:51:38.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.139+0000 7fc65199c640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc64c02dab0 0x7fc64c20c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.139+0000 7fc65199c640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc64c02dab0 0x7fc64c20c400 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.139+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c02dec0 con 0x7fc64c02dab0 2024-04-25T15:51:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.155+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c02dec0 con 0x7fc64c02dab0 2024-04-25T15:51:38.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.159+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c02dec0 con 0x7fc64c02dab0 2024-04-25T15:51:38.163 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.159+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc64c20ce50 0x7fc64c20f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.159+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc64c2121d0 con 0x7fc64c20ce50 2024-04-25T15:51:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.159+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc64c20ce50 0x7fc64c20f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.159+0000 7fc65299e640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc64c20ce50 0x7fc64c20f270 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.163+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c2121d0 con 0x7fc64c20ce50 2024-04-25T15:51:38.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ad1ffb640 1 -- 172.21.15.67:0/2434507780 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v31) ==== 101+0+0 (secure 0 0 0) 0x7f2acc0f6e10 con 0x7f2adc0b3490 2024-04-25T15:51:38.174 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:51:38.175 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:38.168+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac006f430 msgr2=0x7f2ac0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac006f430 0x7f2ac0071940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8000c10 tx=0x7f2ad8001d20 comp rx=0 tx=0).stop 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2adc0b3490 msgr2=0x7f2adc0c3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2adc0b3490 0x7f2adc0c3580 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2acc0af040 tx=0x7f2ac8005840 comp rx=0 tx=0).stop 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 shutdown_connections 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac006f430 0x7f2ac0071940 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2adc0cc830 0x7f2adc14ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2adc0b3490 0x7f2adc0c3580 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 --2- 172.21.15.67:0/2434507780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ac8003f40 0x7f2adc14bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 >> 172.21.15.67:0/2434507780 conn(0x7f2adc0ce1c0 msgr2=0x7f2adc159d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 shutdown_connections 2024-04-25T15:51:38.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.171+0000 7f2ae33a0640 1 -- 172.21.15.67:0/2434507780 wait complete. 2024-04-25T15:51:38.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.179+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c212120 con 0x7fc64c20ce50 2024-04-25T15:51:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.183+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c212120 con 0x7fc64c20ce50 2024-04-25T15:51:38.189 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.183+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc64c211450 0x7fc64c211860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.183+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc64c211eb0 con 0x7fc64c211450 2024-04-25T15:51:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.187+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc64c211450 0x7fc64c211860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.187+0000 7fc65219d640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc64c211450 0x7fc64c211860 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.187+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc64c211eb0 con 0x7fc64c211450 2024-04-25T15:51:38.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.207+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc64c211e30 con 0x7fc64c211450 2024-04-25T15:51:38.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc642ffd640 1 -- 172.21.15.67:0/3580638102 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc64c211e30 con 0x7fc64c211450 2024-04-25T15:51:38.215 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc64c155bc0 msgr2=0x7fc64c15a700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc64c155bc0 0x7fc64c15a700 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc64c159290 msgr2=0x7fc64c1596a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc64c159290 0x7fc64c1596a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc64c063090 msgr2=0x7fc64c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc64c063090 0x7fc64c0634a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc64c063da0 msgr2=0x7fc64c1539f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc64c063da0 0x7fc64c1539f0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc64c154180 msgr2=0x7fc64c02cef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc64c154180 0x7fc64c02cef0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc64c02dab0 msgr2=0x7fc64c20c400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc64c02dab0 0x7fc64c20c400 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc65299e640 1 -- 172.21.15.67:0/3580638102 reap_dead start 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc64c20ce50 msgr2=0x7fc64c20f270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc64c20ce50 0x7fc64c20f270 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc65299e640 1 -- 172.21.15.67:0/3580638102 reap_dead start 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc64c211450 msgr2=0x7fc64c211860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc64c211450 0x7fc64c211860 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc65299e640 1 -- 172.21.15.67:0/3580638102 reap_dead start 2024-04-25T15:51:38.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc65299e640 1 -- 172.21.15.67:0/3580638102 reap_dead start 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc63006f400 msgr2=0x7fc630071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc63006f400 0x7fc630071910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6480ad000 tx=0x7fc648008040 comp rx=0 tx=0).stop 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c0b5660 msgr2=0x7fc64c14fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c0b5660 0x7fc64c14fda0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc63c00b3e0 tx=0x7fc63c00b8a0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 shutdown_connections 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc63006f400 0x7fc630071910 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 --2- 172.21.15.67:0/3580638102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc64c0b5660 0x7fc64c14fda0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 >> 172.21.15.67:0/3580638102 conn(0x7fc64c06e420 msgr2=0x7fc64c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 shutdown_connections 2024-04-25T15:51:38.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.211+0000 7fc654401640 1 -- 172.21.15.67:0/3580638102 wait complete. 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062200107000002, 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937687, 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.276 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212220+0000", 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212245+0000", 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212458+0000", 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021240899999999999 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212488+0000", 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0715e-05 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212836+0000", 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000348096 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213194+0000", 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000357413 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213591+0000", 2024-04-25T15:51:38.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039711599999999998 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224723+0000", 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011132177 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224892+0000", 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016872100000000001 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225031+0000", 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013966 2024-04-25T15:51:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225148+0000", 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116168 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062153805999998, 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013374822, 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.281 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212257+0000", 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212263+0000", 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0900000000000001e-06 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212576+0000", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031227099999999998 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212590+0000", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4133e-05 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214544+0000", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001953939 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214793+0000", 2024-04-25T15:51:38.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024941399999999999 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215181+0000", 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038823800000000001 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224739+0000", 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095574579999999996 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225301+0000", 2024-04-25T15:51:38.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056223199999999997 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225459+0000", 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015786199999999999 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225631+0000", 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172242 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062143392999999, 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013916951, 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8299999999999996e-07 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212272+0000", 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6630000000000001e-06 2024-04-25T15:51:38.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212665+0000", 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039338399999999999 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212687+0000", 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1882000000000002e-05 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215937+0000", 2024-04-25T15:51:38.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032501119999999999 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032469999999999998 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216646+0000", 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038413800000000001 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224753+0000", 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081070039999999993 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225911+0000", 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001158002 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226062+0000", 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015119600000000001 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226184+0000", 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121187 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062135423999997, 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01435588, 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.292 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:51:38.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6000000000000003e-07 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212279+0000", 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1169999999999997e-06 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212723+0000", 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044335799999999998 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212733+0000", 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349e-05 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217516+0000", 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004782606 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217766+0000", 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025045699999999999 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039180399999999999 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224766+0000", 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066076449999999997 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226308+0000", 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00154228 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226483+0000", 2024-04-25T15:51:38.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174938 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226630+0000", 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014766599999999999 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062128444000003, 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014867514, 2024-04-25T15:51:38.297 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2699999999999999e-07 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212287+0000", 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6229999999999996e-06 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212756+0000", 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046907999999999997 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9560000000000003e-06 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218829+0000", 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060638690000000004 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219182+0000", 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035291899999999999 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219507+0000", 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032545399999999998 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224779+0000", 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052717279999999998 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226860+0000", 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020809790000000002 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227036+0000", 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176062 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227149+0000", 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113217 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062120974000003, 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015281197, 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212290+0000", 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3e-07 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212333+0000", 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3251000000000003e-05 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212784+0000", 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045139399999999998 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212793+0000", 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7960000000000001e-06 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220253+0000", 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074595699999999996 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220482+0000", 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000229554 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220879+0000", 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039646400000000002 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225181+0000", 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00430205 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227277+0000", 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020961119999999998 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227449+0000", 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017166800000000001 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227570+0000", 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012170799999999999 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062071494000001, 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016239407000000001, 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212340+0000", 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1200000000000001e-06 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212346+0000", 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.584e-06 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212813+0000", 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046677600000000002 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212821+0000", 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221805+0000", 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089835999999999996 2024-04-25T15:51:38.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222154+0000", 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034940299999999997 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222503+0000", 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034907900000000002 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227840+0000", 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367349999999996 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228310+0000", 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047011999999999998 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228439+0000", 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012899999999999999 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228578+0000", 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138837 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062060283999998, 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016443972000000001, 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212351+0000", 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212357+0000", 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000003e-06 2024-04-25T15:51:38.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212848+0000", 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049076600000000001 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212863+0000", 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5498999999999998e-05 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223300+0000", 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010437093999999999 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223537+0000", 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000236665 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224027+0000", 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048967900000000003 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228184+0000", 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041571209999999997 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228599+0000", 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000415392 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228676+0000", 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7095999999999995e-05 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228794+0000", 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011746700000000001 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062049305000002, 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013451848000000001, 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212362+0000", 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212368+0000", 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6229999999999999e-06 2024-04-25T15:51:38.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212925+0000", 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055690199999999996 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212960+0000", 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5045000000000001e-05 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213123+0000", 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163222 2024-04-25T15:51:38.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213447+0000", 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000323627 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213759+0000", 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031232800000000001 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216671+0000", 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002911145 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223654+0000", 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069831370000000004 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223829+0000", 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174857 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225813+0000", 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019840679999999999 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.062038141999999, 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014242332, 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212373+0000", 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4e-07 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212384+0000", 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1443e-05 2024-04-25T15:51:38.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213026+0000", 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064212500000000001 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213045+0000", 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8621999999999999e-05 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214708+0000", 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016628109999999999 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214970+0000", 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026197699999999999 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215380+0000", 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000410789 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218599+0000", 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032185270000000001 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226296+0000", 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076965269999999999 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226473+0000", 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017755200000000001 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226614+0000", 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014111899999999999 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:4 2.10 2:0f65c9b8:::60a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.027234808000003, 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010394548999999999, 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247176+0000", 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2100000000000001e-06 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247209+0000", 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2945999999999999e-05 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247394+0000", 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018468799999999999 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247417+0000", 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2733000000000001e-05 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249734+0000", 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002317684 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250032+0000", 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029727699999999998 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250453+0000", 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042143500000000001 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255742+0000", 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052886809999999999 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257232+0000", 2024-04-25T15:51:38.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014906260000000001 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257393+0000", 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016088699999999999 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257570+0000", 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017638199999999999 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.332 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943266242, 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094210270000000002, 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:51:38.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6699999999999999e-07 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331165+0000", 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331418+0000", 2024-04-25T15:51:38.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271799999999997 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331428+0000", 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6460000000000005e-06 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333229+0000", 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018009499999999999 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334396+0000", 2024-04-25T15:51:38.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011673269999999999 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334597+0000", 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020135300000000001 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339905+0000", 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053077610000000002 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340210+0000", 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030499999999999999 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340385+0000", 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017450199999999999 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340565+0000", 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018011700000000001 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943241495999999, 2024-04-25T15:51:38.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095762039999999996, 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331174+0000", 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-06 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331467+0000", 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029356 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331478+0000", 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336604+0000", 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051267170000000003 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336735+0000", 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013052699999999999 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:38.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336996+0000", 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260511 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339986+0000", 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002990487 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340579+0000", 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059299899999999998 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340673+0000", 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4241999999999997e-05 2024-04-25T15:51:38.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340745+0000", 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1484999999999997e-05 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:17 2.10 2:0e9da45e:::61b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943234250000003, 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010814343000000001, 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331180+0000", 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6650000000000001e-06 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331501+0000", 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032061600000000002 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331509+0000", 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8240000000000006e-06 2024-04-25T15:51:38.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333383+0000", 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018733020000000001 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333652+0000", 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026890399999999999 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333865+0000", 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021341 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338313+0000", 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044478269999999997 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341868+0000", 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035546729999999999 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341920+0000", 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2507999999999999e-05 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341990+0000", 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0110000000000005e-05 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943224577999999, 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011086621, 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331186+0000", 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.51e-07 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331191+0000", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3970000000000002e-06 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331546+0000", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035447799999999998 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331557+0000", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0791000000000001e-05 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334408+0000", 2024-04-25T15:51:38.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028516520000000001 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334546+0000", 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013802700000000001 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334877+0000", 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000331081 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339910+0000", 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050323449999999997 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342080+0000", 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021703899999999999 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342184+0000", 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103767 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342272+0000", 2024-04-25T15:51:38.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8242e-05 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943216558000003, 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010331643, 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:51:38.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331194+0000", 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331198+0000", 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1400000000000002e-06 2024-04-25T15:51:38.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331608+0000", 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040962799999999999 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331620+0000", 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337545+0000", 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005924726 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338432+0000", 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088720699999999999 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338639+0000", 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020665000000000001 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341318+0000", 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026785860000000002 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341406+0000", 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8344000000000007e-05 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341458+0000", 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805999999999998e-05 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341525+0000", 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7336999999999996e-05 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:2 2.10 2:0daba267:::402.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943210098999998, 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011245289, 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331215+0000", 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331647+0000", 2024-04-25T15:51:38.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043194799999999998 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331656+0000", 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335316+0000", 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036607710000000002 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335504+0000", 2024-04-25T15:51:38.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018793299999999999 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335699+0000", 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194544 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339945+0000", 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042457600000000003 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342333+0000", 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023885730000000002 2024-04-25T15:51:38.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342383+0000", 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9314999999999998e-05 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342445+0000", 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2691e-05 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:51:38.361 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943191337999998, 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087706009999999994, 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331239+0000", 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9653e-05 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331678+0000", 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043868999999999999 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331701+0000", 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3019000000000001e-05 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333753+0000", 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020522050000000001 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334131+0000", 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000378484 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334455+0000", 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032319899999999998 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337268+0000", 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028133199999999998 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339816+0000", 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025479140000000001 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339907+0000", 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1150000000000007e-05 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339989+0000", 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2392000000000007e-05 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:27 2.10 2:091e33df:::502.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943168297, 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011447874, 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:51:38.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4199999999999996e-07 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331249+0000", 2024-04-25T15:51:38.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2259999999999999e-06 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331730+0000", 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048009600000000001 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331739+0000", 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1840000000000005e-06 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336653+0000", 2024-04-25T15:51:38.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049141510000000003 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336830+0000", 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017740099999999999 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337033+0000", 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020295999999999999 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340803+0000", 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037692480000000002 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342567+0000", 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017649250000000001 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342650+0000", 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2748000000000001e-05 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342690+0000", 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9393e-05 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:31 2.10 2:0eee8b97:::202.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.943158439999998, 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012346048, 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.372 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:51:38.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5300000000000004e-07 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331255+0000", 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8380000000000002e-06 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331760+0000", 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050452600000000004 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331768+0000", 2024-04-25T15:51:38.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337613+0000", 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058448700000000003 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337836+0000", 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000223043 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.343598+0000", 2024-04-25T15:51:38.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057612679999999999 2024-04-25T15:51:38.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.376 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.376 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.376 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:38.376 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:38.376 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:38.376 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T15:51:38.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:51:38.377 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:38.272+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:38.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:38.379 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:38.372+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:38.479 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:38.475+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:38.573 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:38.573 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:38.573 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:51:38.573 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:38.580 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:38.576+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a4a18b640 1 Processor -- start 2024-04-25T15:51:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a4a18b640 1 -- start start 2024-04-25T15:51:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a4a18b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 0x7f8a440736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a4a18b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a440caa60 con 0x7f8a441547e0 2024-04-25T15:51:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a4a18b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4415a5f0 con 0x7f8a4415aaf0 2024-04-25T15:51:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a4a18b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a44068590 con 0x7f8a4406d460 2024-04-25T15:51:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a42ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a441547e0 0x7f8a4406c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47050/0 (socket says 172.21.15.67:47050) 2024-04-25T15:51:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a42ffd640 1 -- 172.21.15.67:0/2310154382 learned_addr learned my addr 172.21.15.67:0/2310154382 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a437fe640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 0x7f8a440736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2551671326 0 0) 0x7f8a440caa60 con 0x7f8a441547e0 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a2c003530 con 0x7f8a441547e0 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1407459386 0 0) 0x7f8a2c003530 con 0x7f8a441547e0 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 msgr2=0x7f8a440736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 0x7f8a440736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 >> v1:172.21.15.176:6789/0 conn(0x7f8a4415aaf0 legacy=0x7f8a4415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a440ca100 con 0x7f8a441547e0 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.583+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8a34001a50 con 0x7f8a441547e0 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 >> v1:172.21.15.67:6789/0 conn(0x7f8a441547e0 legacy=0x7f8a4406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a2c0039f0 0x7f8a2c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 0x7f8a2c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c004d50 0x7f8a2c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44068590 con 0x7f8a2c0039f0 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c00ba80 con 0x7f8a2c004d50 2024-04-25T15:51:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2c00bc80 con 0x7f8a4406d460 2024-04-25T15:51:38.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:38.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a437fe640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a2c0039f0 0x7f8a2c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a42ffd640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 0x7f8a2c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a43fff640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c004d50 0x7f8a2c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a437fe640 1 -- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 msgr2=0x7f8a2c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a437fe640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a4406d460 0x7f8a2c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a437fe640 1 -- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c004d50 msgr2=0x7f8a2c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a437fe640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c004d50 0x7f8a2c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a437fe640 1 -- 172.21.15.67:0/2310154382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2c003530 con 0x7f8a2c0039f0 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a437fe640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a2c0039f0 0x7f8a2c003e20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009ae0 tx=0x7f8a3801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd71054a1885503 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380b2070 con 0x7f8a2c0039f0 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a380a5680 con 0x7f8a2c0039f0 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a427fc640 1 -- 172.21.15.67:0/2310154382 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380096b0 con 0x7f8a2c0039f0 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a4a18b640 1 -- 172.21.15.67:0/2310154382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a2c0039f0 msgr2=0x7f8a2c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a4a18b640 1 --2- 172.21.15.67:0/2310154382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a2c0039f0 0x7f8a2c003e20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009ae0 tx=0x7f8a3801ca90 comp rx=0 tx=0).stop 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a43fff640 1 -- 172.21.15.67:0/2310154382 reap_dead start 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a4a18b640 1 -- 172.21.15.67:0/2310154382 shutdown_connections 2024-04-25T15:51:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a4a18b640 1 -- 172.21.15.67:0/2310154382 >> 172.21.15.67:0/2310154382 conn(0x7f8a440ce1c0 msgr2=0x7f8a440ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:38.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a4a18b640 1 -- 172.21.15.67:0/2310154382 shutdown_connections 2024-04-25T15:51:38.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.587+0000 7f8a4a18b640 1 -- 172.21.15.67:0/2310154382 wait complete. 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 Processor -- start 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 -- start start 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c0039f0 0x7f8a441f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a441e3300 0x7f8a441e3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a441e06d0 0x7f8a441e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44068ac0 con 0x7f8a441e06d0 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a440687e0 con 0x7f8a2c0039f0 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a4a18b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a44154bf0 con 0x7f8a441e3300 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a441e06d0 0x7f8a441e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a441e06d0 0x7f8a441e1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47416/0 (socket says 172.21.15.67:47416) 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 -- 172.21.15.67:0/1712184504 learned_addr learned my addr 172.21.15.67:0/1712184504 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a42ffd640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a441e3300 0x7f8a441e3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a437fe640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c0039f0 0x7f8a441f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 -- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a441e3300 msgr2=0x7f8a441e3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a441e3300 0x7f8a441e3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 -- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c0039f0 msgr2=0x7f8a441f0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c0039f0 0x7f8a441f0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 -- 172.21.15.67:0/1712184504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a441e0bb0 con 0x7f8a441e06d0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.591+0000 7f8a43fff640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a441e06d0 0x7f8a441e1280 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c010e40 tx=0x7f8a3c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a2bfff640 1 -- 172.21.15.67:0/1712184504 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00ad70 con 0x7f8a441e06d0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a2bfff640 1 -- 172.21.15.67:0/1712184504 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c019070 con 0x7f8a441e06d0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a441f4aa0 con 0x7f8a441e06d0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a2bfff640 1 -- 172.21.15.67:0/1712184504 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0052b0 con 0x7f8a441e06d0 2024-04-25T15:51:38.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a44157aa0 con 0x7f8a441e06d0 2024-04-25T15:51:38.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a441e0d80 con 0x7f8a441e06d0 2024-04-25T15:51:38.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a2bfff640 1 -- 172.21.15.67:0/1712184504 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8a3c013070 con 0x7f8a441e06d0 2024-04-25T15:51:38.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a2bfff640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a3006f430 0x7f8a30071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a2bfff640 1 -- 172.21.15.67:0/1712184504 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5796+0+0 (secure 0 0 0) 0x7f8a3c0921e0 con 0x7f8a441e06d0 2024-04-25T15:51:38.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.595+0000 7f8a437fe640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a3006f430 0x7f8a30071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.599+0000 7f8a437fe640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a3006f430 0x7f8a30071940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8a380af000 tx=0x7f8a380046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.599+0000 7f8a2bfff640 1 -- 172.21.15.67:0/1712184504 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8a3c05ea20 con 0x7f8a441e06d0 2024-04-25T15:51:38.680 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:38.676+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:38.736 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:51:38.736 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:51:38.736 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:38.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:38.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.767+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f8a4406a480 con 0x7f8a441e06d0 2024-04-25T15:51:38.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a2bfff640 1 -- 172.21.15.67:0/1712184504 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v31) ==== 81+0+34774 (secure 0 0 0) 0x7f8a3c0670f0 con 0x7f8a441e06d0 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 31, 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:51:38.167461+0000", 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:51:28.097961+0000", 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5799936, 2024-04-25T15:51:38.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:51:38.776 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:51:38.777 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:51:38.778 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:51:38.779 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:51:38.780 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.781 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 8, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 8, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.25, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 1, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 1 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:51:38.782 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:51:38.783 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:51:38.784 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:51:38.785 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:51:38.786 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.7400000095367432, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.7400000095367432, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T15:51:38.787 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 1, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 1 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:51:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:51:38.789 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:51:38.790 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.791 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:51:38.792 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.4900000095367432, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.4900000095367432, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.87999999523162842, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.309999942779541, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.309999942779541, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 1, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 1 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.793 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 27, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.794 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.796 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:51:38.797 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.798 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 0, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 18, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.799 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.802 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:51:38.803 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 27, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:51:38.804 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.806 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.807 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:38.808 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 27, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.809 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.810 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.811 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:51:38.812 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.813 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 27, 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.814 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.815 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:51:38.817 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.818 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 27, 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:51:38.819 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.820 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.821 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.822 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 27, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.827 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:51:38.828 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:38.829 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 27, 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:38.832 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:51:38.833 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.834 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 65536, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:38.836 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.837 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [], 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:51:38.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:51:38.840 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:51:38.841 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:51:38.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.843 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a3006f430 msgr2=0x7f8a30071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a3006f430 0x7f8a30071940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8a380af000 tx=0x7f8a380046d0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a441e06d0 msgr2=0x7f8a441e1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a441e06d0 0x7f8a441e1280 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c010e40 tx=0x7f8a3c00e530 comp rx=0 tx=0).stop 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 shutdown_connections 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a3006f430 0x7f8a30071940 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a441e06d0 0x7f8a441e1280 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a441e3300 0x7f8a441e3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 --2- 172.21.15.67:0/1712184504 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a2c0039f0 0x7f8a441f0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 >> 172.21.15.67:0/1712184504 conn(0x7f8a440ce1c0 msgr2=0x7f8a44073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 shutdown_connections 2024-04-25T15:51:38.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:38.771+0000 7f8a4a18b640 1 -- 172.21.15.67:0/1712184504 wait complete. 2024-04-25T15:51:38.846 INFO:tasks.thrashosds.thrasher:j is {'epoch': 31, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:51:38.167461+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:51:28.097961+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 27, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:51:38.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:51:38.846 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:38.839+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.703672165999997, 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:38.919 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:38.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:38.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.703632521000003, 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:38.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:38.926 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.700725499000001, 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:38.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.700572563000001, 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.930 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:38.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:38.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.699056919, 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:38.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.698914272000003, 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:38.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.698811180999996, 2024-04-25T15:51:38.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:38.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.698658430999998, 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.945 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:38.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:38.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 62.696710205000002, 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:38.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.668682574000002, 2024-04-25T15:51:38.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.955 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.666659682999999, 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:38.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:38.959 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.666489134000003, 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.664133638000003, 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:38.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.664098926000001, 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:38.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.664078117000003, 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.975 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.587191070000003, 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:38.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:38.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:38.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:38.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.587166988, 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.587157290999997, 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:38.985 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:38.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.581262354000003, 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.991823697000001, 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:38.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:38.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:38.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:38.998 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:38.999 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:38.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:38.995+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:39.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a59ce1640 1 Processor -- start 2024-04-25T15:51:39.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a59ce1640 1 -- start start 2024-04-25T15:51:39.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a59ce1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a59ce1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a540c6fb0 0x7f9a540c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a59ce1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a54072dc0 con 0x7f9a5415e020 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a59ce1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9a5415b430 con 0x7f9a540c6fb0 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a59ce1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a5415a670 con 0x7f9a54074790 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a4bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9a5415e020 0x7f9a540723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47058/0 (socket says 172.21.15.67:47058) 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a53fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a540c6fb0 0x7f9a540c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a4bfff640 1 -- 172.21.15.67:0/1644526580 learned_addr learned my addr 172.21.15.67:0/1644526580 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a537fe640 1 --2- 172.21.15.67:0/1644526580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a53fff640 1 -- 172.21.15.67:0/1644526580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 msgr2=0x7f9a5415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a53fff640 1 --2- 172.21.15.67:0/1644526580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5415dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a53fff640 1 -- 172.21.15.67:0/1644526580 >> v1:172.21.15.67:6789/0 conn(0x7f9a5415e020 legacy=0x7f9a540723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a53fff640 1 -- 172.21.15.67:0/1644526580 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5415a950 con 0x7f9a540c6fb0 2024-04-25T15:51:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.035+0000 7f9a52ffd640 1 -- 172.21.15.67:0/1644526580 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659110159 0 0) 0x7f9a54072dc0 con 0x7f9a5415e020 2024-04-25T15:51:39.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a53fff640 1 --2- 172.21.15.67:0/1644526580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a540c73c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009d90 tx=0x7f9a4c01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204200a319683c81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:39.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a52ffd640 1 -- 172.21.15.67:0/1644526580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0b4070 con 0x7f9a540c6fb0 2024-04-25T15:51:39.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a52ffd640 1 -- 172.21.15.67:0/1644526580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a4c0af600 con 0x7f9a540c6fb0 2024-04-25T15:51:39.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a52ffd640 1 -- 172.21.15.67:0/1644526580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0af9d0 con 0x7f9a540c6fb0 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1644526580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 msgr2=0x7f9a540c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1644526580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a540c73c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009d90 tx=0x7f9a4c01f150 comp rx=0 tx=0).stop 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1644526580 shutdown_connections 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1644526580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a540c73c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1644526580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5415dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1644526580 >> 172.21.15.67:0/1644526580 conn(0x7f9a5406fe70 msgr2=0x7f9a540cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1644526580 shutdown_connections 2024-04-25T15:51:39.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1644526580 wait complete. 2024-04-25T15:51:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 Processor -- start 2024-04-25T15:51:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.039+0000 7f9a59ce1640 1 -- start start 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5414b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a541f4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a5415e020 0x7f9a54148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9a54073080 con 0x7f9a540c6fb0 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5415af10 con 0x7f9a5415e020 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a54072ab0 con 0x7f9a54074790 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a541f4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a541f4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47438/0 (socket says 172.21.15.67:47438) 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 -- 172.21.15.67:0/1633637157 learned_addr learned my addr 172.21.15.67:0/1633637157 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a537fe640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5414b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a53fff640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a5415e020 0x7f9a54148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 -- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 msgr2=0x7f9a5414b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5414b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 -- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a5415e020 msgr2=0x7f9a54148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a5415e020 0x7f9a54148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 -- 172.21.15.67:0/1633637157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a541f3e90 con 0x7f9a540c6fb0 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a4bfff640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a541f4620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400c4b0 tx=0x7f9a4400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a50ff9640 1 -- 172.21.15.67:0/1633637157 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a44018070 con 0x7f9a540c6fb0 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a50ff9640 1 -- 172.21.15.67:0/1633637157 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a44007500 con 0x7f9a540c6fb0 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a541f0880 con 0x7f9a540c6fb0 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a50ff9640 1 -- 172.21.15.67:0/1633637157 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a44004a30 con 0x7f9a540c6fb0 2024-04-25T15:51:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5415a430 con 0x7f9a540c6fb0 2024-04-25T15:51:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5414baf0 con 0x7f9a540c6fb0 2024-04-25T15:51:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.043+0000 7f9a50ff9640 1 -- 172.21.15.67:0/1633637157 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9a4401b070 con 0x7f9a540c6fb0 2024-04-25T15:51:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.047+0000 7f9a50ff9640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3406f400 0x7f9a34071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.047+0000 7f9a50ff9640 1 -- 172.21.15.67:0/1633637157 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5796+0+0 (secure 0 0 0) 0x7f9a4408e7b0 con 0x7f9a540c6fb0 2024-04-25T15:51:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.047+0000 7f9a537fe640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3406f400 0x7f9a34071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.047+0000 7f9a537fe640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3406f400 0x7f9a34071910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009ac0 tx=0x7f9a3c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.051+0000 7f9a50ff9640 1 -- 172.21.15.67:0/1633637157 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9a4405b160 con 0x7f9a540c6fb0 2024-04-25T15:51:39.100 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:39.095+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:39.156 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:39.156 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:39.156 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:39.157 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:39.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:39.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.211+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9a5414aef0 con 0x7f9a3406f400 2024-04-25T15:51:39.219 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:39.216+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.219+0000 7f9a50ff9640 1 -- 172.21.15.67:0/1633637157 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251959 (secure 0 0 0) 0x7f9a5414aef0 con 0x7f9a3406f400 2024-04-25T15:51:39.225 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:39.226 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:51:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3406f400 msgr2=0x7f9a34071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3406f400 0x7f9a34071910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009ac0 tx=0x7f9a3c009190 comp rx=0 tx=0).stop 2024-04-25T15:51:39.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 msgr2=0x7f9a541f4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a541f4620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400c4b0 tx=0x7f9a4400c970 comp rx=0 tx=0).stop 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 shutdown_connections 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a3406f400 0x7f9a34071910 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a5415e020 0x7f9a54148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a540c6fb0 0x7f9a541f4620 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 --2- 172.21.15.67:0/1633637157 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a54074790 0x7f9a5414b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 >> 172.21.15.67:0/1633637157 conn(0x7f9a5406fe70 msgr2=0x7f9a5414e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 shutdown_connections 2024-04-25T15:51:39.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.223+0000 7f9a59ce1640 1 -- 172.21.15.67:0/1633637157 wait complete. 2024-04-25T15:51:39.246 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-04-25T15:51:38.177060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604666,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":319,"num_write_kb":1704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1532,"num_keys_recovered":12,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":192820,"kb_used_data":3664,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660410060,"statfs":{"total":676457349120,"available":676259901440,"internally_reserved":0,"allocated":3751936,"data_stored":2131533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":193582189},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1532,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.357207"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.395027+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:51:35.395027+0000","last_peered":"2024-04-25T15:51:35.395027+0000","last_clean":"2024-04-25T15:51:35.395027+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:51:35.395027+0000","last_undegraded":"2024-04-25T15:51:35.395027+0000","last_fullsized":"2024-04-25T15:51:35.395027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656778+0000","last_change":"2024-04-25T15:51:29.135531+0000","last_active":"2024-04-25T15:51:35.656778+0000","last_peered":"2024-04-25T15:51:35.656778+0000","last_clean":"2024-04-25T15:51:35.656778+0000","last_became_active":"2024-04-25T15:51:29.134294+0000","last_became_peered":"2024-04-25T15:51:29.134294+0000","last_unstale":"2024-04-25T15:51:35.656778+0000","last_undegraded":"2024-04-25T15:51:35.656778+0000","last_fullsized":"2024-04-25T15:51:35.656778+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:18:08.237765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.655498+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:51:35.655498+0000","last_peered":"2024-04-25T15:51:35.655498+0000","last_clean":"2024-04-25T15:51:35.655498+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:51:35.655498+0000","last_undegraded":"2024-04-25T15:51:35.655498+0000","last_fullsized":"2024-04-25T15:51:35.655498+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544232+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:51:35.544232+0000","last_peered":"2024-04-25T15:51:35.544232+0000","last_clean":"2024-04-25T15:51:35.544232+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:51:35.544232+0000","last_undegraded":"2024-04-25T15:51:35.544232+0000","last_fullsized":"2024-04-25T15:51:35.544232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393291+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:51:35.393291+0000","last_peered":"2024-04-25T15:51:35.393291+0000","last_clean":"2024-04-25T15:51:35.393291+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:51:35.393291+0000","last_undegraded":"2024-04-25T15:51:35.393291+0000","last_fullsized":"2024-04-25T15:51:35.393291+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.543960+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:51:35.543960+0000","last_peered":"2024-04-25T15:51:35.543960+0000","last_clean":"2024-04-25T15:51:35.543960+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:51:35.543960+0000","last_undegraded":"2024-04-25T15:51:35.543960+0000","last_fullsized":"2024-04-25T15:51:35.543960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393786+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:51:35.393786+0000","last_peered":"2024-04-25T15:51:35.393786+0000","last_clean":"2024-04-25T15:51:35.393786+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:51:35.393786+0000","last_undegraded":"2024-04-25T15:51:35.393786+0000","last_fullsized":"2024-04-25T15:51:35.393786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440334+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:51:35.440334+0000","last_peered":"2024-04-25T15:51:35.440334+0000","last_clean":"2024-04-25T15:51:35.440334+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:51:35.440334+0000","last_undegraded":"2024-04-25T15:51:35.440334+0000","last_fullsized":"2024-04-25T15:51:35.440334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.153199+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:51:35.153199+0000","last_peered":"2024-04-25T15:51:35.153199+0000","last_clean":"2024-04-25T15:51:35.153199+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:51:35.153199+0000","last_undegraded":"2024-04-25T15:51:35.153199+0000","last_fullsized":"2024-04-25T15:51:35.153199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.563051+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:51:35.563051+0000","last_peered":"2024-04-25T15:51:35.563051+0000","last_clean":"2024-04-25T15:51:35.563051+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:51:35.563051+0000","last_undegraded":"2024-04-25T15:51:35.563051+0000","last_fullsized":"2024-04-25T15:51:35.563051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.562760+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:51:35.562760+0000","last_peered":"2024-04-25T15:51:35.562760+0000","last_clean":"2024-04-25T15:51:35.562760+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:51:35.562760+0000","last_undegraded":"2024-04-25T15:51:35.562760+0000","last_fullsized":"2024-04-25T15:51:35.562760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152360+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:51:35.152360+0000","last_peered":"2024-04-25T15:51:35.152360+0000","last_clean":"2024-04-25T15:51:35.152360+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:51:35.152360+0000","last_undegraded":"2024-04-25T15:51:35.152360+0000","last_fullsized":"2024-04-25T15:51:35.152360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.441071+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:51:35.441071+0000","last_peered":"2024-04-25T15:51:35.441071+0000","last_clean":"2024-04-25T15:51:35.441071+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:51:35.441071+0000","last_undegraded":"2024-04-25T15:51:35.441071+0000","last_fullsized":"2024-04-25T15:51:35.441071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656063+0000","last_change":"2024-04-25T15:51:29.137669+0000","last_active":"2024-04-25T15:51:35.656063+0000","last_peered":"2024-04-25T15:51:35.656063+0000","last_clean":"2024-04-25T15:51:35.656063+0000","last_became_active":"2024-04-25T15:51:29.137296+0000","last_became_peered":"2024-04-25T15:51:29.137296+0000","last_unstale":"2024-04-25T15:51:35.656063+0000","last_undegraded":"2024-04-25T15:51:35.656063+0000","last_fullsized":"2024-04-25T15:51:35.656063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:20.758522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152108+0000","last_change":"2024-04-25T15:51:29.130420+0000","last_active":"2024-04-25T15:51:35.152108+0000","last_peered":"2024-04-25T15:51:35.152108+0000","last_clean":"2024-04-25T15:51:35.152108+0000","last_became_active":"2024-04-25T15:51:29.129685+0000","last_became_peered":"2024-04-25T15:51:29.129685+0000","last_unstale":"2024-04-25T15:51:35.152108+0000","last_undegraded":"2024-04-25T15:51:35.152108+0000","last_fullsized":"2024-04-25T15:51:35.152108+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:48.724437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.543483+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:51:35.543483+0000","last_peered":"2024-04-25T15:51:35.543483+0000","last_clean":"2024-04-25T15:51:35.543483+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:51:35.543483+0000","last_undegraded":"2024-04-25T15:51:35.543483+0000","last_fullsized":"2024-04-25T15:51:35.543483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544914+0000","last_change":"2024-04-25T15:50:31.198225+0000","last_active":"2024-04-25T15:51:35.544914+0000","last_peered":"2024-04-25T15:51:35.544914+0000","last_clean":"2024-04-25T15:51:35.544914+0000","last_became_active":"2024-04-25T15:50:31.196737+0000","last_became_peered":"2024-04-25T15:50:31.196737+0000","last_unstale":"2024-04-25T15:51:35.544914+0000","last_undegraded":"2024-04-25T15:51:35.544914+0000","last_fullsized":"2024-04-25T15:51:35.544914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:25.652638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.153072+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:51:35.153072+0000","last_peered":"2024-04-25T15:51:35.153072+0000","last_clean":"2024-04-25T15:51:35.153072+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:51:35.153072+0000","last_undegraded":"2024-04-25T15:51:35.153072+0000","last_fullsized":"2024-04-25T15:51:35.153072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393039+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:51:35.393039+0000","last_peered":"2024-04-25T15:51:35.393039+0000","last_clean":"2024-04-25T15:51:35.393039+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:51:35.393039+0000","last_undegraded":"2024-04-25T15:51:35.393039+0000","last_fullsized":"2024-04-25T15:51:35.393039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.561744+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:51:35.561744+0000","last_peered":"2024-04-25T15:51:35.561744+0000","last_clean":"2024-04-25T15:51:35.561744+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:51:35.561744+0000","last_undegraded":"2024-04-25T15:51:35.561744+0000","last_fullsized":"2024-04-25T15:51:35.561744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656267+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:51:35.656267+0000","last_peered":"2024-04-25T15:51:35.656267+0000","last_clean":"2024-04-25T15:51:35.656267+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:51:35.656267+0000","last_undegraded":"2024-04-25T15:51:35.656267+0000","last_fullsized":"2024-04-25T15:51:35.656267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152902+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:51:35.152902+0000","last_peered":"2024-04-25T15:51:35.152902+0000","last_clean":"2024-04-25T15:51:35.152902+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:51:35.152902+0000","last_undegraded":"2024-04-25T15:51:35.152902+0000","last_fullsized":"2024-04-25T15:51:35.152902+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440038+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:51:35.440038+0000","last_peered":"2024-04-25T15:51:35.440038+0000","last_clean":"2024-04-25T15:51:35.440038+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:51:35.440038+0000","last_undegraded":"2024-04-25T15:51:35.440038+0000","last_fullsized":"2024-04-25T15:51:35.440038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.543539+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:51:35.543539+0000","last_peered":"2024-04-25T15:51:35.543539+0000","last_clean":"2024-04-25T15:51:35.543539+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:51:35.543539+0000","last_undegraded":"2024-04-25T15:51:35.543539+0000","last_fullsized":"2024-04-25T15:51:35.543539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440135+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:51:35.440135+0000","last_peered":"2024-04-25T15:51:35.440135+0000","last_clean":"2024-04-25T15:51:35.440135+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:51:35.440135+0000","last_undegraded":"2024-04-25T15:51:35.440135+0000","last_fullsized":"2024-04-25T15:51:35.440135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.563253+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:51:35.563253+0000","last_peered":"2024-04-25T15:51:35.563253+0000","last_clean":"2024-04-25T15:51:35.563253+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:51:35.563253+0000","last_undegraded":"2024-04-25T15:51:35.563253+0000","last_fullsized":"2024-04-25T15:51:35.563253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.441434+0000","last_change":"2024-04-25T15:51:29.143177+0000","last_active":"2024-04-25T15:51:35.441434+0000","last_peered":"2024-04-25T15:51:35.441434+0000","last_clean":"2024-04-25T15:51:35.441434+0000","last_became_active":"2024-04-25T15:51:29.142297+0000","last_became_peered":"2024-04-25T15:51:29.142297+0000","last_unstale":"2024-04-25T15:51:35.441434+0000","last_undegraded":"2024-04-25T15:51:35.441434+0000","last_fullsized":"2024-04-25T15:51:35.441434+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:23.065855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440136+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:51:35.440136+0000","last_peered":"2024-04-25T15:51:35.440136+0000","last_clean":"2024-04-25T15:51:35.440136+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:51:35.440136+0000","last_undegraded":"2024-04-25T15:51:35.440136+0000","last_fullsized":"2024-04-25T15:51:35.440136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656520+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:51:35.656520+0000","last_peered":"2024-04-25T15:51:35.656520+0000","last_clean":"2024-04-25T15:51:35.656520+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:51:35.656520+0000","last_undegraded":"2024-04-25T15:51:35.656520+0000","last_fullsized":"2024-04-25T15:51:35.656520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.543984+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:51:35.543984+0000","last_peered":"2024-04-25T15:51:35.543984+0000","last_clean":"2024-04-25T15:51:35.543984+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:51:35.543984+0000","last_undegraded":"2024-04-25T15:51:35.543984+0000","last_fullsized":"2024-04-25T15:51:35.543984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152789+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:51:35.152789+0000","last_peered":"2024-04-25T15:51:35.152789+0000","last_clean":"2024-04-25T15:51:35.152789+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:51:35.152789+0000","last_undegraded":"2024-04-25T15:51:35.152789+0000","last_fullsized":"2024-04-25T15:51:35.152789+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.562078+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:51:35.562078+0000","last_peered":"2024-04-25T15:51:35.562078+0000","last_clean":"2024-04-25T15:51:35.562078+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:51:35.562078+0000","last_undegraded":"2024-04-25T15:51:35.562078+0000","last_fullsized":"2024-04-25T15:51:35.562078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152127+0000","last_change":"2024-04-25T15:51:29.141670+0000","last_active":"2024-04-25T15:51:35.152127+0000","last_peered":"2024-04-25T15:51:35.152127+0000","last_clean":"2024-04-25T15:51:35.152127+0000","last_became_active":"2024-04-25T15:51:29.141234+0000","last_became_peered":"2024-04-25T15:51:29.141234+0000","last_unstale":"2024-04-25T15:51:35.152127+0000","last_undegraded":"2024-04-25T15:51:35.152127+0000","last_fullsized":"2024-04-25T15:51:35.152127+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:55.877956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.441083+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:51:35.441083+0000","last_peered":"2024-04-25T15:51:35.441083+0000","last_clean":"2024-04-25T15:51:35.441083+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:51:35.441083+0000","last_undegraded":"2024-04-25T15:51:35.441083+0000","last_fullsized":"2024-04-25T15:51:35.441083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.655272+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:51:35.655272+0000","last_peered":"2024-04-25T15:51:35.655272+0000","last_clean":"2024-04-25T15:51:35.655272+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:51:35.655272+0000","last_undegraded":"2024-04-25T15:51:35.655272+0000","last_fullsized":"2024-04-25T15:51:35.655272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.562429+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:51:35.562429+0000","last_peered":"2024-04-25T15:51:35.562429+0000","last_clean":"2024-04-25T15:51:35.562429+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:51:35.562429+0000","last_undegraded":"2024-04-25T15:51:35.562429+0000","last_fullsized":"2024-04-25T15:51:35.562429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440905+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:51:35.440905+0000","last_peered":"2024-04-25T15:51:35.440905+0000","last_clean":"2024-04-25T15:51:35.440905+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:51:35.440905+0000","last_undegraded":"2024-04-25T15:51:35.440905+0000","last_fullsized":"2024-04-25T15:51:35.440905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440484+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:51:35.440484+0000","last_peered":"2024-04-25T15:51:35.440484+0000","last_clean":"2024-04-25T15:51:35.440484+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:51:35.440484+0000","last_undegraded":"2024-04-25T15:51:35.440484+0000","last_fullsized":"2024-04-25T15:51:35.440484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.543696+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:51:35.543696+0000","last_peered":"2024-04-25T15:51:35.543696+0000","last_clean":"2024-04-25T15:51:35.543696+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:51:35.543696+0000","last_undegraded":"2024-04-25T15:51:35.543696+0000","last_fullsized":"2024-04-25T15:51:35.543696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440456+0000","last_change":"2024-04-25T15:51:29.130175+0000","last_active":"2024-04-25T15:51:35.440456+0000","last_peered":"2024-04-25T15:51:35.440456+0000","last_clean":"2024-04-25T15:51:35.440456+0000","last_became_active":"2024-04-25T15:51:29.126559+0000","last_became_peered":"2024-04-25T15:51:29.126559+0000","last_unstale":"2024-04-25T15:51:35.440456+0000","last_undegraded":"2024-04-25T15:51:35.440456+0000","last_fullsized":"2024-04-25T15:51:35.440456+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:19:13.623623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.563056+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:51:35.563056+0000","last_peered":"2024-04-25T15:51:35.563056+0000","last_clean":"2024-04-25T15:51:35.563056+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:51:35.563056+0000","last_undegraded":"2024-04-25T15:51:35.563056+0000","last_fullsized":"2024-04-25T15:51:35.563056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544464+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:51:35.544464+0000","last_peered":"2024-04-25T15:51:35.544464+0000","last_clean":"2024-04-25T15:51:35.544464+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:51:35.544464+0000","last_undegraded":"2024-04-25T15:51:35.544464+0000","last_fullsized":"2024-04-25T15:51:35.544464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544582+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:51:35.544582+0000","last_peered":"2024-04-25T15:51:35.544582+0000","last_clean":"2024-04-25T15:51:35.544582+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:51:35.544582+0000","last_undegraded":"2024-04-25T15:51:35.544582+0000","last_fullsized":"2024-04-25T15:51:35.544582+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.657136+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:51:35.657136+0000","last_peered":"2024-04-25T15:51:35.657136+0000","last_clean":"2024-04-25T15:51:35.657136+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:51:35.657136+0000","last_undegraded":"2024-04-25T15:51:35.657136+0000","last_fullsized":"2024-04-25T15:51:35.657136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544714+0000","last_change":"2024-04-25T15:51:29.163703+0000","last_active":"2024-04-25T15:51:35.544714+0000","last_peered":"2024-04-25T15:51:35.544714+0000","last_clean":"2024-04-25T15:51:35.544714+0000","last_became_active":"2024-04-25T15:51:29.140145+0000","last_became_peered":"2024-04-25T15:51:29.140145+0000","last_unstale":"2024-04-25T15:51:35.544714+0000","last_undegraded":"2024-04-25T15:51:35.544714+0000","last_fullsized":"2024-04-25T15:51:35.544714+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:48.582128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":672,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152488+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:51:35.152488+0000","last_peered":"2024-04-25T15:51:35.152488+0000","last_clean":"2024-04-25T15:51:35.152488+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:51:35.152488+0000","last_undegraded":"2024-04-25T15:51:35.152488+0000","last_fullsized":"2024-04-25T15:51:35.152488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"24'14","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.562462+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:51:35.562462+0000","last_peered":"2024-04-25T15:51:35.562462+0000","last_clean":"2024-04-25T15:51:35.562462+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:51:35.562462+0000","last_undegraded":"2024-04-25T15:51:35.562462+0000","last_fullsized":"2024-04-25T15:51:35.562462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.394383+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:51:35.394383+0000","last_peered":"2024-04-25T15:51:35.394383+0000","last_clean":"2024-04-25T15:51:35.394383+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:51:35.394383+0000","last_undegraded":"2024-04-25T15:51:35.394383+0000","last_fullsized":"2024-04-25T15:51:35.394383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656620+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:51:35.656620+0000","last_peered":"2024-04-25T15:51:35.656620+0000","last_clean":"2024-04-25T15:51:35.656620+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:51:35.656620+0000","last_undegraded":"2024-04-25T15:51:35.656620+0000","last_fullsized":"2024-04-25T15:51:35.656620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656148+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:51:35.656148+0000","last_peered":"2024-04-25T15:51:35.656148+0000","last_clean":"2024-04-25T15:51:35.656148+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:51:35.656148+0000","last_undegraded":"2024-04-25T15:51:35.656148+0000","last_fullsized":"2024-04-25T15:51:35.656148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544710+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:51:35.544710+0000","last_peered":"2024-04-25T15:51:35.544710+0000","last_clean":"2024-04-25T15:51:35.544710+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:51:35.544710+0000","last_undegraded":"2024-04-25T15:51:35.544710+0000","last_fullsized":"2024-04-25T15:51:35.544710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.392989+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:51:35.392989+0000","last_peered":"2024-04-25T15:51:35.392989+0000","last_clean":"2024-04-25T15:51:35.392989+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:51:35.392989+0000","last_undegraded":"2024-04-25T15:51:35.392989+0000","last_fullsized":"2024-04-25T15:51:35.392989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.562056+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:51:35.562056+0000","last_peered":"2024-04-25T15:51:35.562056+0000","last_clean":"2024-04-25T15:51:35.562056+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:51:35.562056+0000","last_undegraded":"2024-04-25T15:51:35.562056+0000","last_fullsized":"2024-04-25T15:51:35.562056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.562150+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:51:35.562150+0000","last_peered":"2024-04-25T15:51:35.562150+0000","last_clean":"2024-04-25T15:51:35.562150+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:51:35.562150+0000","last_undegraded":"2024-04-25T15:51:35.562150+0000","last_fullsized":"2024-04-25T15:51:35.562150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":74,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152597+0000","last_change":"2024-04-25T15:51:29.183932+0000","last_active":"2024-04-25T15:51:35.152597+0000","last_peered":"2024-04-25T15:51:35.152597+0000","last_clean":"2024-04-25T15:51:35.152597+0000","last_became_active":"2024-04-25T15:51:29.151044+0000","last_became_peered":"2024-04-25T15:51:29.151044+0000","last_unstale":"2024-04-25T15:51:35.152597+0000","last_undegraded":"2024-04-25T15:51:35.152597+0000","last_fullsized":"2024-04-25T15:51:35.152597+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:15:25.588052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.392935+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:51:35.392935+0000","last_peered":"2024-04-25T15:51:35.392935+0000","last_clean":"2024-04-25T15:51:35.392935+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:51:35.392935+0000","last_undegraded":"2024-04-25T15:51:35.392935+0000","last_fullsized":"2024-04-25T15:51:35.392935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.153069+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:51:35.153069+0000","last_peered":"2024-04-25T15:51:35.153069+0000","last_clean":"2024-04-25T15:51:35.153069+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:51:35.153069+0000","last_undegraded":"2024-04-25T15:51:35.153069+0000","last_fullsized":"2024-04-25T15:51:35.153069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393039+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:51:35.393039+0000","last_peered":"2024-04-25T15:51:35.393039+0000","last_clean":"2024-04-25T15:51:35.393039+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:51:35.393039+0000","last_undegraded":"2024-04-25T15:51:35.393039+0000","last_fullsized":"2024-04-25T15:51:35.393039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.153227+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:51:35.153227+0000","last_peered":"2024-04-25T15:51:35.153227+0000","last_clean":"2024-04-25T15:51:35.153227+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:51:35.153227+0000","last_undegraded":"2024-04-25T15:51:35.153227+0000","last_fullsized":"2024-04-25T15:51:35.153227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393043+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:51:35.393043+0000","last_peered":"2024-04-25T15:51:35.393043+0000","last_clean":"2024-04-25T15:51:35.393043+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:51:35.393043+0000","last_undegraded":"2024-04-25T15:51:35.393043+0000","last_fullsized":"2024-04-25T15:51:35.393043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152625+0000","last_change":"2024-04-25T15:51:29.154533+0000","last_active":"2024-04-25T15:51:35.152625+0000","last_peered":"2024-04-25T15:51:35.152625+0000","last_clean":"2024-04-25T15:51:35.152625+0000","last_became_active":"2024-04-25T15:51:29.142484+0000","last_became_peered":"2024-04-25T15:51:29.142484+0000","last_unstale":"2024-04-25T15:51:35.152625+0000","last_undegraded":"2024-04-25T15:51:35.152625+0000","last_fullsized":"2024-04-25T15:51:35.152625+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:56.828196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544384+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:51:35.544384+0000","last_peered":"2024-04-25T15:51:35.544384+0000","last_clean":"2024-04-25T15:51:35.544384+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:51:35.544384+0000","last_undegraded":"2024-04-25T15:51:35.544384+0000","last_fullsized":"2024-04-25T15:51:35.544384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152997+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:51:35.152997+0000","last_peered":"2024-04-25T15:51:35.152997+0000","last_clean":"2024-04-25T15:51:35.152997+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:51:35.152997+0000","last_undegraded":"2024-04-25T15:51:35.152997+0000","last_fullsized":"2024-04-25T15:51:35.152997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.394555+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:51:35.394555+0000","last_peered":"2024-04-25T15:51:35.394555+0000","last_clean":"2024-04-25T15:51:35.394555+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:51:35.394555+0000","last_undegraded":"2024-04-25T15:51:35.394555+0000","last_fullsized":"2024-04-25T15:51:35.394555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":130,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393772+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:51:35.393772+0000","last_peered":"2024-04-25T15:51:35.393772+0000","last_clean":"2024-04-25T15:51:35.393772+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:51:35.393772+0000","last_undegraded":"2024-04-25T15:51:35.393772+0000","last_fullsized":"2024-04-25T15:51:35.393772+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544064+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:51:35.544064+0000","last_peered":"2024-04-25T15:51:35.544064+0000","last_clean":"2024-04-25T15:51:35.544064+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:51:35.544064+0000","last_undegraded":"2024-04-25T15:51:35.544064+0000","last_fullsized":"2024-04-25T15:51:35.544064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440806+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:51:35.440806+0000","last_peered":"2024-04-25T15:51:35.440806+0000","last_clean":"2024-04-25T15:51:35.440806+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:51:35.440806+0000","last_undegraded":"2024-04-25T15:51:35.440806+0000","last_fullsized":"2024-04-25T15:51:35.440806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393570+0000","last_change":"2024-04-25T15:51:29.149866+0000","last_active":"2024-04-25T15:51:35.393570+0000","last_peered":"2024-04-25T15:51:35.393570+0000","last_clean":"2024-04-25T15:51:35.393570+0000","last_became_active":"2024-04-25T15:51:29.138800+0000","last_became_peered":"2024-04-25T15:51:29.138800+0000","last_unstale":"2024-04-25T15:51:35.393570+0000","last_undegraded":"2024-04-25T15:51:35.393570+0000","last_fullsized":"2024-04-25T15:51:35.393570+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:40.945354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.655205+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:51:35.655205+0000","last_peered":"2024-04-25T15:51:35.655205+0000","last_clean":"2024-04-25T15:51:35.655205+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:51:35.655205+0000","last_undegraded":"2024-04-25T15:51:35.655205+0000","last_fullsized":"2024-04-25T15:51:35.655205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.153009+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:51:35.153009+0000","last_peered":"2024-04-25T15:51:35.153009+0000","last_clean":"2024-04-25T15:51:35.153009+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:51:35.153009+0000","last_undegraded":"2024-04-25T15:51:35.153009+0000","last_fullsized":"2024-04-25T15:51:35.153009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152571+0000","last_change":"2024-04-25T15:51:29.138989+0000","last_active":"2024-04-25T15:51:35.152571+0000","last_peered":"2024-04-25T15:51:35.152571+0000","last_clean":"2024-04-25T15:51:35.152571+0000","last_became_active":"2024-04-25T15:51:29.138352+0000","last_became_peered":"2024-04-25T15:51:29.138352+0000","last_unstale":"2024-04-25T15:51:35.152571+0000","last_undegraded":"2024-04-25T15:51:35.152571+0000","last_fullsized":"2024-04-25T15:51:35.152571+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:13:22.109973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544325+0000","last_change":"2024-04-25T15:51:29.141352+0000","last_active":"2024-04-25T15:51:35.544325+0000","last_peered":"2024-04-25T15:51:35.544325+0000","last_clean":"2024-04-25T15:51:35.544325+0000","last_became_active":"2024-04-25T15:51:29.134551+0000","last_became_peered":"2024-04-25T15:51:29.134551+0000","last_unstale":"2024-04-25T15:51:35.544325+0000","last_undegraded":"2024-04-25T15:51:35.544325+0000","last_fullsized":"2024-04-25T15:51:35.544325+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:06.995858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152391+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:51:35.152391+0000","last_peered":"2024-04-25T15:51:35.152391+0000","last_clean":"2024-04-25T15:51:35.152391+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:51:35.152391+0000","last_undegraded":"2024-04-25T15:51:35.152391+0000","last_fullsized":"2024-04-25T15:51:35.152391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656651+0000","last_change":"2024-04-25T15:51:29.135172+0000","last_active":"2024-04-25T15:51:35.656651+0000","last_peered":"2024-04-25T15:51:35.656651+0000","last_clean":"2024-04-25T15:51:35.656651+0000","last_became_active":"2024-04-25T15:51:29.134725+0000","last_became_peered":"2024-04-25T15:51:29.134725+0000","last_unstale":"2024-04-25T15:51:35.656651+0000","last_undegraded":"2024-04-25T15:51:35.656651+0000","last_fullsized":"2024-04-25T15:51:35.656651+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:00.115131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.544721+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:51:35.544721+0000","last_peered":"2024-04-25T15:51:35.544721+0000","last_clean":"2024-04-25T15:51:35.544721+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:51:35.544721+0000","last_undegraded":"2024-04-25T15:51:35.544721+0000","last_fullsized":"2024-04-25T15:51:35.544721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.394883+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:51:35.394883+0000","last_peered":"2024-04-25T15:51:35.394883+0000","last_clean":"2024-04-25T15:51:35.394883+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:51:35.394883+0000","last_undegraded":"2024-04-25T15:51:35.394883+0000","last_fullsized":"2024-04-25T15:51:35.394883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152954+0000","last_change":"2024-04-25T15:51:29.145851+0000","last_active":"2024-04-25T15:51:35.152954+0000","last_peered":"2024-04-25T15:51:35.152954+0000","last_clean":"2024-04-25T15:51:35.152954+0000","last_became_active":"2024-04-25T15:51:29.144869+0000","last_became_peered":"2024-04-25T15:51:29.144869+0000","last_unstale":"2024-04-25T15:51:35.152954+0000","last_undegraded":"2024-04-25T15:51:35.152954+0000","last_fullsized":"2024-04-25T15:51:35.152954+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:14.083348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"24'9","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.563221+0000","last_change":"2024-04-25T15:51:29.147114+0000","last_active":"2024-04-25T15:51:35.563221+0000","last_peered":"2024-04-25T15:51:35.563221+0000","last_clean":"2024-04-25T15:51:35.563221+0000","last_became_active":"2024-04-25T15:51:29.126799+0000","last_became_peered":"2024-04-25T15:51:29.126799+0000","last_unstale":"2024-04-25T15:51:35.563221+0000","last_undegraded":"2024-04-25T15:51:35.563221+0000","last_fullsized":"2024-04-25T15:51:35.563221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:18.716996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.441544+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:51:35.441544+0000","last_peered":"2024-04-25T15:51:35.441544+0000","last_clean":"2024-04-25T15:51:35.441544+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:51:35.441544+0000","last_undegraded":"2024-04-25T15:51:35.441544+0000","last_fullsized":"2024-04-25T15:51:35.441544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.563038+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:51:35.563038+0000","last_peered":"2024-04-25T15:51:35.563038+0000","last_clean":"2024-04-25T15:51:35.563038+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:51:35.563038+0000","last_undegraded":"2024-04-25T15:51:35.563038+0000","last_fullsized":"2024-04-25T15:51:35.563038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.656761+0000","last_change":"2024-04-25T15:51:29.136691+0000","last_active":"2024-04-25T15:51:35.656761+0000","last_peered":"2024-04-25T15:51:35.656761+0000","last_clean":"2024-04-25T15:51:35.656761+0000","last_became_active":"2024-04-25T15:51:29.136342+0000","last_became_peered":"2024-04-25T15:51:29.136342+0000","last_unstale":"2024-04-25T15:51:35.656761+0000","last_undegraded":"2024-04-25T15:51:35.656761+0000","last_fullsized":"2024-04-25T15:51:35.656761+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.298937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393439+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:51:35.393439+0000","last_peered":"2024-04-25T15:51:35.393439+0000","last_clean":"2024-04-25T15:51:35.393439+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:51:35.393439+0000","last_undegraded":"2024-04-25T15:51:35.393439+0000","last_fullsized":"2024-04-25T15:51:35.393439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152522+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:51:35.152522+0000","last_peered":"2024-04-25T15:51:35.152522+0000","last_clean":"2024-04-25T15:51:35.152522+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:51:35.152522+0000","last_undegraded":"2024-04-25T15:51:35.152522+0000","last_fullsized":"2024-04-25T15:51:35.152522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.393523+0000","last_change":"2024-04-25T15:51:29.164831+0000","last_active":"2024-04-25T15:51:35.393523+0000","last_peered":"2024-04-25T15:51:35.393523+0000","last_clean":"2024-04-25T15:51:35.393523+0000","last_became_active":"2024-04-25T15:51:29.147881+0000","last_became_peered":"2024-04-25T15:51:29.147881+0000","last_unstale":"2024-04-25T15:51:35.393523+0000","last_undegraded":"2024-04-25T15:51:35.393523+0000","last_fullsized":"2024-04-25T15:51:35.393523+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:23.262523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.440699+0000","last_change":"2024-04-25T15:51:29.132787+0000","last_active":"2024-04-25T15:51:35.440699+0000","last_peered":"2024-04-25T15:51:35.440699+0000","last_clean":"2024-04-25T15:51:35.440699+0000","last_became_active":"2024-04-25T15:51:29.132390+0000","last_became_peered":"2024-04-25T15:51:29.132390+0000","last_unstale":"2024-04-25T15:51:35.440699+0000","last_undegraded":"2024-04-25T15:51:35.440699+0000","last_fullsized":"2024-04-25T15:51:35.440699+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:52:43.265089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.441260+0000","last_change":"2024-04-25T15:51:29.149131+0000","last_active":"2024-04-25T15:51:35.441260+0000","last_peered":"2024-04-25T15:51:35.441260+0000","last_clean":"2024-04-25T15:51:35.441260+0000","last_became_active":"2024-04-25T15:51:29.141582+0000","last_became_peered":"2024-04-25T15:51:29.141582+0000","last_unstale":"2024-04-25T15:51:35.441260+0000","last_undegraded":"2024-04-25T15:51:35.441260+0000","last_fullsized":"2024-04-25T15:51:35.441260+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:14.665877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.152805+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:51:35.152805+0000","last_peered":"2024-04-25T15:51:35.152805+0000","last_clean":"2024-04-25T15:51:35.152805+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:51:35.152805+0000","last_undegraded":"2024-04-25T15:51:35.152805+0000","last_fullsized":"2024-04-25T15:51:35.152805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.394310+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:51:35.394310+0000","last_peered":"2024-04-25T15:51:35.394310+0000","last_clean":"2024-04-25T15:51:35.394310+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:51:35.394310+0000","last_undegraded":"2024-04-25T15:51:35.394310+0000","last_fullsized":"2024-04-25T15:51:35.394310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.561748+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:51:35.561748+0000","last_peered":"2024-04-25T15:51:35.561748+0000","last_clean":"2024-04-25T15:51:35.561748+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:51:35.561748+0000","last_undegraded":"2024-04-25T15:51:35.561748+0000","last_fullsized":"2024-04-25T15:51:35.561748+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.543370+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:51:35.543370+0000","last_peered":"2024-04-25T15:51:35.543370+0000","last_clean":"2024-04-25T15:51:35.543370+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:51:35.543370+0000","last_undegraded":"2024-04-25T15:51:35.543370+0000","last_fullsized":"2024-04-25T15:51:35.543370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.394353+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:51:35.394353+0000","last_peered":"2024-04-25T15:51:35.394353+0000","last_clean":"2024-04-25T15:51:35.394353+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:51:35.394353+0000","last_undegraded":"2024-04-25T15:51:35.394353+0000","last_fullsized":"2024-04-25T15:51:35.394353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.543313+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:51:35.543313+0000","last_peered":"2024-04-25T15:51:35.543313+0000","last_clean":"2024-04-25T15:51:35.543313+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:51:35.543313+0000","last_undegraded":"2024-04-25T15:51:35.543313+0000","last_fullsized":"2024-04-25T15:51:35.543313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.394700+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:51:35.394700+0000","last_peered":"2024-04-25T15:51:35.394700+0000","last_clean":"2024-04-25T15:51:35.394700+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:51:35.394700+0000","last_undegraded":"2024-04-25T15:51:35.394700+0000","last_fullsized":"2024-04-25T15:51:35.394700+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.657016+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:51:35.657016+0000","last_peered":"2024-04-25T15:51:35.657016+0000","last_clean":"2024-04-25T15:51:35.657016+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:51:35.657016+0000","last_undegraded":"2024-04-25T15:51:35.657016+0000","last_fullsized":"2024-04-25T15:51:35.657016+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.441648+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:51:35.441648+0000","last_peered":"2024-04-25T15:51:35.441648+0000","last_clean":"2024-04-25T15:51:35.441648+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:51:35.441648+0000","last_undegraded":"2024-04-25T15:51:35.441648+0000","last_fullsized":"2024-04-25T15:51:35.441648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":32,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.441531+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:51:35.441531+0000","last_peered":"2024-04-25T15:51:35.441531+0000","last_clean":"2024-04-25T15:51:35.441531+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:51:35.441531+0000","last_undegraded":"2024-04-25T15:51:35.441531+0000","last_fullsized":"2024-04-25T15:51:35.441531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-25T15:51:35.657254+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:51:35.657254+0000","last_peered":"2024-04-25T15:51:35.657254+0000","last_clean":"2024-04-25T15:51:35.657254+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:51:35.657254+0000","last_undegraded":"2024-04-25T15:51:35.657254+0000","last_fullsized":"2024-04-25T15:51:35.657254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14298,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":218,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1532,"num_keys_recovered":12,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":30128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":724652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":364544,"data_stored":140115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":136116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":133514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":139721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":133646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":723769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":7978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":8372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:51:39.250 INFO:tasks.thrashosds.thrasher:Setting 3.2f to [6, 2] 2024-04-25T15:51:39.250 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.2f', '6', '2'] 2024-04-25T15:51:39.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.2f 6 2 2024-04-25T15:51:39.322 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:39.323 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:39.334 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:39.331+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:39.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:39.402 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:39.399+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:39.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1ac020c640 1 Processor -- start 2024-04-25T15:51:39.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1ac020c640 1 -- start start 2024-04-25T15:51:39.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1ac020c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1ab80ce260 0x7f1ab80b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1ac020c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ab8156c70 0x7f1ab80cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1ac020c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ab80b79b0 con 0x7f1ab80cd9b0 2024-04-25T15:51:39.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1ac020c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab80b7c20 con 0x7f1ab8156c70 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1ac020c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab80ce840 con 0x7f1ab80ce260 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abd7a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ab8156c70 0x7f1ab80cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abd7a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ab8156c70 0x7f1ab80cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47446/0 (socket says 172.21.15.67:47446) 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3372049428 learned_addr learned my addr 172.21.15.67:0/3372049428 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abdfa8640 1 --2- 172.21.15.67:0/3372049428 >> v2:172.21.15.176:3300/0 conn(0x7f1ab80ce260 0x7f1ab80b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3372049428 >> v2:172.21.15.176:3300/0 conn(0x7f1ab80ce260 msgr2=0x7f1ab80b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abd7a7640 1 --2- 172.21.15.67:0/3372049428 >> v2:172.21.15.176:3300/0 conn(0x7f1ab80ce260 0x7f1ab80b7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3372049428 >> v1:172.21.15.67:6789/0 conn(0x7f1ab80cd9b0 legacy=0x7f1ab815cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abcfa6640 1 -- 172.21.15.67:0/3372049428 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744732377 0 0) 0x7f1ab80b79b0 con 0x7f1ab80cd9b0 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3372049428 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab80ceb20 con 0x7f1ab8156c70 2024-04-25T15:51:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.447+0000 7f1abdfa8640 1 --2- 172.21.15.67:0/3372049428 >> v2:172.21.15.176:3300/0 conn(0x7f1ab80ce260 0x7f1ab80b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:39.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1abd7a7640 1 --2- 172.21.15.67:0/3372049428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab8156c70 0x7f1ab80cd470 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80ccf20 tx=0x7f1aa801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e7c996f329943a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:39.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1abcfa6640 1 -- 172.21.15.67:0/3372049428 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80b5070 con 0x7f1ab8156c70 2024-04-25T15:51:39.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1abcfa6640 1 -- 172.21.15.67:0/3372049428 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa80a8a10 con 0x7f1ab8156c70 2024-04-25T15:51:39.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1abcfa6640 1 -- 172.21.15.67:0/3372049428 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80a8de0 con 0x7f1ab8156c70 2024-04-25T15:51:39.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 -- 172.21.15.67:0/3372049428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab8156c70 msgr2=0x7f1ab80cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3372049428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab8156c70 0x7f1ab80cd470 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80ccf20 tx=0x7f1aa801ed10 comp rx=0 tx=0).stop 2024-04-25T15:51:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 -- 172.21.15.67:0/3372049428 shutdown_connections 2024-04-25T15:51:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3372049428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab8156c70 0x7f1ab80cd470 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3372049428 >> v2:172.21.15.176:3300/0 conn(0x7f1ab80ce260 0x7f1ab80b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 -- 172.21.15.67:0/3372049428 >> 172.21.15.67:0/3372049428 conn(0x7f1ab806e700 msgr2=0x7f1ab806eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 -- 172.21.15.67:0/3372049428 shutdown_connections 2024-04-25T15:51:39.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 -- 172.21.15.67:0/3372049428 wait complete. 2024-04-25T15:51:39.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 Processor -- start 2024-04-25T15:51:39.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.451+0000 7f1ac020c640 1 -- start start 2024-04-25T15:51:39.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ab80cd9b0 0x7f1ab815e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab80ce260 0x7f1ab81539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ab8156c70 0x7f1ab8158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80cf3c0 con 0x7f1ab80ce260 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80cf0e0 con 0x7f1ab8156c70 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab8157080 con 0x7f1ab80cd9b0 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab80ce260 0x7f1ab81539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab80ce260 0x7f1ab81539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47452/0 (socket says 172.21.15.67:47452) 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3128185579 learned_addr learned my addr 172.21.15.67:0/3128185579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abdfa8640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ab80cd9b0 0x7f1ab815e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abe7a9640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ab8156c70 0x7f1ab8158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ab80cd9b0 msgr2=0x7f1ab815e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ab80cd9b0 0x7f1ab815e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ab8156c70 msgr2=0x7f1ab8158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ab8156c70 0x7f1ab8158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:39.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 -- 172.21.15.67:0/3128185579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab815e610 con 0x7f1ab80ce260 2024-04-25T15:51:39.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1abd7a7640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab80ce260 0x7f1ab81539d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b3040 tx=0x7f1aa801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:39.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1aae7fc640 1 -- 172.21.15.67:0/3128185579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80c0070 con 0x7f1ab80ce260 2024-04-25T15:51:39.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1aae7fc640 1 -- 172.21.15.67:0/3128185579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa8004230 con 0x7f1ab80ce260 2024-04-25T15:51:39.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1aae7fc640 1 -- 172.21.15.67:0/3128185579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80b5040 con 0x7f1ab80ce260 2024-04-25T15:51:39.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab815a040 con 0x7f1ab80ce260 2024-04-25T15:51:39.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab81584f0 con 0x7f1ab80ce260 2024-04-25T15:51:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab806a480 con 0x7f1ab80ce260 2024-04-25T15:51:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1aae7fc640 1 -- 172.21.15.67:0/3128185579 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1aa80c3070 con 0x7f1ab80ce260 2024-04-25T15:51:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.455+0000 7f1aae7fc640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1a9c06f400 0x7f1a9c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.459+0000 7f1aae7fc640 1 -- 172.21.15.67:0/3128185579 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5796+0+0 (secure 0 0 0) 0x7f1aa812bd30 con 0x7f1ab80ce260 2024-04-25T15:51:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.459+0000 7f1abdfa8640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1a9c06f400 0x7f1a9c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.459+0000 7f1abdfa8640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1a9c06f400 0x7f1a9c071910 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400ba10 tx=0x7f1ab4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:39.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.463+0000 7f1aae7fc640 1 -- 172.21.15.67:0/3128185579 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1aa8018870 con 0x7f1ab80ce260 2024-04-25T15:51:39.503 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:39.499+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.328821337999997, 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.541 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:39.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:39.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:39.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.326206540999998, 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:39.546 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.550 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.325707643999998, 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:39.554 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.324618397999998, 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:39.558 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.323997902999999, 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.321603676999999, 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:39.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:39.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.292483273999999, 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.570 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.287738539999999, 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.284211038999999, 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.284078557999997, 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.578 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.663139100000002, 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.582 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:39.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:39.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:39.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:39.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.651703968, 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:39.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:39.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.651306532, 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:39.591 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:39.595 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.651196790999997, 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.650696046, 2024-04-25T15:51:39.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:39.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:39.603 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.650320059999999, 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.611721877000001, 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.607 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:39.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.611621618000001, 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:39.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.609516702999997, 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.609141127000001, 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:39.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:39.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:39.621 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:39.622 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:39.623 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:39.619+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:39.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:39.639+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "3.2f", "id": [6, 2]} v 0) -- 0x7f1ab815d730 con 0x7f1ab80ce260 2024-04-25T15:51:39.724 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:39.719+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:39.775 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:39.775 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:39.775 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:39.775 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:39.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:39.843 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:39.840+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:39.926 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:39.923+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:39.941 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:39.942 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:39.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:40.027 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:40.023+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:40.128 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:40.124+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900760757, 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:40.151 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:40.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.156 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900730195999998, 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:40.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.161 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900722365, 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:40.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:40.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:40.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900713435999997, 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:40.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:40.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900707204, 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:40.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.176 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900691793999997, 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:40.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:40.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:40.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900678962000001, 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.185 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.900673748000003, 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.815063631000001, 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:40.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.262533638000001, 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:40.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:40.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:40.199 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.2611408, 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:40.203 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.223120674, 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:40.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.223097348000003, 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:40.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:40.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:40.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.223056808000003, 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.213 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:40.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:40.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:40.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.223048147999997, 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.218 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:40.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.223006474999998, 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:40.223 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.226 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.222997218000003, 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.218859792000003, 2024-04-25T15:51:40.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:40.234 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.218805215000003, 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:40.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.218772006999998, 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:40.238 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:40.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:40.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:40.242 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:40.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:51:40.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:40.244 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:40.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.183+0000 7f1aae7fc640 1 -- 172.21.15.67:0/3128185579 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "3.2f", "id": [6, 2]}]=0 set 3.2f pg_upmap mapping to [6,2] v32) ==== 124+0+0 (secure 0 0 0) 0x7f1aa80f8c50 con 0x7f1ab80ce260 2024-04-25T15:51:40.245 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.2f pg_upmap mapping to [6,2] 2024-04-25T15:51:40.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1a9c06f400 msgr2=0x7f1a9c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1a9c06f400 0x7f1a9c071910 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400ba10 tx=0x7f1ab4008040 comp rx=0 tx=0).stop 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab80ce260 msgr2=0x7f1ab81539d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab80ce260 0x7f1ab81539d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b3040 tx=0x7f1aa801bab0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 shutdown_connections 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1a9c06f400 0x7f1a9c071910 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ab8156c70 0x7f1ab8158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ab80ce260 0x7f1ab81539d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 --2- 172.21.15.67:0/3128185579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ab80cd9b0 0x7f1ab815e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 >> 172.21.15.67:0/3128185579 conn(0x7f1ab806e700 msgr2=0x7f1ab806eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 shutdown_connections 2024-04-25T15:51:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.187+0000 7f1ac020c640 1 -- 172.21.15.67:0/3128185579 wait complete. 2024-04-25T15:51:40.247 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:40.239+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:40.344 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:40.339+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:40.441 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:40.442 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:40.442 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:40.442 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:40.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 Processor -- start 2024-04-25T15:51:40.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 -- start start 2024-04-25T15:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c0caf60 0x7f990c15d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c15d790 0x7f990c06d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f990c073bf0 0x7f990c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f990c0cab60 con 0x7f990c0caf60 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f990c159620 con 0x7f990c073bf0 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9912e64640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f990c0c77a0 con 0x7f990c15d790 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c0caf60 0x7f990c15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c0caf60 0x7f990c15d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47456/0 (socket says 172.21.15.67:47456) 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 -- 172.21.15.67:0/1552733411 learned_addr learned my addr 172.21.15.67:0/1552733411 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f990bfff640 1 --2- 172.21.15.67:0/1552733411 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c15d790 0x7f990c06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9911401640 1 --2- 172.21.15.67:0/1552733411 >> v2:172.21.15.176:3300/0 conn(0x7f990c073bf0 0x7f990c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.446 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:40.439+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 -- 172.21.15.67:0/1552733411 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c15d790 msgr2=0x7f990c06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 --2- 172.21.15.67:0/1552733411 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c15d790 0x7f990c06d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 -- 172.21.15.67:0/1552733411 >> v2:172.21.15.176:3300/0 conn(0x7f990c073bf0 msgr2=0x7f990c06dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 --2- 172.21.15.67:0/1552733411 >> v2:172.21.15.176:3300/0 conn(0x7f990c073bf0 0x7f990c06dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.439+0000 7f9910c00640 1 -- 172.21.15.67:0/1552733411 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c0c7a60 con 0x7f990c0caf60 2024-04-25T15:51:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9910c00640 1 --2- 172.21.15.67:0/1552733411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15d250 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f990c06fd00 tx=0x7f98fc01ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beff096a378d0964 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f990b7fe640 1 -- 172.21.15.67:0/1552733411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00b2d0 con 0x7f990c0caf60 2024-04-25T15:51:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f990b7fe640 1 -- 172.21.15.67:0/1552733411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98fc00b470 con 0x7f990c0caf60 2024-04-25T15:51:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f990b7fe640 1 -- 172.21.15.67:0/1552733411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0a7aa0 con 0x7f990c0caf60 2024-04-25T15:51:40.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 -- 172.21.15.67:0/1552733411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 msgr2=0x7f990c15d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 --2- 172.21.15.67:0/1552733411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15d250 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f990c06fd00 tx=0x7f98fc01ebf0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 -- 172.21.15.67:0/1552733411 shutdown_connections 2024-04-25T15:51:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 --2- 172.21.15.67:0/1552733411 >> v2:172.21.15.176:3300/0 conn(0x7f990c073bf0 0x7f990c06dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f40783b0640 1 Processor -- start 2024-04-25T15:51:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 --2- 172.21.15.67:0/1552733411 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c15d790 0x7f990c06d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 --2- 172.21.15.67:0/1552733411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15d250 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 -- 172.21.15.67:0/1552733411 >> 172.21.15.67:0/1552733411 conn(0x7f990c070db0 msgr2=0x7f990c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:40.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f40783b0640 1 -- start start 2024-04-25T15:51:40.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 -- 172.21.15.67:0/1552733411 shutdown_connections 2024-04-25T15:51:40.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.443+0000 7f9912e64640 1 -- 172.21.15.67:0/1552733411 wait complete. 2024-04-25T15:51:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f40783b0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f407006c640 0x7f407015b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f40783b0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074780 0x7f4070164f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f40783b0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40700c92d0 con 0x7f4070074370 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f40783b0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4070074b90 con 0x7f407006c640 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f40783b0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f407015b560 con 0x7f4070074780 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407614c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f407006c640 0x7f407015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407594b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4070074370 0x7f40700c7fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47078/0 (socket says 172.21.15.67:47078) 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407594b640 1 -- 172.21.15.67:0/1722764716 learned_addr learned my addr 172.21.15.67:0/1722764716 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407614c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f407006c640 0x7f407015b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47466/0 (socket says 172.21.15.67:47466) 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407694d640 1 --2- 172.21.15.67:0/1722764716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074780 0x7f4070164f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407614c640 1 -- 172.21.15.67:0/1722764716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074780 msgr2=0x7f4070164f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407614c640 1 --2- 172.21.15.67:0/1722764716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074780 0x7f4070164f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407614c640 1 -- 172.21.15.67:0/1722764716 >> v1:172.21.15.67:6789/0 conn(0x7f4070074370 legacy=0x7f40700c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407614c640 1 -- 172.21.15.67:0/1722764716 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407006e210 con 0x7f407006c640 2024-04-25T15:51:40.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407694d640 1 --2- 172.21.15.67:0/1722764716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074780 0x7f4070164f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 Processor -- start 2024-04-25T15:51:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 -- start start 2024-04-25T15:51:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c073bf0 0x7f990c1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c15d790 0x7f990c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f990c0733d0 con 0x7f990c0caf60 2024-04-25T15:51:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f990c15a600 con 0x7f990c15d790 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f990c073550 con 0x7f990c073bf0 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47470/0 (socket says 172.21.15.67:47470) 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 -- 172.21.15.67:0/1697451198 learned_addr learned my addr 172.21.15.67:0/1697451198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9910c00640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c073bf0 0x7f990c1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9911401640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c15d790 0x7f990c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 -- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c073bf0 msgr2=0x7f990c1553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c073bf0 0x7f990c1553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 -- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c15d790 msgr2=0x7f990c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c15d790 0x7f990c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 -- 172.21.15.67:0/1697451198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c156910 con 0x7f990c0caf60 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9911401640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c15d790 0x7f990c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:40.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f990bfff640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15ddb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f98f400e870 tx=0x7f98f400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407614c640 1 --2- 172.21.15.67:0/1722764716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407006c640 0x7f407015b020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f406c0097c0 tx=0x7f406c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f9f28e15608a5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f99097fa640 1 -- 172.21.15.67:0/1697451198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4017070 con 0x7f990c0caf60 2024-04-25T15:51:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f99097fa640 1 -- 172.21.15.67:0/1697451198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f400c940 con 0x7f990c0caf60 2024-04-25T15:51:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990c155900 con 0x7f990c0caf60 2024-04-25T15:51:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407514a640 1 -- 172.21.15.67:0/1722764716 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c0b5070 con 0x7f407006c640 2024-04-25T15:51:40.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f99097fa640 1 -- 172.21.15.67:0/1697451198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f400cd30 con 0x7f990c0caf60 2024-04-25T15:51:40.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990c06cf10 con 0x7f990c0caf60 2024-04-25T15:51:40.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407514a640 1 -- 172.21.15.67:0/1722764716 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c0a8990 con 0x7f407006c640 2024-04-25T15:51:40.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.447+0000 7f407514a640 1 -- 172.21.15.67:0/1722764716 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c0a8d60 con 0x7f407006c640 2024-04-25T15:51:40.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- 172.21.15.67:0/1722764716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407006c640 msgr2=0x7f407015b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 --2- 172.21.15.67:0/1722764716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407006c640 0x7f407015b020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f406c0097c0 tx=0x7f406c01ec90 comp rx=0 tx=0).stop 2024-04-25T15:51:40.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990c06a480 con 0x7f990c0caf60 2024-04-25T15:51:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- 172.21.15.67:0/1722764716 shutdown_connections 2024-04-25T15:51:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 --2- 172.21.15.67:0/1722764716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074780 0x7f4070164f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 --2- 172.21.15.67:0/1722764716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f407006c640 0x7f407015b020 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- 172.21.15.67:0/1722764716 >> 172.21.15.67:0/1722764716 conn(0x7f4070070db0 msgr2=0x7f40700ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- 172.21.15.67:0/1722764716 shutdown_connections 2024-04-25T15:51:40.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- 172.21.15.67:0/1722764716 wait complete. 2024-04-25T15:51:40.460 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:40.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f99097fa640 1 -- 172.21.15.67:0/1697451198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f98f401a070 con 0x7f990c0caf60 2024-04-25T15:51:40.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f99097fa640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98ec06f400 0x7f98ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f99097fa640 1 -- 172.21.15.67:0/1697451198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5825+0+0 (secure 0 0 0) 0x7f98f408f520 con 0x7f990c0caf60 2024-04-25T15:51:40.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f9910c00640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98ec06f400 0x7f98ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 Processor -- start 2024-04-25T15:51:40.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- start start 2024-04-25T15:51:40.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f9910c00640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98ec06f400 0x7f98ec071910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0b0000 tx=0x7f98fc0a9070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f407006c640 0x7f4070154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074370 0x7f4070156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4070074780 0x7f407014f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4070073890 con 0x7f4070074780 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f407006e7b0 con 0x7f407006c640 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f40783b0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4070073b70 con 0x7f4070074370 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f407694d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4070074780 0x7f407014f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f407694d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4070074780 0x7f407014f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47484/0 (socket says 172.21.15.67:47484) 2024-04-25T15:51:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.451+0000 7f407694d640 1 -- 172.21.15.67:0/2121190658 learned_addr learned my addr 172.21.15.67:0/2121190658 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f407006c640 0x7f4070154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074370 0x7f4070156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407694d640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074370 msgr2=0x7f4070156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407694d640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074370 0x7f4070156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407694d640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f407006c640 msgr2=0x7f4070154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407694d640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f407006c640 0x7f4070154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407694d640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40701571b0 con 0x7f4070074780 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4070074370 0x7f4070156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f407694d640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4070074780 0x7f407014f050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f406800cd20 tx=0x7f406800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406800f680 con 0x7f4070074780 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406800f840 con 0x7f4070074780 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40701f44a0 con 0x7f4070074780 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406800fc30 con 0x7f4070074780 2024-04-25T15:51:40.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40701f4a60 con 0x7f4070074780 2024-04-25T15:51:40.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f99097fa640 1 -- 172.21.15.67:0/1697451198 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f98f405c420 con 0x7f990c0caf60 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.455+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f407006a480 con 0x7f4070074780 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f40680213f0 con 0x7f4070074780 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f4066ffd640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5825+0+0 (secure 0 0 0) 0x7f4068090000 con 0x7f4070074780 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v32) ==== 54+0+16 (secure 0 0 0) 0x7f4068014030 con 0x7f4070074780 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40701573c0 0x7f4070145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4070073e40 con 0x7f40701573c0 2024-04-25T15:51:40.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40701573c0 0x7f4070145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f406c0b2000 tx=0x7f406c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40701573c0 0x7f4070145310 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.459+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4070073e40 con 0x7f40701573c0 2024-04-25T15:51:40.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.475+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f40700cb070 con 0x7f40701573c0 2024-04-25T15:51:40.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.475+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f40700cb070 con 0x7f40701573c0 2024-04-25T15:51:40.481 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.475+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4070157aa0 0x7f4070063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.475+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4070145d60 con 0x7f4070157aa0 2024-04-25T15:51:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.479+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4070157aa0 0x7f4070063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.479+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4070157aa0 0x7f4070063090 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.479+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4070145d60 con 0x7f4070157aa0 2024-04-25T15:51:40.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.491+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f40700ca860 con 0x7f4070157aa0 2024-04-25T15:51:40.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.495+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f40700ca860 con 0x7f4070157aa0 2024-04-25T15:51:40.499 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:40.499 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:40.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.495+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4070063ae0 0x7f407002c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.495+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f40701f41c0 con 0x7f4070063ae0 2024-04-25T15:51:40.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.495+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4070063ae0 0x7f407002c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.495+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4070063ae0 0x7f407002c9e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.495+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40701f41c0 con 0x7f4070063ae0 2024-04-25T15:51:40.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.511+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f407002cf20 con 0x7f4070063ae0 2024-04-25T15:51:40.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.511+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f407002cf20 con 0x7f4070063ae0 2024-04-25T15:51:40.516 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:40.516 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:40.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.511+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f407002da00 0x7f40702013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.511+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f407002de10 con 0x7f407002da00 2024-04-25T15:51:40.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.511+0000 7f407694d640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f407002da00 0x7f40702013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.511+0000 7f407694d640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f407002da00 0x7f40702013f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.511+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f407002de10 con 0x7f407002da00 2024-04-25T15:51:40.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.527+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4070201b80 con 0x7f407002da00 2024-04-25T15:51:40.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.527+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4070201b80 con 0x7f407002da00 2024-04-25T15:51:40.533 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:40.533 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:40.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.527+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4070202090 0x7f40702044b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.527+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f407002de10 con 0x7f4070202090 2024-04-25T15:51:40.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.527+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4070202090 0x7f40702044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.531+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4070202090 0x7f40702044b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.531+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f407002de10 con 0x7f4070202090 2024-04-25T15:51:40.546 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:40.543+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:40.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.543+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f407002de10 con 0x7f4070202090 2024-04-25T15:51:40.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.551+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f407002de10 con 0x7f4070202090 2024-04-25T15:51:40.554 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:40.555 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:40.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.551+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4070204f00 0x7f4070207320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.551+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f407020a0f0 con 0x7f4070204f00 2024-04-25T15:51:40.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.551+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4070204f00 0x7f4070207320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.551+0000 7f407594b640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4070204f00 0x7f4070207320 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.551+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f407020a0f0 con 0x7f4070204f00 2024-04-25T15:51:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.567+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4070207860 con 0x7f4070204f00 2024-04-25T15:51:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.571+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4070207860 con 0x7f4070204f00 2024-04-25T15:51:40.575 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.571+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4070207ee0 0x7f407020a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.571+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f407020d380 con 0x7f4070207ee0 2024-04-25T15:51:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.571+0000 7f407694d640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4070207ee0 0x7f407020a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.571+0000 7f407694d640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4070207ee0 0x7f407020a300 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.575+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f407020d380 con 0x7f4070207ee0 2024-04-25T15:51:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.587+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4070000960 con 0x7f4070207ee0 2024-04-25T15:51:40.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.591+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4070000960 con 0x7f4070207ee0 2024-04-25T15:51:40.595 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:40.595 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:40.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.591+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f407020c5b0 0x7f407020c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.591+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4070210750 con 0x7f407020c5b0 2024-04-25T15:51:40.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.591+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f407020c5b0 0x7f407020c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.591+0000 7f407614c640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f407020c5b0 0x7f407020c9e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.595+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4070210750 con 0x7f407020c5b0 2024-04-25T15:51:40.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.607+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f407020a960 con 0x7f407020c5b0 2024-04-25T15:51:40.615 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:40.616 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:40.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f4066ffd640 1 -- 172.21.15.67:0/2121190658 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f407020a960 con 0x7f407020c5b0 2024-04-25T15:51:40.616 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40701573c0 msgr2=0x7f4070145310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40701573c0 0x7f4070145310 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4070157aa0 msgr2=0x7f4070063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4070157aa0 0x7f4070063090 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4070063ae0 msgr2=0x7f407002c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4070063ae0 0x7f407002c9e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f407694d640 1 -- 172.21.15.67:0/2121190658 reap_dead start 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f407002da00 msgr2=0x7f40702013f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f407002da00 0x7f40702013f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4070202090 msgr2=0x7f40702044b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4070202090 0x7f40702044b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4070204f00 msgr2=0x7f4070207320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4070204f00 0x7f4070207320 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4070207ee0 msgr2=0x7f407020a300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4070207ee0 0x7f407020a300 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f407020c5b0 msgr2=0x7f407020c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f407020c5b0 0x7f407020c9e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f407694d640 1 -- 172.21.15.67:0/2121190658 reap_dead start 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 msgr2=0x7f4054071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f406c0b2000 tx=0x7f406c008040 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4070074780 msgr2=0x7f407014f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4070074780 0x7f407014f050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f406800cd20 tx=0x7f406800a650 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 shutdown_connections 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 --2- 172.21.15.67:0/2121190658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4070074780 0x7f407014f050 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 >> 172.21.15.67:0/2121190658 conn(0x7f4070070db0 msgr2=0x7f4070151790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 shutdown_connections 2024-04-25T15:51:40.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.611+0000 7f40783b0640 1 -- 172.21.15.67:0/2121190658 wait complete. 2024-04-25T15:51:40.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:40.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:40.635+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f990c159e20 con 0x7f990c0caf60 2024-04-25T15:51:40.679 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:40.676+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:40.748 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:40.743+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:40.777 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:40.778 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:40.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:40.883 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:40.880+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:40.950 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:40.944+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:40.992 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:40.992 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:41.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:41.051 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:41.048+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:41.151 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:41.148+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:41.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.187+0000 7f99097fa640 1 -- 172.21.15.67:0/1697451198 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v33) ==== 87+0+0 (secure 0 0 0) 0x7f98f405bd20 con 0x7f990c0caf60 2024-04-25T15:51:41.190 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:51:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98ec06f400 msgr2=0x7f98ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98ec06f400 0x7f98ec071910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0b0000 tx=0x7f98fc0a9070 comp rx=0 tx=0).stop 2024-04-25T15:51:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 msgr2=0x7f990c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15ddb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f98f400e870 tx=0x7f98f400ec70 comp rx=0 tx=0).stop 2024-04-25T15:51:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 shutdown_connections 2024-04-25T15:51:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98ec06f400 0x7f98ec071910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f990c15d790 0x7f990c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0caf60 0x7f990c15ddb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 --2- 172.21.15.67:0/1697451198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f990c073bf0 0x7f990c1553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 >> 172.21.15.67:0/1697451198 conn(0x7f990c070db0 msgr2=0x7f990c158ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 shutdown_connections 2024-04-25T15:51:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:41.191+0000 7f9912e64640 1 -- 172.21.15.67:0/1697451198 wait complete. 2024-04-25T15:51:41.203 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:41.204 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:41.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:41.263 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:41.260+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:41.352 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:41.347+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:41.406 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:41.406 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:41.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:41.453 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:41.448+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:41.554 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:41.552+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:41.586 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:41.586 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:41.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:41.654 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:41.652+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:41.749 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:41.749 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:41.755 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:41.751+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:41.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:41.857 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:41.851+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:41.957 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:41.956+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:41.966 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:41.966 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:41.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:42.070 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:42.067+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:42.159 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:42.152+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:42.174 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:42.174 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:42.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:42.262 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:42.260+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:42.363 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:42.359+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:42.395 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:42.395 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:42.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:42.498 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:42.495+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:42.565 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:42.560+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:42.609 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:42.609 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:42.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:42.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:42.666 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:42.660+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:42.767 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:42.763+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:42.818 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:42.818 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.815+0000 7f9a8ecd3640 1 Processor -- start 2024-04-25T15:51:42.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.815+0000 7f9a8ecd3640 1 -- start start 2024-04-25T15:51:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8ecd3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9a880b5d70 0x7f9a8806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8ecd3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8815fe00 0x7f9a88160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8ecd3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8806d780 con 0x7f9a8806ca60 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8ecd3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8815d510 con 0x7f9a880b5d70 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8ecd3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a88160be0 con 0x7f9a8815fe00 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a87fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9a880b5d70 0x7f9a8806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8d270640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8815fe00 0x7f9a88160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a87fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9a880b5d70 0x7f9a8806ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:42072/0 (socket says 172.21.15.67:42072) 2024-04-25T15:51:42.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a87fff640 1 -- 172.21.15.67:0/1264277433 learned_addr learned my addr 172.21.15.67:0/1264277433 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8d270640 1 -- 172.21.15.67:0/1264277433 >> v2:172.21.15.176:3300/0 conn(0x7f9a880b5d70 msgr2=0x7f9a8806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8d270640 1 --2- 172.21.15.67:0/1264277433 >> v2:172.21.15.176:3300/0 conn(0x7f9a880b5d70 0x7f9a8806ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8d270640 1 -- 172.21.15.67:0/1264277433 >> v1:172.21.15.176:6789/0 conn(0x7f9a8806ca60 legacy=0x7f9a880b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.819+0000 7f9a8d270640 1 -- 172.21.15.67:0/1264277433 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8815cbe0 con 0x7f9a8815fe00 2024-04-25T15:51:42.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8d270640 1 --2- 172.21.15.67:0/1264277433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8815fe00 0x7f9a88160210 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009dc0 tx=0x7f9a8001f140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de0a1407572ee08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a877fe640 1 -- 172.21.15.67:0/1264277433 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a800b4070 con 0x7f9a8815fe00 2024-04-25T15:51:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a877fe640 1 -- 172.21.15.67:0/1264277433 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a800af600 con 0x7f9a8815fe00 2024-04-25T15:51:42.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a877fe640 1 -- 172.21.15.67:0/1264277433 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a800af9d0 con 0x7f9a8815fe00 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/1264277433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8815fe00 msgr2=0x7f9a88160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/1264277433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8815fe00 0x7f9a88160210 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009dc0 tx=0x7f9a8001f140 comp rx=0 tx=0).stop 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/1264277433 shutdown_connections 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/1264277433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8815fe00 0x7f9a88160210 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/1264277433 >> v2:172.21.15.176:3300/0 conn(0x7f9a880b5d70 0x7f9a8806ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/1264277433 >> 172.21.15.67:0/1264277433 conn(0x7f9a8806e420 msgr2=0x7f9a8806e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/1264277433 shutdown_connections 2024-04-25T15:51:42.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.823+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/1264277433 wait complete. 2024-04-25T15:51:42.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 Processor -- start 2024-04-25T15:51:42.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 -- start start 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8806ca60 0x7f9a881f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880b5d70 0x7f9a8814cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a8815fe00 0x7f9a88155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8815c890 con 0x7f9a880b5d70 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8815d1a0 con 0x7f9a8815fe00 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9a880b6e90 con 0x7f9a8806ca60 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880b5d70 0x7f9a8814cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880b5d70 0x7f9a8814cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33094/0 (socket says 172.21.15.67:33094) 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 -- 172.21.15.67:0/2755233004 learned_addr learned my addr 172.21.15.67:0/2755233004 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8806ca60 0x7f9a881f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8d270640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a8815fe00 0x7f9a88155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8806ca60 msgr2=0x7f9a881f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8806ca60 0x7f9a881f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a8815fe00 msgr2=0x7f9a88155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9a8815fe00 0x7f9a88155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a88076ff0 con 0x7f9a880b5d70 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9a8806ca60 0x7f9a881f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a87fff640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880b5d70 0x7f9a8814cf50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9a7400e8a0 tx=0x7f9a7400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74017070 con 0x7f9a880b5d70 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7400ee90 con 0x7f9a880b5d70 2024-04-25T15:51:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400cb00 con 0x7f9a880b5d70 2024-04-25T15:51:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a881f3900 con 0x7f9a880b5d70 2024-04-25T15:51:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.827+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a88159600 con 0x7f9a880b5d70 2024-04-25T15:51:42.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9a8806a480 con 0x7f9a880b5d70 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9a7401a070 con 0x7f9a880b5d70 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a857fa640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a6c06f400 0x7f9a6c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5825+0+0 (secure 0 0 0) 0x7f9a7408f880 con 0x7f9a880b5d70 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v33) ==== 54+0+16 (secure 0 0 0) 0x7f9a740900c0 con 0x7f9a880b5d70 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a6c06f400 0x7f9a6c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a88159d10 0x7f9a88158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a8806d460 con 0x7f9a88159d10 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a8d270640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a88159d10 0x7f9a88158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a8d270640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a88159d10 0x7f9a88158bf0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a6c06f400 0x7f9a6c071910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000f70 tx=0x7f9a7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.831+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a8806d460 con 0x7f9a88159d10 2024-04-25T15:51:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.847+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a881f7510 con 0x7f9a88159d10 2024-04-25T15:51:42.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.851+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a881f7510 con 0x7f9a88159d10 2024-04-25T15:51:42.855 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.851+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a880c7180 0x7f9a880c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.851+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9a8806d180 con 0x7f9a880c7180 2024-04-25T15:51:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.851+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a880c7180 0x7f9a880c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.851+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a880c7180 0x7f9a880c7590 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.851+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a8806d180 con 0x7f9a880c7180 2024-04-25T15:51:42.868 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:42.864+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:42.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.867+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a881e7a90 con 0x7f9a880c7180 2024-04-25T15:51:42.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.867+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a881e7a90 con 0x7f9a880c7180 2024-04-25T15:51:42.872 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.867+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a88063090 0x7f9a880634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.867+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9a880c7fe0 con 0x7f9a88063090 2024-04-25T15:51:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.867+0000 7f9a87fff640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a88063090 0x7f9a880634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:42.867+0000 7f9a87fff640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a88063090 0x7f9a880634a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.969 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:42.964+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:42.970 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:42.970 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:42.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:43.070 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:43.064+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:43.170 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:43.167+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:43.182 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:43.182 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:43.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:43.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:51:43.272 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:43.268+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:43.373 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:43.368+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.379+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a880c7fe0 con 0x7f9a88063090 2024-04-25T15:51:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.387+0000 7fae2e2c7640 1 Processor -- start 2024-04-25T15:51:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.387+0000 7fae2e2c7640 1 -- start start 2024-04-25T15:51:43.392 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:43.392 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2e2c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae28071bc0 0x7fae280cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2e2c7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae280ca540 0x7fae280ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2e2c7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae2815de10 con 0x7fae280cc100 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2e2c7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fae2815ebb0 con 0x7fae280ca540 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2e2c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae2815e720 con 0x7fae28071bc0 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae280ca540 0x7fae280ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae280ca540 0x7fae280ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33100/0 (socket says 172.21.15.67:33100) 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 -- 172.21.15.67:0/2353933915 learned_addr learned my addr 172.21.15.67:0/2353933915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae27fff640 1 --2- 172.21.15.67:0/2353933915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae28071bc0 0x7fae280cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 -- 172.21.15.67:0/2353933915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae28071bc0 msgr2=0x7fae280cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 --2- 172.21.15.67:0/2353933915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae28071bc0 0x7fae280cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 -- 172.21.15.67:0/2353933915 >> v1:172.21.15.176:6789/0 conn(0x7fae280cc100 legacy=0x7fae2815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 -- 172.21.15.67:0/2353933915 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae28072390 con 0x7fae280ca540 2024-04-25T15:51:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae2c864640 1 --2- 172.21.15.67:0/2353933915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280ca540 0x7fae280ca950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae2806d180 tx=0x7fae2001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee66588cc8e9866f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae26ffd640 1 -- 172.21.15.67:0/2353933915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae200a5760 con 0x7fae280ca540 2024-04-25T15:51:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae26ffd640 1 -- 172.21.15.67:0/2353933915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae200a5900 con 0x7fae280ca540 2024-04-25T15:51:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.391+0000 7fae26ffd640 1 -- 172.21.15.67:0/2353933915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae200a5cd0 con 0x7fae280ca540 2024-04-25T15:51:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- 172.21.15.67:0/2353933915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280ca540 msgr2=0x7fae280ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/2353933915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280ca540 0x7fae280ca950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae2806d180 tx=0x7fae2001c9d0 comp rx=0 tx=0).stop 2024-04-25T15:51:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- 172.21.15.67:0/2353933915 shutdown_connections 2024-04-25T15:51:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/2353933915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280ca540 0x7fae280ca950 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/2353933915 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae28071bc0 0x7fae280cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- 172.21.15.67:0/2353933915 >> 172.21.15.67:0/2353933915 conn(0x7fae280700b0 msgr2=0x7fae280704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- 172.21.15.67:0/2353933915 shutdown_connections 2024-04-25T15:51:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- 172.21.15.67:0/2353933915 wait complete. 2024-04-25T15:51:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 Processor -- start 2024-04-25T15:51:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- start start 2024-04-25T15:51:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae28071bc0 0x7fae2806e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae280ca540 0x7fae2814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280cc100 0x7fae2814a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae28072150 con 0x7fae280cc100 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae28071fd0 con 0x7fae28071bc0 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2e2c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae2815ae30 con 0x7fae280ca540 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2c864640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280cc100 0x7fae2814a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2c864640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280cc100 0x7fae2814a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33108/0 (socket says 172.21.15.67:33108) 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.395+0000 7fae2c864640 1 -- 172.21.15.67:0/4195868242 learned_addr learned my addr 172.21.15.67:0/4195868242 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae27fff640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae28071bc0 0x7fae2806e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2c864640 1 -- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae280ca540 msgr2=0x7fae2814fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2c864640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae280ca540 0x7fae2814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2c864640 1 -- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae28071bc0 msgr2=0x7fae2806e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2c864640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae28071bc0 0x7fae2806e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2c864640 1 -- 172.21.15.67:0/4195868242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae28164440 con 0x7fae280cc100 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae27fff640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae28071bc0 0x7fae2806e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2c864640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280cc100 0x7fae2814a4e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fae2001d270 tx=0x7fae20019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae24ff9640 1 -- 172.21.15.67:0/4195868242 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae200b8670 con 0x7fae280cc100 2024-04-25T15:51:43.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae24ff9640 1 -- 172.21.15.67:0/4195868242 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae200be070 con 0x7fae280cc100 2024-04-25T15:51:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae24ff9640 1 -- 172.21.15.67:0/4195868242 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae200b8a90 con 0x7fae280cc100 2024-04-25T15:51:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a8815b900 con 0x7f9a88063090 2024-04-25T15:51:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2806ef50 con 0x7fae280cc100 2024-04-25T15:51:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae281f9910 con 0x7fae280cc100 2024-04-25T15:51:43.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae281f9460 con 0x7fae280cc100 2024-04-25T15:51:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.399+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a8815b900 con 0x7f9a88063090 2024-04-25T15:51:43.406 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a88157200 0x7f9a88157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9a88063ef0 con 0x7f9a88157200 2024-04-25T15:51:43.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7fae24ff9640 1 -- 172.21.15.67:0/4195868242 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fae200b8c30 con 0x7fae280cc100 2024-04-25T15:51:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7f9a8d270640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a88157200 0x7f9a88157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7fae24ff9640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae1406f430 0x7fae14071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7f9a8d270640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a88157200 0x7f9a88157610 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7fae24ff9640 1 -- 172.21.15.67:0/4195868242 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5825+0+0 (secure 0 0 0) 0x7fae2012ae90 con 0x7fae280cc100 2024-04-25T15:51:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7fae27fff640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae1406f430 0x7fae14071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a88063ef0 con 0x7f9a88157200 2024-04-25T15:51:43.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.403+0000 7fae27fff640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae1406f430 0x7fae14071940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fae1800b9f0 tx=0x7fae18008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.407+0000 7fae24ff9640 1 -- 172.21.15.67:0/4195868242 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fae200165c0 con 0x7fae280cc100 2024-04-25T15:51:43.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.419+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a88063ef0 con 0x7f9a88157200 2024-04-25T15:51:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.423+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a88063ef0 con 0x7f9a88157200 2024-04-25T15:51:43.426 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.423+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a8802c9e0 0x7f9a8802cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.423+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9a88158060 con 0x7f9a8802c9e0 2024-04-25T15:51:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.423+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a8802c9e0 0x7f9a8802cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.423+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a8802c9e0 0x7f9a8802cdf0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.423+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a88158060 con 0x7f9a8802c9e0 2024-04-25T15:51:43.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.439+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a88158060 con 0x7f9a8802c9e0 2024-04-25T15:51:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.443+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a88158060 con 0x7f9a8802c9e0 2024-04-25T15:51:43.448 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.443+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a8802d840 0x7f9a88209aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:43.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.443+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9a88158060 con 0x7f9a8802d840 2024-04-25T15:51:43.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.443+0000 7f9a87fff640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a8802d840 0x7f9a88209aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:43.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.443+0000 7f9a87fff640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a8802d840 0x7f9a88209aa0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:43.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.447+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a88158060 con 0x7f9a8802d840 2024-04-25T15:51:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.463+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a88158060 con 0x7f9a8802d840 2024-04-25T15:51:43.474 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:43.468+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:43.556 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:43.556 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:43.556 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:51:43.556 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:51:43.556 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:43.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:43.574 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:43.568+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:43.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:43.579+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fae2806a480 con 0x7fae280cc100 2024-04-25T15:51:43.674 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:43.671+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:43.740 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:43.740 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:43.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:43.740 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.524173239000007, 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:43.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.524133594000006, 2024-04-25T15:51:43.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.521226572000003, 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:43.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.751 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.521073635999997, 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.519557992000003, 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:43.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:43.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:43.759 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.519415344999999, 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.519312253999999, 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:43.763 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.519159504000001, 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:43.767 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:43.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:43.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.517211278000005, 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:43.771 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:43.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.774 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.489183646999997, 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:43.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:43.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.487160756000002, 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.486990206999998, 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:43.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:43.785 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.484634710999998, 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:43.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:43.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.790 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.484599998999997, 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:43.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.484579189999998, 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:43.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.407692142999998, 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:43.798 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:43.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:43.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.407668061000003, 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.803 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:43.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:43.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.407658364, 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:43.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:43.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.811 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.401763426999999, 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:43.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.812324770000004, 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:43.820 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:43.822 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:43.822 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:43.815+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:43.923 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:43.920+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:43.980 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:43.980 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:43.980 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:43.980 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:43.996 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:44.039 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:44.036+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:44.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:44.119+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:44.190 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:44.190 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:44.190 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:44.190 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:44.190 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:44.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:44.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.215+0000 7fae24ff9640 1 -- 172.21.15.67:0/4195868242 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v34) ==== 97+0+0 (secure 0 0 0) 0x7fae200f7d00 con 0x7fae280cc100 2024-04-25T15:51:44.220 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae1406f430 msgr2=0x7fae14071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae1406f430 0x7fae14071940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fae1800b9f0 tx=0x7fae18008040 comp rx=0 tx=0).stop 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280cc100 msgr2=0x7fae2814a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280cc100 0x7fae2814a4e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fae2001d270 tx=0x7fae20019770 comp rx=0 tx=0).stop 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 shutdown_connections 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae1406f430 0x7fae14071940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae280cc100 0x7fae2814a4e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae280ca540 0x7fae2814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 --2- 172.21.15.67:0/4195868242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae28071bc0 0x7fae2806e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 >> 172.21.15.67:0/4195868242 conn(0x7fae280700b0 msgr2=0x7fae280ca950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 shutdown_connections 2024-04-25T15:51:44.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.219+0000 7fae2e2c7640 1 -- 172.21.15.67:0/4195868242 wait complete. 2024-04-25T15:51:44.251 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:44.248+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.263+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a88158060 con 0x7f9a8802d840 2024-04-25T15:51:44.267 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.263+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a8802dc50 0x7f9a8820c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.263+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9a8820f410 con 0x7f9a8802dc50 2024-04-25T15:51:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.263+0000 7f9a8d270640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a8802dc50 0x7f9a8820c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.263+0000 7f9a8d270640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a8802dc50 0x7f9a8820c500 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.263+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a8820f410 con 0x7f9a8802dc50 2024-04-25T15:51:44.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.279+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a8820ca40 con 0x7f9a8802dc50 2024-04-25T15:51:44.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.283+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a8820ca40 con 0x7f9a8802dc50 2024-04-25T15:51:44.288 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:44.289 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:44.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.283+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a8820d160 0x7f9a8820d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:44.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.283+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9a882127e0 con 0x7f9a8820d160 2024-04-25T15:51:44.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.283+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a8820d160 0x7f9a8820d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:44.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.283+0000 7f9a8ca6f640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a8820d160 0x7f9a8820d570 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:44.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.287+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a882127e0 con 0x7f9a8820d160 2024-04-25T15:51:44.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.299+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9a88211660 con 0x7f9a8820d160 2024-04-25T15:51:44.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a857fa640 1 -- 172.21.15.67:0/2755233004 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9a88211660 con 0x7f9a8820d160 2024-04-25T15:51:44.307 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a88159d10 msgr2=0x7f9a88158bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9a88159d10 0x7f9a88158bf0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a880c7180 msgr2=0x7f9a880c7590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9a880c7180 0x7f9a880c7590 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a88063090 msgr2=0x7f9a880634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9a88063090 0x7f9a880634a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a88157200 msgr2=0x7f9a88157610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9a88157200 0x7f9a88157610 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a8802c9e0 msgr2=0x7f9a8802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8d270640 1 -- 172.21.15.67:0/2755233004 reap_dead start 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9a8802c9e0 0x7f9a8802cdf0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8d270640 1 -- 172.21.15.67:0/2755233004 reap_dead start 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a8802d840 msgr2=0x7f9a88209aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9a8802d840 0x7f9a88209aa0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a8802dc50 msgr2=0x7f9a8820c500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9a8802dc50 0x7f9a8820c500 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a8820d160 msgr2=0x7f9a8820d570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9a8820d160 0x7f9a8820d570 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a6c06f400 msgr2=0x7f9a6c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9a6c06f400 0x7f9a6c071910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c000f70 tx=0x7f9a7c009040 comp rx=0 tx=0).stop 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880b5d70 msgr2=0x7f9a8814cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880b5d70 0x7f9a8814cf50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9a7400e8a0 tx=0x7f9a7400ed60 comp rx=0 tx=0).stop 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8d270640 1 -- 172.21.15.67:0/2755233004 reap_dead start 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 shutdown_connections 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 --2- 172.21.15.67:0/2755233004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880b5d70 0x7f9a8814cf50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 >> 172.21.15.67:0/2755233004 conn(0x7f9a8806e420 msgr2=0x7f9a8815ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:44.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 shutdown_connections 2024-04-25T15:51:44.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:44.303+0000 7f9a8ecd3640 1 -- 172.21.15.67:0/2755233004 wait complete. 2024-04-25T15:51:44.327 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:44.323+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.144645558999997, 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.357 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:44.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:44.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:44.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:44.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.142030762000005, 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:44.362 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:44.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.141531865000005, 2024-04-25T15:51:44.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:44.370 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.140442618999998, 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:44.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:44.373 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.139822124000005, 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.137427897999999, 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.378 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:44.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:44.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:44.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.108307494999998, 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.382 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:44.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:44.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.103562760999999, 2024-04-25T15:51:44.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:44.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:44.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.100035259999999, 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.099902778999997, 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.394 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:44.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:44.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.478963321000002, 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.398 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.467528188999999, 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.403 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.467130752999999, 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.407 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:44.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:44.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.467021012000004, 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.411 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:44.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:44.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.466520267, 2024-04-25T15:51:44.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:44.419 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.466144280999998, 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 40.427546098000001, 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:44.423 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:44.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:44.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:44.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 40.427445839000001, 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.427 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:44.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:44.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 40.425340923999997, 2024-04-25T15:51:44.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.434 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 40.424965348000001, 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:44.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:44.438 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:44.439 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:44.440 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:44.435+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:44.540 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:44.536+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:44.635 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:44.636 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:44.641 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:44.636+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:44.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:44.741 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:44.736+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:44.839 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:44.839 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:44.839 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:44.839 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:44.839 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:44.842 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:44.839+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:44.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:44.943 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:44.939+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:45.044 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:45.039+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:45.051 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801291112999998, 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:45.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:45.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.056 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801260552000002, 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:45.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:45.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:45.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.061 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801252720999997, 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:45.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:45.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:45.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:45.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:45.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801243792000001, 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801237559999997, 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:45.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:45.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.076 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801222150000001, 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:45.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:45.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801209317999998, 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.801204104, 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:45.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.715593986999998, 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:45.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.163063993999998, 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:45.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.161671155999997, 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.100 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:45.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.123651029999998, 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.104 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:45.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:45.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.123627704, 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:45.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:45.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.113 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.123587164, 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:45.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.123578504000001, 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:45.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:45.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.123536831000003, 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:45.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:45.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:45.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.123527574000001, 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.130 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:45.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:45.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.119390148000001, 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.119335571000001, 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.138 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.119302363000003, 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.142 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:45.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:45.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.146 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:45.147 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:45.148 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:45.144+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:45.247 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:45.247 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:45.247 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-04-25T15:51:45.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-04-25T15:51:45.248 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:45.243+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:45.344 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:45.344 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:45.344 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:45.344 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:45.349 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:45.344+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:45.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.407+0000 7fb1f1bce640 1 Processor -- start 2024-04-25T15:51:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.407+0000 7fb1f1bce640 1 -- start start 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1f1bce640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1ec071bc0 0x7fb1ec0cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1f1bce640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb1ec0cc100 0x7fb1ec15a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1f1bce640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec15de10 con 0x7fb1ec15ac60 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1f1bce640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb1ec15ebb0 con 0x7fb1ec071bc0 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1f1bce640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb1ec15e720 con 0x7fb1ec0cc100 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1ec071bc0 0x7fb1ec0cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eb7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1ec071bc0 0x7fb1ec0cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33120/0 (socket says 172.21.15.67:33120) 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3070265969 learned_addr learned my addr 172.21.15.67:0/3070265969 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eaffd640 1 --2- 172.21.15.67:0/3070265969 >> v2:172.21.15.176:3300/0 conn(0x7fb1ec0cc100 0x7fb1ec15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3070265969 >> v2:172.21.15.176:3300/0 conn(0x7fb1ec0cc100 msgr2=0x7fb1ec15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eb7fe640 1 --2- 172.21.15.67:0/3070265969 >> v2:172.21.15.176:3300/0 conn(0x7fb1ec0cc100 0x7fb1ec15a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3070265969 >> v1:172.21.15.176:6789/0 conn(0x7fb1ec15ac60 legacy=0x7fb1ec0ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:45.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3070265969 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec072390 con 0x7fb1ec071bc0 2024-04-25T15:51:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.411+0000 7fb1eaffd640 1 --2- 172.21.15.67:0/3070265969 >> v2:172.21.15.176:3300/0 conn(0x7fb1ec0cc100 0x7fb1ec15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1eb7fe640 1 --2- 172.21.15.67:0/3070265969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec0cbbc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec071000 tx=0x7fb1dc01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995a1bf5705d1e4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1ea7fc640 1 -- 172.21.15.67:0/3070265969 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0a57d0 con 0x7fb1ec071bc0 2024-04-25T15:51:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1ea7fc640 1 -- 172.21.15.67:0/3070265969 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc0a5970 con 0x7fb1ec071bc0 2024-04-25T15:51:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1ea7fc640 1 -- 172.21.15.67:0/3070265969 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0a5d40 con 0x7fb1ec071bc0 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3070265969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 msgr2=0x7fb1ec0cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3070265969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec0cbbc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec071000 tx=0x7fb1dc01c9f0 comp rx=0 tx=0).stop 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3070265969 shutdown_connections 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3070265969 >> v2:172.21.15.176:3300/0 conn(0x7fb1ec0cc100 0x7fb1ec15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3070265969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec0cbbc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3070265969 >> 172.21.15.67:0/3070265969 conn(0x7fb1ec0700b0 msgr2=0x7fb1ec0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3070265969 shutdown_connections 2024-04-25T15:51:45.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3070265969 wait complete. 2024-04-25T15:51:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 Processor -- start 2024-04-25T15:51:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.415+0000 7fb1f1bce640 1 -- start start 2024-04-25T15:51:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec15b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0cc100 0x7fb1ec14d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1ec15ac60 0x7fb1ec14f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec072150 con 0x7fb1ec071bc0 2024-04-25T15:51:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec071fd0 con 0x7fb1ec0cc100 2024-04-25T15:51:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec0cac50 con 0x7fb1ec15ac60 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec15b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec15b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33134/0 (socket says 172.21.15.67:33134) 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3620331711 learned_addr learned my addr 172.21.15.67:0/3620331711 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eaffd640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0cc100 0x7fb1ec14d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1ebfff640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1ec15ac60 0x7fb1ec14f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1ec15ac60 msgr2=0x7fb1ec14f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1ec15ac60 0x7fb1ec14f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0cc100 msgr2=0x7fb1ec14d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0cc100 0x7fb1ec14d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 -- 172.21.15.67:0/3620331711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec15cc60 con 0x7fb1ec071bc0 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1ebfff640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1ec15ac60 0x7fb1ec14f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eb7fe640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec15b880 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc01a620 tx=0x7fb1dc019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1d3fff640 1 -- 172.21.15.67:0/3620331711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0b8670 con 0x7fb1ec071bc0 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1d3fff640 1 -- 172.21.15.67:0/3620331711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc0be070 con 0x7fb1ec071bc0 2024-04-25T15:51:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec15cf40 con 0x7fb1ec071bc0 2024-04-25T15:51:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1d3fff640 1 -- 172.21.15.67:0/3620331711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0b8a70 con 0x7fb1ec071bc0 2024-04-25T15:51:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec15d500 con 0x7fb1ec071bc0 2024-04-25T15:51:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ec06a480 con 0x7fb1ec071bc0 2024-04-25T15:51:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1d3fff640 1 -- 172.21.15.67:0/3620331711 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb1dc0b8c10 con 0x7fb1ec071bc0 2024-04-25T15:51:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1d3fff640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb1d806f430 0x7fb1d8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1d3fff640 1 -- 172.21.15.67:0/3620331711 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5825+0+0 (secure 0 0 0) 0x7fb1dc129d40 con 0x7fb1ec071bc0 2024-04-25T15:51:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.419+0000 7fb1eaffd640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb1d806f430 0x7fb1d8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.423+0000 7fb1eaffd640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb1d806f430 0x7fb1d8071940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec15aa20 tx=0x7fb1e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:45.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.427+0000 7fb1d3fff640 1 -- 172.21.15.67:0/3620331711 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb1dc0c7020 con 0x7fb1ec071bc0 2024-04-25T15:51:45.449 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:45.444+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:45.550 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:45.547+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:45.553 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:45.554 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:45.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:45.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:45.599+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd in", "ids": ["1"]} v 0) -- 0x7fb1ec0c6c10 con 0x7fb1ec071bc0 2024-04-25T15:51:45.603 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:51:45.599+0000 7fa4fdaf0640 -1 mon.a@0(leader).osd e34 definitely_dead 0 2024-04-25T15:51:45.651 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:45.647+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:45.752 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:45.747+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:45.764 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:45.764 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:45.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:45.764 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:45.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.764 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.550930042999994, 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:45.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:51:45.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.550900779000003, 2024-04-25T15:51:45.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:51:45.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.550891491000002, 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.774 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:51:45.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:51:45.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:51:45.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:51:45.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.550883915, 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.779 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:51:45.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:51:45.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.550875282000007, 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.784 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:51:45.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:51:45.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.550825449000001, 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:51:45.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.793 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.550817570999996, 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:45.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:51:45.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:45.797 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.514389563999998, 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:45.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.514358899999998, 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:51:45.802 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:51:45.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:51:45.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.514346369999998, 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:51:45.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.514335738, 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:51:45.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:51:45.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.514316192999999, 2024-04-25T15:51:45.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.514307301000002, 2024-04-25T15:51:45.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:45.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:51:45.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.510744684999999, 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:51:45.826 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:45.830 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 63.510624073000002, 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:51:45.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.834 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.875626052999998, 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:45.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:51:45.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.835056655000002, 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.839 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:45.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:51:45.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.835051147000001, 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.844 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:51:45.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:51:45.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:51:45.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.832636633, 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.849 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:51:45.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:51:45.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.831674747000001, 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:45.853 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:45.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:51:45.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:51:45.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:45.857 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:45.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:45.859 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:45.855+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:45.959 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:45.956+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:46.013 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:46.013 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:46.013 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:46.014 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:46.031 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:46.075 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:46.072+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:46.162 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:46.155+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:46.225 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:46.225 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:46.225 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:46.225 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:46.225 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:46.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.231+0000 7fb1d3fff640 1 -- 172.21.15.67:0/3620331711 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["1"]}]=0 marked in osd.1. v35) ==== 85+0+0 (secure 0 0 0) 0x7fb1dc0165e0 con 0x7fb1ec071bc0 2024-04-25T15:51:46.236 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.1. 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb1d806f430 msgr2=0x7fb1d8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb1d806f430 0x7fb1d8071940 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec15aa20 tx=0x7fb1e0008040 comp rx=0 tx=0).stop 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 msgr2=0x7fb1ec15b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec15b880 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc01a620 tx=0x7fb1dc019790 comp rx=0 tx=0).stop 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 shutdown_connections 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb1d806f430 0x7fb1d8071940 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec15aa20 tx=0x7fb1e0008040 comp rx=0 tx=0).stop 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1ec15ac60 0x7fb1ec14f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0cc100 0x7fb1ec14d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 --2- 172.21.15.67:0/3620331711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ec071bc0 0x7fb1ec15b880 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc01a620 tx=0x7fb1dc019790 comp rx=0 tx=0).stop 2024-04-25T15:51:46.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.235+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 >> 172.21.15.67:0/3620331711 conn(0x7fb1ec0700b0 msgr2=0x7fb1ec0c7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:46.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.239+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 shutdown_connections 2024-04-25T15:51:46.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.239+0000 7fb1f1bce640 1 -- 172.21.15.67:0/3620331711 wait complete. 2024-04-25T15:51:46.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:46.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:51:46.277 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-04-25T15:51:46.295 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:46.292+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:46.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:46.374 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:46.371+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:46.399 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:46.399 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:46.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:46.399 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:46.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.399 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.399 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.297435320999995, 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:51:46.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:51:46.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:51:46.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.833834506000002, 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01409815, 2024-04-25T15:51:46.405 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 25 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562066+0000", 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562072+0000", 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.703e-06 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562156+0000", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3486000000000007e-05 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562165+0000", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.568966+0000", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006801372 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569110+0000", 2024-04-25T15:51:46.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014385799999999999 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569344+0000", 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023376500000000001 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.573357+0000", 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040127870000000003 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.575955+0000", 2024-04-25T15:51:46.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025982969999999998 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576077+0000", 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121966 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576163+0000", 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6058999999999997e-05 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.707029070000004, 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013368089999999999, 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688872+0000", 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0470000000000001e-06 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688892+0000", 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9143e-05 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688940+0000", 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8343000000000001e-05 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688961+0000", 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0925999999999999e-05 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694131+0000", 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051698639999999997 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694283+0000", 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015229800000000001 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694506+0000", 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222995 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.698950+0000", 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044438300000000002 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702065+0000", 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031151719999999998 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702159+0000", 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4089e-05 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702238+0000", 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9382999999999995e-05 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706902518000007, 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013712703999999999, 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.416 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 44 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688998+0000", 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689301+0000", 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000302684 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689729+0000", 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000428486 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689746+0000", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6690000000000001e-05 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.695997+0000", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062505779999999997 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696164+0000", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167045 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696383+0000", 2024-04-25T15:51:46.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.701919+0000", 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055357399999999999 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702410+0000", 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00049136500000000001 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702581+0000", 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017146899999999999 2024-04-25T15:51:46.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702710+0000", 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012822199999999999 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706590176999995, 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013745535999999999, 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.421 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 45 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689310+0000", 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.127e-06 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689317+0000", 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.973e-06 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689793+0000", 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000475468 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689804+0000", 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.082e-05 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697359+0000", 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075556 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697536+0000", 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176658 2024-04-25T15:51:46.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697753+0000", 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021705499999999999 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702734+0000", 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049807779999999999 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702850+0000", 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116109 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702934+0000", 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4429000000000001e-05 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703055+0000", 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120519 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:51:46.426 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706578131000001, 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013995720999999999, 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 46 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689322+0000", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8999999999999996e-07 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689329+0000", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1899999999999998e-06 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689831+0000", 2024-04-25T15:51:46.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050143900000000001 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689839+0000", 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4069999999999993e-06 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699042+0000", 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092026810000000007 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699214+0000", 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017275100000000001 2024-04-25T15:51:46.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699442+0000", 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022718200000000001 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702745+0000", 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003303266 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703161+0000", 2024-04-25T15:51:46.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000416253 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703250+0000", 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8806000000000003e-05 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703317+0000", 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7056000000000003e-05 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.431 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706566863999996, 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014300548, 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689491+0000", 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015768600000000001 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689857+0000", 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000366313 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689865+0000", 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1470000000000001e-06 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700215+0000", 2024-04-25T15:51:46.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010350177 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700471+0000", 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521499999999999 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700879+0000", 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040780599999999998 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703339+0000", 2024-04-25T15:51:46.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024602140000000001 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703443+0000", 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104393 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703524+0000", 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0745e-05 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703633+0000", 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109226 2024-04-25T15:51:46.436 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706402585999996, 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019412740000000001, 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.437 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 48 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689498+0000", 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5999999999999991e-07 2024-04-25T15:51:46.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689505+0000", 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1339999999999998e-06 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689885+0000", 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000380553 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689894+0000", 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0660000000000004e-06 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703735+0000", 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013841509 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703880+0000", 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014450899999999999 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704121+0000", 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024184799999999999 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708286+0000", 2024-04-25T15:51:46.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041643369999999997 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708530+0000", 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024477400000000001 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708710+0000", 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179715 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708910+0000", 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199335 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.441 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706390576000004, 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019597107999999999, 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 49 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.442 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4700000000000001e-07 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689515+0000", 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3770000000000004e-06 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689911+0000", 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039616199999999998 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689920+0000", 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2939999999999994e-06 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704840+0000", 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014919412999999999 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704990+0000", 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150388 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.705238+0000", 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000248392 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708485+0000", 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032464740000000001 2024-04-25T15:51:46.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708936+0000", 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045072599999999998 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709037+0000", 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101482 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709106+0000", 2024-04-25T15:51:46.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8752999999999997e-05 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706381019000005, 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:51:46.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:51:46.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706232396999994, 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.452 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:51:46.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.456 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.706221318000004, 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:51:46.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:51:46.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.399382837999994, 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019231747, 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.462 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 69 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996518+0000", 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6840000000000001e-06 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996645+0000", 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012677899999999999 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996696+0000", 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0912999999999999e-05 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996718+0000", 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1931999999999999e-05 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000632+0000", 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003914509 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000756+0000", 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012338099999999999 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000879+0000", 2024-04-25T15:51:46.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012301599999999999 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009110+0000", 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082309819999999995 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015495+0000", 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063848969999999996 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015700+0000", 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020508399999999999 2024-04-25T15:51:46.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015748+0000", 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8569999999999997e-05 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.399130795999994, 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019263037, 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.467 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 70 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996770+0000", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7060000000000001e-06 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996786+0000", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5497000000000001e-05 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996990+0000", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000203938 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997014+0000", 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4617000000000002e-05 2024-04-25T15:51:46.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002337+0000", 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053230450000000002 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002417+0000", 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9548999999999993e-05 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002579+0000", 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162385 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009114+0000", 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00653442 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015848+0000", 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067339909999999999 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015951+0000", 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102796 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016032+0000", 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1093000000000004e-05 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.399104895999997, 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019380720000000001, 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.472 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 71 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996796+0000", 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.063e-06 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996810+0000", 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4100000000000001e-05 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997061+0000", 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000251797 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997076+0000", 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4970000000000001e-05 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003008+0000", 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005931487 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003148+0000", 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140097 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003371+0000", 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022273400000000001 2024-04-25T15:51:46.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009117+0000", 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057466929999999998 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016106+0000", 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069880919999999996 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016144+0000", 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8686000000000002e-05 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016175+0000", 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1000999999999997e-05 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.399082230000005, 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019495743999999999, 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.477 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996818+0000", 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.046e-06 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996944+0000", 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012572199999999999 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997139+0000", 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019529899999999999 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997151+0000", 2024-04-25T15:51:46.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2116000000000001e-05 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004407+0000", 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072560519999999998 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004474+0000", 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6810000000000006e-05 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004576+0000", 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101431 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009121+0000", 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045451099999999998 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016239+0000", 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071178359999999998 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016280+0000", 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0914000000000002e-05 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016313+0000", 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3408000000000003e-05 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.398702963999995, 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019262988000000002, 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.482 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 73 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997198+0000", 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.347e-06 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997208+0000", 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0427e-05 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997419+0000", 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021052399999999999 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997433+0000", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4687e-05 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005473+0000", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080393610000000001 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005619+0000", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146157 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005848+0000", 2024-04-25T15:51:46.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229001 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009124+0000", 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003276288 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016366+0000", 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072413759999999999 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016408+0000", 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2119000000000001e-05 2024-04-25T15:51:46.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016459+0000", 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1700999999999998e-05 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.398685490000005, 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019348008999999999, 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.487 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 74 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997215+0000", 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997220+0000", 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7200000000000003e-06 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997480+0000", 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026014000000000001 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997493+0000", 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2306999999999999e-05 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006512+0000", 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090195669999999992 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006591+0000", 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9058000000000001e-05 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006724+0000", 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013295000000000001 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009130+0000", 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024053659999999999 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073654790000000003 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016533+0000", 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7432999999999999e-05 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016562+0000", 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9379000000000001e-05 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.398675683999997, 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020191183000000001, 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.492 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997376+0000", 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015132899999999999 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997528+0000", 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152275 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997538+0000", 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0137e-05 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007159+0000", 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096209090000000004 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007248+0000", 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9078000000000003e-05 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007394+0000", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014618699999999999 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009133+0000", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001739008 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017311+0000", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081775159999999993 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017360+0000", 2024-04-25T15:51:46.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9202000000000001e-05 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017415+0000", 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4932000000000002e-05 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 86.398518279000001, 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:51:46.497 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:46.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:51:46.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:46.502 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:46.504 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:46.500+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:46.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.403+0000 7fc716ae1640 1 Processor -- start 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.403+0000 7fc716ae1640 1 -- start start 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.403+0000 7fc716ae1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc71015a720 0x7fc71015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.403+0000 7fc716ae1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc71015de10 con 0x7fc710071bc0 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.403+0000 7fc716ae1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc710072390 con 0x7fc7100ca540 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.403+0000 7fc716ae1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc71015ebb0 con 0x7fc71015a720 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc71487d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc710071bc0 0x7fc7100cbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40244/0 (socket says 172.21.15.67:40244) 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc71487d640 1 -- 172.21.15.67:0/2748372283 learned_addr learned my addr 172.21.15.67:0/2748372283 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70ffff640 1 --2- 172.21.15.67:0/2748372283 >> v2:172.21.15.176:3300/0 conn(0x7fc71015a720 0x7fc71015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2800537666 0 0) 0x7fc71015de10 con 0x7fc710071bc0 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6f8003530 con 0x7fc710071bc0 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1750622094 0 0) 0x7fc6f8003530 con 0x7fc710071bc0 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 >> v2:172.21.15.176:3300/0 conn(0x7fc71015a720 msgr2=0x7fc71015ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 --2- 172.21.15.67:0/2748372283 >> v2:172.21.15.176:3300/0 conn(0x7fc71015a720 0x7fc71015ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 >> v1:172.21.15.176:6789/0 conn(0x7fc7100ca540 legacy=0x7fc710164fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71015e0d0 con 0x7fc710071bc0 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fc7000016a0 con 0x7fc710071bc0 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 >> v1:172.21.15.67:6789/0 conn(0x7fc710071bc0 legacy=0x7fc7100cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6f80039c0 0x7fc6f8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc6f8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6f80051d0 0x7fc6f800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc71015ebb0 con 0x7fc6f8004810 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f800c170 con 0x7fc6f80039c0 2024-04-25T15:51:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f800c370 con 0x7fc6f80051d0 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70ffff640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc6f8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc71507e640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6f80051d0 0x7fc6f800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc71487d640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6f80039c0 0x7fc6f8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70ffff640 1 -- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6f80051d0 msgr2=0x7fc6f800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70ffff640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6f80051d0 0x7fc6f800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70ffff640 1 -- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6f80039c0 msgr2=0x7fc6f8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70ffff640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6f80039c0 0x7fc6f8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc70ffff640 1 -- 172.21.15.67:0/2748372283 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f8003530 con 0x7fc6f8004810 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc71487d640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6f80039c0 0x7fc6f8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc71507e640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6f80051d0 0x7fc6f800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.407+0000 7fc71507e640 1 -- 172.21.15.67:0/2748372283 reap_dead start 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc70ffff640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc6f8004c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc704009ab0 tx=0x7fc70401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e087aa625381f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7040b2070 con 0x7fc6f8004810 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7040a5640 con 0x7fc6f8004810 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc70f7fe640 1 -- 172.21.15.67:0/2748372283 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7040096b0 con 0x7fc6f8004810 2024-04-25T15:51:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc716ae1640 1 -- 172.21.15.67:0/2748372283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 msgr2=0x7fc6f8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc716ae1640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc6f8004c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc704009ab0 tx=0x7fc70401ca50 comp rx=0 tx=0).stop 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc716ae1640 1 -- 172.21.15.67:0/2748372283 shutdown_connections 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc716ae1640 1 --2- 172.21.15.67:0/2748372283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc6f8004c20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc716ae1640 1 -- 172.21.15.67:0/2748372283 >> 172.21.15.67:0/2748372283 conn(0x7fc7100700b0 msgr2=0x7fc7100704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc716ae1640 1 -- 172.21.15.67:0/2748372283 shutdown_connections 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.411+0000 7fc716ae1640 1 -- 172.21.15.67:0/2748372283 wait complete. 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.415+0000 7fc716ae1640 1 Processor -- start 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.415+0000 7fc716ae1640 1 -- start start 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc7101f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc710152060 0x7fc7101538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7100c7880 0x7fc71014e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7100c93b0 con 0x7fc6f8004810 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc71015e720 con 0x7fc7100c7880 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc710071fd0 con 0x7fc710152060 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc7101f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc7101f2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33144/0 (socket says 172.21.15.67:33144) 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 -- 172.21.15.67:0/3929120981 learned_addr learned my addr 172.21.15.67:0/3929120981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc70ffff640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc710152060 0x7fc7101538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71507e640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7100c7880 0x7fc71014e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 -- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc710152060 msgr2=0x7fc7101538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc710152060 0x7fc7101538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 -- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7100c7880 msgr2=0x7fc71014e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7100c7880 0x7fc71014e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 -- 172.21.15.67:0/3929120981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7101f2ec0 con 0x7fc6f8004810 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc71487d640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc7101f2980 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc700010c50 tx=0x7fc70000e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc70d7fa640 1 -- 172.21.15.67:0/3929120981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70001b070 con 0x7fc6f8004810 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc70d7fa640 1 -- 172.21.15.67:0/3929120981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc700011040 con 0x7fc6f8004810 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc70d7fa640 1 -- 172.21.15.67:0/3929120981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc700010e60 con 0x7fc6f8004810 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7101f4b00 con 0x7fc6f8004810 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7101f50c0 con 0x7fc6f8004810 2024-04-25T15:51:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.419+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc71006a480 con 0x7fc6f8004810 2024-04-25T15:51:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.423+0000 7fc70d7fa640 1 -- 172.21.15.67:0/3929120981 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc7000145a0 con 0x7fc6f8004810 2024-04-25T15:51:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.423+0000 7fc70d7fa640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc6fc06f430 0x7fc6fc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.423+0000 7fc70d7fa640 1 -- 172.21.15.67:0/3929120981 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5825+0+0 (secure 0 0 0) 0x7fc700091f20 con 0x7fc6f8004810 2024-04-25T15:51:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.423+0000 7fc70ffff640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc6fc06f430 0x7fc6fc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.423+0000 7fc70ffff640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc6fc06f430 0x7fc6fc071940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc7040af000 tx=0x7fc704008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.427+0000 7fc70d7fa640 1 -- 172.21.15.67:0/3929120981 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc70005ed90 con 0x7fc6f8004810 2024-04-25T15:51:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4c01030640 1 Processor -- start 2024-04-25T15:51:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4c01030640 1 -- start start 2024-04-25T15:51:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4c01030640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4bfc160800 0x7f4bfc06d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4c01030640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc06d8e0 con 0x7f4bfc0c96e0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4c01030640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc073c70 con 0x7f4bfc0c92d0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4c01030640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc06dbc0 con 0x7f4bfc160800 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfb59e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4bfc0c96e0 0x7f4bfc167180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40260/0 (socket says 172.21.15.67:40260) 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1116687851 learned_addr learned my addr 172.21.15.67:0/1116687851 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4bfc160800 0x7f4bfc06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866375167 0 0) 0x7f4bfc06d8e0 con 0x7f4bfc0c96e0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be8003530 con 0x7f4bfc0c96e0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2402257122 0 0) 0x7f4be8003530 con 0x7f4bfc0c96e0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4bfc160800 msgr2=0x7f4bfc06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4bfc160800 0x7f4bfc06d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 >> v1:172.21.15.176:6789/0 conn(0x7f4bfc0c92d0 legacy=0x7f4bfc072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc073f50 con 0x7f4bfc0c96e0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f4bf4001a00 con 0x7f4bfc0c96e0 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 >> v1:172.21.15.67:6789/0 conn(0x7f4bfc0c96e0 legacy=0x7f4bfc167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be80039f0 0x7f4be8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be8004810 0x7f4be8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be8005160 0x7f4be800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc06dbc0 con 0x7f4be80039f0 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4be800c0d0 con 0x7f4be8005160 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4be800c2d0 con 0x7f4be8004810 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4bfc160800 0x7f4bfc06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.475+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be80039f0 0x7f4be8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be8005160 0x7f4be800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be8004810 0x7f4be8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfad9d640 1 -- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be8004810 msgr2=0x7f4be8004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be8004810 0x7f4be8004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfad9d640 1 -- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be8005160 msgr2=0x7f4be800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be8005160 0x7f4be800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfad9d640 1 -- 172.21.15.67:0/1116687851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be8003530 con 0x7f4be80039f0 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be8004810 0x7f4be8004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:46.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be80039f0 0x7f4be8003e20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc071dd0 tx=0x7f4bec01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4560ddfd1f99efe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec0097f0 con 0x7f4be80039f0 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bec004030 con 0x7f4be80039f0 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfa59c640 1 -- 172.21.15.67:0/1116687851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec004400 con 0x7f4be80039f0 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4c01030640 1 -- 172.21.15.67:0/1116687851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be80039f0 msgr2=0x7f4be8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4c01030640 1 --2- 172.21.15.67:0/1116687851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4be80039f0 0x7f4be8003e20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc071dd0 tx=0x7f4bec01cab0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1116687851 reap_dead start 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4c01030640 1 -- 172.21.15.67:0/1116687851 shutdown_connections 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4c01030640 1 -- 172.21.15.67:0/1116687851 >> 172.21.15.67:0/1116687851 conn(0x7f4bfc070db0 msgr2=0x7f4bfc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4c01030640 1 -- 172.21.15.67:0/1116687851 shutdown_connections 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.479+0000 7f4c01030640 1 -- 172.21.15.67:0/1116687851 wait complete. 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 Processor -- start 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 -- start start 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be80039f0 0x7f4bfc1f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4bfc1f9490 0x7f4bfc1523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bfc14ad60 0x7f4bfc14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc160d90 con 0x7f4bfc14ad60 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc160c10 con 0x7f4bfc1f9490 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0734d0 con 0x7f4be80039f0 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bfc14ad60 0x7f4bfc14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bfc14ad60 0x7f4bfc14b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33162/0 (socket says 172.21.15.67:33162) 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1863285127 learned_addr learned my addr 172.21.15.67:0/1863285127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be80039f0 0x7f4bfc1f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4bfc1f9490 0x7f4bfc1523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be80039f0 msgr2=0x7f4bfc1f5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be80039f0 0x7f4bfc1f5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4bfc1f9490 msgr2=0x7f4bfc1523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4bfc1f9490 0x7f4bfc1523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc1f5f50 con 0x7f4bfc14ad60 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be80039f0 0x7f4bfc1f5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bfc14ad60 0x7f4bfc14b170 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4001340 tx=0x7f4bf4007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf400e9e0 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf4018070 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc1f5510 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf400edc0 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.483+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc15ea80 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4bfc06a480 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4bf401d460 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bf3fff640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4bdc06f400 0x7f4bdc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5825+0+0 (secure 0 0 0) 0x7f4bf408ff90 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v35) ==== 54+0+16 (secure 0 0 0) 0x7f4bf4090420 con 0x7f4bfc14ad60 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4bdc06f400 0x7f4bdc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4bfc1f6120 0x7f4bfc1ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc0c8c20 con 0x7f4bfc1f6120 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4bfc1f6120 0x7f4bfc1ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4bfc1f6120 0x7f4bfc1ec090 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4bdc06f400 0x7f4bdc071910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0ad000 tx=0x7f4bec0af040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.487+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc0c8c20 con 0x7f4bfc1f6120 2024-04-25T15:51:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.503+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc155bd0 con 0x7f4bfc1f6120 2024-04-25T15:51:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.507+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc155bd0 con 0x7f4bfc1f6120 2024-04-25T15:51:46.525 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.507+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4bfc0cdc90 0x7f4bfc0ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.507+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc0ce5e0 con 0x7f4bfc0cdc90 2024-04-25T15:51:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.507+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4bfc0cdc90 0x7f4bfc0ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.507+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4bfc0cdc90 0x7f4bfc0ce0a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.507+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc0ce5e0 con 0x7f4bfc0cdc90 2024-04-25T15:51:46.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.523+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc155eb0 con 0x7f4bfc0cdc90 2024-04-25T15:51:46.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.523+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc155eb0 con 0x7f4bfc0cdc90 2024-04-25T15:51:46.528 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:46.529 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:46.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.523+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4bfc0635a0 0x7f4bfc0639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.523+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc14b820 con 0x7f4bfc0635a0 2024-04-25T15:51:46.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.523+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4bfc0635a0 0x7f4bfc0639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.523+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4bfc0635a0 0x7f4bfc0639b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.527+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc14b820 con 0x7f4bfc0635a0 2024-04-25T15:51:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.539+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc152920 con 0x7f4bfc0635a0 2024-04-25T15:51:46.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.543+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc152920 con 0x7f4bfc0635a0 2024-04-25T15:51:46.548 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:46.548 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:46.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.543+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4bfc153970 0x7f4bfc02c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.543+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc152b30 con 0x7f4bfc153970 2024-04-25T15:51:46.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.543+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4bfc153970 0x7f4bfc02c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.543+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4bfc153970 0x7f4bfc02c9e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.543+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc152b30 con 0x7f4bfc153970 2024-04-25T15:51:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.559+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc152b30 con 0x7f4bfc153970 2024-04-25T15:51:46.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.559+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc152b30 con 0x7f4bfc153970 2024-04-25T15:51:46.565 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:46.565 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.559+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4bfc02d7f0 0x7f4bfc209e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.559+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc152b30 con 0x7f4bfc02d7f0 2024-04-25T15:51:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.559+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4bfc02d7f0 0x7f4bfc209e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.563+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4bfc02d7f0 0x7f4bfc209e00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.563+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc152b30 con 0x7f4bfc02d7f0 2024-04-25T15:51:46.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.579+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc02dc00 con 0x7f4bfc02d7f0 2024-04-25T15:51:46.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.583+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc02dc00 con 0x7f4bfc02d7f0 2024-04-25T15:51:46.587 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.583+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4bfc02dc00 0x7f4bfc20c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.583+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc20f7f0 con 0x7f4bfc02dc00 2024-04-25T15:51:46.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.583+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4bfc02dc00 0x7f4bfc20c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.583+0000 7f4bf259c640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4bfc02dc00 0x7f4bfc20c860 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.587+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc20f7f0 con 0x7f4bfc02dc00 2024-04-25T15:51:46.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.599+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc20e530 con 0x7f4bfc02dc00 2024-04-25T15:51:46.605 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:46.600+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.603+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fc7101f25a0 con 0x7fc6f8004810 2024-04-25T15:51:46.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.603+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc20e530 con 0x7f4bfc02dc00 2024-04-25T15:51:46.609 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.603+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4bfc20ec50 0x7f4bfc20f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.603+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc212c40 con 0x7f4bfc20ec50 2024-04-25T15:51:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.607+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4bfc20ec50 0x7f4bfc20f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.607+0000 7f4bfb59e640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4bfc20ec50 0x7f4bfc20f060 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.607+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc212c40 con 0x7f4bfc20ec50 2024-04-25T15:51:46.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.623+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc211ac0 con 0x7f4bfc20ec50 2024-04-25T15:51:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.627+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc211ac0 con 0x7f4bfc20ec50 2024-04-25T15:51:46.631 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.627+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4bfc2121e0 0x7f4bfc2125f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.627+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4bfc215fe0 con 0x7f4bfc2121e0 2024-04-25T15:51:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.627+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4bfc2121e0 0x7f4bfc2125f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:46.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.627+0000 7f4bfad9d640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4bfc2121e0 0x7f4bfc2125f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:46.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.631+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bfc215fe0 con 0x7f4bfc2121e0 2024-04-25T15:51:46.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.647+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4bfc20cdf0 con 0x7f4bfc2121e0 2024-04-25T15:51:46.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4bf3fff640 1 -- 172.21.15.67:0/1863285127 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4bfc20cdf0 con 0x7f4bfc2121e0 2024-04-25T15:51:46.655 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4bfc1f6120 msgr2=0x7f4bfc1ec090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4bfc1f6120 0x7f4bfc1ec090 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4bfc0cdc90 msgr2=0x7f4bfc0ce0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4bfc0cdc90 0x7f4bfc0ce0a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4bfc0635a0 msgr2=0x7f4bfc0639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4bfc0635a0 0x7f4bfc0639b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4bfc153970 msgr2=0x7f4bfc02c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4bfc153970 0x7f4bfc02c9e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1863285127 reap_dead start 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4bfc02d7f0 msgr2=0x7f4bfc209e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4bfc02d7f0 0x7f4bfc209e00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1863285127 reap_dead start 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4bfc02dc00 msgr2=0x7f4bfc20c860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4bfc02dc00 0x7f4bfc20c860 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4bfc20ec50 msgr2=0x7f4bfc20f060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4bfc20ec50 0x7f4bfc20f060 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4bfc2121e0 msgr2=0x7f4bfc2125f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4bfc2121e0 0x7f4bfc2125f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4bdc06f400 msgr2=0x7f4bdc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4bdc06f400 0x7f4bdc071910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0ad000 tx=0x7f4bec0af040 comp rx=0 tx=0).stop 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bfc14ad60 msgr2=0x7f4bfc14b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4bfb59e640 1 -- 172.21.15.67:0/1863285127 reap_dead start 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bfc14ad60 0x7f4bfc14b170 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4bf4001340 tx=0x7f4bf4007b00 comp rx=0 tx=0).stop 2024-04-25T15:51:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 shutdown_connections 2024-04-25T15:51:46.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 --2- 172.21.15.67:0/1863285127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bfc14ad60 0x7f4bfc14b170 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:46.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 >> 172.21.15.67:0/1863285127 conn(0x7f4bfc070db0 msgr2=0x7f4bfc0c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:46.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 shutdown_connections 2024-04-25T15:51:46.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:46.651+0000 7f4c01030640 1 -- 172.21.15.67:0/1863285127 wait complete. 2024-04-25T15:51:46.665 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:46.665 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:46.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:46.706 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:46.704+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:46.806 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:46.804+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:46.876 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:46.876 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:46.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:46.907 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:46.904+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:47.008 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:47.003+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:47.048 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:47.048 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:47.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:47.108 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:47.104+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:47.208 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:47.204+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.235+0000 7fc70d7fa640 1 -- 172.21.15.67:0/3929120981 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v36) ==== 91+0+0 (secure 0 0 0) 0x7fc70005e690 con 0x7fc6f8004810 2024-04-25T15:51:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:51:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc6fc06f430 msgr2=0x7fc6fc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc6fc06f430 0x7fc6fc071940 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc7040af000 tx=0x7fc704008040 comp rx=0 tx=0).stop 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 msgr2=0x7fc7101f2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc7101f2980 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc700010c50 tx=0x7fc70000e4d0 comp rx=0 tx=0).stop 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 shutdown_connections 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc6fc06f430 0x7fc6fc071940 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7100c7880 0x7fc71014e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc710152060 0x7fc7101538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 --2- 172.21.15.67:0/3929120981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6f8004810 0x7fc7101f2980 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 >> 172.21.15.67:0/3929120981 conn(0x7fc7100700b0 msgr2=0x7fc7100704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.239+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 shutdown_connections 2024-04-25T15:51:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:47.243+0000 7fc716ae1640 1 -- 172.21.15.67:0/3929120981 wait complete. 2024-04-25T15:51:47.275 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:47.275 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:47.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:47.346 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:47.343+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:47.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:47.407+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:47.489 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:47.490 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:47.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:47.511 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:47.508+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:47.612 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:47.612+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:47.704 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:47.704 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:47.713 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:47.707+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:47.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:47.814 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:47.811+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:47.911 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:47.912 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:47.915 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:47.911+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:47.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:48.016 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:48.012+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:48.117 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:48.111+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:48.121 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:48.121 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:48.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:48.218 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:48.212+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:48.319 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:48.315+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:48.337 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:48.337 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:48.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:48.446 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:48.443+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:48.521 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:48.515+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:48.554 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:48.554 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:48.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:48.621 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:48.620+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:48.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:48.722 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:48.720+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:48.762 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:48.763 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:48.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:48.822 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:48.816+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:48.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.823+0000 7f8c5e9be640 1 Processor -- start 2024-04-25T15:51:48.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.823+0000 7f8c5e9be640 1 -- start start 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c5e9be640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c58156f10 0x7f8c580cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c5e9be640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8c5815d210 0x7f8c5815d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c5e9be640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c58160fa0 con 0x7f8c580cc600 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c5e9be640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8c58157320 con 0x7f8c58156f10 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c5e9be640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8c580c6d10 con 0x7f8c5815d210 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c57fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c58156f10 0x7f8c580cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c5cf5b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c580cc600 0x7f8c58073c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40276/0 (socket says 172.21.15.67:40276) 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c57fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c58156f10 0x7f8c580cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33174/0 (socket says 172.21.15.67:33174) 2024-04-25T15:51:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c5cf5b640 1 -- 172.21.15.67:0/127172176 learned_addr learned my addr 172.21.15.67:0/127172176 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c577fe640 1 --2- 172.21.15.67:0/127172176 >> v2:172.21.15.176:3300/0 conn(0x7f8c5815d210 0x7f8c5815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c57fff640 1 -- 172.21.15.67:0/127172176 >> v2:172.21.15.176:3300/0 conn(0x7f8c5815d210 msgr2=0x7f8c5815d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c57fff640 1 --2- 172.21.15.67:0/127172176 >> v2:172.21.15.176:3300/0 conn(0x7f8c5815d210 0x7f8c5815d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c57fff640 1 -- 172.21.15.67:0/127172176 >> v1:172.21.15.67:6789/0 conn(0x7f8c580cc600 legacy=0x7f8c58073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c57fff640 1 -- 172.21.15.67:0/127172176 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c58160420 con 0x7f8c58156f10 2024-04-25T15:51:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.827+0000 7f8c57fff640 1 --2- 172.21.15.67:0/127172176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c580cbfc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8c580cd1c0 tx=0x7f8c4801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d50d5fa36df24c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c56ffd640 1 -- 172.21.15.67:0/127172176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c480b0070 con 0x7f8c58156f10 2024-04-25T15:51:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c56ffd640 1 -- 172.21.15.67:0/127172176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c480ab600 con 0x7f8c58156f10 2024-04-25T15:51:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c56ffd640 1 -- 172.21.15.67:0/127172176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c480ab9d0 con 0x7f8c58156f10 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 -- 172.21.15.67:0/127172176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 msgr2=0x7f8c580cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/127172176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c580cbfc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8c580cd1c0 tx=0x7f8c4801c480 comp rx=0 tx=0).stop 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 -- 172.21.15.67:0/127172176 shutdown_connections 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/127172176 >> v2:172.21.15.176:3300/0 conn(0x7f8c5815d210 0x7f8c5815d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/127172176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c580cbfc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 -- 172.21.15.67:0/127172176 >> 172.21.15.67:0/127172176 conn(0x7f8c58068c70 msgr2=0x7f8c58071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 -- 172.21.15.67:0/127172176 shutdown_connections 2024-04-25T15:51:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 -- 172.21.15.67:0/127172176 wait complete. 2024-04-25T15:51:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 Processor -- start 2024-04-25T15:51:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.831+0000 7f8c5e9be640 1 -- start start 2024-04-25T15:51:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c580cc600 0x7f8c581f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c5815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c5815d210 0x7f8c58151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c580cce20 con 0x7f8c58156f10 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5815ccf0 con 0x7f8c5815d210 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c581600c0 con 0x7f8c580cc600 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c5815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c5815e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33184/0 (socket says 172.21.15.67:33184) 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 -- 172.21.15.67:0/2620521850 learned_addr learned my addr 172.21.15.67:0/2620521850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c580cc600 0x7f8c581f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5cf5b640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c5815d210 0x7f8c58151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c580cc600 msgr2=0x7f8c581f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c580cc600 0x7f8c581f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c5815d210 msgr2=0x7f8c58151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c5815d210 0x7f8c58151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5815ec00 con 0x7f8c58156f10 2024-04-25T15:51:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c577fe640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c5815e6c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00d620 tx=0x7f8c4c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c00b620 con 0x7f8c58156f10 2024-04-25T15:51:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c4c019070 con 0x7f8c58156f10 2024-04-25T15:51:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c58151dc0 con 0x7f8c58156f10 2024-04-25T15:51:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c00b9e0 con 0x7f8c58156f10 2024-04-25T15:51:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c5815dec0 con 0x7f8c58156f10 2024-04-25T15:51:48.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8c581522e0 con 0x7f8c58156f10 2024-04-25T15:51:48.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8c4c00bb80 con 0x7f8c58156f10 2024-04-25T15:51:48.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c54ff9640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c4406f400 0x7f8c44071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5825+0+0 (secure 0 0 0) 0x7f8c4c08f730 con 0x7f8c58156f10 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v36) ==== 54+0+16 (secure 0 0 0) 0x7f8c4c095170 con 0x7f8c58156f10 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.835+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c4406f400 0x7f8c44071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.839+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c581466f0 0x7f8c58146b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.839+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c5815dcb0 con 0x7f8c581466f0 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.839+0000 7f8c5cf5b640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c581466f0 0x7f8c58146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.839+0000 7f8c5cf5b640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c581466f0 0x7f8c58146b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.839+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c4406f400 0x7f8c44071910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c480ad000 tx=0x7f8c48008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.839+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c5815dcb0 con 0x7f8c581466f0 2024-04-25T15:51:48.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.855+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c5806a480 con 0x7f8c581466f0 2024-04-25T15:51:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.855+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c5806a480 con 0x7f8c581466f0 2024-04-25T15:51:48.861 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:48.861 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:48.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.855+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c581fa200 0x7f8c581fa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.855+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8c581fab50 con 0x7f8c581fa200 2024-04-25T15:51:48.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.855+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c581fa200 0x7f8c581fa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.855+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c581fa200 0x7f8c581fa610 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.859+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c581fab50 con 0x7f8c581fa200 2024-04-25T15:51:48.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.871+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c580cb260 con 0x7f8c581fa200 2024-04-25T15:51:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.875+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c580cb260 con 0x7f8c581fa200 2024-04-25T15:51:48.880 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.875+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c580635a0 0x7f8c580639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.875+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8c5814a010 con 0x7f8c580635a0 2024-04-25T15:51:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.875+0000 7f8c577fe640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c580635a0 0x7f8c580639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.875+0000 7f8c577fe640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c580635a0 0x7f8c580639b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.879+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c5814a010 con 0x7f8c580635a0 2024-04-25T15:51:48.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.891+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c5814a010 con 0x7f8c580635a0 2024-04-25T15:51:48.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.895+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c5814a010 con 0x7f8c580635a0 2024-04-25T15:51:48.899 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:48.899 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:48.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.895+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c5802cef0 0x7f8c5802d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.895+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8c5806ee60 con 0x7f8c5802cef0 2024-04-25T15:51:48.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.895+0000 7f8c5cf5b640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c5802cef0 0x7f8c5802d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.895+0000 7f8c5cf5b640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c5802cef0 0x7f8c5802d300 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.895+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c5806ee60 con 0x7f8c5802cef0 2024-04-25T15:51:48.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.911+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c58149d30 con 0x7f8c5802cef0 2024-04-25T15:51:48.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.911+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c58149d30 con 0x7f8c5802cef0 2024-04-25T15:51:48.916 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:48.916 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:48.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.911+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c5806d750 0x7f8c5806db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:48.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.911+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8c58149a50 con 0x7f8c5806d750 2024-04-25T15:51:48.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.911+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c5806d750 0x7f8c5806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:48.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.911+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c5806d750 0x7f8c5806db60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:48.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.915+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c58149a50 con 0x7f8c5806d750 2024-04-25T15:51:48.923 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:48.919+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:48.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.927+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c58149a50 con 0x7f8c5806d750 2024-04-25T15:51:48.937 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:48.937 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:48.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:48.937 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:48.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.937 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:48.937 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.722735241999999, 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.722695596999998, 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:48.941 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:48.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.944 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.719788574999995, 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:48.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.719635639000003, 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.948 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.718119994999995, 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.717977348000005, 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.717874257000005, 2024-04-25T15:51:48.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.717721506999993, 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.964 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:48.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.715773280999997, 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:48.971 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.687745649999997, 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:48.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:48.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:48.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.685722759000001, 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:48.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.685552209999997, 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:48.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:48.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.683196714000005, 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:48.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.683162002000003, 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:48.993 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:48.993 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.683141192999997, 2024-04-25T15:51:48.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:48.993 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:48.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.606254145999998, 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:48.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:49.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:49.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.606230064000002, 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:49.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:49.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:49.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.606220366999999, 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:49.007 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:49.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:49.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:49.010 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 56.600325429999998, 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:49.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:49.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:49.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:49.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.010886773000003, 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:49.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:49.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:49.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:49.019 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:49.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.931+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c58149a50 con 0x7f8c5806d750 2024-04-25T15:51:49.020 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.931+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c5806e5b0 0x7f8c5806e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.931+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8c58151ae0 con 0x7f8c5806e5b0 2024-04-25T15:51:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.935+0000 7f8c577fe640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c5806e5b0 0x7f8c5806e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.935+0000 7f8c577fe640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c5806e5b0 0x7f8c5806e9c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.935+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c58151ae0 con 0x7f8c5806e5b0 2024-04-25T15:51:49.022 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:49.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.951+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c58151ae0 con 0x7f8c5806e5b0 2024-04-25T15:51:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.955+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c58151ae0 con 0x7f8c5806e5b0 2024-04-25T15:51:49.023 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:49.024 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:49.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.955+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c5820d5e0 0x7f8c5820fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.955+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8c5802de20 con 0x7f8c5820d5e0 2024-04-25T15:51:49.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.955+0000 7f8c5cf5b640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c5820d5e0 0x7f8c5820fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.955+0000 7f8c5cf5b640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c5820d5e0 0x7f8c5820fa00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:49.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.959+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c5802de20 con 0x7f8c5820d5e0 2024-04-25T15:51:49.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.971+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c580008e0 con 0x7f8c5820d5e0 2024-04-25T15:51:49.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.975+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c580008e0 con 0x7f8c5820d5e0 2024-04-25T15:51:49.025 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:49.026 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:49.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.975+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c58210450 0x7f8c58212870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.975+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8c582157a0 con 0x7f8c58210450 2024-04-25T15:51:49.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.975+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c58210450 0x7f8c58212870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.979+0000 7f8c57fff640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c58210450 0x7f8c58212870 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:49.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.979+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c582157a0 con 0x7f8c58210450 2024-04-25T15:51:49.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.991+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c58000960 con 0x7f8c58210450 2024-04-25T15:51:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c54ff9640 1 -- 172.21.15.67:0/2620521850 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c58000960 con 0x7f8c58210450 2024-04-25T15:51:49.027 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:49.028 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c581466f0 msgr2=0x7f8c58146b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c581466f0 0x7f8c58146b00 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c581fa200 msgr2=0x7f8c581fa610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c581fa200 0x7f8c581fa610 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c580635a0 msgr2=0x7f8c580639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c580635a0 0x7f8c580639b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c5802cef0 msgr2=0x7f8c5802d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c5802cef0 0x7f8c5802d300 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5cf5b640 1 -- 172.21.15.67:0/2620521850 reap_dead start 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c5806d750 msgr2=0x7f8c5806db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c5806d750 0x7f8c5806db60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5cf5b640 1 -- 172.21.15.67:0/2620521850 reap_dead start 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c5806e5b0 msgr2=0x7f8c5806e9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c5806e5b0 0x7f8c5806e9c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c5820d5e0 msgr2=0x7f8c5820fa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c5820d5e0 0x7f8c5820fa00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c58210450 msgr2=0x7f8c58212870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c58210450 0x7f8c58212870 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c4406f400 msgr2=0x7f8c44071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c4406f400 0x7f8c44071910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8c480ad000 tx=0x7f8c48008040 comp rx=0 tx=0).stop 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 msgr2=0x7f8c5815e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 --2- 172.21.15.67:0/2620521850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c58156f10 0x7f8c5815e6c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00d620 tx=0x7f8c4c00dae0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5cf5b640 1 -- 172.21.15.67:0/2620521850 reap_dead start 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 shutdown_connections 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 >> 172.21.15.67:0/2620521850 conn(0x7f8c58068c70 msgr2=0x7f8c580ca260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 shutdown_connections 2024-04-25T15:51:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:48.999+0000 7f8c5e9be640 1 -- 172.21.15.67:0/2620521850 wait complete. 2024-04-25T15:51:49.070 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:49.067+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:49.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:49.123+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:49.182 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:49.183 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:49.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:49.247 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:49.244+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:49.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:51:49.330 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:49.327+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:49.348 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:49.349 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:49.349 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:49.349 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:49.349 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:49.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:49.428 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:49.423+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 Processor -- start 2024-04-25T15:51:49.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 -- start start 2024-04-25T15:51:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9241609d0 0x7fc9240cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc9240ca5b0 0x7fc92415c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9240ca9c0 0x7fc924071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc924160de0 con 0x7fc9240ca9c0 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc92415d1a0 con 0x7fc9240ca5b0 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc929664640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc92415d480 con 0x7fc9241609d0 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9240ca9c0 0x7fc924071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9240ca9c0 0x7fc924071b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33190/0 (socket says 172.21.15.67:33190) 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 -- 172.21.15.67:0/3251207239 learned_addr learned my addr 172.21.15.67:0/3251207239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9227fc640 1 --2- 172.21.15.67:0/3251207239 >> v2:172.21.15.176:3300/0 conn(0x7fc9240ca5b0 0x7fc92415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc922ffd640 1 --2- 172.21.15.67:0/3251207239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9241609d0 0x7fc9240cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 -- 172.21.15.67:0/3251207239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9241609d0 msgr2=0x7fc9240cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 --2- 172.21.15.67:0/3251207239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9241609d0 0x7fc9240cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 -- 172.21.15.67:0/3251207239 >> v2:172.21.15.176:3300/0 conn(0x7fc9240ca5b0 msgr2=0x7fc92415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 --2- 172.21.15.67:0/3251207239 >> v2:172.21.15.176:3300/0 conn(0x7fc9240ca5b0 0x7fc92415c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.443+0000 7fc9237fe640 1 -- 172.21.15.67:0/3251207239 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92415fef0 con 0x7fc9240ca9c0 2024-04-25T15:51:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc9237fe640 1 --2- 172.21.15.67:0/3251207239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9240ca9c0 0x7fc924071b50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc9240d0ba0 tx=0x7fc91c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aad9e2b6e2254de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc921ffb640 1 -- 172.21.15.67:0/3251207239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c0a5670 con 0x7fc9240ca9c0 2024-04-25T15:51:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc921ffb640 1 -- 172.21.15.67:0/3251207239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c0a5810 con 0x7fc9240ca9c0 2024-04-25T15:51:49.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc921ffb640 1 -- 172.21.15.67:0/3251207239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c0a5be0 con 0x7fc9240ca9c0 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 -- 172.21.15.67:0/3251207239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9240ca9c0 msgr2=0x7fc924071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 --2- 172.21.15.67:0/3251207239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9240ca9c0 0x7fc924071b50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc9240d0ba0 tx=0x7fc91c01c9f0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 -- 172.21.15.67:0/3251207239 shutdown_connections 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 --2- 172.21.15.67:0/3251207239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9240ca9c0 0x7fc924071b50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 --2- 172.21.15.67:0/3251207239 >> v2:172.21.15.176:3300/0 conn(0x7fc9240ca5b0 0x7fc92415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 --2- 172.21.15.67:0/3251207239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9241609d0 0x7fc9240cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 -- 172.21.15.67:0/3251207239 >> 172.21.15.67:0/3251207239 conn(0x7fc9240700b0 msgr2=0x7fc9240704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 -- 172.21.15.67:0/3251207239 shutdown_connections 2024-04-25T15:51:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.447+0000 7fc929664640 1 -- 172.21.15.67:0/3251207239 wait complete. 2024-04-25T15:51:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 Processor -- start 2024-04-25T15:51:49.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 -- start start 2024-04-25T15:51:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9240ca5b0 0x7fc92414f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9240ca9c0 0x7fc92414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9241609d0 0x7fc924150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc92415cf40 con 0x7fc9241609d0 2024-04-25T15:51:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc92415cdc0 con 0x7fc9240ca9c0 2024-04-25T15:51:49.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9240d02e0 con 0x7fc9240ca5b0 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9241609d0 0x7fc924150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9241609d0 0x7fc924150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33204/0 (socket says 172.21.15.67:33204) 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 -- 172.21.15.67:0/1699445936 learned_addr learned my addr 172.21.15.67:0/1699445936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc922ffd640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9240ca5b0 0x7fc92414f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9227fc640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9240ca9c0 0x7fc92414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 -- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9240ca5b0 msgr2=0x7fc92414f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9240ca5b0 0x7fc92414f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 -- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9240ca9c0 msgr2=0x7fc92414cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9240ca9c0 0x7fc92414cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 -- 172.21.15.67:0/1699445936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92414fb60 con 0x7fc9241609d0 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc9237fe640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9241609d0 0x7fc924150370 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0af040 tx=0x7fc91c019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc91b7fe640 1 -- 172.21.15.67:0/1699445936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c0a5670 con 0x7fc9241609d0 2024-04-25T15:51:49.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc91b7fe640 1 -- 172.21.15.67:0/1699445936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c0be070 con 0x7fc9241609d0 2024-04-25T15:51:49.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9241f7070 con 0x7fc9241609d0 2024-04-25T15:51:49.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc91b7fe640 1 -- 172.21.15.67:0/1699445936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c009510 con 0x7fc9241609d0 2024-04-25T15:51:49.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92414e420 con 0x7fc9241609d0 2024-04-25T15:51:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.451+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9241f6dc0 con 0x7fc9241609d0 2024-04-25T15:51:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.455+0000 7fc91b7fe640 1 -- 172.21.15.67:0/1699445936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc91c0b88d0 con 0x7fc9241609d0 2024-04-25T15:51:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.455+0000 7fc91b7fe640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc90406f400 0x7fc904071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.455+0000 7fc91b7fe640 1 -- 172.21.15.67:0/1699445936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5825+0+0 (secure 0 0 0) 0x7fc91c12a0a0 con 0x7fc9241609d0 2024-04-25T15:51:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.455+0000 7fc922ffd640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc90406f400 0x7fc904071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.455+0000 7fc922ffd640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc90406f400 0x7fc904071910 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc90c009ac0 tx=0x7fc90c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:49.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.459+0000 7fc91b7fe640 1 -- 172.21.15.67:0/1699445936 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc91c0165e0 con 0x7fc9241609d0 2024-04-25T15:51:49.528 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:49.524+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.350179639999993, 2024-04-25T15:51:49.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:49.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:49.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:49.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:49.569 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:49.569 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.347564843000001, 2024-04-25T15:51:49.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:49.569 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.569 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.569 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:49.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.347065946000001, 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.573 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:49.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.345976699999994, 2024-04-25T15:51:49.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.580 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.345356205000002, 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:49.584 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.342961978999995, 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.313841576000002, 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.588 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:49.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:49.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.309096842000002, 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:49.592 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:49.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:49.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.595 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.305569340999995, 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:49.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.30543686, 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:49.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:49.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:49.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:49.603 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.684497401999998, 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:49.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:49.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.673062270000003, 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.608 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:49.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.672664834000003, 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.615 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.672555093, 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:49.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.672054348000003, 2024-04-25T15:51:49.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:49.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:49.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.671678362000002, 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.623 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 45.633080178999997, 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:49.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 45.632979919999997, 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:49.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:49.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.634 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 45.630875005, 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:49.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:49.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 45.630499428999997, 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:49.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:49.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:49.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:49.641 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:49.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:49.631+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fc92406a480 con 0x7fc9241609d0 2024-04-25T15:51:49.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:49.643 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:49.639+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:49.744 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:49.740+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:49.840 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:49.840 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:49.844 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:49.840+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:49.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:49.945 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:49.939+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:50.046 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:50.040+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:50.047 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:50.047 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:50.047 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:50.047 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:50.047 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:50.065 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:50.147 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:50.147+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:50.248 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:50.244+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015738002999996, 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.266 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:50.271 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015707441999993, 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:50.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015699611000002, 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.281 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015690681999999, 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:50.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015684449999995, 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.287 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:50.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:50.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015669040000006, 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.292 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:50.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:50.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.297 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015656207999996, 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:50.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:50.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:50.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.015650993999998, 2024-04-25T15:51:50.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:50.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:50.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.930040877000003, 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.305 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:50.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:50.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:50.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.377510884000003, 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.310 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:50.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.376118046000002, 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.315 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:50.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.338097920000003, 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.319 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:50.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:50.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.338074593999998, 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:50.324 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:50.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.338034053999998, 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.338025393999999, 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.334 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:50.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:50.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.337983721000001, 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:50.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.342 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.337974463999998, 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:50.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.333837037999999, 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.346 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:50.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:50.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.333782460999998, 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.353 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 46.333749253000001, 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:50.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.357 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:50.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.287+0000 7fc91b7fe640 1 -- 172.21.15.67:0/1699445936 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v37) ==== 101+0+0 (secure 0 0 0) 0x7fc91c0f6960 con 0x7fc9241609d0 2024-04-25T15:51:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:51:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc90406f400 msgr2=0x7fc904071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc90406f400 0x7fc904071910 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc90c009ac0 tx=0x7fc90c009190 comp rx=0 tx=0).stop 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9241609d0 msgr2=0x7fc924150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9241609d0 0x7fc924150370 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0af040 tx=0x7fc91c019790 comp rx=0 tx=0).stop 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 shutdown_connections 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc90406f400 0x7fc904071910 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9241609d0 0x7fc924150370 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9240ca9c0 0x7fc92414cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 --2- 172.21.15.67:0/1699445936 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9240ca5b0 0x7fc92414f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 >> 172.21.15.67:0/1699445936 conn(0x7fc9240700b0 msgr2=0x7fc9240ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 shutdown_connections 2024-04-25T15:51:50.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:50.291+0000 7fc929664640 1 -- 172.21.15.67:0/1699445936 wait complete. 2024-04-25T15:51:50.361 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:50.355+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:50.459 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:50.456+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:50.544 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:50.544 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:50.544 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:50.545 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:50.560 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:50.555+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:50.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:50.661 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:50.656+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:50.743 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:50.743 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:50.743 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:50.743 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:50.743 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:50.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:50.803 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:50.800+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:50.863 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:50.859+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:50.953 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:50.953 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:50.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:50.953 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:50.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.739788165999997, 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.739758902000005, 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.739749614000004, 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.964 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:51:50.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.739742038000003, 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:51:50.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:51:50.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:51:50.974 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.739733404999996, 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:51:50.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:51:50.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:51:50.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.739683572000004, 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.980 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:51:50.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:51:50.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.739675693999999, 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.703247687000001, 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:51:50.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:51:50.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.703217022999993, 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.994 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:51:50.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:51:50.996 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:51:50.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.703204493000001, 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:50.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:51:51.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:51:51.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:51:51.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.703193861000003, 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.004 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:51.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:51.009 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.703174316000002, 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:51.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:51.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.014 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.703165424000005, 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:51:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:51:51.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:51:51.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:51:51.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.699602807999995, 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:51:51.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:51:51.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 68.699482196000005, 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:51.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:51:51.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:51:51.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.027 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 53.064484176000001, 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:51:51.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:51:51.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:51:51.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.023914777999998, 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:51:51.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:51:51.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.023909269999997, 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.038 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:51:51.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:51.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:51:51.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.021494756000003, 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.043 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.020532869999997, 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:51.047 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:51:51.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:51.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:51.051 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:51.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:51.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:51.053 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:51.048+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:51.153 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:51.148+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5262dc4640 1 Processor -- start 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5262dc4640 1 -- start start 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5262dc4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f525c06ca60 0x7f525c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5262dc4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f525c15fe00 0x7f525c160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5262dc4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525c06d780 con 0x7f525c0b5660 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5262dc4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f525c15d510 con 0x7f525c15fe00 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5262dc4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f525c160be0 con 0x7f525c06ca60 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5261361640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f525c15fe00 0x7f525c160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5261361640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f525c15fe00 0x7f525c160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33212/0 (socket says 172.21.15.67:33212) 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5261361640 1 -- 172.21.15.67:0/133176807 learned_addr learned my addr 172.21.15.67:0/133176807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5260b60640 1 --2- 172.21.15.67:0/133176807 >> v2:172.21.15.176:3300/0 conn(0x7f525c06ca60 0x7f525c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5261361640 1 -- 172.21.15.67:0/133176807 >> v2:172.21.15.176:3300/0 conn(0x7f525c06ca60 msgr2=0x7f525c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5261361640 1 --2- 172.21.15.67:0/133176807 >> v2:172.21.15.176:3300/0 conn(0x7f525c06ca60 0x7f525c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5261361640 1 -- 172.21.15.67:0/133176807 >> v1:172.21.15.176:6789/0 conn(0x7f525c0b5660 legacy=0x7f525c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.211+0000 7f5261361640 1 -- 172.21.15.67:0/133176807 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c15cbe0 con 0x7f525c15fe00 2024-04-25T15:51:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5261361640 1 --2- 172.21.15.67:0/133176807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c160210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5254009e50 tx=0x7f525401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e28480e94b15ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f525b7fe640 1 -- 172.21.15.67:0/133176807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540a57a0 con 0x7f525c15fe00 2024-04-25T15:51:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f525b7fe640 1 -- 172.21.15.67:0/133176807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52540a5940 con 0x7f525c15fe00 2024-04-25T15:51:51.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f525b7fe640 1 -- 172.21.15.67:0/133176807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540a5d10 con 0x7f525c15fe00 2024-04-25T15:51:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 -- 172.21.15.67:0/133176807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 msgr2=0x7f525c160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 --2- 172.21.15.67:0/133176807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c160210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5254009e50 tx=0x7f525401c9f0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 -- 172.21.15.67:0/133176807 shutdown_connections 2024-04-25T15:51:51.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 --2- 172.21.15.67:0/133176807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c160210 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 --2- 172.21.15.67:0/133176807 >> v2:172.21.15.176:3300/0 conn(0x7f525c06ca60 0x7f525c06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 -- 172.21.15.67:0/133176807 >> 172.21.15.67:0/133176807 conn(0x7f525c06e420 msgr2=0x7f525c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 -- 172.21.15.67:0/133176807 shutdown_connections 2024-04-25T15:51:51.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.215+0000 7f5262dc4640 1 -- 172.21.15.67:0/133176807 wait complete. 2024-04-25T15:51:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 Processor -- start 2024-04-25T15:51:51.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 -- start start 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f525c06ca60 0x7f525c1f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f525c0b5660 0x7f525c0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c1531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0b7010 con 0x7f525c15fe00 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0b6e90 con 0x7f525c0b5660 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f525c160750 con 0x7f525c06ca60 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c1531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c1531e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33216/0 (socket says 172.21.15.67:33216) 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 -- 172.21.15.67:0/1349712434 learned_addr learned my addr 172.21.15.67:0/1349712434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:51.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f525c06ca60 0x7f525c1f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f525c0b5660 0x7f525c0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f525c06ca60 msgr2=0x7f525c1f5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f525c06ca60 0x7f525c1f5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f525c0b5660 msgr2=0x7f525c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f525c0b5660 0x7f525c0cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c1f63e0 con 0x7f525c15fe00 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f525c0b5660 0x7f525c0cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5261361640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c1531e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f52540af040 tx=0x7f5254019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540bc020 con 0x7f525c15fe00 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52540b8670 con 0x7f525c15fe00 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c159120 con 0x7f525c15fe00 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540b8a20 con 0x7f525c15fe00 2024-04-25T15:51:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c1596a0 con 0x7f525c15fe00 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.219+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f525c06a480 con 0x7f525c15fe00 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f52540bf020 con 0x7f525c15fe00 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f52597fa640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f524806f400 0x7f5248071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5825+0+0 (secure 0 0 0) 0x7f525412a660 con 0x7f525c15fe00 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v37) ==== 54+0+16 (secure 0 0 0) 0x7f52540c6030 con 0x7f525c15fe00 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f524806f400 0x7f5248071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f525c1598e0 0x7f525c0ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f525c06d010 con 0x7f525c1598e0 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f525c1598e0 0x7f525c0ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f525c1598e0 0x7f525c0ce5b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f524806f400 0x7f5248071910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f525c0cd350 tx=0x7f5250008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.223+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c06d010 con 0x7f525c1598e0 2024-04-25T15:51:51.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.239+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c0008e0 con 0x7f525c1598e0 2024-04-25T15:51:51.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.239+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c0008e0 con 0x7f525c1598e0 2024-04-25T15:51:51.245 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.239+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f525c1f9980 0x7f525c15f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.239+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f525c1f65f0 con 0x7f525c1f9980 2024-04-25T15:51:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.243+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f525c1f9980 0x7f525c15f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.243+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f525c1f9980 0x7f525c15f580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.243+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c1f65f0 con 0x7f525c1f9980 2024-04-25T15:51:51.253 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:51.253 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:51.253 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:51.253 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:51.253 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:51.247+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:51.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.255+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c0cbd00 con 0x7f525c1f9980 2024-04-25T15:51:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.259+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c0cbd00 con 0x7f525c1f9980 2024-04-25T15:51:51.263 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.259+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f525c157880 0x7f525c157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.259+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f525c15a580 con 0x7f525c157880 2024-04-25T15:51:51.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.259+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f525c157880 0x7f525c157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.259+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f525c157880 0x7f525c157c90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.259+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c15a580 con 0x7f525c157880 2024-04-25T15:51:51.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:51.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:51.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:51.277 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T15:51:51.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:51:51.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.271+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c15a9e0 con 0x7f525c157880 2024-04-25T15:51:51.280 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.275+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c15a9e0 con 0x7f525c157880 2024-04-25T15:51:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.275+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f525c063090 0x7f525c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.275+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f525c15b4c0 con 0x7f525c063090 2024-04-25T15:51:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.275+0000 7f5261361640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f525c063090 0x7f525c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.275+0000 7f5261361640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f525c063090 0x7f525c0634a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.275+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c15b4c0 con 0x7f525c063090 2024-04-25T15:51:51.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.291+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c063cb0 con 0x7f525c063090 2024-04-25T15:51:51.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.291+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c063cb0 con 0x7f525c063090 2024-04-25T15:51:51.298 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.295+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f525c02cef0 0x7f525c02d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.295+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f525c15b4c0 con 0x7f525c02cef0 2024-04-25T15:51:51.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.295+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f525c02cef0 0x7f525c02d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.295+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f525c02cef0 0x7f525c02d300 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.295+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c15b4c0 con 0x7f525c02cef0 2024-04-25T15:51:51.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.311+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c15b4c0 con 0x7f525c02cef0 2024-04-25T15:51:51.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.311+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c15b4c0 con 0x7f525c02cef0 2024-04-25T15:51:51.317 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:51.317 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:51.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.311+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f525c20a340 0x7f525c20c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.311+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f525c02de90 con 0x7f525c20a340 2024-04-25T15:51:51.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.311+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f525c20a340 0x7f525c20c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.315+0000 7f525bfff640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f525c20a340 0x7f525c20c760 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.315+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c02de90 con 0x7f525c20a340 2024-04-25T15:51:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.327+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c02de60 con 0x7f525c20a340 2024-04-25T15:51:51.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.331+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c02de60 con 0x7f525c20a340 2024-04-25T15:51:51.335 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.331+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f525c20d1b0 0x7f525c20f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.331+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f525c2123a0 con 0x7f525c20d1b0 2024-04-25T15:51:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.331+0000 7f5261361640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f525c20d1b0 0x7f525c20f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.331+0000 7f5261361640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f525c20d1b0 0x7f525c20f5d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.331+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c2123a0 con 0x7f525c20d1b0 2024-04-25T15:51:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.347+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c20fb10 con 0x7f525c20d1b0 2024-04-25T15:51:51.354 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:51.351+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.351+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c20fb10 con 0x7f525c20d1b0 2024-04-25T15:51:51.354 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.351+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f525c210020 0x7f525c212510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.351+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f525c215560 con 0x7f525c210020 2024-04-25T15:51:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.351+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f525c210020 0x7f525c212510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.351+0000 7f5260b60640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f525c210020 0x7f525c212510 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.351+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f525c215560 con 0x7f525c210020 2024-04-25T15:51:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.367+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f525c215530 con 0x7f525c210020 2024-04-25T15:51:51.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f52597fa640 1 -- 172.21.15.67:0/1349712434 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f525c215530 con 0x7f525c210020 2024-04-25T15:51:51.376 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:51.377 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:51.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f525c1598e0 msgr2=0x7f525c0ce5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f525c1598e0 0x7f525c0ce5b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f525c1f9980 msgr2=0x7f525c15f580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f525c1f9980 0x7f525c15f580 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f525c157880 msgr2=0x7f525c157c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f525c157880 0x7f525c157c90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f525c063090 msgr2=0x7f525c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f525c063090 0x7f525c0634a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5261361640 1 -- 172.21.15.67:0/1349712434 reap_dead start 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f525c02cef0 msgr2=0x7f525c02d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f525c02cef0 0x7f525c02d300 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5261361640 1 -- 172.21.15.67:0/1349712434 reap_dead start 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f525c20a340 msgr2=0x7f525c20c760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f525c20a340 0x7f525c20c760 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f525c20d1b0 msgr2=0x7f525c20f5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f525c20d1b0 0x7f525c20f5d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f525c210020 msgr2=0x7f525c212510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f525c210020 0x7f525c212510 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f524806f400 msgr2=0x7f5248071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f524806f400 0x7f5248071910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f525c0cd350 tx=0x7f5250008040 comp rx=0 tx=0).stop 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 msgr2=0x7f525c1531e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 --2- 172.21.15.67:0/1349712434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f525c15fe00 0x7f525c1531e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f52540af040 tx=0x7f5254019790 comp rx=0 tx=0).stop 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5261361640 1 -- 172.21.15.67:0/1349712434 reap_dead start 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 shutdown_connections 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 >> 172.21.15.67:0/1349712434 conn(0x7f525c06e420 msgr2=0x7f525c0c6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 shutdown_connections 2024-04-25T15:51:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.371+0000 7f5262dc4640 1 -- 172.21.15.67:0/1349712434 wait complete. 2024-04-25T15:51:51.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8bf0da640 1 Processor -- start 2024-04-25T15:51:51.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8bf0da640 1 -- start start 2024-04-25T15:51:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8bf0da640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 0x7ff8b8068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8bf0da640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8b8160990 con 0x7ff8b81569a0 2024-04-25T15:51:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8bf0da640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8b8160c70 con 0x7ff8b8156db0 2024-04-25T15:51:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8bf0da640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8b815e110 con 0x7ff8b815dd00 2024-04-25T15:51:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8b7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8b81569a0 0x7ff8b806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40288/0 (socket says 172.21.15.67:40288) 2024-04-25T15:51:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8b7fff640 1 -- 172.21.15.67:0/2984482339 learned_addr learned my addr 172.21.15.67:0/2984482339 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:51.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8bce76640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 0x7ff8b8068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702125768 0 0) 0x7ff8b8160990 con 0x7ff8b81569a0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.419+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8a8003530 con 0x7ff8b81569a0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3111731839 0 0) 0x7ff8b8160c70 con 0x7ff8b8156db0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8b8160990 con 0x7ff8b8156db0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 394867362 0 0) 0x7ff8a8003530 con 0x7ff8b81569a0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 msgr2=0x7ff8b8068830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 0x7ff8b8068830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 >> v1:172.21.15.176:6789/0 conn(0x7ff8b8156db0 legacy=0x7ff8b8167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b80c3bf0 con 0x7ff8b81569a0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7ff8a4001800 con 0x7ff8b81569a0 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 >> v1:172.21.15.67:6789/0 conn(0x7ff8b81569a0 legacy=0x7ff8b806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a8003b80 0x7ff8a8003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 0x7ff8a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8004d50 0x7ff8a8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b815e110 con 0x7ff8a8003b80 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a800bad0 con 0x7ff8a8004d50 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a800bcd0 con 0x7ff8b815dd00 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bce76640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a8003b80 0x7ff8a8003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 0x7ff8a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bd677640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8004d50 0x7ff8a8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bce76640 1 -- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 msgr2=0x7ff8a8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bce76640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815dd00 0x7ff8a8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bce76640 1 -- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8004d50 msgr2=0x7ff8a8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bce76640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8004d50 0x7ff8a8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bce76640 1 -- 172.21.15.67:0/2984482339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a8003530 con 0x7ff8a8003b80 2024-04-25T15:51:51.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bce76640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a8003b80 0x7ff8a8003fd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009ae0 tx=0x7ff8ac01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15ec0a8e1fd410b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0096b0 con 0x7ff8a8003b80 2024-04-25T15:51:51.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ac0a6470 con 0x7ff8a8003b80 2024-04-25T15:51:51.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8b77fe640 1 -- 172.21.15.67:0/2984482339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0a6860 con 0x7ff8a8003b80 2024-04-25T15:51:51.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bf0da640 1 -- 172.21.15.67:0/2984482339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a8003b80 msgr2=0x7ff8a8003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bf0da640 1 --2- 172.21.15.67:0/2984482339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a8003b80 0x7ff8a8003fd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009ae0 tx=0x7ff8ac01ca90 comp rx=0 tx=0).stop 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bd677640 1 -- 172.21.15.67:0/2984482339 reap_dead start 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bf0da640 1 -- 172.21.15.67:0/2984482339 shutdown_connections 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.423+0000 7ff8bf0da640 1 -- 172.21.15.67:0/2984482339 >> 172.21.15.67:0/2984482339 conn(0x7ff8b80ce1c0 msgr2=0x7ff8b80ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 -- 172.21.15.67:0/2984482339 shutdown_connections 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 -- 172.21.15.67:0/2984482339 wait complete. 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 Processor -- start 2024-04-25T15:51:51.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 -- start start 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8003b80 0x7ff8b8168210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80b1140 0x7ff8b81651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b814c840 0x7ff8b814f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b806d660 con 0x7ff8b80b1140 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b8068c40 con 0x7ff8a8003b80 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bf0da640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8b806d7e0 con 0x7ff8b814c840 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80b1140 0x7ff8b81651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80b1140 0x7ff8b81651a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33240/0 (socket says 172.21.15.67:33240) 2024-04-25T15:51:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 -- 172.21.15.67:0/1791582764 learned_addr learned my addr 172.21.15.67:0/1791582764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bd677640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b814c840 0x7ff8b814f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8bce76640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8003b80 0x7ff8b8168210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 -- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b814c840 msgr2=0x7ff8b814f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b814c840 0x7ff8b814f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 -- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8003b80 msgr2=0x7ff8b8168210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8003b80 0x7ff8b8168210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 -- 172.21.15.67:0/1791582764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b8163790 con 0x7ff8b80b1140 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80b1140 0x7ff8b81651a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400e7e0 tx=0x7ff8a400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.427+0000 7ff8b57fa640 1 -- 172.21.15.67:0/1791582764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a400ee90 con 0x7ff8b80b1140 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8b57fa640 1 -- 172.21.15.67:0/1791582764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8a4018070 con 0x7ff8b80b1140 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b81f6e00 con 0x7ff8b80b1140 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8b57fa640 1 -- 172.21.15.67:0/1791582764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4007520 con 0x7ff8b80b1140 2024-04-25T15:51:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b81f6320 con 0x7ff8b80b1140 2024-04-25T15:51:51.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b806a480 con 0x7ff8b80b1140 2024-04-25T15:51:51.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8b57fa640 1 -- 172.21.15.67:0/1791582764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff8a40057b0 con 0x7ff8b80b1140 2024-04-25T15:51:51.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8b57fa640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89c06f5a0 0x7ff89c071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8b57fa640 1 -- 172.21.15.67:0/1791582764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5825+0+0 (secure 0 0 0) 0x7ff8a408f040 con 0x7ff8b80b1140 2024-04-25T15:51:51.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8bce76640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89c06f5a0 0x7ff89c071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.431+0000 7ff8bce76640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89c06f5a0 0x7ff89c071ab0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0ad000 tx=0x7ff8ac0046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.435+0000 7ff8b57fa640 1 -- 172.21.15.67:0/1791582764 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff8a405bfd0 con 0x7ff8b80b1140 2024-04-25T15:51:51.455 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:51.451+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:51.455 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:51.455 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:51.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:51.556 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:51.551+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:51.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.603+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7ff8b8163960 con 0x7ff8b80b1140 2024-04-25T15:51:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8b57fa640 1 -- 172.21.15.67:0/1791582764 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v37) ==== 81+0+34746 (secure 0 0 0) 0x7ff8a405b8d0 con 0x7ff8b80b1140 2024-04-25T15:51:51.610 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:51.610 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:51.610 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 37, 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:51:50.279310+0000", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:51:46.224511+0000", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5799936, 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:51:51.611 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:51:51.612 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:51:51.613 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:51:51.614 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:51:51.615 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:51:51.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 8, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 8, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.25, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 1, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 1 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.617 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:51:51.618 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:51:51.619 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:51:51.620 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:51:51.621 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.5, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.5, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-04-25T15:51:51.622 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 1, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 1 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:51:51.623 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:51:51.624 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:51:51.625 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.626 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:51:51.627 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.25, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.25, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.25, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.25, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 1, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 1 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.628 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.630 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:51:51.631 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.634 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:51:51.636 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:51:51.637 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:51:51.638 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.640 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.641 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.642 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.643 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:51:51.645 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.646 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.648 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.649 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:51:51.650 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:51:51.652 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.653 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.655 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.657 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 32, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.658 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.659 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.660 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.662 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:51:51.663 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:51.667 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.668 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.669 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.671 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.672 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2f", 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: 6, 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:51:51.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:51:51.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:51:51.675 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:51:51.676 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:51:51.677 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:51:51.677 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:51:51.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:51.677 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:51.678 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:51.679 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:51.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89c06f5a0 msgr2=0x7ff89c071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89c06f5a0 0x7ff89c071ab0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0ad000 tx=0x7ff8ac0046d0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80b1140 msgr2=0x7ff8b81651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80b1140 0x7ff8b81651a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400e7e0 tx=0x7ff8a400eca0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 shutdown_connections 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89c06f5a0 0x7ff89c071ab0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b814c840 0x7ff8b814f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80b1140 0x7ff8b81651a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 --2- 172.21.15.67:0/1791582764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8a8003b80 0x7ff8b8168210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 >> 172.21.15.67:0/1791582764 conn(0x7ff8b80ce1c0 msgr2=0x7ff8b8154b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 shutdown_connections 2024-04-25T15:51:51.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.607+0000 7ff8bf0da640 1 -- 172.21.15.67:0/1791582764 wait complete. 2024-04-25T15:51:51.681 INFO:tasks.thrashosds.thrasher:j is {'epoch': 37, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:51:50.279310+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:51:46.224511+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.2f', 'osds': [6, 2]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:51:51.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:51:51.682 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:51.675+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:51.778 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:51.775+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:51.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.867+0000 7f9c463a1640 1 Processor -- start 2024-04-25T15:51:51.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c463a1640 1 -- start start 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c463a1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9c40160800 0x7f9c4006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c463a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 0x7f9c40167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c463a1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c4006d8e0 con 0x7f9c400c92d0 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c463a1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9c4006dbc0 con 0x7f9c40160800 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c463a1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9c40073c70 con 0x7f9c400c96e0 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9c400c92d0 0x7f9c40072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40294/0 (socket says 172.21.15.67:40294) 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3f7fe640 1 -- 172.21.15.67:0/1824493393 learned_addr learned my addr 172.21.15.67:0/1824493393 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3ffff640 1 --2- 172.21.15.67:0/1824493393 >> v2:172.21.15.176:3300/0 conn(0x7f9c40160800 0x7f9c4006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c4493e640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 0x7f9c40167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2299183660 0 0) 0x7f9c4006d8e0 con 0x7f9c400c92d0 2024-04-25T15:51:51.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c28003530 con 0x7f9c400c92d0 2024-04-25T15:51:51.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 216116170 0 0) 0x7f9c28003530 con 0x7f9c400c92d0 2024-04-25T15:51:51.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 msgr2=0x7f9c40167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 0x7f9c40167180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 >> v2:172.21.15.176:3300/0 conn(0x7f9c40160800 msgr2=0x7f9c4006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 --2- 172.21.15.67:0/1824493393 >> v2:172.21.15.176:3300/0 conn(0x7f9c40160800 0x7f9c4006d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c40073f50 con 0x7f9c400c92d0 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f9c34001680 con 0x7f9c400c92d0 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 >> v1:172.21.15.67:6789/0 conn(0x7f9c400c92d0 legacy=0x7f9c40072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c280039c0 0x7f9c28003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 0x7f9c28004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c28004d50 0x7f9c28005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4006dbc0 con 0x7f9c28004d50 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9c40073c70 con 0x7f9c280039c0 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9c2800ba50 con 0x7f9c400c96e0 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c4493e640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c28004d50 0x7f9c28005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3ffff640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c280039c0 0x7f9c28003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3f7fe640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 0x7f9c28004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c4493e640 1 -- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 msgr2=0x7f9c28004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c4493e640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 0x7f9c28004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c4493e640 1 -- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c280039c0 msgr2=0x7f9c28003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c4493e640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c280039c0 0x7f9c28003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c4493e640 1 -- 172.21.15.67:0/1824493393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c28003530 con 0x7f9c28004d50 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.871+0000 7f9c3f7fe640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c400c96e0 0x7f9c28004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:51.880 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c4493e640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c28004d50 0x7f9c28005160 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9c400c8c20 tx=0x7f9c3801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760b266b53e3dee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c380b2070 con 0x7f9c28004d50 2024-04-25T15:51:51.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c380a5c50 con 0x7f9c28004d50 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c3effd640 1 -- 172.21.15.67:0/1824493393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c38009c10 con 0x7f9c28004d50 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c463a1640 1 -- 172.21.15.67:0/1824493393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c28004d50 msgr2=0x7f9c28005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c463a1640 1 --2- 172.21.15.67:0/1824493393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c28004d50 0x7f9c28005160 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9c400c8c20 tx=0x7f9c3801cc40 comp rx=0 tx=0).stop 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c4493e640 1 -- 172.21.15.67:0/1824493393 reap_dead start 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c463a1640 1 -- 172.21.15.67:0/1824493393 shutdown_connections 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c463a1640 1 -- 172.21.15.67:0/1824493393 >> 172.21.15.67:0/1824493393 conn(0x7f9c40070db0 msgr2=0x7f9c400ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c463a1640 1 -- 172.21.15.67:0/1824493393 shutdown_connections 2024-04-25T15:51:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.875+0000 7f9c463a1640 1 -- 172.21.15.67:0/1824493393 wait complete. 2024-04-25T15:51:51.883 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:51.880+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:51.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.879+0000 7f9c463a1640 1 Processor -- start 2024-04-25T15:51:51.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.879+0000 7f9c463a1640 1 -- start start 2024-04-25T15:51:51.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c40158010 0x7f9c4015fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4014f180 0x7f9c40158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c4014f590 0x7f9c40158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c40074a30 con 0x7f9c4014f180 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9c40074510 con 0x7f9c40158010 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4006d570 con 0x7f9c4014f590 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4014f180 0x7f9c40158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4014f180 0x7f9c40158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33260/0 (socket says 172.21.15.67:33260) 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 -- 172.21.15.67:0/1179534900 learned_addr learned my addr 172.21.15.67:0/1179534900 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c4493e640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c4014f590 0x7f9c40158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3ffff640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c40158010 0x7f9c4015fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 -- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c4014f590 msgr2=0x7f9c40158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c4014f590 0x7f9c40158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 -- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c40158010 msgr2=0x7f9c4015fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c40158010 0x7f9c4015fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 -- 172.21.15.67:0/1179534900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c401f2d00 con 0x7f9c4014f180 2024-04-25T15:51:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3f7fe640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4014f180 0x7f9c40158560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9c34010bc0 tx=0x7f9c3400e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3cff9640 1 -- 172.21.15.67:0/1179534900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3401b070 con 0x7f9c4014f180 2024-04-25T15:51:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3cff9640 1 -- 172.21.15.67:0/1179534900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c34011040 con 0x7f9c4014f180 2024-04-25T15:51:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c40157c70 con 0x7f9c4014f180 2024-04-25T15:51:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c3cff9640 1 -- 172.21.15.67:0/1179534900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c340143a0 con 0x7f9c4014f180 2024-04-25T15:51:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c401f32a0 con 0x7f9c4014f180 2024-04-25T15:51:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.883+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c4006a480 con 0x7f9c4014f180 2024-04-25T15:51:51.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.887+0000 7f9c3cff9640 1 -- 172.21.15.67:0/1179534900 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9c3401e070 con 0x7f9c4014f180 2024-04-25T15:51:51.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.887+0000 7f9c3cff9640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c2c06f400 0x7f9c2c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.887+0000 7f9c3cff9640 1 -- 172.21.15.67:0/1179534900 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5825+0+0 (secure 0 0 0) 0x7f9c340921e0 con 0x7f9c4014f180 2024-04-25T15:51:51.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.887+0000 7f9c3ffff640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c2c06f400 0x7f9c2c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.887+0000 7f9c3ffff640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c2c06f400 0x7f9c2c071910 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009ab0 tx=0x7f9c30008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:51.891+0000 7f9c3cff9640 1 -- 172.21.15.67:0/1179534900 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c34025020 con 0x7f9c4014f180 2024-04-25T15:51:51.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:51.984 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:51.980+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:52.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.051+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9c400c8c50 con 0x7f9c2c06f400 2024-04-25T15:51:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c3cff9640 1 -- 172.21.15.67:0/1179534900 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252057 (secure 0 0 0) 0x7f9c400c8c50 con 0x7f9c2c06f400 2024-04-25T15:51:52.076 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:51:52.077 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:51:52.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c2c06f400 msgr2=0x7f9c2c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c2c06f400 0x7f9c2c071910 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009ab0 tx=0x7f9c30008040 comp rx=0 tx=0).stop 2024-04-25T15:51:52.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4014f180 msgr2=0x7f9c40158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4014f180 0x7f9c40158560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9c34010bc0 tx=0x7f9c3400e4d0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 shutdown_connections 2024-04-25T15:51:52.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c2c06f400 0x7f9c2c071910 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c4014f590 0x7f9c40158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c4014f180 0x7f9c40158560 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 --2- 172.21.15.67:0/1179534900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c40158010 0x7f9c4015fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 >> 172.21.15.67:0/1179534900 conn(0x7f9c40070db0 msgr2=0x7f9c40152840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 shutdown_connections 2024-04-25T15:51:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.071+0000 7f9c463a1640 1 -- 172.21.15.67:0/1179534900 wait complete. 2024-04-25T15:51:52.084 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:52.084 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:52.085 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:52.080+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:52.099 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-25T15:51:51.827316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604666,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":319,"num_write_kb":1704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2232,"num_keys_recovered":12,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220712,"kb_used_data":4508,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754008,"statfs":{"total":773094113280,"available":772868104192,"internally_reserved":0,"allocated":4616192,"data_stored":2542950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002324"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247512+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:51:47.247512+0000","last_peered":"2024-04-25T15:51:47.247512+0000","last_clean":"2024-04-25T15:51:47.247512+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:51:47.247512+0000","last_undegraded":"2024-04-25T15:51:47.247512+0000","last_fullsized":"2024-04-25T15:51:47.247512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.268757+0000","last_change":"2024-04-25T15:51:47.264956+0000","last_active":"2024-04-25T15:51:47.268757+0000","last_peered":"2024-04-25T15:51:47.268757+0000","last_clean":"2024-04-25T15:51:47.268757+0000","last_became_active":"2024-04-25T15:51:47.264571+0000","last_became_peered":"2024-04-25T15:51:47.264571+0000","last_unstale":"2024-04-25T15:51:47.268757+0000","last_undegraded":"2024-04-25T15:51:47.268757+0000","last_fullsized":"2024-04-25T15:51:47.268757+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:39.343444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247607+0000","last_change":"2024-04-25T15:50:31.180125+0000","last_active":"2024-04-25T15:51:47.247607+0000","last_peered":"2024-04-25T15:51:47.247607+0000","last_clean":"2024-04-25T15:51:47.247607+0000","last_became_active":"2024-04-25T15:50:31.179272+0000","last_became_peered":"2024-04-25T15:50:31.179272+0000","last_unstale":"2024-04-25T15:51:47.247607+0000","last_undegraded":"2024-04-25T15:51:47.247607+0000","last_fullsized":"2024-04-25T15:51:47.247607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:29:18.770678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.260456+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:51:47.260456+0000","last_peered":"2024-04-25T15:51:47.260456+0000","last_clean":"2024-04-25T15:51:47.260456+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:51:47.260456+0000","last_undegraded":"2024-04-25T15:51:47.260456+0000","last_fullsized":"2024-04-25T15:51:47.260456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.246432+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:51:47.246432+0000","last_peered":"2024-04-25T15:51:47.246432+0000","last_clean":"2024-04-25T15:51:47.246432+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:51:47.246432+0000","last_undegraded":"2024-04-25T15:51:47.246432+0000","last_fullsized":"2024-04-25T15:51:47.246432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259655+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:51:47.259655+0000","last_peered":"2024-04-25T15:51:47.259655+0000","last_clean":"2024-04-25T15:51:47.259655+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:51:47.259655+0000","last_undegraded":"2024-04-25T15:51:47.259655+0000","last_fullsized":"2024-04-25T15:51:47.259655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.245269+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:51:47.245269+0000","last_peered":"2024-04-25T15:51:47.245269+0000","last_clean":"2024-04-25T15:51:47.245269+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:51:47.245269+0000","last_undegraded":"2024-04-25T15:51:47.245269+0000","last_fullsized":"2024-04-25T15:51:47.245269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.253112+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:51:47.253112+0000","last_peered":"2024-04-25T15:51:47.253112+0000","last_clean":"2024-04-25T15:51:47.253112+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:51:47.253112+0000","last_undegraded":"2024-04-25T15:51:47.253112+0000","last_fullsized":"2024-04-25T15:51:47.253112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":38,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.244302+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:51:47.244302+0000","last_peered":"2024-04-25T15:51:47.244302+0000","last_clean":"2024-04-25T15:51:47.244302+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:51:47.244302+0000","last_undegraded":"2024-04-25T15:51:47.244302+0000","last_fullsized":"2024-04-25T15:51:47.244302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256436+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:51:47.256436+0000","last_peered":"2024-04-25T15:51:47.256436+0000","last_clean":"2024-04-25T15:51:47.256436+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:51:47.256436+0000","last_undegraded":"2024-04-25T15:51:47.256436+0000","last_fullsized":"2024-04-25T15:51:47.256436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.255792+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:51:47.255792+0000","last_peered":"2024-04-25T15:51:47.255792+0000","last_clean":"2024-04-25T15:51:47.255792+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:51:47.255792+0000","last_undegraded":"2024-04-25T15:51:47.255792+0000","last_fullsized":"2024-04-25T15:51:47.255792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.250096+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:51:47.250096+0000","last_peered":"2024-04-25T15:51:47.250096+0000","last_clean":"2024-04-25T15:51:47.250096+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:51:47.250096+0000","last_undegraded":"2024-04-25T15:51:47.250096+0000","last_fullsized":"2024-04-25T15:51:47.250096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.252179+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:51:47.252179+0000","last_peered":"2024-04-25T15:51:47.252179+0000","last_clean":"2024-04-25T15:51:47.252179+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:51:47.252179+0000","last_undegraded":"2024-04-25T15:51:47.252179+0000","last_fullsized":"2024-04-25T15:51:47.252179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":54,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.268314+0000","last_change":"2024-04-25T15:51:47.268314+0000","last_active":"2024-04-25T15:51:47.268314+0000","last_peered":"2024-04-25T15:51:47.268314+0000","last_clean":"2024-04-25T15:51:47.268314+0000","last_became_active":"2024-04-25T15:51:47.267664+0000","last_became_peered":"2024-04-25T15:51:47.267664+0000","last_unstale":"2024-04-25T15:51:47.268314+0000","last_undegraded":"2024-04-25T15:51:47.268314+0000","last_fullsized":"2024-04-25T15:51:47.268314+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:00:55.974276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.267885+0000","last_change":"2024-04-25T15:51:47.265508+0000","last_active":"2024-04-25T15:51:47.267885+0000","last_peered":"2024-04-25T15:51:47.267885+0000","last_clean":"2024-04-25T15:51:47.267885+0000","last_became_active":"2024-04-25T15:51:47.264990+0000","last_became_peered":"2024-04-25T15:51:47.264990+0000","last_unstale":"2024-04-25T15:51:47.267885+0000","last_undegraded":"2024-04-25T15:51:47.267885+0000","last_fullsized":"2024-04-25T15:51:47.267885+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:00.837180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.258914+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:51:47.258914+0000","last_peered":"2024-04-25T15:51:47.258914+0000","last_clean":"2024-04-25T15:51:47.258914+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:51:47.258914+0000","last_undegraded":"2024-04-25T15:51:47.258914+0000","last_fullsized":"2024-04-25T15:51:47.258914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256018+0000","last_change":"2024-04-25T15:51:41.970074+0000","last_active":"2024-04-25T15:51:47.256018+0000","last_peered":"2024-04-25T15:51:47.256018+0000","last_clean":"2024-04-25T15:51:47.256018+0000","last_became_active":"2024-04-25T15:51:41.969256+0000","last_became_peered":"2024-04-25T15:51:41.969256+0000","last_unstale":"2024-04-25T15:51:47.256018+0000","last_undegraded":"2024-04-25T15:51:47.256018+0000","last_fullsized":"2024-04-25T15:51:47.256018+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:46.412980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.243602+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:51:47.243602+0000","last_peered":"2024-04-25T15:51:47.243602+0000","last_clean":"2024-04-25T15:51:47.243602+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:51:47.243602+0000","last_undegraded":"2024-04-25T15:51:47.243602+0000","last_fullsized":"2024-04-25T15:51:47.243602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.248202+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:51:47.248202+0000","last_peered":"2024-04-25T15:51:47.248202+0000","last_clean":"2024-04-25T15:51:47.248202+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:51:47.248202+0000","last_undegraded":"2024-04-25T15:51:47.248202+0000","last_fullsized":"2024-04-25T15:51:47.248202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.255749+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:51:47.255749+0000","last_peered":"2024-04-25T15:51:47.255749+0000","last_clean":"2024-04-25T15:51:47.255749+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:51:47.255749+0000","last_undegraded":"2024-04-25T15:51:47.255749+0000","last_fullsized":"2024-04-25T15:51:47.255749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.249200+0000","last_change":"2024-04-25T15:50:31.198107+0000","last_active":"2024-04-25T15:51:47.249200+0000","last_peered":"2024-04-25T15:51:47.249200+0000","last_clean":"2024-04-25T15:51:47.249200+0000","last_became_active":"2024-04-25T15:50:31.196894+0000","last_became_peered":"2024-04-25T15:50:31.196894+0000","last_unstale":"2024-04-25T15:51:47.249200+0000","last_undegraded":"2024-04-25T15:51:47.249200+0000","last_fullsized":"2024-04-25T15:51:47.249200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:08:38.443995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":38,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.242869+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:51:47.242869+0000","last_peered":"2024-04-25T15:51:47.242869+0000","last_clean":"2024-04-25T15:51:47.242869+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:51:47.242869+0000","last_undegraded":"2024-04-25T15:51:47.242869+0000","last_fullsized":"2024-04-25T15:51:47.242869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.254789+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:51:47.254789+0000","last_peered":"2024-04-25T15:51:47.254789+0000","last_clean":"2024-04-25T15:51:47.254789+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:51:47.254789+0000","last_undegraded":"2024-04-25T15:51:47.254789+0000","last_fullsized":"2024-04-25T15:51:47.254789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259036+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:51:47.259036+0000","last_peered":"2024-04-25T15:51:47.259036+0000","last_clean":"2024-04-25T15:51:47.259036+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:51:47.259036+0000","last_undegraded":"2024-04-25T15:51:47.259036+0000","last_fullsized":"2024-04-25T15:51:47.259036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259234+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:51:47.259234+0000","last_peered":"2024-04-25T15:51:47.259234+0000","last_clean":"2024-04-25T15:51:47.259234+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:51:47.259234+0000","last_undegraded":"2024-04-25T15:51:47.259234+0000","last_fullsized":"2024-04-25T15:51:47.259234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256449+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:51:47.256449+0000","last_peered":"2024-04-25T15:51:47.256449+0000","last_clean":"2024-04-25T15:51:47.256449+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:51:47.256449+0000","last_undegraded":"2024-04-25T15:51:47.256449+0000","last_fullsized":"2024-04-25T15:51:47.256449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.274145+0000","last_change":"2024-04-25T15:51:47.271333+0000","last_active":"2024-04-25T15:51:47.274145+0000","last_peered":"2024-04-25T15:51:47.274145+0000","last_clean":"2024-04-25T15:51:47.274145+0000","last_became_active":"2024-04-25T15:51:47.270971+0000","last_became_peered":"2024-04-25T15:51:47.270971+0000","last_unstale":"2024-04-25T15:51:47.274145+0000","last_undegraded":"2024-04-25T15:51:47.274145+0000","last_fullsized":"2024-04-25T15:51:47.274145+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:23.860435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.253601+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:51:47.253601+0000","last_peered":"2024-04-25T15:51:47.253601+0000","last_clean":"2024-04-25T15:51:47.253601+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:51:47.253601+0000","last_undegraded":"2024-04-25T15:51:47.253601+0000","last_fullsized":"2024-04-25T15:51:47.253601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.249406+0000","last_change":"2024-04-25T15:50:31.204179+0000","last_active":"2024-04-25T15:51:47.249406+0000","last_peered":"2024-04-25T15:51:47.249406+0000","last_clean":"2024-04-25T15:51:47.249406+0000","last_became_active":"2024-04-25T15:50:31.203643+0000","last_became_peered":"2024-04-25T15:50:31.203643+0000","last_unstale":"2024-04-25T15:51:47.249406+0000","last_undegraded":"2024-04-25T15:51:47.249406+0000","last_fullsized":"2024-04-25T15:51:47.249406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:28.814460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259997+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:51:47.259997+0000","last_peered":"2024-04-25T15:51:47.259997+0000","last_clean":"2024-04-25T15:51:47.259997+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:51:47.259997+0000","last_undegraded":"2024-04-25T15:51:47.259997+0000","last_fullsized":"2024-04-25T15:51:47.259997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":54,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.243460+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:51:47.243460+0000","last_peered":"2024-04-25T15:51:47.243460+0000","last_clean":"2024-04-25T15:51:47.243460+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:51:47.243460+0000","last_undegraded":"2024-04-25T15:51:47.243460+0000","last_fullsized":"2024-04-25T15:51:47.243460+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.254970+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:51:47.254970+0000","last_peered":"2024-04-25T15:51:47.254970+0000","last_clean":"2024-04-25T15:51:47.254970+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:51:47.254970+0000","last_undegraded":"2024-04-25T15:51:47.254970+0000","last_fullsized":"2024-04-25T15:51:47.254970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":54,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.268608+0000","last_change":"2024-04-25T15:51:47.268608+0000","last_active":"2024-04-25T15:51:47.268608+0000","last_peered":"2024-04-25T15:51:47.268608+0000","last_clean":"2024-04-25T15:51:47.268608+0000","last_became_active":"2024-04-25T15:51:47.267582+0000","last_became_peered":"2024-04-25T15:51:47.267582+0000","last_unstale":"2024-04-25T15:51:47.268608+0000","last_undegraded":"2024-04-25T15:51:47.268608+0000","last_fullsized":"2024-04-25T15:51:47.268608+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:08.075848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":52,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.252339+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:51:47.252339+0000","last_peered":"2024-04-25T15:51:47.252339+0000","last_clean":"2024-04-25T15:51:47.252339+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:51:47.252339+0000","last_undegraded":"2024-04-25T15:51:47.252339+0000","last_fullsized":"2024-04-25T15:51:47.252339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.249613+0000","last_change":"2024-04-25T15:50:31.198849+0000","last_active":"2024-04-25T15:51:47.249613+0000","last_peered":"2024-04-25T15:51:47.249613+0000","last_clean":"2024-04-25T15:51:47.249613+0000","last_became_active":"2024-04-25T15:50:31.198255+0000","last_became_peered":"2024-04-25T15:50:31.198255+0000","last_unstale":"2024-04-25T15:51:47.249613+0000","last_undegraded":"2024-04-25T15:51:47.249613+0000","last_fullsized":"2024-04-25T15:51:47.249613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:02.329507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256829+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:51:47.256829+0000","last_peered":"2024-04-25T15:51:47.256829+0000","last_clean":"2024-04-25T15:51:47.256829+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:51:47.256829+0000","last_undegraded":"2024-04-25T15:51:47.256829+0000","last_fullsized":"2024-04-25T15:51:47.256829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":49,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.255014+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:51:47.255014+0000","last_peered":"2024-04-25T15:51:47.255014+0000","last_clean":"2024-04-25T15:51:47.255014+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:51:47.255014+0000","last_undegraded":"2024-04-25T15:51:47.255014+0000","last_fullsized":"2024-04-25T15:51:47.255014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.253184+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:51:47.253184+0000","last_peered":"2024-04-25T15:51:47.253184+0000","last_clean":"2024-04-25T15:51:47.253184+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:51:47.253184+0000","last_undegraded":"2024-04-25T15:51:47.253184+0000","last_fullsized":"2024-04-25T15:51:47.253184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":49,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259348+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:51:47.259348+0000","last_peered":"2024-04-25T15:51:47.259348+0000","last_clean":"2024-04-25T15:51:47.259348+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:51:47.259348+0000","last_undegraded":"2024-04-25T15:51:47.259348+0000","last_fullsized":"2024-04-25T15:51:47.259348+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.274075+0000","last_change":"2024-04-25T15:51:47.271089+0000","last_active":"2024-04-25T15:51:47.274075+0000","last_peered":"2024-04-25T15:51:47.274075+0000","last_clean":"2024-04-25T15:51:47.274075+0000","last_became_active":"2024-04-25T15:51:47.270705+0000","last_became_peered":"2024-04-25T15:51:47.270705+0000","last_unstale":"2024-04-25T15:51:47.274075+0000","last_undegraded":"2024-04-25T15:51:47.274075+0000","last_fullsized":"2024-04-25T15:51:47.274075+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:08.380566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":51,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.255701+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:51:47.255701+0000","last_peered":"2024-04-25T15:51:47.255701+0000","last_clean":"2024-04-25T15:51:47.255701+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:51:47.255701+0000","last_undegraded":"2024-04-25T15:51:47.255701+0000","last_fullsized":"2024-04-25T15:51:47.255701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.261359+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:51:47.261359+0000","last_peered":"2024-04-25T15:51:47.261359+0000","last_clean":"2024-04-25T15:51:47.261359+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:51:47.261359+0000","last_undegraded":"2024-04-25T15:51:47.261359+0000","last_fullsized":"2024-04-25T15:51:47.261359+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":50,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.261593+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:51:47.261593+0000","last_peered":"2024-04-25T15:51:47.261593+0000","last_clean":"2024-04-25T15:51:47.261593+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:51:47.261593+0000","last_undegraded":"2024-04-25T15:51:47.261593+0000","last_fullsized":"2024-04-25T15:51:47.261593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259729+0000","last_change":"2024-04-25T15:50:31.195782+0000","last_active":"2024-04-25T15:51:47.259729+0000","last_peered":"2024-04-25T15:51:47.259729+0000","last_clean":"2024-04-25T15:51:47.259729+0000","last_became_active":"2024-04-25T15:50:31.192546+0000","last_became_peered":"2024-04-25T15:50:31.192546+0000","last_unstale":"2024-04-25T15:51:47.259729+0000","last_undegraded":"2024-04-25T15:51:47.259729+0000","last_fullsized":"2024-04-25T15:51:47.259729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:01.279402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.303575+0000","last_change":"2024-04-25T15:51:47.302272+0000","last_active":"2024-04-25T15:51:47.303575+0000","last_peered":"2024-04-25T15:51:47.303575+0000","last_clean":"2024-04-25T15:51:47.303575+0000","last_became_active":"2024-04-25T15:51:47.266158+0000","last_became_peered":"2024-04-25T15:51:47.266158+0000","last_unstale":"2024-04-25T15:51:47.303575+0000","last_undegraded":"2024-04-25T15:51:47.303575+0000","last_fullsized":"2024-04-25T15:51:47.303575+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:28.386086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":672,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.250341+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:51:47.250341+0000","last_peered":"2024-04-25T15:51:47.250341+0000","last_clean":"2024-04-25T15:51:47.250341+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:51:47.250341+0000","last_undegraded":"2024-04-25T15:51:47.250341+0000","last_fullsized":"2024-04-25T15:51:47.250341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"24'14","reported_seq":60,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256077+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:51:47.256077+0000","last_peered":"2024-04-25T15:51:47.256077+0000","last_clean":"2024-04-25T15:51:47.256077+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:51:47.256077+0000","last_undegraded":"2024-04-25T15:51:47.256077+0000","last_fullsized":"2024-04-25T15:51:47.256077+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247370+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:51:47.247370+0000","last_peered":"2024-04-25T15:51:47.247370+0000","last_clean":"2024-04-25T15:51:47.247370+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:51:47.247370+0000","last_undegraded":"2024-04-25T15:51:47.247370+0000","last_fullsized":"2024-04-25T15:51:47.247370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":55,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.249837+0000","last_change":"2024-04-25T15:50:30.185576+0000","last_active":"2024-04-25T15:51:47.249837+0000","last_peered":"2024-04-25T15:51:47.249837+0000","last_clean":"2024-04-25T15:51:47.249837+0000","last_became_active":"2024-04-25T15:50:30.185142+0000","last_became_peered":"2024-04-25T15:50:30.185142+0000","last_unstale":"2024-04-25T15:51:47.249837+0000","last_undegraded":"2024-04-25T15:51:47.249837+0000","last_fullsized":"2024-04-25T15:51:47.249837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:29.203553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.248686+0000","last_change":"2024-04-25T15:50:31.195550+0000","last_active":"2024-04-25T15:51:47.248686+0000","last_peered":"2024-04-25T15:51:47.248686+0000","last_clean":"2024-04-25T15:51:47.248686+0000","last_became_active":"2024-04-25T15:50:31.194495+0000","last_became_peered":"2024-04-25T15:50:31.194495+0000","last_unstale":"2024-04-25T15:51:47.248686+0000","last_undegraded":"2024-04-25T15:51:47.248686+0000","last_fullsized":"2024-04-25T15:51:47.248686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:18:56.408869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":51,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.260018+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:51:47.260018+0000","last_peered":"2024-04-25T15:51:47.260018+0000","last_clean":"2024-04-25T15:51:47.260018+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:51:47.260018+0000","last_undegraded":"2024-04-25T15:51:47.260018+0000","last_fullsized":"2024-04-25T15:51:47.260018+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.246520+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:51:47.246520+0000","last_peered":"2024-04-25T15:51:47.246520+0000","last_clean":"2024-04-25T15:51:47.246520+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:51:47.246520+0000","last_undegraded":"2024-04-25T15:51:47.246520+0000","last_fullsized":"2024-04-25T15:51:47.246520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":52,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.255311+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:51:47.255311+0000","last_peered":"2024-04-25T15:51:47.255311+0000","last_clean":"2024-04-25T15:51:47.255311+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:51:47.255311+0000","last_undegraded":"2024-04-25T15:51:47.255311+0000","last_fullsized":"2024-04-25T15:51:47.255311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.255287+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:51:47.255287+0000","last_peered":"2024-04-25T15:51:47.255287+0000","last_clean":"2024-04-25T15:51:47.255287+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:51:47.255287+0000","last_undegraded":"2024-04-25T15:51:47.255287+0000","last_fullsized":"2024-04-25T15:51:47.255287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":109,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.287562+0000","last_change":"2024-04-25T15:51:47.287562+0000","last_active":"2024-04-25T15:51:47.287562+0000","last_peered":"2024-04-25T15:51:47.287562+0000","last_clean":"2024-04-25T15:51:47.287562+0000","last_became_active":"2024-04-25T15:51:47.264513+0000","last_became_peered":"2024-04-25T15:51:47.264513+0000","last_unstale":"2024-04-25T15:51:47.287562+0000","last_undegraded":"2024-04-25T15:51:47.287562+0000","last_fullsized":"2024-04-25T15:51:47.287562+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:51:11.555766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.246103+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:51:47.246103+0000","last_peered":"2024-04-25T15:51:47.246103+0000","last_clean":"2024-04-25T15:51:47.246103+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:51:47.246103+0000","last_undegraded":"2024-04-25T15:51:47.246103+0000","last_fullsized":"2024-04-25T15:51:47.246103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.251534+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:51:47.251534+0000","last_peered":"2024-04-25T15:51:47.251534+0000","last_clean":"2024-04-25T15:51:47.251534+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:51:47.251534+0000","last_undegraded":"2024-04-25T15:51:47.251534+0000","last_fullsized":"2024-04-25T15:51:47.251534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247164+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:51:47.247164+0000","last_peered":"2024-04-25T15:51:47.247164+0000","last_clean":"2024-04-25T15:51:47.247164+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:51:47.247164+0000","last_undegraded":"2024-04-25T15:51:47.247164+0000","last_fullsized":"2024-04-25T15:51:47.247164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":51,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.243917+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:51:47.243917+0000","last_peered":"2024-04-25T15:51:47.243917+0000","last_clean":"2024-04-25T15:51:47.243917+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:51:47.243917+0000","last_undegraded":"2024-04-25T15:51:47.243917+0000","last_fullsized":"2024-04-25T15:51:47.243917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247680+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:51:47.247680+0000","last_peered":"2024-04-25T15:51:47.247680+0000","last_clean":"2024-04-25T15:51:47.247680+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:51:47.247680+0000","last_undegraded":"2024-04-25T15:51:47.247680+0000","last_fullsized":"2024-04-25T15:51:47.247680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.307054+0000","last_change":"2024-04-25T15:51:47.305459+0000","last_active":"2024-04-25T15:51:47.307054+0000","last_peered":"2024-04-25T15:51:47.307054+0000","last_clean":"2024-04-25T15:51:47.307054+0000","last_became_active":"2024-04-25T15:51:47.266603+0000","last_became_peered":"2024-04-25T15:51:47.266603+0000","last_unstale":"2024-04-25T15:51:47.307054+0000","last_undegraded":"2024-04-25T15:51:47.307054+0000","last_fullsized":"2024-04-25T15:51:47.307054+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:12.124748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.261002+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:51:47.261002+0000","last_peered":"2024-04-25T15:51:47.261002+0000","last_clean":"2024-04-25T15:51:47.261002+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:51:47.261002+0000","last_undegraded":"2024-04-25T15:51:47.261002+0000","last_fullsized":"2024-04-25T15:51:47.261002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":52,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.243224+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:51:47.243224+0000","last_peered":"2024-04-25T15:51:47.243224+0000","last_clean":"2024-04-25T15:51:47.243224+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:51:47.243224+0000","last_undegraded":"2024-04-25T15:51:47.243224+0000","last_fullsized":"2024-04-25T15:51:47.243224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247639+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:51:47.247639+0000","last_peered":"2024-04-25T15:51:47.247639+0000","last_clean":"2024-04-25T15:51:47.247639+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:51:47.247639+0000","last_undegraded":"2024-04-25T15:51:47.247639+0000","last_fullsized":"2024-04-25T15:51:47.247639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":142,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.245674+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:51:47.245674+0000","last_peered":"2024-04-25T15:51:47.245674+0000","last_clean":"2024-04-25T15:51:47.245674+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:51:47.245674+0000","last_undegraded":"2024-04-25T15:51:47.245674+0000","last_fullsized":"2024-04-25T15:51:47.245674+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":52,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259559+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:51:47.259559+0000","last_peered":"2024-04-25T15:51:47.259559+0000","last_clean":"2024-04-25T15:51:47.259559+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:51:47.259559+0000","last_undegraded":"2024-04-25T15:51:47.259559+0000","last_fullsized":"2024-04-25T15:51:47.259559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256020+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:51:47.256020+0000","last_peered":"2024-04-25T15:51:47.256020+0000","last_clean":"2024-04-25T15:51:47.256020+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:51:47.256020+0000","last_undegraded":"2024-04-25T15:51:47.256020+0000","last_fullsized":"2024-04-25T15:51:47.256020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":68,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.292263+0000","last_change":"2024-04-25T15:51:47.292263+0000","last_active":"2024-04-25T15:51:47.292263+0000","last_peered":"2024-04-25T15:51:47.292263+0000","last_clean":"2024-04-25T15:51:47.292263+0000","last_became_active":"2024-04-25T15:51:47.264780+0000","last_became_peered":"2024-04-25T15:51:47.264780+0000","last_unstale":"2024-04-25T15:51:47.292263+0000","last_undegraded":"2024-04-25T15:51:47.292263+0000","last_fullsized":"2024-04-25T15:51:47.292263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:40.269146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247414+0000","last_change":"2024-04-25T15:50:31.198447+0000","last_active":"2024-04-25T15:51:47.247414+0000","last_peered":"2024-04-25T15:51:47.247414+0000","last_clean":"2024-04-25T15:51:47.247414+0000","last_became_active":"2024-04-25T15:50:31.197951+0000","last_became_peered":"2024-04-25T15:50:31.197951+0000","last_unstale":"2024-04-25T15:51:47.247414+0000","last_undegraded":"2024-04-25T15:51:47.247414+0000","last_fullsized":"2024-04-25T15:51:47.247414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:23.264761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":52,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.243549+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:51:47.243549+0000","last_peered":"2024-04-25T15:51:47.243549+0000","last_clean":"2024-04-25T15:51:47.243549+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:51:47.243549+0000","last_undegraded":"2024-04-25T15:51:47.243549+0000","last_fullsized":"2024-04-25T15:51:47.243549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.265958+0000","last_change":"2024-04-25T15:51:47.255396+0000","last_active":"2024-04-25T15:51:47.265958+0000","last_peered":"2024-04-25T15:51:47.265958+0000","last_clean":"2024-04-25T15:51:47.265958+0000","last_became_active":"2024-04-25T15:51:47.254596+0000","last_became_peered":"2024-04-25T15:51:47.254596+0000","last_unstale":"2024-04-25T15:51:47.265958+0000","last_undegraded":"2024-04-25T15:51:47.265958+0000","last_fullsized":"2024-04-25T15:51:47.265958+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:06:53.573154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":73,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.301818+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:51:47.301818+0000","last_peered":"2024-04-25T15:51:47.301818+0000","last_clean":"2024-04-25T15:51:47.301818+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:51:47.301818+0000","last_undegraded":"2024-04-25T15:51:47.301818+0000","last_fullsized":"2024-04-25T15:51:47.301818+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.251722+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:51:47.251722+0000","last_peered":"2024-04-25T15:51:47.251722+0000","last_clean":"2024-04-25T15:51:47.251722+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:51:47.251722+0000","last_undegraded":"2024-04-25T15:51:47.251722+0000","last_fullsized":"2024-04-25T15:51:47.251722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.267732+0000","last_change":"2024-04-25T15:51:47.263905+0000","last_active":"2024-04-25T15:51:47.267732+0000","last_peered":"2024-04-25T15:51:47.267732+0000","last_clean":"2024-04-25T15:51:47.267732+0000","last_became_active":"2024-04-25T15:51:47.263546+0000","last_became_peered":"2024-04-25T15:51:47.263546+0000","last_unstale":"2024-04-25T15:51:47.267732+0000","last_undegraded":"2024-04-25T15:51:47.267732+0000","last_fullsized":"2024-04-25T15:51:47.267732+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:14.832346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.260675+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:51:47.260675+0000","last_peered":"2024-04-25T15:51:47.260675+0000","last_clean":"2024-04-25T15:51:47.260675+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:51:47.260675+0000","last_undegraded":"2024-04-25T15:51:47.260675+0000","last_fullsized":"2024-04-25T15:51:47.260675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":50,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.248506+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:51:47.248506+0000","last_peered":"2024-04-25T15:51:47.248506+0000","last_clean":"2024-04-25T15:51:47.248506+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:51:47.248506+0000","last_undegraded":"2024-04-25T15:51:47.248506+0000","last_fullsized":"2024-04-25T15:51:47.248506+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.267664+0000","last_change":"2024-04-25T15:51:47.263756+0000","last_active":"2024-04-25T15:51:47.267664+0000","last_peered":"2024-04-25T15:51:47.267664+0000","last_clean":"2024-04-25T15:51:47.267664+0000","last_became_active":"2024-04-25T15:51:47.263130+0000","last_became_peered":"2024-04-25T15:51:47.263130+0000","last_unstale":"2024-04-25T15:51:47.267664+0000","last_undegraded":"2024-04-25T15:51:47.267664+0000","last_fullsized":"2024-04-25T15:51:47.267664+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:38.024802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'9","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.314999+0000","last_change":"2024-04-25T15:51:47.314396+0000","last_active":"2024-04-25T15:51:47.314999+0000","last_peered":"2024-04-25T15:51:47.314999+0000","last_clean":"2024-04-25T15:51:47.314999+0000","last_became_active":"2024-04-25T15:51:47.274984+0000","last_became_peered":"2024-04-25T15:51:47.274984+0000","last_unstale":"2024-04-25T15:51:47.314999+0000","last_undegraded":"2024-04-25T15:51:47.314999+0000","last_fullsized":"2024-04-25T15:51:47.314999+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:53.936857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259540+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:51:47.259540+0000","last_peered":"2024-04-25T15:51:47.259540+0000","last_clean":"2024-04-25T15:51:47.259540+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:51:47.259540+0000","last_undegraded":"2024-04-25T15:51:47.259540+0000","last_fullsized":"2024-04-25T15:51:47.259540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":53,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256183+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:51:47.256183+0000","last_peered":"2024-04-25T15:51:47.256183+0000","last_clean":"2024-04-25T15:51:47.256183+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:51:47.256183+0000","last_undegraded":"2024-04-25T15:51:47.256183+0000","last_fullsized":"2024-04-25T15:51:47.256183+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":54,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.264115+0000","last_change":"2024-04-25T15:51:47.264115+0000","last_active":"2024-04-25T15:51:47.264115+0000","last_peered":"2024-04-25T15:51:47.264115+0000","last_clean":"2024-04-25T15:51:47.264115+0000","last_became_active":"2024-04-25T15:51:47.263768+0000","last_became_peered":"2024-04-25T15:51:47.263768+0000","last_unstale":"2024-04-25T15:51:47.264115+0000","last_undegraded":"2024-04-25T15:51:47.264115+0000","last_fullsized":"2024-04-25T15:51:47.264115+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:13.362212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.245128+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:51:47.245128+0000","last_peered":"2024-04-25T15:51:47.245128+0000","last_clean":"2024-04-25T15:51:47.245128+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:51:47.245128+0000","last_undegraded":"2024-04-25T15:51:47.245128+0000","last_fullsized":"2024-04-25T15:51:47.245128+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.250848+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:51:47.250848+0000","last_peered":"2024-04-25T15:51:47.250848+0000","last_clean":"2024-04-25T15:51:47.250848+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:51:47.250848+0000","last_undegraded":"2024-04-25T15:51:47.250848+0000","last_fullsized":"2024-04-25T15:51:47.250848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":20,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.282800+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:51:47.282800+0000","last_peered":"2024-04-25T15:51:47.282800+0000","last_clean":"2024-04-25T15:51:47.282800+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:51:47.282800+0000","last_undegraded":"2024-04-25T15:51:47.282800+0000","last_fullsized":"2024-04-25T15:51:47.282800+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":54,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.270024+0000","last_change":"2024-04-25T15:51:47.270024+0000","last_active":"2024-04-25T15:51:47.270024+0000","last_peered":"2024-04-25T15:51:47.270024+0000","last_clean":"2024-04-25T15:51:47.270024+0000","last_became_active":"2024-04-25T15:51:47.269733+0000","last_became_peered":"2024-04-25T15:51:47.269733+0000","last_unstale":"2024-04-25T15:51:47.270024+0000","last_undegraded":"2024-04-25T15:51:47.270024+0000","last_fullsized":"2024-04-25T15:51:47.270024+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:23.505183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":16,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.287706+0000","last_change":"2024-04-25T15:51:47.286937+0000","last_active":"2024-04-25T15:51:47.287706+0000","last_peered":"2024-04-25T15:51:47.287706+0000","last_clean":"2024-04-25T15:51:47.287706+0000","last_became_active":"2024-04-25T15:51:47.266040+0000","last_became_peered":"2024-04-25T15:51:47.266040+0000","last_unstale":"2024-04-25T15:51:47.287706+0000","last_undegraded":"2024-04-25T15:51:47.287706+0000","last_fullsized":"2024-04-25T15:51:47.287706+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:48.792187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":38,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.242620+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:51:47.242620+0000","last_peered":"2024-04-25T15:51:47.242620+0000","last_clean":"2024-04-25T15:51:47.242620+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:51:47.242620+0000","last_undegraded":"2024-04-25T15:51:47.242620+0000","last_fullsized":"2024-04-25T15:51:47.242620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":50,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.246827+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:51:47.246827+0000","last_peered":"2024-04-25T15:51:47.246827+0000","last_clean":"2024-04-25T15:51:47.246827+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:51:47.246827+0000","last_undegraded":"2024-04-25T15:51:47.246827+0000","last_fullsized":"2024-04-25T15:51:47.246827+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.256744+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:51:47.256744+0000","last_peered":"2024-04-25T15:51:47.256744+0000","last_clean":"2024-04-25T15:51:47.256744+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:51:47.256744+0000","last_undegraded":"2024-04-25T15:51:47.256744+0000","last_fullsized":"2024-04-25T15:51:47.256744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.260388+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:51:47.260388+0000","last_peered":"2024-04-25T15:51:47.260388+0000","last_clean":"2024-04-25T15:51:47.260388+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:51:47.260388+0000","last_undegraded":"2024-04-25T15:51:47.260388+0000","last_fullsized":"2024-04-25T15:51:47.260388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":51,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247030+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:51:47.247030+0000","last_peered":"2024-04-25T15:51:47.247030+0000","last_clean":"2024-04-25T15:51:47.247030+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:51:47.247030+0000","last_undegraded":"2024-04-25T15:51:47.247030+0000","last_fullsized":"2024-04-25T15:51:47.247030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.260750+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:51:47.260750+0000","last_peered":"2024-04-25T15:51:47.260750+0000","last_clean":"2024-04-25T15:51:47.260750+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:51:47.260750+0000","last_undegraded":"2024-04-25T15:51:47.260750+0000","last_fullsized":"2024-04-25T15:51:47.260750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":58,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.247900+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:51:47.247900+0000","last_peered":"2024-04-25T15:51:47.247900+0000","last_clean":"2024-04-25T15:51:47.247900+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:51:47.247900+0000","last_undegraded":"2024-04-25T15:51:47.247900+0000","last_fullsized":"2024-04-25T15:51:47.247900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259494+0000","last_change":"2024-04-25T15:50:31.197791+0000","last_active":"2024-04-25T15:51:47.259494+0000","last_peered":"2024-04-25T15:51:47.259494+0000","last_clean":"2024-04-25T15:51:47.259494+0000","last_became_active":"2024-04-25T15:50:31.195994+0000","last_became_peered":"2024-04-25T15:50:31.195994+0000","last_unstale":"2024-04-25T15:51:47.259494+0000","last_undegraded":"2024-04-25T15:51:47.259494+0000","last_fullsized":"2024-04-25T15:51:47.259494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:05.534377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259243+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:51:47.259243+0000","last_peered":"2024-04-25T15:51:47.259243+0000","last_clean":"2024-04-25T15:51:47.259243+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:51:47.259243+0000","last_undegraded":"2024-04-25T15:51:47.259243+0000","last_fullsized":"2024-04-25T15:51:47.259243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259103+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:51:47.259103+0000","last_peered":"2024-04-25T15:51:47.259103+0000","last_clean":"2024-04-25T15:51:47.259103+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:51:47.259103+0000","last_undegraded":"2024-04-25T15:51:47.259103+0000","last_fullsized":"2024-04-25T15:51:47.259103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":59,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-25T15:51:47.259955+0000","last_change":"2024-04-25T15:50:30.181509+0000","last_active":"2024-04-25T15:51:47.259955+0000","last_peered":"2024-04-25T15:51:47.259955+0000","last_clean":"2024-04-25T15:51:47.259955+0000","last_became_active":"2024-04-25T15:50:30.180582+0000","last_became_peered":"2024-04-25T15:50:30.180582+0000","last_unstale":"2024-04-25T15:51:47.259955+0000","last_undegraded":"2024-04-25T15:51:47.259955+0000","last_fullsized":"2024-04-25T15:51:47.259955+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:27.828638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14298,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":218,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2232,"num_keys_recovered":12,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":28596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1032192,"data_stored":761323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":434176,"data_stored":176198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":450560,"data_stored":172787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":425984,"data_stored":170185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":438272,"data_stored":176232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27436,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344404,"statfs":{"total":96636764160,"available":96608669696,"internally_reserved":0,"allocated":421888,"data_stored":164236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":162333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1024000,"data_stored":759656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":7818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":7784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:51:52.103 INFO:tasks.thrashosds.thrasher:Setting 3.13 to [1, 6, 0, 5] 2024-04-25T15:51:52.103 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.13', '1', '6', '0', '5'] 2024-04-25T15:51:52.103 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.13 1 6 0 5 2024-04-25T15:51:52.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:52.186 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:52.184+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:52.285 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:52.285 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:52.286 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:52.280+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:52.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31abc4f640 1 Processor -- start 2024-04-25T15:51:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31abc4f640 1 -- start start 2024-04-25T15:51:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31abc4f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31a4164590 0x7f31a41669b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31abc4f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a406f220 con 0x7f31a40c95f0 2024-04-25T15:51:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31abc4f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a406f4e0 con 0x7f31a40c9a00 2024-04-25T15:51:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31abc4f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f31a415a810 con 0x7f31a4164590 2024-04-25T15:51:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31a99eb640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f31a40c95f0 0x7f31a40cac00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40302/0 (socket says 172.21.15.67:40302) 2024-04-25T15:51:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31aa1ec640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f31a4164590 0x7f31a41669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31a99eb640 1 -- 172.21.15.67:0/417970948 learned_addr learned my addr 172.21.15.67:0/417970948 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31aa1ec640 1 -- 172.21.15.67:0/417970948 >> v1:172.21.15.176:6789/0 conn(0x7f31a40c9a00 legacy=0x7f31a4162e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31aa1ec640 1 -- 172.21.15.67:0/417970948 >> v1:172.21.15.67:6789/0 conn(0x7f31a40c95f0 legacy=0x7f31a40cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31aa1ec640 1 -- 172.21.15.67:0/417970948 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a415aaf0 con 0x7f31a4164590 2024-04-25T15:51:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.299+0000 7f31a89e9640 1 -- 172.21.15.67:0/417970948 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387929968 0 0) 0x7f31a406f220 con 0x7f31a40c95f0 2024-04-25T15:51:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31aa1ec640 1 --2- 172.21.15.67:0/417970948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a4164590 0x7f31a41669b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f319c009d90 tx=0x7f319c01f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5542bac7ce4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31a89e9640 1 -- 172.21.15.67:0/417970948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319c0b4070 con 0x7f31a4164590 2024-04-25T15:51:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31a89e9640 1 -- 172.21.15.67:0/417970948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f319c0af600 con 0x7f31a4164590 2024-04-25T15:51:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31a89e9640 1 -- 172.21.15.67:0/417970948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319c0af9d0 con 0x7f31a4164590 2024-04-25T15:51:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31abc4f640 1 -- 172.21.15.67:0/417970948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a4164590 msgr2=0x7f31a41669b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31abc4f640 1 --2- 172.21.15.67:0/417970948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a4164590 0x7f31a41669b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f319c009d90 tx=0x7f319c01f120 comp rx=0 tx=0).stop 2024-04-25T15:51:52.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31abc4f640 1 -- 172.21.15.67:0/417970948 shutdown_connections 2024-04-25T15:51:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31abc4f640 1 --2- 172.21.15.67:0/417970948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a4164590 0x7f31a41669b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31abc4f640 1 -- 172.21.15.67:0/417970948 >> 172.21.15.67:0/417970948 conn(0x7f31a40cebf0 msgr2=0x7f31a40cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31abc4f640 1 -- 172.21.15.67:0/417970948 shutdown_connections 2024-04-25T15:51:52.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.303+0000 7f31abc4f640 1 -- 172.21.15.67:0/417970948 wait complete. 2024-04-25T15:51:52.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 Processor -- start 2024-04-25T15:51:52.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 -- start start 2024-04-25T15:51:52.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31a40c95f0 0x7f31a413da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a40c9a00 0x7f31a41f1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31a4164590 0x7f31a4142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4143b00 con 0x7f31a40c9a00 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f31a415a500 con 0x7f31a4164590 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f31a4143c80 con 0x7f31a40c95f0 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a40c9a00 0x7f31a41f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a40c9a00 0x7f31a41f1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33282/0 (socket says 172.21.15.67:33282) 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 -- 172.21.15.67:0/2691288730 learned_addr learned my addr 172.21.15.67:0/2691288730 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a99eb640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31a40c95f0 0x7f31a413da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31aa1ec640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31a4164590 0x7f31a4142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 -- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31a40c95f0 msgr2=0x7f31a413da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31a40c95f0 0x7f31a413da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 -- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31a4164590 msgr2=0x7f31a4142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31a4164590 0x7f31a4142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 -- 172.21.15.67:0/2691288730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a415c920 con 0x7f31a40c9a00 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31aa1ec640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31a4164590 0x7f31a4142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a99eb640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31a40c95f0 0x7f31a413da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31a91ea640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a40c9a00 0x7f31a41f1670 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f319800b3e0 tx=0x7f319800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31967fc640 1 -- 172.21.15.67:0/2691288730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198015020 con 0x7f31a40c9a00 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31967fc640 1 -- 172.21.15.67:0/2691288730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3198011850 con 0x7f31a40c9a00 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a41f1bb0 con 0x7f31a40c9a00 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31967fc640 1 -- 172.21.15.67:0/2691288730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198011c40 con 0x7f31a40c9a00 2024-04-25T15:51:52.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a40c71a0 con 0x7f31a40c9a00 2024-04-25T15:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.307+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a40c6f00 con 0x7f31a40c9a00 2024-04-25T15:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.311+0000 7f31967fc640 1 -- 172.21.15.67:0/2691288730 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3198018020 con 0x7f31a40c9a00 2024-04-25T15:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.311+0000 7f31967fc640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f318806f400 0x7f3188071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.311+0000 7f31967fc640 1 -- 172.21.15.67:0/2691288730 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5825+0+0 (secure 0 0 0) 0x7f319808f410 con 0x7f31a40c9a00 2024-04-25T15:51:52.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.311+0000 7f31a99eb640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f318806f400 0x7f3188071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.311+0000 7f31a99eb640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f318806f400 0x7f3188071910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f31a0007fd0 tx=0x7f31a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.315+0000 7f31967fc640 1 -- 172.21.15.67:0/2691288730 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f31a40c6f00 con 0x7f31a40c9a00 2024-04-25T15:51:52.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:51:52.410 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:52.407+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:52.441 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:52.442 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:52.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:52.488 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:52.484+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.487+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [1, 6, 0, 5]} v 0) -- 0x7f31a406a480 con 0x7f31a40c9a00 2024-04-25T15:51:52.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d41c06640 1 Processor -- start 2024-04-25T15:51:52.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d41c06640 1 -- start start 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d41c06640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c156d30 0x7f8d3c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d41c06640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d3c0c5720 0x7f8d3c0c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d41c06640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d3c0cb800 con 0x7f8d3c1603e0 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d41c06640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8d3c0cb5b0 con 0x7f8d3c0c5720 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d41c06640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d3c15ff50 con 0x7f8d3c156d30 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d32ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d3c0c5720 0x7f8d3c0c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d32ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d3c0c5720 0x7f8d3c0c5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33294/0 (socket says 172.21.15.67:33294) 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d32ffd640 1 -- 172.21.15.67:0/1192865748 learned_addr learned my addr 172.21.15.67:0/1192865748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d3b7fe640 1 --2- 172.21.15.67:0/1192865748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c156d30 0x7f8d3c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d32ffd640 1 -- 172.21.15.67:0/1192865748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c156d30 msgr2=0x7f8d3c0ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d32ffd640 1 --2- 172.21.15.67:0/1192865748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c156d30 0x7f8d3c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d32ffd640 1 -- 172.21.15.67:0/1192865748 >> v1:172.21.15.176:6789/0 conn(0x7f8d3c1603e0 legacy=0x7f8d3c1607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.495+0000 7f8d32ffd640 1 -- 172.21.15.67:0/1192865748 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3c0cbae0 con 0x7f8d3c0c5720 2024-04-25T15:51:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d32ffd640 1 --2- 172.21.15.67:0/1192865748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c0c5720 0x7f8d3c0c5b30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0d0bb0 tx=0x7f8d2401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9387acb512c1a72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d3affd640 1 -- 172.21.15.67:0/1192865748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d240b5070 con 0x7f8d3c0c5720 2024-04-25T15:51:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d3affd640 1 -- 172.21.15.67:0/1192865748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d240a89b0 con 0x7f8d3c0c5720 2024-04-25T15:51:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d3affd640 1 -- 172.21.15.67:0/1192865748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d240a8d80 con 0x7f8d3c0c5720 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 -- 172.21.15.67:0/1192865748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c0c5720 msgr2=0x7f8d3c0c5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1192865748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c0c5720 0x7f8d3c0c5b30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0d0bb0 tx=0x7f8d2401ecf0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 -- 172.21.15.67:0/1192865748 shutdown_connections 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1192865748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c0c5720 0x7f8d3c0c5b30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1192865748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c156d30 0x7f8d3c0ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 -- 172.21.15.67:0/1192865748 >> 172.21.15.67:0/1192865748 conn(0x7f8d3c0700b0 msgr2=0x7f8d3c0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 -- 172.21.15.67:0/1192865748 shutdown_connections 2024-04-25T15:51:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 -- 172.21.15.67:0/1192865748 wait complete. 2024-04-25T15:51:52.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 Processor -- start 2024-04-25T15:51:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.499+0000 7f8d41c06640 1 -- start start 2024-04-25T15:51:52.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c0c5720 0x7f8d3c159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d3c156d30 0x7f8d3c1519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c1603e0 0x7f8d3c1e6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c0cc380 con 0x7f8d3c1603e0 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3c0cc0a0 con 0x7f8d3c156d30 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d3c0caa80 con 0x7f8d3c0c5720 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c1603e0 0x7f8d3c1e6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c1603e0 0x7f8d3c1e6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33304/0 (socket says 172.21.15.67:33304) 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 -- 172.21.15.67:0/1162434558 learned_addr learned my addr 172.21.15.67:0/1162434558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d32ffd640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d3c156d30 0x7f8d3c1519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3b7fe640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c0c5720 0x7f8d3c159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 -- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c0c5720 msgr2=0x7f8d3c159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c0c5720 0x7f8d3c159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 -- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d3c156d30 msgr2=0x7f8d3c1519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d3c156d30 0x7f8d3c1519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 -- 172.21.15.67:0/1162434558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3c151f20 con 0x7f8d3c1603e0 2024-04-25T15:51:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d3bfff640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c1603e0 0x7f8d3c1e6720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400be60 tx=0x7f8d3400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d38ff9640 1 -- 172.21.15.67:0/1162434558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34019070 con 0x7f8d3c1603e0 2024-04-25T15:51:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d38ff9640 1 -- 172.21.15.67:0/1162434558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d34014440 con 0x7f8d3c1603e0 2024-04-25T15:51:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3c1524a0 con 0x7f8d3c1603e0 2024-04-25T15:51:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d38ff9640 1 -- 172.21.15.67:0/1162434558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34014870 con 0x7f8d3c1603e0 2024-04-25T15:51:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d3c1508c0 con 0x7f8d3c1603e0 2024-04-25T15:51:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d3c1521f0 con 0x7f8d3c1603e0 2024-04-25T15:51:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.503+0000 7f8d38ff9640 1 -- 172.21.15.67:0/1162434558 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8d3401c070 con 0x7f8d3c1603e0 2024-04-25T15:51:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.507+0000 7f8d38ff9640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d1c06f430 0x7f8d1c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.507+0000 7f8d38ff9640 1 -- 172.21.15.67:0/1162434558 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5825+0+0 (secure 0 0 0) 0x7f8d3408f200 con 0x7f8d3c1603e0 2024-04-25T15:51:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.507+0000 7f8d3b7fe640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d1c06f430 0x7f8d1c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.507+0000 7f8d3b7fe640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d1c06f430 0x7f8d1c071940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00a8d0 tx=0x7f8d2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.511+0000 7f8d38ff9640 1 -- 172.21.15.67:0/1162434558 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8d3405c100 con 0x7f8d3c1603e0 2024-04-25T15:51:52.589 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:52.583+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:52.645 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:52.645 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:52.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:52.689 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:52.684+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:52.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:52.703+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f8d3c06a480 con 0x7f8d3c1603e0 2024-04-25T15:51:52.790 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:52.787+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:52.853 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:52.854 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:52.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:52.914 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:52.911+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:52.993 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:52.992+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:53.063 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:53.063 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:53.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:53.126 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:53.123+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:53.194 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:53.188+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:53.224 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:53.225 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:53.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.295+0000 7f31967fc640 1 -- 172.21.15.67:0/2691288730 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [1, 6, 0, 5]}]=0 set 3.13 pg_upmap_items mapping to [1->6,0->5] v38) ==== 148+0+0 (secure 0 0 0) 0x7f319801f020 con 0x7f31a40c9a00 2024-04-25T15:51:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.13 pg_upmap_items mapping to [1->6,0->5] 2024-04-25T15:51:53.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.295+0000 7f8d38ff9640 1 -- 172.21.15.67:0/1162434558 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v38) ==== 87+0+0 (secure 0 0 0) 0x7f8d3405ba00 con 0x7f8d3c1603e0 2024-04-25T15:51:53.302 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:51:53.302 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:53.295+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d1c06f430 msgr2=0x7f8d1c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d1c06f430 0x7f8d1c071940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00a8d0 tx=0x7f8d2c008040 comp rx=0 tx=0).stop 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c1603e0 msgr2=0x7f8d3c1e6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c1603e0 0x7f8d3c1e6720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400be60 tx=0x7f8d3400bf50 comp rx=0 tx=0).stop 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 shutdown_connections 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d1c06f430 0x7f8d1c071940 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d3c1603e0 0x7f8d3c1e6720 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d3c156d30 0x7f8d3c1519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 --2- 172.21.15.67:0/1162434558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d3c0c5720 0x7f8d3c159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 >> 172.21.15.67:0/1162434558 conn(0x7f8d3c0700b0 msgr2=0x7f8d3c068ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 shutdown_connections 2024-04-25T15:51:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f8d41c06640 1 -- 172.21.15.67:0/1162434558 wait complete. 2024-04-25T15:51:53.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f318806f400 msgr2=0x7f3188071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f318806f400 0x7f3188071910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f31a0007fd0 tx=0x7f31a0009040 comp rx=0 tx=0).stop 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a40c9a00 msgr2=0x7f31a41f1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a40c9a00 0x7f31a41f1670 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f319800b3e0 tx=0x7f319800b8a0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 shutdown_connections 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f318806f400 0x7f3188071910 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31a4164590 0x7f31a4142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31a40c9a00 0x7f31a41f1670 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 --2- 172.21.15.67:0/2691288730 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31a40c95f0 0x7f31a413da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 >> 172.21.15.67:0/2691288730 conn(0x7f31a40cebf0 msgr2=0x7f31a40736e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 shutdown_connections 2024-04-25T15:51:53.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.299+0000 7f31abc4f640 1 -- 172.21.15.67:0/2691288730 wait complete. 2024-04-25T15:51:53.396 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:53.392+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:53.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:53.427 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:53.427 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:53.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:53.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe081fa3640 1 Processor -- start 2024-04-25T15:51:53.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe081fa3640 1 -- start start 2024-04-25T15:51:53.531 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:53.527+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe081fa3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 0x7fe07c0b7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe081fa3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe07c15cb40 0x7fe07c15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe081fa3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe07c0b79b0 con 0x7fe07c156c70 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe081fa3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe07c0b7c20 con 0x7fe07c15cb40 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe081fa3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe07c0ce840 con 0x7fe07c0ce260 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe07c15cb40 0x7fe07c15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe07c15cb40 0x7fe07c15cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47768/0 (socket says 172.21.15.67:47768) 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07bfff640 1 -- 172.21.15.67:0/819343642 learned_addr learned my addr 172.21.15.67:0/819343642 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/819343642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 0x7fe07c0b7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07bfff640 1 -- 172.21.15.67:0/819343642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 msgr2=0x7fe07c0b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07bfff640 1 --2- 172.21.15.67:0/819343642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 0x7fe07c0b7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07bfff640 1 -- 172.21.15.67:0/819343642 >> v1:172.21.15.176:6789/0 conn(0x7fe07c156c70 legacy=0x7fe07c0cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:53.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.527+0000 7fe07bfff640 1 -- 172.21.15.67:0/819343642 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07c0ceb20 con 0x7fe07c15cb40 2024-04-25T15:51:53.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe07bfff640 1 --2- 172.21.15.67:0/819343642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15cb40 0x7fe07c15cf50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe074009e50 tx=0x7fe07401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846f511d3eb19cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe07affd640 1 -- 172.21.15.67:0/819343642 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0740b4070 con 0x7fe07c15cb40 2024-04-25T15:51:53.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe07affd640 1 -- 172.21.15.67:0/819343642 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0740af400 con 0x7fe07c15cb40 2024-04-25T15:51:53.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe07affd640 1 -- 172.21.15.67:0/819343642 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0740af7d0 con 0x7fe07c15cb40 2024-04-25T15:51:53.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 -- 172.21.15.67:0/819343642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15cb40 msgr2=0x7fe07c15cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 --2- 172.21.15.67:0/819343642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15cb40 0x7fe07c15cf50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe074009e50 tx=0x7fe07401f1e0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 -- 172.21.15.67:0/819343642 shutdown_connections 2024-04-25T15:51:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 --2- 172.21.15.67:0/819343642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15cb40 0x7fe07c15cf50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 --2- 172.21.15.67:0/819343642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 0x7fe07c0b7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 -- 172.21.15.67:0/819343642 >> 172.21.15.67:0/819343642 conn(0x7fe07c06e700 msgr2=0x7fe07c06eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 -- 172.21.15.67:0/819343642 shutdown_connections 2024-04-25T15:51:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 -- 172.21.15.67:0/819343642 wait complete. 2024-04-25T15:51:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 Processor -- start 2024-04-25T15:51:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.531+0000 7fe081fa3640 1 -- start start 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 0x7fe07c14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c156c70 0x7fe07c1f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe07c15cb40 0x7fe07c0b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c0cf3c0 con 0x7fe07c156c70 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c0cf0e0 con 0x7fe07c15cb40 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe07c15d490 con 0x7fe07c0ce260 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c156c70 0x7fe07c1f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c156c70 0x7fe07c1f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47774/0 (socket says 172.21.15.67:47774) 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 -- 172.21.15.67:0/4102111940 learned_addr learned my addr 172.21.15.67:0/4102111940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe07bfff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe07c15cb40 0x7fe07c0b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 0x7fe07c14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 msgr2=0x7fe07c14b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe07c0ce260 0x7fe07c14b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe07c15cb40 msgr2=0x7fe07c0b62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe07c15cb40 0x7fe07c0b62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07c151b20 con 0x7fe07c156c70 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe073fff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c156c70 0x7fe07c1f62a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe06400b440 tx=0x7fe06400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064015020 con 0x7fe07c156c70 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe064011830 con 0x7fe07c156c70 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07c151dc0 con 0x7fe07c156c70 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064011c20 con 0x7fe07c156c70 2024-04-25T15:51:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe07c1518b0 con 0x7fe07c156c70 2024-04-25T15:51:53.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe07c06a480 con 0x7fe07c156c70 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.535+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe064018020 con 0x7fe07c156c70 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe078ff9640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe05c06f430 0x7fe05c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5825+0+0 (secure 0 0 0) 0x7fe06408f270 con 0x7fe07c156c70 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v38) ==== 54+0+16 (secure 0 0 0) 0x7fe06401f020 con 0x7fe07c156c70 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe05c06f430 0x7fe05c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe07c0c9970 0x7fe07c0cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe07c15add0 con 0x7fe07c0c9970 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe07bfff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe07c0c9970 0x7fe07c0cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe07bfff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe07c0c9970 0x7fe07c0cc8b0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe05c06f430 0x7fe05c071940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00a8d0 tx=0x7fe06c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.539+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c15add0 con 0x7fe07c0c9970 2024-04-25T15:51:53.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.559+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c15d9e0 con 0x7fe07c0c9970 2024-04-25T15:51:53.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.563+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c15d9e0 con 0x7fe07c0c9970 2024-04-25T15:51:53.567 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.563+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe07c1544a0 0x7fe07c1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.563+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe07c0cb1d0 con 0x7fe07c1544a0 2024-04-25T15:51:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.563+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe07c1544a0 0x7fe07c1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.563+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe07c1544a0 0x7fe07c1548b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.563+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c0cb1d0 con 0x7fe07c1544a0 2024-04-25T15:51:53.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.579+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c1f67e0 con 0x7fe07c1544a0 2024-04-25T15:51:53.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.583+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c1f67e0 con 0x7fe07c1544a0 2024-04-25T15:51:53.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.583+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe07c063090 0x7fe07c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.583+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe07c155300 con 0x7fe07c063090 2024-04-25T15:51:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.583+0000 7fe073fff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe07c063090 0x7fe07c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.583+0000 7fe073fff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe07c063090 0x7fe07c0634a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.583+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c155300 con 0x7fe07c063090 2024-04-25T15:51:53.598 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:53.595+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:53.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.595+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c155300 con 0x7fe07c063090 2024-04-25T15:51:53.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.599+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c155300 con 0x7fe07c063090 2024-04-25T15:51:53.603 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:53.604 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:53.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.599+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe07c063da0 0x7fe07c157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.599+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe07c155300 con 0x7fe07c063da0 2024-04-25T15:51:53.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.599+0000 7fe07bfff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe07c063da0 0x7fe07c157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.599+0000 7fe07bfff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe07c063da0 0x7fe07c157950 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.599+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c155300 con 0x7fe07c063da0 2024-04-25T15:51:53.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.615+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c155300 con 0x7fe07c063da0 2024-04-25T15:51:53.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.615+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c155300 con 0x7fe07c063da0 2024-04-25T15:51:53.621 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:53.622 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:53.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.615+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe07c02c9e0 0x7fe07c02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.619+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe07c0cbc10 con 0x7fe07c02c9e0 2024-04-25T15:51:53.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.619+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe07c02c9e0 0x7fe07c02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.619+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe07c02c9e0 0x7fe07c02cdf0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.619+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c0cbc10 con 0x7fe07c02c9e0 2024-04-25T15:51:53.633 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:53.633 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:53.633 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:51:53.633 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:53.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.635+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c0cbc10 con 0x7fe07c02c9e0 2024-04-25T15:51:53.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.639+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c0cbc10 con 0x7fe07c02c9e0 2024-04-25T15:51:53.643 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.639+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe07c02d840 0x7fe07c2096e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.639+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe07c0ce020 con 0x7fe07c02d840 2024-04-25T15:51:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.639+0000 7fe073fff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe07c02d840 0x7fe07c2096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.639+0000 7fe073fff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe07c02d840 0x7fe07c2096e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.643+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c0ce020 con 0x7fe07c02d840 2024-04-25T15:51:53.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:53.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.655+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c02dc50 con 0x7fe07c02d840 2024-04-25T15:51:53.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.659+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c02dc50 con 0x7fe07c02d840 2024-04-25T15:51:53.665 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.659+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe07c02dc50 0x7fe07c20c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.659+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe07c20f120 con 0x7fe07c02dc50 2024-04-25T15:51:53.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.659+0000 7fe07bfff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe07c02dc50 0x7fe07c20c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.663+0000 7fe07bfff640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe07c02dc50 0x7fe07c20c140 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.663+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c20f120 con 0x7fe07c02dc50 2024-04-25T15:51:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.675+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c20de60 con 0x7fe07c02dc50 2024-04-25T15:51:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.683+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c20de60 con 0x7fe07c02dc50 2024-04-25T15:51:53.687 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.683+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe07c20e580 0x7fe07c20e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:53.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.683+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe07c2124f0 con 0x7fe07c20e580 2024-04-25T15:51:53.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.683+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe07c20e580 0x7fe07c20e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:53.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.683+0000 7fe07b7fe640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe07c20e580 0x7fe07c20e990 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.683+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe07c2124f0 con 0x7fe07c20e580 2024-04-25T15:51:53.699 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:53.692+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:53.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.699+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe07c211370 con 0x7fe07c20e580 2024-04-25T15:51:53.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe078ff9640 1 -- 172.21.15.67:0/4102111940 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe07c211370 con 0x7fe07c20e580 2024-04-25T15:51:53.708 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe07c0c9970 msgr2=0x7fe07c0cc8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe07c0c9970 0x7fe07c0cc8b0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe07c1544a0 msgr2=0x7fe07c1548b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe07c1544a0 0x7fe07c1548b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe07c063090 msgr2=0x7fe07c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe07c063090 0x7fe07c0634a0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe07c063da0 msgr2=0x7fe07c157950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe07c063da0 0x7fe07c157950 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe07c02c9e0 msgr2=0x7fe07c02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe07c02c9e0 0x7fe07c02cdf0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe07c02d840 msgr2=0x7fe07c2096e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe07c02d840 0x7fe07c2096e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe07c02dc50 msgr2=0x7fe07c20c140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe07c02dc50 0x7fe07c20c140 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe07c20e580 msgr2=0x7fe07c20e990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe07c20e580 0x7fe07c20e990 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe07bfff640 1 -- 172.21.15.67:0/4102111940 reap_dead start 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe07bfff640 1 -- 172.21.15.67:0/4102111940 reap_dead start 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe07bfff640 1 -- 172.21.15.67:0/4102111940 reap_dead start 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe07bfff640 1 -- 172.21.15.67:0/4102111940 reap_dead start 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe07bfff640 1 -- 172.21.15.67:0/4102111940 reap_dead start 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe05c06f430 msgr2=0x7fe05c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe05c06f430 0x7fe05c071940 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00a8d0 tx=0x7fe06c008040 comp rx=0 tx=0).stop 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c156c70 msgr2=0x7fe07c1f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c156c70 0x7fe07c1f62a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe06400b440 tx=0x7fe06400b900 comp rx=0 tx=0).stop 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 shutdown_connections 2024-04-25T15:51:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe05c06f430 0x7fe05c071940 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 --2- 172.21.15.67:0/4102111940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c156c70 0x7fe07c1f62a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 >> 172.21.15.67:0/4102111940 conn(0x7fe07c06e700 msgr2=0x7fe07c15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 shutdown_connections 2024-04-25T15:51:53.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:53.703+0000 7fe081fa3640 1 -- 172.21.15.67:0/4102111940 wait complete. 2024-04-25T15:51:53.800 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:53.796+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:53.800 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:53.800 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:53.800 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:51:53.800 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:51:53.800 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:53.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:53.900 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:53.896+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:53.962 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.746183403000003, 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:53.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:53.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.966 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.746143758000002, 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.743236736, 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.970 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:53.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.743083799999994, 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:53.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:53.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:53.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.977 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.741568156, 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.741425508999995, 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:53.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:53.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.741322417999996, 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:53.989 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.741169667999998, 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:53.993 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 77.739221442000002, 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.996 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.711193811000001, 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:53.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:53.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.709170920000005, 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:54.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.709000371000002, 2024-04-25T15:51:54.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:54.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:54.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.706644874999995, 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.007 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:54.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:54.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:54.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.706610162999993, 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.706589354000002, 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:54.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.019 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.629702307000002, 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:54.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.629678224999999, 2024-04-25T15:51:54.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:54.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:54.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.629668528000003, 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:54.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 61.623773591000003, 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:54.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:54.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.036 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.034334934, 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:54.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:54.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:54.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:54.041 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:54.042 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:54.043 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:54.039+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:54.143 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:54.140+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:54.201 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:54.201 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:54.201 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:54.201 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:54.218 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:54.259 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:54.256+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:54.344 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:54.339+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:54.414 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:54.414 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:54.431 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:54.445 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:54.439+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:54.546 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:54.540+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:54.585 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.373252291, 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:54.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.590 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.370637493999993, 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:54.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:54.594 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.370138596999993, 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:54.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.369049351000001, 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:54.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.368428855999994, 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.601 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:54.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:54.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.366034630000001, 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:54.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:54.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 72.336914226999994, 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.612 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 72.332169492999995, 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:54.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 72.328641992000001, 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.619 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 72.328509510999993, 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:54.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.707570052999998, 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.623 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:54.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:54.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:54.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.696134921000002, 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.628 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.695737485000002, 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:54.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.695627743999999, 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:54.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.695126999000003, 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:54.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.694751013000001, 2024-04-25T15:51:54.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.656152830000003, 2024-04-25T15:51:54.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.656052571000004, 2024-04-25T15:51:54.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:54.655 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.653947656, 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.653572080000004, 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:54.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:54.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:54.662 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:54.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:54.664 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:54.663+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:54.764 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:54.759+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:54.862 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:54.862 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:54.862 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:54.862 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:54.865 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:54.860+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:54.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:54.966 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:54.963+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:55.067 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:55.060+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:55.074 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:55.075 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:55.075 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:55.075 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:55.075 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:55.092 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:51:55.168 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:55.163+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:55.269 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:55.264+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:55.291 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:55.291 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040983209000004, 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.292 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:51:55.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:51:55.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040952648000001, 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.298 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:51:55.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:51:55.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:51:55.304 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040944816999996, 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.305 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:51:55.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:51:55.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:51:55.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.311 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040935888000007, 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.312 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:55.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:51:55.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.318 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040929656000003, 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:51:55.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:51:55.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.325 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040914246, 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:51:55.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:51:55.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040901414000004, 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.040896200000006, 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:55.337 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:51:55.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.955286082999997, 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:51:55.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.402756089999997, 2024-04-25T15:51:55.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:51:55.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:51:55.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.354 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.401363252000003, 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:55.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:51:55.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.363343125999997, 2024-04-25T15:51:55.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:51:55.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:51:55.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.366 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.363319799999999, 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:51:55.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:51:55.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.372 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.363279259999999, 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:51:55.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:51:55.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.3632706, 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:51:55.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:51:55.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:51:55.384 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.363228927000002, 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:51:55.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:55.389 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.363219669999999, 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:51:55.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.359082244, 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:55.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:51:55.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.359027666999999, 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.399 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:51:55.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.403 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.358994459000002, 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:51:55.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:55.408 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:55.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:51:55.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:51:55.411 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:55.407+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:55.511 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:55.508+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.583+0000 7ff338920640 1 Processor -- start 2024-04-25T15:51:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.583+0000 7ff338920640 1 -- start start 2024-04-25T15:51:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff338920640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff3340caf60 0x7ff33415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff338920640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff33415d790 0x7ff33406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff338920640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff334073bf0 0x7ff33406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff338920640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff3340cab60 con 0x7ff33415d790 2024-04-25T15:51:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff338920640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff334159620 con 0x7ff3340caf60 2024-04-25T15:51:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff338920640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3340c77a0 con 0x7ff334073bf0 2024-04-25T15:51:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff33415d790 0x7ff33406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff33415d790 0x7ff33406d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47786/0 (socket says 172.21.15.67:47786) 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 -- 172.21.15.67:0/1943454786 learned_addr learned my addr 172.21.15.67:0/1943454786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff332d9d640 1 --2- 172.21.15.67:0/1943454786 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff334073bf0 0x7ff33406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff33259c640 1 --2- 172.21.15.67:0/1943454786 >> v2:172.21.15.176:3300/0 conn(0x7ff3340caf60 0x7ff33415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 -- 172.21.15.67:0/1943454786 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff334073bf0 msgr2=0x7ff33406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 --2- 172.21.15.67:0/1943454786 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff334073bf0 0x7ff33406dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 -- 172.21.15.67:0/1943454786 >> v2:172.21.15.176:3300/0 conn(0x7ff3340caf60 msgr2=0x7ff33415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 --2- 172.21.15.67:0/1943454786 >> v2:172.21.15.176:3300/0 conn(0x7ff3340caf60 0x7ff33415d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.587+0000 7ff331d9b640 1 -- 172.21.15.67:0/1943454786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3340c7a60 con 0x7ff33415d790 2024-04-25T15:51:55.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff331d9b640 1 --2- 172.21.15.67:0/1943454786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff33415d790 0x7ff33406d750 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff3340720b0 tx=0x7ff31c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58a8c42bfd549b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff33159a640 1 -- 172.21.15.67:0/1943454786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0b0070 con 0x7ff33415d790 2024-04-25T15:51:55.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff33159a640 1 -- 172.21.15.67:0/1943454786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff31c0ab600 con 0x7ff33415d790 2024-04-25T15:51:55.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff33159a640 1 -- 172.21.15.67:0/1943454786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0ab9d0 con 0x7ff33415d790 2024-04-25T15:51:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 -- 172.21.15.67:0/1943454786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff33415d790 msgr2=0x7ff33406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 --2- 172.21.15.67:0/1943454786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff33415d790 0x7ff33406d750 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff3340720b0 tx=0x7ff31c01c520 comp rx=0 tx=0).stop 2024-04-25T15:51:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 -- 172.21.15.67:0/1943454786 shutdown_connections 2024-04-25T15:51:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 --2- 172.21.15.67:0/1943454786 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff334073bf0 0x7ff33406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 --2- 172.21.15.67:0/1943454786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff33415d790 0x7ff33406d750 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 --2- 172.21.15.67:0/1943454786 >> v2:172.21.15.176:3300/0 conn(0x7ff3340caf60 0x7ff33415d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 -- 172.21.15.67:0/1943454786 >> 172.21.15.67:0/1943454786 conn(0x7ff334070db0 msgr2=0x7ff3340ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 -- 172.21.15.67:0/1943454786 shutdown_connections 2024-04-25T15:51:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.591+0000 7ff338920640 1 -- 172.21.15.67:0/1943454786 wait complete. 2024-04-25T15:51:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 Processor -- start 2024-04-25T15:51:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 -- start start 2024-04-25T15:51:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff334073bf0 0x7ff3341f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3340caf60 0x7ff3340c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff33415d790 0x7ff33406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3340733d0 con 0x7ff3340caf60 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff33415a600 con 0x7ff334073bf0 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff334073550 con 0x7ff33415d790 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3340caf60 0x7ff3340c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3340caf60 0x7ff3340c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47798/0 (socket says 172.21.15.67:47798) 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 -- 172.21.15.67:0/361925826 learned_addr learned my addr 172.21.15.67:0/361925826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff33259c640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff334073bf0 0x7ff3341f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff332d9d640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff33415d790 0x7ff33406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 -- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff33415d790 msgr2=0x7ff33406c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff33415d790 0x7ff33406c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 -- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff334073bf0 msgr2=0x7ff3341f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff334073bf0 0x7ff3341f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 -- 172.21.15.67:0/361925826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33414aa20 con 0x7ff3340caf60 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff331d9b640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3340caf60 0x7ff3340c51c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0ae040 tx=0x7ff31c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff32affd640 1 -- 172.21.15.67:0/361925826 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0ba070 con 0x7ff3340caf60 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff32affd640 1 -- 172.21.15.67:0/361925826 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff31c0b4620 con 0x7ff3340caf60 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3341f7ae0 con 0x7ff3340caf60 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff32affd640 1 -- 172.21.15.67:0/361925826 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0b0040 con 0x7ff3340caf60 2024-04-25T15:51:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33414e560 con 0x7ff3340caf60 2024-04-25T15:51:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.595+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33414a770 con 0x7ff3340caf60 2024-04-25T15:51:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.599+0000 7ff32affd640 1 -- 172.21.15.67:0/361925826 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff31c0bd070 con 0x7ff3340caf60 2024-04-25T15:51:55.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.599+0000 7ff32affd640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff31406f430 0x7ff314071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.599+0000 7ff32affd640 1 -- 172.21.15.67:0/361925826 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5825+0+0 (secure 0 0 0) 0x7ff31c1284b0 con 0x7ff3340caf60 2024-04-25T15:51:55.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.599+0000 7ff33259c640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff31406f430 0x7ff314071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.599+0000 7ff33259c640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff31406f430 0x7ff314071940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff324007440 tx=0x7ff32400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.605 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:55.606 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:55.606 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:55.606 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:55.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.603+0000 7ff32affd640 1 -- 172.21.15.67:0/361925826 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff31c016110 con 0x7ff3340caf60 2024-04-25T15:51:55.612 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:55.608+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:55.623 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:51:55.713 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:55.707+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:55.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:51:55.813 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:55.807+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:55.819 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:55.819 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:55.819 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:55.819 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:55.819 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:55.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.827+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff33406a480 con 0x7ff3340caf60 2024-04-25T15:51:55.837 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:51:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f676ae06640 1 Processor -- start 2024-04-25T15:51:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f676ae06640 1 -- start start 2024-04-25T15:51:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f676ae06640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6764073bf0 0x7f676406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f676ae06640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f676406dc90 0x7f6764163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f676ae06640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67640c77a0 con 0x7f67640caf60 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f676ae06640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f67640cab60 con 0x7f6764073bf0 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f676ae06640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6764159620 con 0x7f676406dc90 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f6763fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6764073bf0 0x7f676406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f6763fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6764073bf0 0x7f676406d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:58718/0 (socket says 172.21.15.67:58718) 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f6763fff640 1 -- 172.21.15.67:0/322457339 learned_addr learned my addr 172.21.15.67:0/322457339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f67693a3640 1 --2- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f676406dc90 0x7f6764163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f6763fff640 1 -- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f676406dc90 msgr2=0x7f6764163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f6763fff640 1 --2- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f676406dc90 0x7f6764163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f6763fff640 1 -- 172.21.15.67:0/322457339 >> v1:172.21.15.176:6789/0 conn(0x7f67640caf60 legacy=0x7f676415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f6763fff640 1 -- 172.21.15.67:0/322457339 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67640c7a60 con 0x7f6764073bf0 2024-04-25T15:51:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.879+0000 7f67693a3640 1 --2- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f676406dc90 0x7f6764163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:51:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.883+0000 7ff32affd640 1 -- 172.21.15.67:0/361925826 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v39) ==== 97+0+0 (secure 0 0 0) 0x7ff31c0f5320 con 0x7ff3340caf60 2024-04-25T15:51:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:51:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.883+0000 7f6763fff640 1 --2- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6764073bf0 0x7f676406d750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6750009870 tx=0x7f675001c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4997698bc106043 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff31406f430 msgr2=0x7ff314071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff31406f430 0x7ff314071940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff324007440 tx=0x7ff32400b040 comp rx=0 tx=0).stop 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3340caf60 msgr2=0x7ff3340c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3340caf60 0x7ff3340c51c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0ae040 tx=0x7ff31c0192c0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 shutdown_connections 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff31406f430 0x7ff314071940 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff33415d790 0x7ff33406c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3340caf60 0x7ff3340c51c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 --2- 172.21.15.67:0/361925826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff334073bf0 0x7ff3341f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 >> 172.21.15.67:0/361925826 conn(0x7ff334070db0 msgr2=0x7ff334153740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 shutdown_connections 2024-04-25T15:51:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7ff338920640 1 -- 172.21.15.67:0/361925826 wait complete. 2024-04-25T15:51:55.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f67637fe640 1 -- 172.21.15.67:0/322457339 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67500b2070 con 0x7f6764073bf0 2024-04-25T15:51:55.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f67637fe640 1 -- 172.21.15.67:0/322457339 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67500a9070 con 0x7f6764073bf0 2024-04-25T15:51:55.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f67637fe640 1 -- 172.21.15.67:0/322457339 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67500ad880 con 0x7f6764073bf0 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f676ae06640 1 -- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6764073bf0 msgr2=0x7f676406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f676ae06640 1 --2- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6764073bf0 0x7f676406d750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6750009870 tx=0x7f675001c560 comp rx=0 tx=0).stop 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f676ae06640 1 -- 172.21.15.67:0/322457339 shutdown_connections 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f676ae06640 1 --2- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f676406dc90 0x7f6764163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f676ae06640 1 --2- 172.21.15.67:0/322457339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6764073bf0 0x7f676406d750 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f676ae06640 1 -- 172.21.15.67:0/322457339 >> 172.21.15.67:0/322457339 conn(0x7f6764070db0 msgr2=0x7f67640ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.887+0000 7f676ae06640 1 -- 172.21.15.67:0/322457339 shutdown_connections 2024-04-25T15:51:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 -- 172.21.15.67:0/322457339 wait complete. 2024-04-25T15:51:55.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 Processor -- start 2024-04-25T15:51:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 -- start start 2024-04-25T15:51:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f676406dc90 0x7f67641569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764073bf0 0x7f67640cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67640caf60 0x7f676414cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f676415a570 con 0x7f6764073bf0 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f676415a3f0 con 0x7f676406dc90 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f676ae06640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6764166c20 con 0x7f67640caf60 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f6763fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764073bf0 0x7f67640cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f6763fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764073bf0 0x7f67640cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47804/0 (socket says 172.21.15.67:47804) 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f6763fff640 1 -- 172.21.15.67:0/1565086414 learned_addr learned my addr 172.21.15.67:0/1565086414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f67693a3640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67640caf60 0x7f676414cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f676406dc90 0x7f67641569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f6763fff640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67640caf60 msgr2=0x7f676414cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.891+0000 7f6763fff640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67640caf60 0x7f676414cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f6763fff640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f676406dc90 msgr2=0x7f67641569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f6763fff640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f676406dc90 0x7f67641569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f6763fff640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67640cdc40 con 0x7f6764073bf0 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f6763fff640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764073bf0 0x7f67640cd700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f67500b0040 tx=0x7f6750019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67500bc070 con 0x7f6764073bf0 2024-04-25T15:51:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67500add80 con 0x7f6764073bf0 2024-04-25T15:51:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67641f3f70 con 0x7f6764073bf0 2024-04-25T15:51:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67500b2040 con 0x7f6764073bf0 2024-04-25T15:51:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67640cc940 con 0x7f6764073bf0 2024-04-25T15:51:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f676406a480 con 0x7f6764073bf0 2024-04-25T15:51:55.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f67500bf070 con 0x7f6764073bf0 2024-04-25T15:51:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67617fa640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f674806f430 0x7f6748071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5825+0+0 (secure 0 0 0) 0x7f6750128050 con 0x7f6764073bf0 2024-04-25T15:51:55.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v39) ==== 54+0+16 (secure 0 0 0) 0x7f6750016150 con 0x7f6764073bf0 2024-04-25T15:51:55.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f674806f430 0x7f6748071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6764163f70 0x7f67640ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67640c8a40 con 0x7f6764163f70 2024-04-25T15:51:55.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.895+0000 7f67693a3640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6764163f70 0x7f67640ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.899+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f674806f430 0x7f6748071940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6758007fd0 tx=0x7f6758009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.899+0000 7f67693a3640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6764163f70 0x7f67640ccb10 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.899+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67640c8a40 con 0x7f6764163f70 2024-04-25T15:51:55.914 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:55.911+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:55.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.915+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67640cad40 con 0x7f6764163f70 2024-04-25T15:51:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.919+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67640cad40 con 0x7f6764163f70 2024-04-25T15:51:55.922 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:55.923 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:55.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.919+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f67640cd050 0x7f676414bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.919+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f67640cbeb0 con 0x7f67640cd050 2024-04-25T15:51:55.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.919+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f67640cd050 0x7f676414bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.919+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f67640cd050 0x7f676414bed0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.919+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67640cbeb0 con 0x7f67640cd050 2024-04-25T15:51:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.935+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f676406d7f0 con 0x7f67640cd050 2024-04-25T15:51:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.935+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f676406d7f0 con 0x7f67640cd050 2024-04-25T15:51:55.941 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.935+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f67641f77c0 0x7f67641f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.935+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f67641f8110 con 0x7f67641f77c0 2024-04-25T15:51:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.935+0000 7f6763fff640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f67641f77c0 0x7f67641f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.935+0000 7f6763fff640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f67641f77c0 0x7f67641f7bd0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.939+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67641f8110 con 0x7f67641f77c0 2024-04-25T15:51:55.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.951+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67640c51b0 con 0x7f67641f77c0 2024-04-25T15:51:55.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.955+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67640c51b0 con 0x7f67641f77c0 2024-04-25T15:51:55.959 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:55.959 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:55.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.955+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6764063960 0x7f6764152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.955+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6764063d70 con 0x7f6764063960 2024-04-25T15:51:55.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.955+0000 7f67693a3640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6764063960 0x7f6764152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.955+0000 7f67693a3640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6764063960 0x7f6764152070 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.955+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6764063d70 con 0x7f6764063960 2024-04-25T15:51:55.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.971+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67641525b0 con 0x7f6764063960 2024-04-25T15:51:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.971+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67641525b0 con 0x7f6764063960 2024-04-25T15:51:55.976 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.971+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f676402c9e0 0x7f676402cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.971+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f67641525b0 con 0x7f676402c9e0 2024-04-25T15:51:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.971+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f676402c9e0 0x7f676402cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.975+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f676402c9e0 0x7f676402cdf0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:55.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.975+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67641525b0 con 0x7f676402c9e0 2024-04-25T15:51:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.991+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f67641525b0 con 0x7f676402c9e0 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 79.782499610000002, 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:55.996 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:51:55.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.000 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 79.782470345999997, 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:51:56.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 79.782461057999996, 2024-04-25T15:51:56.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:56.010 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 79.782453481999994, 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:56.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 79.782444849000001, 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:51:56.015 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:51:56.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 79.782395015999995, 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:51:56.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 79.782387138000004, 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:51:56.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.745959131000006, 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:51:56.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.745928466999999, 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:51:56.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:51:56.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:56.037 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.745915937000007, 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:51:56.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.745905304999994, 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:51:56.042 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.745885759999993, 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:51:56.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:51:56.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.745876867999996, 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:51:56.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.742314252, 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:51:56.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:51:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.742193639999996, 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 58.107195619999999, 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.064 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:51:56.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:51:56.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.068 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.066626221999996, 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:51:56.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.066620714000003, 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:51:56.073 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:51:56.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:51:56.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.077 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.064206200000001, 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:51:56.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:56.081 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.063244314000002, 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:56.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:51:56.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:56.085 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:56.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.995+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f67641525b0 con 0x7f676402c9e0 2024-04-25T15:51:56.086 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:56.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:51:56.087 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:56.080+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.995+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f676402d840 0x7f676420a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.995+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f676402de40 con 0x7f676402d840 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.995+0000 7f6763fff640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f676402d840 0x7f676420a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.995+0000 7f6763fff640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f676402d840 0x7f676420a640 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:55.995+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f676402de40 con 0x7f676402d840 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.011+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f676402de10 con 0x7f676402d840 2024-04-25T15:51:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.015+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f676402de10 con 0x7f676402d840 2024-04-25T15:51:56.089 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.015+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f676420b090 0x7f676420d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.015+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6764210430 con 0x7f676420b090 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.015+0000 7f67693a3640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f676420b090 0x7f676420d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.015+0000 7f67693a3640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f676420b090 0x7f676420d4b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.015+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6764210430 con 0x7f676420b090 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.031+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6764210460 con 0x7f676420b090 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.035+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6764210460 con 0x7f676420b090 2024-04-25T15:51:56.090 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:56.091 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:56.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.035+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f676420f710 0x7f676420fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:56.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.035+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6764213880 con 0x7f676420f710 2024-04-25T15:51:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.035+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f676420f710 0x7f676420fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.035+0000 7f6768ba2640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f676420f710 0x7f676420fb60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.039+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6764213880 con 0x7f676420f710 2024-04-25T15:51:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.051+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f6764212700 con 0x7f676420f710 2024-04-25T15:51:56.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f67617fa640 1 -- 172.21.15.67:0/1565086414 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6764212700 con 0x7f676420f710 2024-04-25T15:51:56.092 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6764163f70 msgr2=0x7f67640ccb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6764163f70 0x7f67640ccb10 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f67640cd050 msgr2=0x7f676414bed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f67640cd050 0x7f676414bed0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f67641f77c0 msgr2=0x7f67641f7bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f67641f77c0 0x7f67641f7bd0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6764063960 msgr2=0x7f6764152070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6764063960 0x7f6764152070 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f67693a3640 1 -- 172.21.15.67:0/1565086414 reap_dead start 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f676402c9e0 msgr2=0x7f676402cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f676402c9e0 0x7f676402cdf0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f67693a3640 1 -- 172.21.15.67:0/1565086414 reap_dead start 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f676402d840 msgr2=0x7f676420a640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f676402d840 0x7f676420a640 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f676420b090 msgr2=0x7f676420d4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f676420b090 0x7f676420d4b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f676420f710 msgr2=0x7f676420fb60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f676420f710 0x7f676420fb60 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f674806f430 msgr2=0x7f6748071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f674806f430 0x7f6748071940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6758007fd0 tx=0x7f6758009040 comp rx=0 tx=0).stop 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764073bf0 msgr2=0x7f67640cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 --2- 172.21.15.67:0/1565086414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764073bf0 0x7f67640cd700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f67500b0040 tx=0x7f6750019300 comp rx=0 tx=0).stop 2024-04-25T15:51:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f67693a3640 1 -- 172.21.15.67:0/1565086414 reap_dead start 2024-04-25T15:51:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 shutdown_connections 2024-04-25T15:51:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 >> 172.21.15.67:0/1565086414 conn(0x7f6764070db0 msgr2=0x7f6764159830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 shutdown_connections 2024-04-25T15:51:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:56.055+0000 7f676ae06640 1 -- 172.21.15.67:0/1565086414 wait complete. 2024-04-25T15:51:56.187 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:56.184+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:56.284 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:56.284 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:56.288 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:56.284+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:56.302 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:51:56.389 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:56.383+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:56.490 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:56.484+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:56.494 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:56.494 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:56.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:51:56.599 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:56.596+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:56.691 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:56.687+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:56.704 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:56.704 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:56.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:51:56.791 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:56.787+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:56.877 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:56.877 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:56.892 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:56.887+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:56.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:51:56.993 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:56.988+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:57.074 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:57.074 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:57.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:51:57.093 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:57.088+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:57.193 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:57.187+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:57.230 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:57.230 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:57.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:51:57.294 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:57.288+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:57.395 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:57.388+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:57.443 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:57.443 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:57.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:51:57.510 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:57.507+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:57.597 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:57.592+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:57.614 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:57.614 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:57.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:51:57.698 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:51:57.696+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:51:57.799 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:57.795+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:57.824 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:57.824 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:57.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:51:57.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:51:57.926 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:57.923+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:58.001 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:57.996+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:58.033 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:58.033 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:58.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:51:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c3266b640 1 Processor -- start 2024-04-25T15:51:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c3266b640 1 -- start start 2024-04-25T15:51:58.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c3266b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0c6c10 0x7f4c2c156c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c3266b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c2c0c9bc0 0x7f4c2c0d2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c3266b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2c15cc60 con 0x7f4c2c0d2540 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c3266b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2c15cf40 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c3266b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2c15d220 con 0x7f4c2c0c6c10 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c30c08640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c2c0d2540 0x7f4c2c0c8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54690/0 (socket says 172.21.15.67:54690) 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c30c08640 1 -- 172.21.15.67:0/2320515417 learned_addr learned my addr 172.21.15.67:0/2320515417 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c23fff640 1 --2- 172.21.15.67:0/2320515417 >> v2:172.21.15.67:3300/0 conn(0x7f4c2c0c9bc0 0x7f4c2c0d2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/2320515417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0c6c10 0x7f4c2c156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c23fff640 1 -- 172.21.15.67:0/2320515417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0c6c10 msgr2=0x7f4c2c156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c23fff640 1 --2- 172.21.15.67:0/2320515417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0c6c10 0x7f4c2c156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c23fff640 1 -- 172.21.15.67:0/2320515417 >> v1:172.21.15.67:6789/0 conn(0x7f4c2c0d2540 legacy=0x7f4c2c0c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.063+0000 7f4c23fff640 1 -- 172.21.15.67:0/2320515417 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c15d500 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c23fff640 1 --2- 172.21.15.67:0/2320515417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c0d2000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4c140097c0 tx=0x7f4c1401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30525e131df39f5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/2320515417 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140b5070 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/2320515417 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c140a8990 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c2b7fe640 1 -- 172.21.15.67:0/2320515417 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140a8d60 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 -- 172.21.15.67:0/2320515417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 msgr2=0x7f4c2c0d2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 --2- 172.21.15.67:0/2320515417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c0d2000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4c140097c0 tx=0x7f4c1401ec90 comp rx=0 tx=0).stop 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 -- 172.21.15.67:0/2320515417 shutdown_connections 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 --2- 172.21.15.67:0/2320515417 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c0d2000 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 --2- 172.21.15.67:0/2320515417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0c6c10 0x7f4c2c156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 -- 172.21.15.67:0/2320515417 >> 172.21.15.67:0/2320515417 conn(0x7f4c2c068760 msgr2=0x7f4c2c068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 -- 172.21.15.67:0/2320515417 shutdown_connections 2024-04-25T15:51:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.067+0000 7f4c3266b640 1 -- 172.21.15.67:0/2320515417 wait complete. 2024-04-25T15:51:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 Processor -- start 2024-04-25T15:51:58.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 -- start start 2024-04-25T15:51:58.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c2c0c6c10 0x7f4c2c159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0d2540 0x7f4c2c0b6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c15c8b0 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c0c7ce0 con 0x7f4c2c0c6c10 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c3266b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2c1f2950 con 0x7f4c2c0d2540 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c23fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47836/0 (socket says 172.21.15.67:47836) 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c23fff640 1 -- 172.21.15.67:0/67838214 learned_addr learned my addr 172.21.15.67:0/67838214 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c30c08640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0d2540 0x7f4c2c0b6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c2c0c6c10 0x7f4c2c159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.071+0000 7f4c23fff640 1 -- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0d2540 msgr2=0x7f4c2c0b6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c23fff640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0d2540 0x7f4c2c0b6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c23fff640 1 -- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c2c0c6c10 msgr2=0x7f4c2c159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c23fff640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c2c0c6c10 0x7f4c2c159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c23fff640 1 -- 172.21.15.67:0/67838214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c0b75b0 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c23fff640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c156e30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4c140b3040 tx=0x7f4c1401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c297fa640 1 -- 172.21.15.67:0/67838214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140bf070 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c297fa640 1 -- 172.21.15.67:0/67838214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c140b07a0 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2c0b7890 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c297fa640 1 -- 172.21.15.67:0/67838214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140b5040 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c2c160890 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c2c06a480 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c297fa640 1 -- 172.21.15.67:0/67838214 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4c140c2070 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c297fa640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4c0c06f400 0x7f4c0c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c297fa640 1 -- 172.21.15.67:0/67838214 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5825+0+0 (secure 0 0 0) 0x7f4c1412ba70 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.075+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4c0c06f400 0x7f4c0c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.079+0000 7f4c2bfff640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4c0c06f400 0x7f4c0c071910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00b930 tx=0x7f4c1c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.083+0000 7f4c297fa640 1 -- 172.21.15.67:0/67838214 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4c140187f0 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:51:58.142 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:58.139+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:58.202 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:58.199+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:58.214 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:58.215 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:58.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:51:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f589e561640 1 Processor -- start 2024-04-25T15:51:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f589e561640 1 -- start start 2024-04-25T15:51:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f589e561640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 0x7f5898154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f589e561640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58980749d0 con 0x7f589815e4a0 2024-04-25T15:51:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f589815e4a0 0x7f58980c7fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54692/0 (socket says 172.21.15.67:54692) 2024-04-25T15:51:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f589e561640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589806e510 con 0x7f589815e8b0 2024-04-25T15:51:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 -- 172.21.15.67:0/2713329359 learned_addr learned my addr 172.21.15.67:0/2713329359 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f589e561640 1 -- 172.21.15.67:0/2713329359 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f589815dca0 con 0x7f589806c640 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5897fff640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 0x7f5898154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395400885 0 0) 0x7f58980749d0 con 0x7f589815e4a0 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5880003530 con 0x7f589815e4a0 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1892332314 0 0) 0x7f5880003530 con 0x7f589815e4a0 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 msgr2=0x7f5898154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 0x7f5898154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 >> v1:172.21.15.176:6789/0 conn(0x7f589815e8b0 legacy=0x7f5898164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589815df80 con 0x7f589815e4a0 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f588c001680 con 0x7f589815e4a0 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 >> v1:172.21.15.67:6789/0 conn(0x7f589815e4a0 legacy=0x7f58980c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 0x7f58800039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f5880004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5880004d50 0x7f5880005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f589815dca0 con 0x7f5880003f70 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f588000bc10 con 0x7f5880004d50 2024-04-25T15:51:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5880010220 con 0x7f589806c640 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f5880004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f5897fff640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 0x7f58800039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f589cafe640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5880004d50 0x7f5880005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 -- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 msgr2=0x7f58800039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f589806c640 0x7f58800039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 -- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5880004d50 msgr2=0x7f5880005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5880004d50 0x7f5880005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.239+0000 7f58977fe640 1 -- 172.21.15.67:0/2713329359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5880003530 con 0x7f5880003f70 2024-04-25T15:51:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.243+0000 7f58977fe640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f5880004810 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f58980cacb0 tx=0x7f588c01cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdc060baeb3abca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.243+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c007d40 con 0x7f5880003f70 2024-04-25T15:51:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.243+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c0a9070 con 0x7f5880003f70 2024-04-25T15:51:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.243+0000 7f5896ffd640 1 -- 172.21.15.67:0/2713329359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c005d20 con 0x7f5880003f70 2024-04-25T15:51:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 -- 172.21.15.67:0/2713329359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 msgr2=0x7f5880004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 --2- 172.21.15.67:0/2713329359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f5880004810 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f58980cacb0 tx=0x7f588c01cb90 comp rx=0 tx=0).stop 2024-04-25T15:51:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589cafe640 1 -- 172.21.15.67:0/2713329359 reap_dead start 2024-04-25T15:51:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 -- 172.21.15.67:0/2713329359 shutdown_connections 2024-04-25T15:51:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 -- 172.21.15.67:0/2713329359 >> 172.21.15.67:0/2713329359 conn(0x7f5898070db0 msgr2=0x7f58980ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 -- 172.21.15.67:0/2713329359 shutdown_connections 2024-04-25T15:51:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 -- 172.21.15.67:0/2713329359 wait complete. 2024-04-25T15:51:58.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 Processor -- start 2024-04-25T15:51:58.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.247+0000 7f589e561640 1 -- start start 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f589815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f589815a8f0 0x7f58980cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58980cdb70 0x7f58981f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58980cad60 con 0x7f5880003f70 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f58980c86d0 con 0x7f589815a8f0 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f589815d430 con 0x7f58980cdb70 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f589815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f589815ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47852/0 (socket says 172.21.15.67:47852) 2024-04-25T15:51:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 -- 172.21.15.67:0/513679653 learned_addr learned my addr 172.21.15.67:0/513679653 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f589815a8f0 0x7f58980cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589cafe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58980cdb70 0x7f58981f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58980cdb70 msgr2=0x7f58981f1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58980cdb70 0x7f58981f1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f589815a8f0 msgr2=0x7f58980cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f589815a8f0 0x7f58980cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58981f6470 con 0x7f5880003f70 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f589815ca20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f58880115e0 tx=0x7f5888011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888015870 con 0x7f5880003f70 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589815cf60 con 0x7f5880003f70 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58981f6190 con 0x7f5880003f70 2024-04-25T15:51:58.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5888015a10 con 0x7f5880003f70 2024-04-25T15:51:58.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888015de0 con 0x7f5880003f70 2024-04-25T15:51:58.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f589806a480 con 0x7f5880003f70 2024-04-25T15:51:58.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.251+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f588800f480 con 0x7f5880003f70 2024-04-25T15:51:58.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f5894ff9640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f588406f400 0x7f5884071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5825+0+0 (secure 0 0 0) 0x7f588808fad0 con 0x7f5880003f70 2024-04-25T15:51:58.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v39) ==== 54+0+16 (secure 0 0 0) 0x7f5888019030 con 0x7f5880003f70 2024-04-25T15:51:58.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f588406f400 0x7f5884071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f58980cdf80 0x7f589815c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f589815c690 con 0x7f58980cdf80 2024-04-25T15:51:58.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f589cafe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f58980cdf80 0x7f589815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f589cafe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f58980cdf80 0x7f589815c150 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f588406f400 0x7f5884071910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f588c0ab000 tx=0x7f588c0b1040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.255+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58900019e0 con 0x7f58980cdf80 2024-04-25T15:51:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.263+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f4c2c0c8da0 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.271+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f58980cc240 con 0x7f58980cdf80 2024-04-25T15:51:58.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.271+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f58980cc240 con 0x7f58980cdf80 2024-04-25T15:51:58.276 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:51:58.277 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:51:58.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.271+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f58981f2500 0x7f58981f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.271+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f58981f2e50 con 0x7f58981f2500 2024-04-25T15:51:58.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.271+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f58981f2500 0x7f58981f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.271+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f58981f2500 0x7f58981f2910 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.275+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58981f2e50 con 0x7f58981f2500 2024-04-25T15:51:58.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.287+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f58980c5a60 con 0x7f58981f2500 2024-04-25T15:51:58.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f58980c5a60 con 0x7f58981f2500 2024-04-25T15:51:58.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 0x7f58980639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5898063ef0 con 0x7f58980635a0 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 0x7f58980639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f58977fe640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 msgr2=0x7f58980639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 23 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f58977fe640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 msgr2=0x7f58980639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 0x7f58980639b0 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:51:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.291+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 0x7f58980639b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:51:58.303 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:58.300+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:58.329 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:51:58.329 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:51:58.329 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-25T15:51:58.329 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-04-25T15:51:58.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0 2024-04-25T15:51:58.384 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:58.384 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:58.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:51:58.404 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:58.400+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7fa005c67640 1 Processor -- start 2024-04-25T15:51:58.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7fa005c67640 1 -- start start 2024-04-25T15:51:58.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7fa005c67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa000160800 0x7fa00006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7fa005c67640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0000c96e0 0x7fa000167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7fa005c67640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa00006d8e0 con 0x7fa0000c92d0 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7fa005c67640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa00006dbc0 con 0x7fa0000c96e0 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7fa005c67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa000073c70 con 0x7fa000160800 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffeffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa0000c92d0 0x7fa000072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54700/0 (socket says 172.21.15.67:54700) 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffeffd640 1 -- 172.21.15.67:0/2439318210 learned_addr learned my addr 172.21.15.67:0/2439318210 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffffff640 1 --2- 172.21.15.67:0/2439318210 >> v2:172.21.15.67:3300/0 conn(0x7fa0000c96e0 0x7fa000167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9fff7fe640 1 --2- 172.21.15.67:0/2439318210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa000160800 0x7fa00006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffe7fc640 1 -- 172.21.15.67:0/2439318210 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737038260 0 0) 0x7fa00006d8e0 con 0x7fa0000c92d0 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffe7fc640 1 -- 172.21.15.67:0/2439318210 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fe8003530 con 0x7fa0000c92d0 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffffff640 1 -- 172.21.15.67:0/2439318210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa000160800 msgr2=0x7fa00006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffffff640 1 --2- 172.21.15.67:0/2439318210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa000160800 0x7fa00006d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffffff640 1 -- 172.21.15.67:0/2439318210 >> v1:172.21.15.67:6789/0 conn(0x7fa0000c92d0 legacy=0x7fa000072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.463+0000 7f9ffffff640 1 -- 172.21.15.67:0/2439318210 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa000073f50 con 0x7fa0000c96e0 2024-04-25T15:51:58.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7f9ffffff640 1 --2- 172.21.15.67:0/2439318210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0000c96e0 0x7fa000167180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa00006dd40 tx=0x7f9ff801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2048af2886a4ff9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7f9ffe7fc640 1 -- 172.21.15.67:0/2439318210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80b2070 con 0x7fa0000c96e0 2024-04-25T15:51:58.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7f9ffe7fc640 1 -- 172.21.15.67:0/2439318210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ff80a5870 con 0x7fa0000c96e0 2024-04-25T15:51:58.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7f9ffe7fc640 1 -- 172.21.15.67:0/2439318210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80a5c20 con 0x7fa0000c96e0 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 -- 172.21.15.67:0/2439318210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0000c96e0 msgr2=0x7fa000167180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 --2- 172.21.15.67:0/2439318210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0000c96e0 0x7fa000167180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa00006dd40 tx=0x7f9ff801c8c0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 -- 172.21.15.67:0/2439318210 shutdown_connections 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 --2- 172.21.15.67:0/2439318210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0000c96e0 0x7fa000167180 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 --2- 172.21.15.67:0/2439318210 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa000160800 0x7fa00006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 -- 172.21.15.67:0/2439318210 >> 172.21.15.67:0/2439318210 conn(0x7fa000070db0 msgr2=0x7fa0000ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 -- 172.21.15.67:0/2439318210 shutdown_connections 2024-04-25T15:51:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 -- 172.21.15.67:0/2439318210 wait complete. 2024-04-25T15:51:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.467+0000 7fa005c67640 1 Processor -- start 2024-04-25T15:51:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 -- start start 2024-04-25T15:51:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0000c92d0 0x7fa000158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0000c96e0 0x7fa00015c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa000160800 0x7fa0000c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa000160c10 con 0x7fa000160800 2024-04-25T15:51:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa00006d570 con 0x7fa0000c96e0 2024-04-25T15:51:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa000160d90 con 0x7fa0000c92d0 2024-04-25T15:51:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa000160800 0x7fa0000c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa000160800 0x7fa0000c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47866/0 (socket says 172.21.15.67:47866) 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 -- 172.21.15.67:0/4205673091 learned_addr learned my addr 172.21.15.67:0/4205673091 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9fff7fe640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0000c92d0 0x7fa000158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffeffd640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0000c96e0 0x7fa00015c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 -- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0000c92d0 msgr2=0x7fa000158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0000c92d0 0x7fa000158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 -- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0000c96e0 msgr2=0x7fa00015c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0000c96e0 0x7fa00015c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 -- 172.21.15.67:0/4205673091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00015cc40 con 0x7fa000160800 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9fff7fe640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0000c92d0 0x7fa000158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9ffffff640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa000160800 0x7fa0000c7700 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80b0040 tx=0x7f9ff8019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9fe7fff640 1 -- 172.21.15.67:0/4205673091 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80b2050 con 0x7fa000160800 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9fe7fff640 1 -- 172.21.15.67:0/4205673091 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ff80be070 con 0x7fa000160800 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7f9fe7fff640 1 -- 172.21.15.67:0/4205673091 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8003db0 con 0x7fa000160800 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa000159740 con 0x7fa000160800 2024-04-25T15:51:58.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.471+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa000159d00 con 0x7fa000160800 2024-04-25T15:51:58.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.475+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa00006a480 con 0x7fa000160800 2024-04-25T15:51:58.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.475+0000 7f9fe7fff640 1 -- 172.21.15.67:0/4205673091 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9ff8004040 con 0x7fa000160800 2024-04-25T15:51:58.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.475+0000 7f9fe7fff640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9fec06f430 0x7f9fec071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.475+0000 7f9fe7fff640 1 -- 172.21.15.67:0/4205673091 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5825+0+0 (secure 0 0 0) 0x7f9ff812a300 con 0x7fa000160800 2024-04-25T15:51:58.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.475+0000 7f9fff7fe640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9fec06f430 0x7f9fec071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.475+0000 7f9fff7fe640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9fec06f430 0x7f9fec071940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009b10 tx=0x7f9ff0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.479+0000 7f9fe7fff640 1 -- 172.21.15.67:0/4205673091 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9ff8016540 con 0x7fa000160800 2024-04-25T15:51:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.495+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 0x7f58980639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.495+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 0x7f58980639b0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.495+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5898063ef0 con 0x7f58980635a0 2024-04-25T15:51:58.505 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:58.500+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:58.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.523+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5898063ef0 con 0x7f58980635a0 2024-04-25T15:51:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.527+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5898063ef0 con 0x7f58980635a0 2024-04-25T15:51:58.531 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:51:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:51:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.527+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f58981e64c0 0x7f58981e68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.527+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f58981e6e10 con 0x7f58981e64c0 2024-04-25T15:51:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.527+0000 7f589cafe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f58981e64c0 0x7f58981e68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.527+0000 7f589cafe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f58981e64c0 0x7f58981e68d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.527+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58981e6e10 con 0x7f58981e64c0 2024-04-25T15:51:58.547 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:51:58.547 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:51:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.543+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f58981e6e10 con 0x7f58981e64c0 2024-04-25T15:51:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.547+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f58981e6e10 con 0x7f58981e64c0 2024-04-25T15:51:58.552 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:51:58.552 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:51:58.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.547+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f589802cef0 0x7f589802d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.547+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f58980caee0 con 0x7f589802cef0 2024-04-25T15:51:58.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.547+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f589802cef0 0x7f589802d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.547+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f589802cef0 0x7f589802d300 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.551+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58980caee0 con 0x7f589802cef0 2024-04-25T15:51:58.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:51:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.567+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f58980caee0 con 0x7f589802cef0 2024-04-25T15:51:58.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.571+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f58980caee0 con 0x7f589802cef0 2024-04-25T15:51:58.576 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:51:58.576 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:51:58.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.571+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f58982068f0 0x7f5898208d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.571+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f58980cb050 con 0x7f58982068f0 2024-04-25T15:51:58.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.571+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f58982068f0 0x7f5898208d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.571+0000 7f58977fe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f58982068f0 0x7f5898208d10 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.575+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58980cb050 con 0x7f58982068f0 2024-04-25T15:51:58.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.591+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f58980cb050 con 0x7f58982068f0 2024-04-25T15:51:58.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.595+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f58980cb050 con 0x7f58982068f0 2024-04-25T15:51:58.598 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:51:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:51:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.595+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5898209760 0x7f589820bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.595+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f589815ad00 con 0x7f5898209760 2024-04-25T15:51:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.595+0000 7f589cafe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5898209760 0x7f589820bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.595+0000 7f589cafe640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5898209760 0x7f589820bb80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.599+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f589815ad00 con 0x7f5898209760 2024-04-25T15:51:58.606 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:51:58.603+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:51:58.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:51:58.612 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:51:58.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.611+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589815ad00 con 0x7f5898209760 2024-04-25T15:51:58.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.615+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f589815ad00 con 0x7f5898209760 2024-04-25T15:51:58.620 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:51:58.620 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:51:58.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.615+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f589820c5d0 0x7f589820e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.615+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f58982117c0 con 0x7f589820c5d0 2024-04-25T15:51:58.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.615+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f589820c5d0 0x7f589820e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.615+0000 7f5897fff640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f589820c5d0 0x7f589820e9f0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.619+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58982117c0 con 0x7f589820c5d0 2024-04-25T15:51:58.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.631+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f589820ef30 con 0x7f589820c5d0 2024-04-25T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f5894ff9640 1 -- 172.21.15.67:0/513679653 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f589820ef30 con 0x7f589820c5d0 2024-04-25T15:51:58.641 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:51:58.642 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f58980cdf80 msgr2=0x7f589815c150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f58980cdf80 0x7f589815c150 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f58981f2500 msgr2=0x7f58981f2910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f58981f2500 0x7f58981f2910 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 msgr2=0x7f58980639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f58980635a0 0x7f58980639b0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f58981e64c0 msgr2=0x7f58981e68d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f58981e64c0 0x7f58981e68d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f589802cef0 msgr2=0x7f589802d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f589802cef0 0x7f589802d300 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f58982068f0 msgr2=0x7f5898208d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589cafe640 1 -- 172.21.15.67:0/513679653 reap_dead start 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f58982068f0 0x7f5898208d10 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589cafe640 1 -- 172.21.15.67:0/513679653 reap_dead start 2024-04-25T15:51:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.635+0000 7f589cafe640 1 -- 172.21.15.67:0/513679653 reap_dead start 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5898209760 msgr2=0x7f589820bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5898209760 0x7f589820bb80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f589820c5d0 msgr2=0x7f589820e9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f589820c5d0 0x7f589820e9f0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f588406f400 msgr2=0x7f5884071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f588406f400 0x7f5884071910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f588c0ab000 tx=0x7f588c0b1040 comp rx=0 tx=0).stop 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 msgr2=0x7f589815ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 --2- 172.21.15.67:0/513679653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5880003f70 0x7f589815ca20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f58880115e0 tx=0x7f5888011aa0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589cafe640 1 -- 172.21.15.67:0/513679653 reap_dead start 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 shutdown_connections 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 >> 172.21.15.67:0/513679653 conn(0x7f5898070db0 msgr2=0x7f5898150110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 shutdown_connections 2024-04-25T15:51:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.639+0000 7f589e561640 1 -- 172.21.15.67:0/513679653 wait complete. 2024-04-25T15:51:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.675+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} v 0) -- 0x7fa000155140 con 0x7fa000160800 2024-04-25T15:51:58.707 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:58.700+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:58.707 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:58.707 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:51:58.707 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:51:58.708 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:51:58.708 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:58.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:51:58.808 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:58.808+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c297fa640 1 -- 172.21.15.67:0/67838214 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v40) ==== 91+0+0 (secure 0 0 0) 0x7f4c140f8970 con 0x7f4c2c0c9bc0 2024-04-25T15:51:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:51:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f9fe7fff640 1 -- 172.21.15.67:0/4205673091 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]=0 set osd.0 primary-affinity to 0 (802) v40) ==== 129+0+0 (secure 0 0 0) 0x7f9ff80f7200 con 0x7fa000160800 2024-04-25T15:51:58.908 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:58.909 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 0 (802) 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4c0c06f400 msgr2=0x7f4c0c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4c0c06f400 0x7f4c0c071910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00b930 tx=0x7f4c1c008040 comp rx=0 tx=0).stop 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 msgr2=0x7f4c2c156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c156e30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4c140b3040 tx=0x7f4c1401ba30 comp rx=0 tx=0).stop 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 shutdown_connections 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4c0c06f400 0x7f4c0c071910 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c2c0d2540 0x7f4c2c0b6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.903+0000 7f4c3266b640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c2c0c9bc0 0x7f4c2c156e30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.907+0000 7f4c3266b640 1 --2- 172.21.15.67:0/67838214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c2c0c6c10 0x7f4c2c159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.907+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 >> 172.21.15.67:0/67838214 conn(0x7f4c2c068760 msgr2=0x7f4c2c0cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.907+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 shutdown_connections 2024-04-25T15:51:58.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.907+0000 7f4c3266b640 1 -- 172.21.15.67:0/67838214 wait complete. 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.700407729000005, 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.916 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:51:58.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:51:58.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.700368084000004, 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:58.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:58.923 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.697461062000002, 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:51:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.697308125999996, 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.927 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:51:58.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:51:58.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.695792482000002, 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:51:58.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.695649834999998, 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:58.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.695546743999998, 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:51:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:51:58.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:58.942 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.695393994, 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 82.693445768000004, 2024-04-25T15:51:58.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:51:58.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.949 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.665418137000003, 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:51:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.952 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.663395245999993, 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:51:58.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.663224697000004, 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:51:58.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:51:58.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:51:58.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:58.960 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.660869200999997, 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:51:58.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:51:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.660834488999996, 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.965 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:51:58.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.660813680000004, 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:51:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:51:58.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.583926633000004, 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:51:58.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.583902550999994, 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:51:58.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.583892853999998, 2024-04-25T15:51:58.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:51:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.577997917000005, 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:51:58.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:58.988 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.988559260000002, 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:51:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:51:58.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:51:58.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:51:58.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:51:58.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:51:58.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:51:58.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:51:58.993 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9fec06f430 msgr2=0x7f9fec071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9fec06f430 0x7f9fec071940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009b10 tx=0x7f9ff0008040 comp rx=0 tx=0).stop 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa000160800 msgr2=0x7fa0000c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa000160800 0x7fa0000c7700 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80b0040 tx=0x7f9ff8019660 comp rx=0 tx=0).stop 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 shutdown_connections 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9fec06f430 0x7f9fec071940 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa000160800 0x7fa0000c7700 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0000c96e0 0x7fa00015c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 --2- 172.21.15.67:0/4205673091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0000c92d0 0x7fa000158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 >> 172.21.15.67:0/4205673091 conn(0x7fa000070db0 msgr2=0x7fa00015f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 shutdown_connections 2024-04-25T15:51:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:51:58.911+0000 7fa005c67640 1 -- 172.21.15.67:0/4205673091 wait complete. 2024-04-25T15:51:58.995 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:51:59.043 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:59.040+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:59.110 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:51:59.104+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:51:59.154 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:59.154 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:59.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:51:59.215 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:59.212+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:59.312 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:59.308+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:59.364 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:51:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:51:59.364 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:59.381 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:51:59.413 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:51:59.407+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:51:59.514 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:51:59.508+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:51:59.576 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:59.576 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:51:59.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:51:59.576 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:51:59.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.576 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:51:59.576 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.363792158999999, 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:51:59.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.361177362000006, 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.581 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:51:59.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.360678465000007, 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.359589219, 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:51:59.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.358968723999993, 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:51:59.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.356574498000001, 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:59.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:51:59.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:51:59.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.599 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.327454094999993, 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.322709360999994, 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:51:59.607 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:59.608 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.31918186, 2024-04-25T15:51:59.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:51:59.608 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.608 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.608 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.609 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.319049379000006, 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:51:59.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:51:59.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.698109920999997, 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.686674789000001, 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:51:59.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.625 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.686277353000001, 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:51:59.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:59.629 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.686167611999998, 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:59.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.685666867000002, 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:51:59.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:51:59.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.637 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.685290881, 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:59.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:51:59.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:51:59.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 55.646692698000003, 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 55.646592439000003, 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:51:59.646 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:51:59.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:51:59.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:51:59.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 55.644487523999999, 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:51:59.650 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 55.644111948000003, 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:51:59.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:51:59.657 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:59.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:51:59.659 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:51:59.652+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:51:59.759 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:59.755+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:51:59.852 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:51:59.852 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:51:59.852 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:51:59.853 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:51:59.860 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:51:59.855+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:51:59.872 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:51:59.960 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:51:59.955+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:00.028 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:00.028 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:00.028 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:00.028 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:00.028 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:00.045 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:00.061 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:00.055+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:00.162 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:00.156+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945512085999994, 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.196 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:52:00.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:52:00.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945481525000005, 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:52:00.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945473694, 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:52:00.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945464764999997, 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:00.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:52:00.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:52:00.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945458532999993, 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:00.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:52:00.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:52:00.220 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945443123000004, 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:00.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:52:00.225 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945430290999994, 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:52:00.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.229 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.945425076999996, 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:52:00.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.859814959999994, 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.233 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:52:00.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.307284967000001, 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:52:00.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.305892129, 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.267872003000001, 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:52:00.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:52:00.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:52:00.251 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.267848677000003, 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.267808137000003, 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.257 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:52:00.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:52:00.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.267799476999997, 2024-04-25T15:52:00.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:52:00.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:52:00.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.267757803999999, 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.267 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:52:00.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.270 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.267748546999997, 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:00.274 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.263611120999997, 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:00.278 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.263556543999997, 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:52:00.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:52:00.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.263523335999999, 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:52:00.282 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:52:00.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:52:00.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.286 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:00.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:00.288 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:00.284+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:00.388 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:00.384+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:00.484 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:00.484 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:00.484 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:00.484 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:00.489 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:00.483+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:00.502 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:00.590 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:00.587+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:00.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:00.689 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:00.689 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:00.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:00.707 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:00.703+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:00.791 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:00.788+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:00.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f42183640 1 Processor -- start 2024-04-25T15:52:00.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f42183640 1 -- start start 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f42183640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c160510 0x7f1f3c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f42183640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f3c06d780 con 0x7f1f3c06ca60 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f42183640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f3c160920 con 0x7f1f3c0b5d70 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f42183640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1f3c15d510 con 0x7f1f3c160510 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1f3c06ca60 0x7f1f3c0b5660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54708/0 (socket says 172.21.15.67:54708) 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3b7fe640 1 -- 172.21.15.67:0/628373394 learned_addr learned my addr 172.21.15.67:0/628373394 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c160510 0x7f1f3c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793478747 0 0) 0x7f1f3c06d780 con 0x7f1f3c06ca60 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f28003530 con 0x7f1f3c06ca60 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3423190776 0 0) 0x7f1f28003530 con 0x7f1f3c06ca60 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c160510 msgr2=0x7f1f3c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c160510 0x7f1f3c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 >> v1:172.21.15.176:6789/0 conn(0x7f1f3c0b5d70 legacy=0x7f1f3c15fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3c160be0 con 0x7f1f3c06ca60 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f1f2c0016a0 con 0x7f1f3c06ca60 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 >> v1:172.21.15.67:6789/0 conn(0x7f1f3c06ca60 legacy=0x7f1f3c0b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c160510 0x7f1f280039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f28003f70 0x7f1f28004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f28004d50 0x7f1f28005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c15d510 con 0x7f1f28004d50 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2800bbf0 con 0x7f1f28003f70 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1f2800bdf0 con 0x7f1f3c160510 2024-04-25T15:52:00.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f28004d50 0x7f1f28005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f33fff640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f28003f70 0x7f1f28004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3bfff640 1 -- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c160510 msgr2=0x7f1f280039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:00.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c160510 0x7f1f280039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3bfff640 1 -- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f28003f70 msgr2=0x7f1f28004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:00.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f28003f70 0x7f1f28004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.811+0000 7f1f3bfff640 1 -- 172.21.15.67:0/628373394 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f28003530 con 0x7f1f28004d50 2024-04-25T15:52:00.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f28004d50 0x7f1f28005160 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1f34009fc0 tx=0x7f1f3401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e95962bc32f1d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f340b2070 con 0x7f1f28004d50 2024-04-25T15:52:00.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f340a5cd0 con 0x7f1f28004d50 2024-04-25T15:52:00.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f3affd640 1 -- 172.21.15.67:0/628373394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f34009c10 con 0x7f1f28004d50 2024-04-25T15:52:00.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f42183640 1 -- 172.21.15.67:0/628373394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f28004d50 msgr2=0x7f1f28005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f42183640 1 --2- 172.21.15.67:0/628373394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f28004d50 0x7f1f28005160 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1f34009fc0 tx=0x7f1f3401ccc0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f3bfff640 1 -- 172.21.15.67:0/628373394 reap_dead start 2024-04-25T15:52:00.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f42183640 1 -- 172.21.15.67:0/628373394 shutdown_connections 2024-04-25T15:52:00.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f42183640 1 -- 172.21.15.67:0/628373394 >> 172.21.15.67:0/628373394 conn(0x7f1f3c06e420 msgr2=0x7f1f3c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:00.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f42183640 1 -- 172.21.15.67:0/628373394 shutdown_connections 2024-04-25T15:52:00.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.815+0000 7f1f42183640 1 -- 172.21.15.67:0/628373394 wait complete. 2024-04-25T15:52:00.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 Processor -- start 2024-04-25T15:52:00.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 -- start start 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f3c1571c0 0x7f1f3c1f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f3c157cd0 0x7f1f3c1f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c0d1cb0 0x7f1f3c0cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c15d1a0 con 0x7f1f3c157cd0 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c15cec0 con 0x7f1f3c1571c0 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f42183640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1f3c15c890 con 0x7f1f3c0d1cb0 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f3c157cd0 0x7f1f3c1f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f3c157cd0 0x7f1f3c1f6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47878/0 (socket says 172.21.15.67:47878) 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f33fff640 1 -- 172.21.15.67:0/2263726840 learned_addr learned my addr 172.21.15.67:0/2263726840 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.819+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f3c1571c0 0x7f1f3c1f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c0d1cb0 0x7f1f3c0cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f33fff640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c0d1cb0 msgr2=0x7f1f3c0cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f33fff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f3c0d1cb0 0x7f1f3c0cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f33fff640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f3c1571c0 msgr2=0x7f1f3c1f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f33fff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f3c1571c0 0x7f1f3c1f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f33fff640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3c1f2d10 con 0x7f1f3c157cd0 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f33fff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f3c157cd0 0x7f1f3c1f6160 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400d540 tx=0x7f1f2400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f24005480 con 0x7f1f3c157cd0 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f24017070 con 0x7f1f3c157cd0 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f3c1580e0 con 0x7f1f3c157cd0 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f24003820 con 0x7f1f3c157cd0 2024-04-25T15:52:00.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f3c1f2f80 con 0x7f1f3c157cd0 2024-04-25T15:52:00.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1f3c06a480 con 0x7f1f3c157cd0 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1f240039c0 con 0x7f1f3c157cd0 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f38ff9640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1f1c06f400 0x7f1f1c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5857+0+0 (secure 0 0 0) 0x7f1f2408f150 con 0x7f1f3c157cd0 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1f1c06f400 0x7f1f1c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v41) ==== 54+0+16 (secure 0 0 0) 0x7f1f240917d0 con 0x7f1f3c157cd0 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1f3c0cb940 0x7f1f3c0cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c0cd380 con 0x7f1f3c0cb940 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.823+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1f3c0cb940 0x7f1f3c0cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.827+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1f1c06f400 0x7f1f1c071910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c001890 tx=0x7f1f2c001640 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.827+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1f3c0cb940 0x7f1f3c0cf9c0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.827+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c0cd380 con 0x7f1f3c0cb940 2024-04-25T15:52:00.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.839+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c0cc1b0 con 0x7f1f3c0cb940 2024-04-25T15:52:00.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.843+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c0cc1b0 con 0x7f1f3c0cb940 2024-04-25T15:52:00.848 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.843+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1f3c1e7020 0x7f1f3c1e7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.843+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c1e7970 con 0x7f1f3c1e7020 2024-04-25T15:52:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.843+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1f3c1e7020 0x7f1f3c1e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.843+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1f3c1e7020 0x7f1f3c1e7430 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.843+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c1e7970 con 0x7f1f3c1e7020 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.644313073999996, 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.858 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:52:00.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.644283810000005, 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.863 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:52:00.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:52:00.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.644274522000003, 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.868 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:52:00.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.872 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.644266946000002, 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:52:00.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:52:00.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.644258312999995, 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:52:00.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.644208480000003, 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:52:00.883 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.644200601999998, 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.892 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.607772595, 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:52:00.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.607741931000007, 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.607729401, 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.902 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:52:00.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:52:00.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:52:00.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:52:00.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.607718769000002, 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.907 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:52:00.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:52:00.911 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.607699224000001, 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:52:00.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.916 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.607690332000004, 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:52:00.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:52:00.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.921 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.604127715999994, 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:52:00.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 78.604007104000004, 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.926 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.969009084, 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.930 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:52:00.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:52:00.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:00.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:00.935 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.928439685999997, 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:52:00.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.928434178000003, 2024-04-25T15:52:00.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:00.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:52:00.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.926019664000002, 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.945 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.949 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.925057778000003, 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:52:00.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:52:00.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.953 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.954 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:00.954 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:00.954 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.859+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c1e8810 con 0x7f1f3c1e7020 2024-04-25T15:52:00.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.863+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c1e8810 con 0x7f1f3c1e7020 2024-04-25T15:52:00.955 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:00.956 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:00.956 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:00.952+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.863+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1f3c0635a0 0x7f1f3c0639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.863+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c0c9870 con 0x7f1f3c0635a0 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.863+0000 7f1f33fff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1f3c0635a0 0x7f1f3c0639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.863+0000 7f1f33fff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1f3c0635a0 0x7f1f3c0639b0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.863+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c0c9870 con 0x7f1f3c0635a0 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.879+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c1f31c0 con 0x7f1f3c0635a0 2024-04-25T15:52:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.879+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c1f31c0 con 0x7f1f3c0635a0 2024-04-25T15:52:00.958 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.879+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1f3c15afd0 0x7f1f3c02c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.879+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c1f31c0 con 0x7f1f3c15afd0 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.879+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1f3c15afd0 0x7f1f3c02c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.879+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1f3c15afd0 0x7f1f3c02c9e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.879+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c1f31c0 con 0x7f1f3c15afd0 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.895+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c1f31c0 con 0x7f1f3c15afd0 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.899+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c1f31c0 con 0x7f1f3c15afd0 2024-04-25T15:52:00.959 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.899+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1f3c02d430 0x7f1f3c02d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.899+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c1f31c0 con 0x7f1f3c02d430 2024-04-25T15:52:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.899+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1f3c02d430 0x7f1f3c02d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.899+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1f3c02d430 0x7f1f3c02d840 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.899+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c1f31c0 con 0x7f1f3c02d430 2024-04-25T15:52:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.915+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c1f31c0 con 0x7f1f3c02d430 2024-04-25T15:52:00.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.915+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c1f31c0 con 0x7f1f3c02d430 2024-04-25T15:52:00.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.919+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1f3c207aa0 0x7f1f3c209ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.919+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c02dec0 con 0x7f1f3c207aa0 2024-04-25T15:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.919+0000 7f1f33fff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1f3c207aa0 0x7f1f3c209ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.919+0000 7f1f33fff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1f3c207aa0 0x7f1f3c209ec0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.919+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c02dec0 con 0x7f1f3c207aa0 2024-04-25T15:52:00.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.935+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c20a400 con 0x7f1f3c207aa0 2024-04-25T15:52:00.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.939+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c20a400 con 0x7f1f3c207aa0 2024-04-25T15:52:00.963 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:00.963 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:00.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.939+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1f3c20ab20 0x7f1f3c20af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.939+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c20ff60 con 0x7f1f3c20ab20 2024-04-25T15:52:00.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.943+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1f3c20ab20 0x7f1f3c20af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.943+0000 7f1f3bfff640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1f3c20ab20 0x7f1f3c20af30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.943+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c20ff60 con 0x7f1f3c20ab20 2024-04-25T15:52:00.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.959+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c000960 con 0x7f1f3c20ab20 2024-04-25T15:52:00.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.963+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c000960 con 0x7f1f3c20ab20 2024-04-25T15:52:00.967 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:00.968 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:00.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.963+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1f3c20f1e0 0x7f1f3c20f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:00.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.963+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1f3c213330 con 0x7f1f3c20f1e0 2024-04-25T15:52:00.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.963+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1f3c20f1e0 0x7f1f3c20f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:00.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.963+0000 7f1f3b7fe640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1f3c20f1e0 0x7f1f3c20f630 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:00.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.967+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f3c213330 con 0x7f1f3c20f1e0 2024-04-25T15:52:00.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.979+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1f3c20b580 con 0x7f1f3c20f1e0 2024-04-25T15:52:00.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f38ff9640 1 -- 172.21.15.67:0/2263726840 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1f3c20b580 con 0x7f1f3c20f1e0 2024-04-25T15:52:00.989 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1f3c0cb940 msgr2=0x7f1f3c0cf9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1f3c0cb940 0x7f1f3c0cf9c0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1f3c1e7020 msgr2=0x7f1f3c1e7430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1f3c1e7020 0x7f1f3c1e7430 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1f3c0635a0 msgr2=0x7f1f3c0639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1f3c0635a0 0x7f1f3c0639b0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1f3c15afd0 msgr2=0x7f1f3c02c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f3bfff640 1 -- 172.21.15.67:0/2263726840 reap_dead start 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1f3c15afd0 0x7f1f3c02c9e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1f3c02d430 msgr2=0x7f1f3c02d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1f3c02d430 0x7f1f3c02d840 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1f3c207aa0 msgr2=0x7f1f3c209ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1f3c207aa0 0x7f1f3c209ec0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1f3c20ab20 msgr2=0x7f1f3c20af30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1f3c20ab20 0x7f1f3c20af30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1f3c20f1e0 msgr2=0x7f1f3c20f630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1f3c20f1e0 0x7f1f3c20f630 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f3bfff640 1 -- 172.21.15.67:0/2263726840 reap_dead start 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1f1c06f400 msgr2=0x7f1f1c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1f1c06f400 0x7f1f1c071910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c001890 tx=0x7f1f2c001640 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f3c157cd0 msgr2=0x7f1f3c1f6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.983+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f3c157cd0 0x7f1f3c1f6160 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400d540 tx=0x7f1f2400da00 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.987+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 shutdown_connections 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.987+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1f1c06f400 0x7f1f1c071910 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.987+0000 7f1f42183640 1 --2- 172.21.15.67:0/2263726840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f3c157cd0 0x7f1f3c1f6160 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.987+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 >> 172.21.15.67:0/2263726840 conn(0x7f1f3c06e420 msgr2=0x7f1f3c156860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:00.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.987+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 shutdown_connections 2024-04-25T15:52:00.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:00.987+0000 7f1f42183640 1 -- 172.21.15.67:0/2263726840 wait complete. 2024-04-25T15:52:00.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:52:01.056 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:01.051+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:01.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.139+0000 7fb7fbe84640 1 Processor -- start 2024-04-25T15:52:01.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.139+0000 7fb7fbe84640 1 -- start start 2024-04-25T15:52:01.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fbe84640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb7f406ca60 0x7fb7f406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fbe84640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f406d780 con 0x7fb7f415fe00 2024-04-25T15:52:01.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fbe84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f4160920 con 0x7fb7f40b5660 2024-04-25T15:52:01.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fbe84640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb7f415d510 con 0x7fb7f406ca60 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fa421640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7f415fe00 0x7fb7f4160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54720/0 (socket says 172.21.15.67:54720) 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fa421640 1 -- 172.21.15.67:0/2374601215 learned_addr learned my addr 172.21.15.67:0/2374601215 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/2374601215 >> v2:172.21.15.176:3300/0 conn(0x7fb7f406ca60 0x7fb7f406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759159663 0 0) 0x7fb7f406d780 con 0x7fb7f415fe00 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e0003530 con 0x7fb7f415fe00 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3483574260 0 0) 0x7fb7e0003530 con 0x7fb7f415fe00 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 >> v2:172.21.15.176:3300/0 conn(0x7fb7f406ca60 msgr2=0x7fb7f406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 --2- 172.21.15.67:0/2374601215 >> v2:172.21.15.176:3300/0 conn(0x7fb7f406ca60 0x7fb7f406ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 >> v1:172.21.15.176:6789/0 conn(0x7fb7f40b5660 legacy=0x7fb7f40b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f4160be0 con 0x7fb7f415fe00 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fb7ec0016a0 con 0x7fb7f415fe00 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 >> v1:172.21.15.67:6789/0 conn(0x7fb7f415fe00 legacy=0x7fb7f4160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e00039c0 0x7fb7e0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e0004810 0x7fb7e0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e00051d0 0x7fb7e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f415d510 con 0x7fb7e00039c0 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e000c170 con 0x7fb7e00051d0 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7e000c370 con 0x7fb7e0004810 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e00039c0 0x7fb7e0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f9c20640 1 -- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e0004810 msgr2=0x7fb7e0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f941f640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e0004810 0x7fb7e0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fa421640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e00051d0 0x7fb7e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e0004810 0x7fb7e0004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f9c20640 1 -- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e00051d0 msgr2=0x7fb7e000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7e00051d0 0x7fb7e000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7fa421640 1 -- 172.21.15.67:0/2374601215 reap_dead start 2024-04-25T15:52:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.143+0000 7fb7f9c20640 1 -- 172.21.15.67:0/2374601215 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e0003530 con 0x7fb7e00039c0 2024-04-25T15:52:01.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e00039c0 0x7fb7e0003e30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009f90 tx=0x7fb7f001cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c7c5ec9a089c80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00b2070 con 0x7fb7e00039c0 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7f00a5b20 con 0x7fb7e00039c0 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7f8c1e640 1 -- 172.21.15.67:0/2374601215 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0009c10 con 0x7fb7e00039c0 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 -- 172.21.15.67:0/2374601215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e00039c0 msgr2=0x7fb7e0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e00039c0 0x7fb7e0003e30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009f90 tx=0x7fb7f001cb50 comp rx=0 tx=0).stop 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 -- 172.21.15.67:0/2374601215 shutdown_connections 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/2374601215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e00039c0 0x7fb7e0003e30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 -- 172.21.15.67:0/2374601215 >> 172.21.15.67:0/2374601215 conn(0x7fb7f406e420 msgr2=0x7fb7f406e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 -- 172.21.15.67:0/2374601215 shutdown_connections 2024-04-25T15:52:01.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 -- 172.21.15.67:0/2374601215 wait complete. 2024-04-25T15:52:01.154 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:01.154 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:01.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 Processor -- start 2024-04-25T15:52:01.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.147+0000 7fb7fbe84640 1 -- start start 2024-04-25T15:52:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e00039c0 0x7fb7f4151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f41f6b30 0x7fb7f41582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7f414b2f0 0x7fb7f414b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f415c890 con 0x7fb7f41f6b30 2024-04-25T15:52:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f406d280 con 0x7fb7f414b2f0 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f4160620 con 0x7fb7e00039c0 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f41f6b30 0x7fb7f41582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f41f6b30 0x7fb7f41582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47906/0 (socket says 172.21.15.67:47906) 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 -- 172.21.15.67:0/441514084 learned_addr learned my addr 172.21.15.67:0/441514084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e00039c0 0x7fb7f4151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fa421640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7f414b2f0 0x7fb7f414b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 -- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e00039c0 msgr2=0x7fb7f4151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e00039c0 0x7fb7f4151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 -- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7f414b2f0 msgr2=0x7fb7f414b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7f414b2f0 0x7fb7f414b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 -- 172.21.15.67:0/441514084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f41eab20 con 0x7fb7f41f6b30 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7f941f640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f41f6b30 0x7fb7f41582f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d540 tx=0x7fb7e400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/441514084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4005480 con 0x7fb7f41f6b30 2024-04-25T15:52:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/441514084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e4017070 con 0x7fb7f41f6b30 2024-04-25T15:52:01.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f415a070 con 0x7fb7f41f6b30 2024-04-25T15:52:01.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/441514084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4003820 con 0x7fb7f41f6b30 2024-04-25T15:52:01.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f4157730 con 0x7fb7f41f6b30 2024-04-25T15:52:01.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f406a480 con 0x7fb7f41f6b30 2024-04-25T15:52:01.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/441514084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb7e40039c0 con 0x7fb7f41f6b30 2024-04-25T15:52:01.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.151+0000 7fb7ea7fc640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb7d806f430 0x7fb7d8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:01.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.155+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/441514084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5857+0+0 (secure 0 0 0) 0x7fb7e408f0c0 con 0x7fb7f41f6b30 2024-04-25T15:52:01.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.155+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb7d806f430 0x7fb7d8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:01.159 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:01.155+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:01.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.155+0000 7fb7f9c20640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb7d806f430 0x7fb7d8071940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00af000 tx=0x7fb7f00a7040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:01.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.159+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/441514084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb7e405c030 con 0x7fb7f41f6b30 2024-04-25T15:52:01.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:01.259 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:01.255+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:01.319 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:01.320 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:01.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:01.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.343+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fb7f41f6750 con 0x7fb7f41f6b30 2024-04-25T15:52:01.360 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:01.355+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:01.461 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:01.456+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:01.487 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:01.487 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:01.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:01.561 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:01.556+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:01.662 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:01.656+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:01.669 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:01.669 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:01.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:01.763 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:01.759+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:01.835 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:01.836 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:01.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:01.894 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:01.891+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:01.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.919+0000 7fb7ea7fc640 1 -- 172.21.15.67:0/441514084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v42) ==== 101+0+0 (secure 0 0 0) 0x7fb7e405b930 con 0x7fb7f41f6b30 2024-04-25T15:52:01.924 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:52:01.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb7d806f430 msgr2=0x7fb7d8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:01.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb7d806f430 0x7fb7d8071940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00af000 tx=0x7fb7f00a7040 comp rx=0 tx=0).stop 2024-04-25T15:52:01.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f41f6b30 msgr2=0x7fb7f41582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:01.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f41f6b30 0x7fb7f41582f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d540 tx=0x7fb7e400da00 comp rx=0 tx=0).stop 2024-04-25T15:52:01.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 shutdown_connections 2024-04-25T15:52:01.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb7d806f430 0x7fb7d8071940 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7f414b2f0 0x7fb7f414b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7f41f6b30 0x7fb7f41582f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 --2- 172.21.15.67:0/441514084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7e00039c0 0x7fb7f4151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:01.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 >> 172.21.15.67:0/441514084 conn(0x7fb7f406e420 msgr2=0x7fb7f4150a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:01.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 shutdown_connections 2024-04-25T15:52:01.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:01.923+0000 7fb7fbe84640 1 -- 172.21.15.67:0/441514084 wait complete. 2024-04-25T15:52:01.964 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:01.960+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:02.047 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:02.048 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:02.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:02.066 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:02.059+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:02.165 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:02.159+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:02.266 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:02.263+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:02.267 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:02.267 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:02.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:02.374 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:02.371+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:02.468 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:02.463+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:02.484 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:02.484 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:02.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:02.568 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:02.568+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:02.669 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:02.664+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:02.710 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:02.710 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:02.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:02.774 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:02.771+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:02.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:02.867+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:02.874 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:02.874 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:02.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:02.978 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:02.975+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:03.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:03.072 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:03.068+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:03.079 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:03.079 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:03.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:03.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.143+0000 7f374fe79640 1 Processor -- start 2024-04-25T15:52:03.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.143+0000 7f374fe79640 1 -- start start 2024-04-25T15:52:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374fe79640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37480caf60 0x7f374815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374fe79640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f374815d790 0x7f374806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374fe79640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37480c77a0 con 0x7f3748073bf0 2024-04-25T15:52:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374fe79640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f37480cab60 con 0x7f37480caf60 2024-04-25T15:52:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374fe79640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3748159620 con 0x7f374815d790 2024-04-25T15:52:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374dc15640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37480caf60 0x7f374815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374e416640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3748073bf0 0x7f3748072bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48678/0 (socket says 172.21.15.67:48678) 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374e416640 1 -- 172.21.15.67:0/1801815297 learned_addr learned my addr 172.21.15.67:0/1801815297 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374d414640 1 --2- 172.21.15.67:0/1801815297 >> v2:172.21.15.176:3300/0 conn(0x7f374815d790 0x7f374806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374cc13640 1 -- 172.21.15.67:0/1801815297 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3532322743 0 0) 0x7f37480c77a0 con 0x7f3748073bf0 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374cc13640 1 -- 172.21.15.67:0/1801815297 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3734003530 con 0x7f3748073bf0 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374dc15640 1 -- 172.21.15.67:0/1801815297 >> v2:172.21.15.176:3300/0 conn(0x7f374815d790 msgr2=0x7f374806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374dc15640 1 --2- 172.21.15.67:0/1801815297 >> v2:172.21.15.176:3300/0 conn(0x7f374815d790 0x7f374806d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374dc15640 1 -- 172.21.15.67:0/1801815297 >> v1:172.21.15.67:6789/0 conn(0x7f3748073bf0 legacy=0x7f3748072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374cc13640 1 -- 172.21.15.67:0/1801815297 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 786457443 0 0) 0x7f3734003530 con 0x7f3748073bf0 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374dc15640 1 -- 172.21.15.67:0/1801815297 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37480c7a60 con 0x7f37480caf60 2024-04-25T15:52:03.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.147+0000 7f374d414640 1 --2- 172.21.15.67:0/1801815297 >> v2:172.21.15.176:3300/0 conn(0x7f374815d790 0x7f374806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374dc15640 1 --2- 172.21.15.67:0/1801815297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374815d250 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f37480cf8d0 tx=0x7f374401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0c26ee51720d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374cc13640 1 -- 172.21.15.67:0/1801815297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37440b0070 con 0x7f37480caf60 2024-04-25T15:52:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374cc13640 1 -- 172.21.15.67:0/1801815297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37440ab400 con 0x7f37480caf60 2024-04-25T15:52:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374cc13640 1 -- 172.21.15.67:0/1801815297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3740001a00 con 0x7f37480caf60 2024-04-25T15:52:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 -- 172.21.15.67:0/1801815297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 msgr2=0x7f374815d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 --2- 172.21.15.67:0/1801815297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374815d250 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f37480cf8d0 tx=0x7f374401c3a0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 -- 172.21.15.67:0/1801815297 shutdown_connections 2024-04-25T15:52:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 --2- 172.21.15.67:0/1801815297 >> v2:172.21.15.176:3300/0 conn(0x7f374815d790 0x7f374806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 --2- 172.21.15.67:0/1801815297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374815d250 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 -- 172.21.15.67:0/1801815297 >> 172.21.15.67:0/1801815297 conn(0x7f3748070db0 msgr2=0x7f37480ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 -- 172.21.15.67:0/1801815297 shutdown_connections 2024-04-25T15:52:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.151+0000 7f374fe79640 1 -- 172.21.15.67:0/1801815297 wait complete. 2024-04-25T15:52:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 Processor -- start 2024-04-25T15:52:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 -- start start 2024-04-25T15:52:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3748073bf0 0x7f374814d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374814e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f374815d790 0x7f37481541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3748165050 con 0x7f37480caf60 2024-04-25T15:52:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3748164ed0 con 0x7f374815d790 2024-04-25T15:52:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f37481651d0 con 0x7f3748073bf0 2024-04-25T15:52:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374814e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374814e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60496/0 (socket says 172.21.15.67:60496) 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 -- 172.21.15.67:0/3632564484 learned_addr learned my addr 172.21.15.67:0/3632564484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3748073bf0 0x7f374814d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374e416640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f374815d790 0x7f37481541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3748073bf0 msgr2=0x7f374814d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3748073bf0 0x7f374814d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f374815d790 msgr2=0x7f37481541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f374815d790 0x7f37481541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3748075d30 con 0x7f37480caf60 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374d414640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374814e8c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f373800d830 tx=0x7f373800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37380096a0 con 0x7f37480caf60 2024-04-25T15:52:03.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3738019070 con 0x7f37480caf60 2024-04-25T15:52:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374814daf0 con 0x7f37480caf60 2024-04-25T15:52:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738009a80 con 0x7f37480caf60 2024-04-25T15:52:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3748155fb0 con 0x7f37480caf60 2024-04-25T15:52:03.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.155+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f374806a480 con 0x7f37480caf60 2024-04-25T15:52:03.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f373801e460 con 0x7f37480caf60 2024-04-25T15:52:03.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f373e7fc640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f372c06f430 0x7f372c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5857+0+0 (secure 0 0 0) 0x7f373808f850 con 0x7f37480caf60 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v42) ==== 54+0+16 (secure 0 0 0) 0x7f3738094240 con 0x7f37480caf60 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f372c06f430 0x7f372c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3748155660 0x7f3748156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f374814f1e0 con 0x7f3748155660 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f374e416640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3748155660 0x7f3748156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f374e416640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3748155660 0x7f3748156290 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f372c06f430 0x7f372c071940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f37440ad000 tx=0x7f3744008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.159+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f374814f1e0 con 0x7f3748155660 2024-04-25T15:52:03.173 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:03.168+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.175+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f37480c8a40 con 0x7f3748155660 2024-04-25T15:52:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.179+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f37480c8a40 con 0x7f3748155660 2024-04-25T15:52:03.183 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.179+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f37480c52c0 0x7f37481426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.179+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3748142be0 con 0x7f37480c52c0 2024-04-25T15:52:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.179+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f37480c52c0 0x7f37481426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.179+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f37480c52c0 0x7f37481426a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.179+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3748142be0 con 0x7f37480c52c0 2024-04-25T15:52:03.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.195+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3748072bc0 con 0x7f37480c52c0 2024-04-25T15:52:03.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.199+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3748072bc0 con 0x7f37480c52c0 2024-04-25T15:52:03.202 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.199+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f37481f7d40 0x7f37481f8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.199+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f37481f8690 con 0x7f37481f7d40 2024-04-25T15:52:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.199+0000 7f374d414640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f37481f7d40 0x7f37481f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.199+0000 7f374d414640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f37481f7d40 0x7f37481f8150 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.199+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f373807ce50 con 0x7f37481f7d40 2024-04-25T15:52:03.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.215+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3748156940 con 0x7f37481f7d40 2024-04-25T15:52:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.215+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3748156940 con 0x7f37481f7d40 2024-04-25T15:52:03.222 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.215+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3748063960 0x7f37480cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.219+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3748063d70 con 0x7f3748063960 2024-04-25T15:52:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.219+0000 7f374e416640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3748063960 0x7f37480cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.219+0000 7f374e416640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3748063960 0x7f37480cc730 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.219+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3748063d70 con 0x7f3748063960 2024-04-25T15:52:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.235+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f37480ccec0 con 0x7f3748063960 2024-04-25T15:52:03.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.235+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f37480ccec0 con 0x7f3748063960 2024-04-25T15:52:03.241 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.235+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f374802c9e0 0x7f374802cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.235+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3748064000 con 0x7f374802c9e0 2024-04-25T15:52:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.235+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f374802c9e0 0x7f374802cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.239+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f374802c9e0 0x7f374802cdf0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.239+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3748064000 con 0x7f374802c9e0 2024-04-25T15:52:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.255+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3748063d70 con 0x7f374802c9e0 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046183510999995, 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937687, 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.260 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212220+0000", 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212245+0000", 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212458+0000", 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021240899999999999 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212488+0000", 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0715e-05 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212836+0000", 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000348096 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213194+0000", 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000357413 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213591+0000", 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039711599999999998 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224723+0000", 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011132177 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224892+0000", 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016872100000000001 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225031+0000", 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013966 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225148+0000", 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116168 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046137209999998, 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013374822, 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212257+0000", 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:03.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212263+0000", 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0900000000000001e-06 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212576+0000", 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031227099999999998 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212590+0000", 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4133e-05 2024-04-25T15:52:03.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214544+0000", 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001953939 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214793+0000", 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024941399999999999 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215181+0000", 2024-04-25T15:52:03.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038823800000000001 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224739+0000", 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095574579999999996 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225301+0000", 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056223199999999997 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225459+0000", 2024-04-25T15:52:03.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015786199999999999 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225631+0000", 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172242 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:03.270 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046126796999999, 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013916951, 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8299999999999996e-07 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212272+0000", 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6630000000000001e-06 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212665+0000", 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039338399999999999 2024-04-25T15:52:03.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212687+0000", 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1882000000000002e-05 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215937+0000", 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032501119999999999 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032469999999999998 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216646+0000", 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038413800000000001 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224753+0000", 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081070039999999993 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225911+0000", 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001158002 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226062+0000", 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015119600000000001 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226184+0000", 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121187 2024-04-25T15:52:03.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046118828000004, 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01435588, 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:03.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6000000000000003e-07 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212279+0000", 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1169999999999997e-06 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212723+0000", 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044335799999999998 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212733+0000", 2024-04-25T15:52:03.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349e-05 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217516+0000", 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004782606 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217766+0000", 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025045699999999999 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039180399999999999 2024-04-25T15:52:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224766+0000", 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066076449999999997 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226308+0000", 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00154228 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226483+0000", 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174938 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226630+0000", 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014766599999999999 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046111847999995, 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014867514, 2024-04-25T15:52:03.281 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2699999999999999e-07 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212287+0000", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6229999999999996e-06 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212756+0000", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046907999999999997 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:03.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9560000000000003e-06 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218829+0000", 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060638690000000004 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219182+0000", 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035291899999999999 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219507+0000", 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032545399999999998 2024-04-25T15:52:03.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224779+0000", 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052717279999999998 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226860+0000", 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020809790000000002 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227036+0000", 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176062 2024-04-25T15:52:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227149+0000", 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113217 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046104377999995, 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015281197, 2024-04-25T15:52:03.286 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212290+0000", 2024-04-25T15:52:03.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3e-07 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212333+0000", 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3251000000000003e-05 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212784+0000", 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045139399999999998 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212793+0000", 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7960000000000001e-06 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220253+0000", 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074595699999999996 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220482+0000", 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000229554 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220879+0000", 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039646400000000002 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225181+0000", 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00430205 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227277+0000", 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020961119999999998 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227449+0000", 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017166800000000001 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227570+0000", 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012170799999999999 2024-04-25T15:52:03.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046054897999994, 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016239407000000001, 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:03.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212340+0000", 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1200000000000001e-06 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212346+0000", 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.584e-06 2024-04-25T15:52:03.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212813+0000", 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046677600000000002 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212821+0000", 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221805+0000", 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089835999999999996 2024-04-25T15:52:03.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222154+0000", 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034940299999999997 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222503+0000", 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034907900000000002 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227840+0000", 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367349999999996 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228310+0000", 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047011999999999998 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228439+0000", 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012899999999999999 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228578+0000", 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138837 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.295 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046043687999997, 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016443972000000001, 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212351+0000", 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212357+0000", 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000003e-06 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212848+0000", 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049076600000000001 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212863+0000", 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5498999999999998e-05 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223300+0000", 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010437093999999999 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223537+0000", 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000236665 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224027+0000", 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048967900000000003 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228184+0000", 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041571209999999997 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228599+0000", 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000415392 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228676+0000", 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7095999999999995e-05 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228794+0000", 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011746700000000001 2024-04-25T15:52:03.300 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046032709000002, 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013451848000000001, 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:03.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212362+0000", 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212368+0000", 2024-04-25T15:52:03.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6229999999999999e-06 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212925+0000", 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055690199999999996 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212960+0000", 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5045000000000001e-05 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213123+0000", 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163222 2024-04-25T15:52:03.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213447+0000", 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000323627 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213759+0000", 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031232800000000001 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216671+0000", 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002911145 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223654+0000", 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069831370000000004 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223829+0000", 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174857 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225813+0000", 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019840679999999999 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.305 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.046021546000006, 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014242332, 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212373+0000", 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4e-07 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212384+0000", 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1443e-05 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213026+0000", 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064212500000000001 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213045+0000", 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8621999999999999e-05 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214708+0000", 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016628109999999999 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214970+0000", 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026197699999999999 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215380+0000", 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000410789 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218599+0000", 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032185270000000001 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226296+0000", 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076965269999999999 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226473+0000", 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017755200000000001 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226614+0000", 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:4 2.10 2:0f65c9b8:::60a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.011218212000003, 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010394548999999999, 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247176+0000", 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2100000000000001e-06 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247209+0000", 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2945999999999999e-05 2024-04-25T15:52:03.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247394+0000", 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018468799999999999 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247417+0000", 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2733000000000001e-05 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249734+0000", 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002317684 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250032+0000", 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029727699999999998 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250453+0000", 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042143500000000001 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255742+0000", 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052886809999999999 2024-04-25T15:52:03.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257232+0000", 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014906260000000001 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257393+0000", 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016088699999999999 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257570+0000", 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017638199999999999 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927249646000007, 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094210270000000002, 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6699999999999999e-07 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331165+0000", 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331418+0000", 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271799999999997 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331428+0000", 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6460000000000005e-06 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333229+0000", 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018009499999999999 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334396+0000", 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011673269999999999 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334597+0000", 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020135300000000001 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339905+0000", 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053077610000000002 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340210+0000", 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030499999999999999 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340385+0000", 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017450199999999999 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340565+0000", 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018011700000000001 2024-04-25T15:52:03.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927224899999999, 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095762039999999996, 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:03.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331174+0000", 2024-04-25T15:52:03.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-06 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331467+0000", 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029356 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331478+0000", 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336604+0000", 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051267170000000003 2024-04-25T15:52:03.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336735+0000", 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013052699999999999 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336996+0000", 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260511 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339986+0000", 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002990487 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340579+0000", 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059299899999999998 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340673+0000", 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4241999999999997e-05 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340745+0000", 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1484999999999997e-05 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.325 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:17 2.10 2:0e9da45e:::61b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927217654000003, 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010814343000000001, 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:03.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331180+0000", 2024-04-25T15:52:03.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6650000000000001e-06 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331501+0000", 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032061600000000002 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331509+0000", 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8240000000000006e-06 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333383+0000", 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018733020000000001 2024-04-25T15:52:03.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333652+0000", 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026890399999999999 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333865+0000", 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021341 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338313+0000", 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044478269999999997 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341868+0000", 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035546729999999999 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341920+0000", 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2507999999999999e-05 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341990+0000", 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0110000000000005e-05 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.330 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927207981999999, 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011086621, 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331186+0000", 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.51e-07 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331191+0000", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3970000000000002e-06 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331546+0000", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035447799999999998 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331557+0000", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0791000000000001e-05 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334408+0000", 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028516520000000001 2024-04-25T15:52:03.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334546+0000", 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013802700000000001 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334877+0000", 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000331081 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339910+0000", 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050323449999999997 2024-04-25T15:52:03.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342080+0000", 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021703899999999999 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342184+0000", 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103767 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342272+0000", 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8242e-05 2024-04-25T15:52:03.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927199962000003, 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010331643, 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331194+0000", 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331198+0000", 2024-04-25T15:52:03.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1400000000000002e-06 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331608+0000", 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040962799999999999 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331620+0000", 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337545+0000", 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005924726 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338432+0000", 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088720699999999999 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338639+0000", 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020665000000000001 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341318+0000", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026785860000000002 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341406+0000", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8344000000000007e-05 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341458+0000", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805999999999998e-05 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341525+0000", 2024-04-25T15:52:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7336999999999996e-05 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:2 2.10 2:0daba267:::402.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927193502999998, 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011245289, 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:03.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331215+0000", 2024-04-25T15:52:03.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331647+0000", 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043194799999999998 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331656+0000", 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335316+0000", 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036607710000000002 2024-04-25T15:52:03.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335504+0000", 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018793299999999999 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335699+0000", 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194544 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339945+0000", 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042457600000000003 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342333+0000", 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023885730000000002 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342383+0000", 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9314999999999998e-05 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342445+0000", 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2691e-05 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927174742000005, 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087706009999999994, 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331239+0000", 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9653e-05 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331678+0000", 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043868999999999999 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331701+0000", 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3019000000000001e-05 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333753+0000", 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020522050000000001 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334131+0000", 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000378484 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334455+0000", 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032319899999999998 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337268+0000", 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028133199999999998 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339816+0000", 2024-04-25T15:52:03.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025479140000000001 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339907+0000", 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1150000000000007e-05 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339989+0000", 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2392000000000007e-05 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:27 2.10 2:091e33df:::502.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927151701, 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011447874, 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:03.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4199999999999996e-07 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331249+0000", 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2259999999999999e-06 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331730+0000", 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048009600000000001 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331739+0000", 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1840000000000005e-06 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336653+0000", 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049141510000000003 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336830+0000", 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017740099999999999 2024-04-25T15:52:03.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337033+0000", 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020295999999999999 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340803+0000", 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037692480000000002 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342567+0000", 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017649250000000001 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342650+0000", 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2748000000000001e-05 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342690+0000", 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9393e-05 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:31 2.10 2:0eee8b97:::202.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:03.355 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.927141844000005, 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012346048, 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5300000000000004e-07 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331255+0000", 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8380000000000002e-06 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331760+0000", 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050452600000000004 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331768+0000", 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337613+0000", 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058448700000000003 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337836+0000", 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000223043 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.343598+0000", 2024-04-25T15:52:03.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057612679999999999 2024-04-25T15:52:03.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.359 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.359 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.359 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.359 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.259+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3748063d70 con 0x7f374802c9e0 2024-04-25T15:52:03.360 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.259+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f374802d840 0x7f374820abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.259+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3748064050 con 0x7f374802d840 2024-04-25T15:52:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.259+0000 7f374d414640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f374802d840 0x7f374820abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.259+0000 7f374d414640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f374802d840 0x7f374820abc0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.263+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37481f8690 con 0x7f374802d840 2024-04-25T15:52:03.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.283+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f37481f8690 con 0x7f374802d840 2024-04-25T15:52:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.287+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f37481f8690 con 0x7f374802d840 2024-04-25T15:52:03.363 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.287+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f374802dc50 0x7f374820d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.287+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f37482103f0 con 0x7f374802dc50 2024-04-25T15:52:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.287+0000 7f374e416640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f374802dc50 0x7f374820d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.287+0000 7f374e416640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f374802dc50 0x7f374820d620 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.291+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37482103f0 con 0x7f374802dc50 2024-04-25T15:52:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.303+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f374820db60 con 0x7f374802dc50 2024-04-25T15:52:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.307+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f374820db60 con 0x7f374802dc50 2024-04-25T15:52:03.364 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.311+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f374820e1e0 0x7f374820e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.311+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3748213760 con 0x7f374820e1e0 2024-04-25T15:52:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.311+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f374820e1e0 0x7f374820e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.311+0000 7f374dc15640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f374820e1e0 0x7f374820e5f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.311+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3748213760 con 0x7f374820e1e0 2024-04-25T15:52:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.327+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f374820ec40 con 0x7f374820e1e0 2024-04-25T15:52:03.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f373e7fc640 1 -- 172.21.15.67:0/3632564484 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f374820ec40 con 0x7f374820e1e0 2024-04-25T15:52:03.366 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:03.367 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3748155660 msgr2=0x7f3748156290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3748155660 0x7f3748156290 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f37480c52c0 msgr2=0x7f37481426a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f37480c52c0 0x7f37481426a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f37481f7d40 msgr2=0x7f37481f8150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f37481f7d40 0x7f37481f8150 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3748063960 msgr2=0x7f37480cc730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3748063960 0x7f37480cc730 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f374802c9e0 msgr2=0x7f374802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f374802c9e0 0x7f374802cdf0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374e416640 1 -- 172.21.15.67:0/3632564484 reap_dead start 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374e416640 1 -- 172.21.15.67:0/3632564484 reap_dead start 2024-04-25T15:52:03.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f374802d840 msgr2=0x7f374820abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f374802d840 0x7f374820abc0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f374802dc50 msgr2=0x7f374820d620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f374802dc50 0x7f374820d620 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f374820e1e0 msgr2=0x7f374820e5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f374820e1e0 0x7f374820e5f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f372c06f430 msgr2=0x7f372c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f372c06f430 0x7f372c071940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f37440ad000 tx=0x7f3744008040 comp rx=0 tx=0).stop 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 msgr2=0x7f374814e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 --2- 172.21.15.67:0/3632564484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37480caf60 0x7f374814e8c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f373800d830 tx=0x7f373800dcf0 comp rx=0 tx=0).stop 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374e416640 1 -- 172.21.15.67:0/3632564484 reap_dead start 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 shutdown_connections 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 >> 172.21.15.67:0/3632564484 conn(0x7f3748070db0 msgr2=0x7f37480ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 shutdown_connections 2024-04-25T15:52:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:03.331+0000 7f374fe79640 1 -- 172.21.15.67:0/3632564484 wait complete. 2024-04-25T15:52:03.370 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:03.355+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:03.460 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:03.455+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:03.544 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:03.544 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:03.545 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:03.545 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:03.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:03.562 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:03.556+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:03.662 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:03.659+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:03.762 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:03.762 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:03.763 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:03.763 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:03.763 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:03.759+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:03.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:03.863 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:03.864+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:03.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:52:03.972 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.756948346000001, 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:03.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:52:03.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:52:03.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:03.976 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.756908701, 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.754001678999998, 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:52:03.983 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.753848743000006, 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.752333098999998, 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:52:03.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.752190451999994, 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.752087360999994, 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:03.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:52:03.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:03.999 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.751934610999996, 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.749986385, 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:52:04.003 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:52:04.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:04.006 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.721958753999999, 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:52:04.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:04.009 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.719935863000003, 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:52:04.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.719765314, 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.013 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.717409817999993, 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.017 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:52:04.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.021 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.717375106000006, 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.025 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.717354297, 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:52:04.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:52:04.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.64046725, 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:52:04.029 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:52:04.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:52:04.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.640443168000004, 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:52:04.034 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:04.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.640433470999994, 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:52:04.039 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:52:04.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 71.634538534000001, 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:52:04.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:52:04.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:52:04.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.045099876999998, 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:04.047 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:52:04.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:52:04.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:52:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:04.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:04.052 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:04.052 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:03.967+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:04.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:04.053 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:04.054 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:04.054 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-25T15:52:04.054 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:52:04.054 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:52:04.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-25T15:52:04.094 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:04.091+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.095+0000 7f991b6a6640 1 Processor -- start 2024-04-25T15:52:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.095+0000 7f991b6a6640 1 -- start start 2024-04-25T15:52:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f991b6a6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 0x7f99140cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f991b6a6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f991415de10 con 0x7f99140cc100 2024-04-25T15:52:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f991b6a6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9914072390 con 0x7f99140ca540 2024-04-25T15:52:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f991b6a6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f991415ebb0 con 0x7f9914071bc0 2024-04-25T15:52:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9918c41640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f99140cc100 0x7f991415a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48694/0 (socket says 172.21.15.67:48694) 2024-04-25T15:52:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9918c41640 1 -- 172.21.15.67:0/7833041 learned_addr learned my addr 172.21.15.67:0/7833041 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919442640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 0x7f99140cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231825111 0 0) 0x7f991415de10 con 0x7f99140cc100 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9900003530 con 0x7f99140cc100 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 516532254 0 0) 0x7f9900003530 con 0x7f99140cc100 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 msgr2=0x7f99140cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 0x7f99140cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 >> v1:172.21.15.176:6789/0 conn(0x7f99140ca540 legacy=0x7f9914164fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991415e0d0 con 0x7f99140cc100 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f99040016a0 con 0x7f99140cc100 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 >> v1:172.21.15.67:6789/0 conn(0x7f99140cc100 legacy=0x7f991415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99000039f0 0x7f9900003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9900004810 0x7f9900004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 0x7f9900005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f991415ebb0 con 0x7f99000039f0 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f990000ba80 con 0x7f9900004810 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f990000bc80 con 0x7f9914071bc0 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9918c41640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9900004810 0x7f9900004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919442640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99000039f0 0x7f9900003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919c43640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 0x7f9900005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919442640 1 -- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 msgr2=0x7f9900005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919442640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 0x7f9900005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919442640 1 -- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9900004810 msgr2=0x7f9900004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919442640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9900004810 0x7f9900004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919442640 1 -- 172.21.15.67:0/7833041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9900003530 con 0x7f99000039f0 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9919c43640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9914071bc0 0x7f9900005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.099+0000 7f9918c41640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9900004810 0x7f9900004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f9919442640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99000039f0 0x7f9900003e20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9910009ae0 tx=0x7f991001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa10dd15ea05fd00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99100b2070 con 0x7f99000039f0 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99100a5570 con 0x7f99000039f0 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f990bfff640 1 -- 172.21.15.67:0/7833041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99100096b0 con 0x7f99000039f0 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f991b6a6640 1 -- 172.21.15.67:0/7833041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99000039f0 msgr2=0x7f9900003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f991b6a6640 1 --2- 172.21.15.67:0/7833041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99000039f0 0x7f9900003e20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9910009ae0 tx=0x7f991001ca90 comp rx=0 tx=0).stop 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f9919c43640 1 -- 172.21.15.67:0/7833041 reap_dead start 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f991b6a6640 1 -- 172.21.15.67:0/7833041 shutdown_connections 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f991b6a6640 1 -- 172.21.15.67:0/7833041 >> 172.21.15.67:0/7833041 conn(0x7f99140700b0 msgr2=0x7f99140704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f991b6a6640 1 -- 172.21.15.67:0/7833041 shutdown_connections 2024-04-25T15:52:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.103+0000 7f991b6a6640 1 -- 172.21.15.67:0/7833041 wait complete. 2024-04-25T15:52:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 Processor -- start 2024-04-25T15:52:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 -- start start 2024-04-25T15:52:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99140b56b0 0x7f991415d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9914154cf0 0x7f99141f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9914155100 0x7f99141f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9914071fd0 con 0x7f9914154cf0 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f991415e720 con 0x7f9914155100 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9914072150 con 0x7f99140b56b0 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9914154cf0 0x7f99141f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9914154cf0 0x7f99141f5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60512/0 (socket says 172.21.15.67:60512) 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 -- 172.21.15.67:0/3678567729 learned_addr learned my addr 172.21.15.67:0/3678567729 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9919442640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99140b56b0 0x7f991415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9919c43640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9914155100 0x7f99141f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 -- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99140b56b0 msgr2=0x7f991415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99140b56b0 0x7f991415d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 -- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9914155100 msgr2=0x7f99141f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9914155100 0x7f99141f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 -- 172.21.15.67:0/3678567729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99141f1bd0 con 0x7f9914154cf0 2024-04-25T15:52:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9918c41640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9914154cf0 0x7f99141f5660 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9904007400 tx=0x7f9904004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9909ffb640 1 -- 172.21.15.67:0/3678567729 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990400fb80 con 0x7f9914154cf0 2024-04-25T15:52:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9909ffb640 1 -- 172.21.15.67:0/3678567729 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9904018070 con 0x7f9914154cf0 2024-04-25T15:52:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99141f1e70 con 0x7f9914154cf0 2024-04-25T15:52:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f9909ffb640 1 -- 172.21.15.67:0/3678567729 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040133f0 con 0x7f9914154cf0 2024-04-25T15:52:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99141f2430 con 0x7f9914154cf0 2024-04-25T15:52:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.107+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f991406a480 con 0x7f9914154cf0 2024-04-25T15:52:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.111+0000 7f9909ffb640 1 -- 172.21.15.67:0/3678567729 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9904013590 con 0x7f9914154cf0 2024-04-25T15:52:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.111+0000 7f9909ffb640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98f806f400 0x7f98f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.111+0000 7f9909ffb640 1 -- 172.21.15.67:0/3678567729 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5857+0+0 (secure 0 0 0) 0x7f990408f4c0 con 0x7f9914154cf0 2024-04-25T15:52:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.111+0000 7f9919442640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98f806f400 0x7f98f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.111+0000 7f9919442640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98f806f400 0x7f98f8071910 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9910009ae0 tx=0x7f99100046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.115+0000 7f9909ffb640 1 -- 172.21.15.67:0/3678567729 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f990405bcc0 con 0x7f9914154cf0 2024-04-25T15:52:04.166 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:04.163+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:04.213 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:04.213 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:04.213 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:04.213 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:04.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:04.242 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:04.242 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "" 2024-04-25T15:52:04.242 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:04.275 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:04.272+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.291+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f99140705f0 con 0x7f9914154cf0 2024-04-25T15:52:04.367 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:04.364+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:04.378 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:04.378 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:04.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:04.468 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:04.463+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:04.569 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:04.563+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:04.591 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 88.379091299999999, 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:52:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:52:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:52:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:52:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 88.376476503000006, 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 88.375977606000006, 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 88.37488836, 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:52:04.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:52:04.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 88.374267864999993, 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:04.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:52:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:52:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 88.371873639, 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.342753235999993, 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:52:04.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.338008501999994, 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.334481001, 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.624 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:52:04.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:52:04.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:52:04.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.334348520000006, 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:52:04.628 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:04.632 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.713409061999997, 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:52:04.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:04.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.701973929999994, 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.701576493999994, 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.642 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:52:04.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.701466753000005, 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.646 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:52:04.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:52:04.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:52:04.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.700966007999995, 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.650 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:52:04.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.700590022, 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.654 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:04.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 60.661991839000002, 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 60.661891580000002, 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:52:04.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:52:04.665 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 60.659786664999999, 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:52:04.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:04.669 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 60.659411089000002, 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:04.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:52:04.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:52:04.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.673 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.673 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.673 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:04.673 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:04.673 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:04.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:04.675 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:04.671+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:04.775 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:04.771+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:04.826 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:04.826 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:04.826 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:04.826 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:04.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:04.876 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:04.871+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.935+0000 7f9909ffb640 1 -- 172.21.15.67:0/3678567729 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v43) ==== 87+0+0 (secure 0 0 0) 0x7f990405ffc0 con 0x7f9914154cf0 2024-04-25T15:52:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:52:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98f806f400 msgr2=0x7f98f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98f806f400 0x7f98f8071910 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9910009ae0 tx=0x7f99100046d0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9914154cf0 msgr2=0x7f99141f5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9914154cf0 0x7f99141f5660 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9904007400 tx=0x7f9904004f70 comp rx=0 tx=0).stop 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 shutdown_connections 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f98f806f400 0x7f98f8071910 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9914155100 0x7f99141f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9914154cf0 0x7f99141f5660 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 --2- 172.21.15.67:0/3678567729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99140b56b0 0x7f991415d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 >> 172.21.15.67:0/3678567729 conn(0x7f99140700b0 msgr2=0x7f99140d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 shutdown_connections 2024-04-25T15:52:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:04.939+0000 7f991b6a6640 1 -- 172.21.15.67:0/3678567729 wait complete. 2024-04-25T15:52:04.989 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:04.989 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:04.989 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:04.989 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:04.989 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:05.006 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:05.003+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:05.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:05.077 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:05.071+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:05.161 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910666860000006, 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:52:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:52:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:52:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910636299000004, 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:52:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:52:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910628467999999, 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:52:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910619538999995, 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910613307000006, 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910597897000002, 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:52:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910585065000006, 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:52:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:52:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.910579850999994, 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:52:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 72.824969734000007, 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:52:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.272439740999999, 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:52:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:52:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:52:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.271046902999998, 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.233026776999999, 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.233003451000002, 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:52:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.232962911000001, 2024-04-25T15:52:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.232954251000002, 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.232912577999997, 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:52:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.232903321000002, 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.228765895000002, 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.228711318000002, 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:52:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:52:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 61.228678109999997, 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:05.254 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:05.255 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:05.255 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:05.252+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:05.356 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:05.352+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:05.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:05.450 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:05.450 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:05.451 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:05.451 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:05.457 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:05.451+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:05.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97aa3d9640 1 Processor -- start 2024-04-25T15:52:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97aa3d9640 1 -- start start 2024-04-25T15:52:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97aa3d9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97a406db50 0x7f97a40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97aa3d9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f97a4153980 0x7f97a40cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97aa3d9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a40c5770 con 0x7f97a40c9140 2024-04-25T15:52:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97aa3d9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f97a40c8620 con 0x7f97a406db50 2024-04-25T15:52:05.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97aa3d9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f97a40ca0d0 con 0x7f97a4153980 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97a406db50 0x7f97a40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97a406db50 0x7f97a40cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60526/0 (socket says 172.21.15.67:60526) 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a3fff640 1 -- 172.21.15.67:0/1237930095 learned_addr learned my addr 172.21.15.67:0/1237930095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a37fe640 1 --2- 172.21.15.67:0/1237930095 >> v2:172.21.15.176:3300/0 conn(0x7f97a4153980 0x7f97a40cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a3fff640 1 -- 172.21.15.67:0/1237930095 >> v2:172.21.15.176:3300/0 conn(0x7f97a4153980 msgr2=0x7f97a40cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a3fff640 1 --2- 172.21.15.67:0/1237930095 >> v2:172.21.15.176:3300/0 conn(0x7f97a4153980 0x7f97a40cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a3fff640 1 -- 172.21.15.67:0/1237930095 >> v1:172.21.15.176:6789/0 conn(0x7f97a40c9140 legacy=0x7f97a40c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.523+0000 7f97a3fff640 1 -- 172.21.15.67:0/1237930095 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a40ca3b0 con 0x7f97a406db50 2024-04-25T15:52:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97a3fff640 1 --2- 172.21.15.67:0/1237930095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a406db50 0x7f97a40cae00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f97a406fb70 tx=0x7f979401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ed40c58abb15dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97a2ffd640 1 -- 172.21.15.67:0/1237930095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940ab430 con 0x7f97a406db50 2024-04-25T15:52:05.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97a2ffd640 1 -- 172.21.15.67:0/1237930095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97940b0070 con 0x7f97a406db50 2024-04-25T15:52:05.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97a2ffd640 1 -- 172.21.15.67:0/1237930095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940ab7f0 con 0x7f97a406db50 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 -- 172.21.15.67:0/1237930095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a406db50 msgr2=0x7f97a40cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/1237930095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a406db50 0x7f97a40cae00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f97a406fb70 tx=0x7f979401c520 comp rx=0 tx=0).stop 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 -- 172.21.15.67:0/1237930095 shutdown_connections 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/1237930095 >> v2:172.21.15.176:3300/0 conn(0x7f97a4153980 0x7f97a40cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/1237930095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a406db50 0x7f97a40cae00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 -- 172.21.15.67:0/1237930095 >> 172.21.15.67:0/1237930095 conn(0x7f97a40700c0 msgr2=0x7f97a40704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 -- 172.21.15.67:0/1237930095 shutdown_connections 2024-04-25T15:52:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.527+0000 7f97aa3d9640 1 -- 172.21.15.67:0/1237930095 wait complete. 2024-04-25T15:52:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.531+0000 7f97aa3d9640 1 Processor -- start 2024-04-25T15:52:05.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.531+0000 7f97aa3d9640 1 -- start start 2024-04-25T15:52:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a406db50 0x7f97a4148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a40c9140 0x7f97a415a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a4153980 0x7f97a41511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4166890 con 0x7f97a4153980 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f97a40c8dc0 con 0x7f97a40c9140 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97a4166a10 con 0x7f97a406db50 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a4153980 0x7f97a41511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a4153980 0x7f97a41511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60536/0 (socket says 172.21.15.67:60536) 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 learned_addr learned my addr 172.21.15.67:0/3669896193 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a406db50 0x7f97a4148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a37fe640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a40c9140 0x7f97a415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a406db50 msgr2=0x7f97a4148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a406db50 0x7f97a4148770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a40c9140 msgr2=0x7f97a415a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a40c9140 0x7f97a415a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a415ad90 con 0x7f97a4153980 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a8976640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a4153980 0x7f97a41511e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f979c00c760 tx=0x7f979c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c003ba0 con 0x7f97a4153980 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f979c003d40 con 0x7f97a4153980 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c005310 con 0x7f97a4153980 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a41478b0 con 0x7f97a4153980 2024-04-25T15:52:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a4152c90 con 0x7f97a4153980 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.535+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f97a406a480 con 0x7f97a4153980 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f979c0054b0 con 0x7f97a4153980 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a0ff9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979006f400 0x7f9790071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5857+0+0 (secure 0 0 0) 0x7f979c08fa30 con 0x7f97a4153980 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v43) ==== 54+0+16 (secure 0 0 0) 0x7f979c08fe20 con 0x7f97a4153980 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979006f400 0x7f9790071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f97a4152e60 0x7f97a413a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97a414d850 con 0x7f97a4152e60 2024-04-25T15:52:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a37fe640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f97a4152e60 0x7f97a413a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979006f400 0x7f9790071910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97940af000 tx=0x7f97940a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a37fe640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f97a4152e60 0x7f97a413a7b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.539+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97a414d850 con 0x7f97a4152e60 2024-04-25T15:52:05.558 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:05.555+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:05.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.559+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a40706b0 con 0x7f97a4152e60 2024-04-25T15:52:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.559+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a40706b0 con 0x7f97a4152e60 2024-04-25T15:52:05.565 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.559+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f97a415af60 0x7f97a4137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.559+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f97a40738c0 con 0x7f97a415af60 2024-04-25T15:52:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.563+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f97a415af60 0x7f97a4137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.563+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f97a415af60 0x7f97a4137db0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.563+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97a40738c0 con 0x7f97a415af60 2024-04-25T15:52:05.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.575+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a415c030 con 0x7f97a415af60 2024-04-25T15:52:05.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.579+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a415c030 con 0x7f97a415af60 2024-04-25T15:52:05.583 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.579+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f97a41f6240 0x7f97a41f6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.579+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f97a413baf0 con 0x7f97a41f6240 2024-04-25T15:52:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.579+0000 7f97a37fe640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f97a41f6240 0x7f97a41f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.579+0000 7f97a37fe640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f97a41f6240 0x7f97a41f6650 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.579+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97a413baf0 con 0x7f97a41f6240 2024-04-25T15:52:05.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.595+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a413c000 con 0x7f97a41f6240 2024-04-25T15:52:05.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.595+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a413c000 con 0x7f97a41f6240 2024-04-25T15:52:05.601 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:05.601 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:05.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.595+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f97a4063090 0x7f97a40634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.595+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f97a40639e0 con 0x7f97a4063090 2024-04-25T15:52:05.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.595+0000 7f97a8976640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f97a4063090 0x7f97a40634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.595+0000 7f97a8976640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f97a4063090 0x7f97a40634a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.599+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97a40639e0 con 0x7f97a4063090 2024-04-25T15:52:05.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.611+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a4063ec0 con 0x7f97a4063090 2024-04-25T15:52:05.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.611+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a4063ec0 con 0x7f97a4063090 2024-04-25T15:52:05.617 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:05.618 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:05.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.611+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f97a402cef0 0x7f97a402d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.611+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f97a40639e0 con 0x7f97a402cef0 2024-04-25T15:52:05.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.611+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f97a402cef0 0x7f97a402d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.615+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f97a402cef0 0x7f97a402d300 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.615+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97a40639e0 con 0x7f97a402cef0 2024-04-25T15:52:05.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.631+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a4063cb0 con 0x7f97a402cef0 2024-04-25T15:52:05.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.635+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a4063cb0 con 0x7f97a402cef0 2024-04-25T15:52:05.638 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.635+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f97a4206f40 0x7f97a4209360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.635+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f97a4063ea0 con 0x7f97a4206f40 2024-04-25T15:52:05.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.635+0000 7f97a37fe640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f97a4206f40 0x7f97a4209360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.635+0000 7f97a37fe640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f97a4206f40 0x7f97a4209360 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.635+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97a4063ea0 con 0x7f97a4206f40 2024-04-25T15:52:05.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.651+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a402dd80 con 0x7f97a4206f40 2024-04-25T15:52:05.658 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:05.655+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:05.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.655+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a402dd80 con 0x7f97a4206f40 2024-04-25T15:52:05.660 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.655+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f97a4209db0 0x7f97a420c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.655+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f97a420f0e0 con 0x7f97a4209db0 2024-04-25T15:52:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.655+0000 7f97a8976640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f97a4209db0 0x7f97a420c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.655+0000 7f97a8976640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f97a4209db0 0x7f97a420c1d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.659+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97a420f0e0 con 0x7f97a4209db0 2024-04-25T15:52:05.666 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:05.666 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:05.666 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:05.666 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:05.666 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:05.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.671+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a420f030 con 0x7f97a4209db0 2024-04-25T15:52:05.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.679+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a420f030 con 0x7f97a4209db0 2024-04-25T15:52:05.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.679+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f97a420cc20 0x7f97a420d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.679+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f97a420ede0 con 0x7f97a420cc20 2024-04-25T15:52:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.679+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f97a420cc20 0x7f97a420d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.679+0000 7f97a3fff640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f97a420cc20 0x7f97a420d030 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.683+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97940d0040 con 0x7f97a420cc20 2024-04-25T15:52:05.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.695+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f97a4211330 con 0x7f97a420cc20 2024-04-25T15:52:05.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97a0ff9640 1 -- 172.21.15.67:0/3669896193 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f97a4211330 con 0x7f97a420cc20 2024-04-25T15:52:05.702 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:05.703 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:05.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f97a4152e60 msgr2=0x7f97a413a7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f97a4152e60 0x7f97a413a7b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f97a415af60 msgr2=0x7f97a4137db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f97a415af60 0x7f97a4137db0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f97a41f6240 msgr2=0x7f97a41f6650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f97a41f6240 0x7f97a41f6650 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f97a4063090 msgr2=0x7f97a40634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f97a4063090 0x7f97a40634a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f97a402cef0 msgr2=0x7f97a402d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f97a402cef0 0x7f97a402d300 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f97a4206f40 msgr2=0x7f97a4209360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f97a4206f40 0x7f97a4209360 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f97a4209db0 msgr2=0x7f97a420c1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f97a4209db0 0x7f97a420c1d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f97a420cc20 msgr2=0x7f97a420d030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f97a420cc20 0x7f97a420d030 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 reap_dead start 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 reap_dead start 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 reap_dead start 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 reap_dead start 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97a8976640 1 -- 172.21.15.67:0/3669896193 reap_dead start 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979006f400 msgr2=0x7f9790071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979006f400 0x7f9790071910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97940af000 tx=0x7f97940a7000 comp rx=0 tx=0).stop 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a4153980 msgr2=0x7f97a41511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a4153980 0x7f97a41511e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f979c00c760 tx=0x7f979c00cb60 comp rx=0 tx=0).stop 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 shutdown_connections 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979006f400 0x7f9790071910 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 --2- 172.21.15.67:0/3669896193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97a4153980 0x7f97a41511e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 >> 172.21.15.67:0/3669896193 conn(0x7f97a40700c0 msgr2=0x7f97a4072cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 shutdown_connections 2024-04-25T15:52:05.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:05.699+0000 7f97aa3d9640 1 -- 172.21.15.67:0/3669896193 wait complete. 2024-04-25T15:52:05.762 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:05.759+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:05.829 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:05.829 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:05.846 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:05.860 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:05.855+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:05.961 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:05.956+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:05.998 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:05.999 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:06.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:06.061 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:06.056+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:06.161 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:06.156+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:06.210 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:06.211 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:06.228 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:06.262 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:06.259+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:06.364 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:06.360+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:06.421 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:06.421 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:06.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:06.482 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:06.479+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:06.566 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:06.560+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:06.607 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:06.608 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:06.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:06.666 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:06.660+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:06.767 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:06.764+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:06.835 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:06.835 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:06.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:06.867 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:06.864+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:06.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:52:07.004 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:07.004 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:07.005 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:06.999+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:07.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:07.070 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:07.064+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:07.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.103+0000 7f5c720ac640 1 Processor -- start 2024-04-25T15:52:07.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.103+0000 7f5c720ac640 1 -- start start 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c720ac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c6c0c96e0 0x7f5c6c168190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c720ac640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6c06d8e0 con 0x7f5c6c160800 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c720ac640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6c073c70 con 0x7f5c6c0c92d0 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c720ac640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5c6c06dbc0 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c6c0c96e0 0x7f5c6c168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c6c160800 0x7f5c6c06d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48704/0 (socket says 172.21.15.67:48704) 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6b7fe640 1 -- 172.21.15.67:0/2138475970 learned_addr learned my addr 172.21.15.67:0/2138475970 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6bfff640 1 -- 172.21.15.67:0/2138475970 >> v1:172.21.15.176:6789/0 conn(0x7f5c6c0c92d0 legacy=0x7f5c6c072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6bfff640 1 -- 172.21.15.67:0/2138475970 >> v1:172.21.15.67:6789/0 conn(0x7f5c6c160800 legacy=0x7f5c6c06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6a7fc640 1 -- 172.21.15.67:0/2138475970 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 818496566 0 0) 0x7f5c6c06d8e0 con 0x7f5c6c160800 2024-04-25T15:52:07.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6bfff640 1 -- 172.21.15.67:0/2138475970 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c073f50 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.107+0000 7f5c6bfff640 1 --2- 172.21.15.67:0/2138475970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c168190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009d90 tx=0x7f5c6401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30af3a39bd733801 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c6a7fc640 1 -- 172.21.15.67:0/2138475970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c640b2070 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c6a7fc640 1 -- 172.21.15.67:0/2138475970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c640a5850 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c6a7fc640 1 -- 172.21.15.67:0/2138475970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c640a5c20 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 -- 172.21.15.67:0/2138475970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 msgr2=0x7f5c6c168190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 --2- 172.21.15.67:0/2138475970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c168190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009d90 tx=0x7f5c6401c8a0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 -- 172.21.15.67:0/2138475970 shutdown_connections 2024-04-25T15:52:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 --2- 172.21.15.67:0/2138475970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c168190 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 -- 172.21.15.67:0/2138475970 >> 172.21.15.67:0/2138475970 conn(0x7f5c6c070db0 msgr2=0x7f5c6c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 -- 172.21.15.67:0/2138475970 shutdown_connections 2024-04-25T15:52:07.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 -- 172.21.15.67:0/2138475970 wait complete. 2024-04-25T15:52:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 Processor -- start 2024-04-25T15:52:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.111+0000 7f5c720ac640 1 -- start start 2024-04-25T15:52:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c6c0c92d0 0x7f5c6c15f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c15e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c6c160800 0x7f5c6c15e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c15e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c15e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60564/0 (socket says 172.21.15.67:60564) 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 -- 172.21.15.67:0/3744512243 learned_addr learned my addr 172.21.15.67:0/3744512243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c160d90 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c160c10 con 0x7f5c6c160800 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c0734d0 con 0x7f5c6c0c92d0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6bfff640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c6c160800 0x7f5c6c15e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 -- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c6c0c92d0 msgr2=0x7f5c6c15f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c6c0c92d0 0x7f5c6c15f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 -- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c6c160800 msgr2=0x7f5c6c15e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c6c160800 0x7f5c6c15e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c15ef20 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c6affd640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c15e4a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000b440 tx=0x7f5c6000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c53fff640 1 -- 172.21.15.67:0/3744512243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60015020 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c53fff640 1 -- 172.21.15.67:0/3744512243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c60011890 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c15f9a0 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c53fff640 1 -- 172.21.15.67:0/3744512243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60011c80 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c0c73e0 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c6c06a480 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c53fff640 1 -- 172.21.15.67:0/3744512243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5c60018020 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.115+0000 7f5c53fff640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c5806f430 0x7f5c58071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.119+0000 7f5c53fff640 1 -- 172.21.15.67:0/3744512243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5857+0+0 (secure 0 0 0) 0x7f5c6008f110 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.119+0000 7f5c6b7fe640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c5806f430 0x7f5c58071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.119+0000 7f5c6b7fe640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c5806f430 0x7f5c58071940 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0015a0 tx=0x7f5c5c007970 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.123+0000 7f5c53fff640 1 -- 172.21.15.67:0/3744512243 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5c6001f030 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.170 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:07.164+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:07.220 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:07.221 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:07.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:07.278 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:07.275+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.295+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f5c6c15a4d0 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.372 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:07.367+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:07.414 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:07.414 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:07.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:07.478 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:07.475+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:07.574 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:07.571+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:07.621 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:07.621 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:07.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:07.678 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:07.675+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:07.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:07.775 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:07.771+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:07.816 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:07.816 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:07.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:07.876 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:07.876+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.879+0000 7f859515a640 1 Processor -- start 2024-04-25T15:52:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.879+0000 7f859515a640 1 -- start start 2024-04-25T15:52:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.879+0000 7f859515a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8590162ec0 0x7f85901652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.879+0000 7f859515a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85900c77a0 con 0x7f8590073bf0 2024-04-25T15:52:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.879+0000 7f859515a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85900c7a60 con 0x7f85900caf60 2024-04-25T15:52:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.879+0000 7f859515a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f85900cab60 con 0x7f8590162ec0 2024-04-25T15:52:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8590073bf0 0x7f859006d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48712/0 (socket says 172.21.15.67:48712) 2024-04-25T15:52:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858e59c640 1 -- 172.21.15.67:0/4034479593 learned_addr learned my addr 172.21.15.67:0/4034479593 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858f59e640 1 --2- 172.21.15.67:0/4034479593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8590162ec0 0x7f85901652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858dd9b640 1 -- 172.21.15.67:0/4034479593 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622357652 0 0) 0x7f85900c77a0 con 0x7f8590073bf0 2024-04-25T15:52:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858dd9b640 1 -- 172.21.15.67:0/4034479593 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8578003530 con 0x7f8590073bf0 2024-04-25T15:52:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858f59e640 1 -- 172.21.15.67:0/4034479593 >> v1:172.21.15.176:6789/0 conn(0x7f85900caf60 legacy=0x7f859015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858f59e640 1 -- 172.21.15.67:0/4034479593 >> v1:172.21.15.67:6789/0 conn(0x7f8590073bf0 legacy=0x7f859006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.883+0000 7f858f59e640 1 -- 172.21.15.67:0/4034479593 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85900c7d40 con 0x7f8590162ec0 2024-04-25T15:52:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f858f59e640 1 --2- 172.21.15.67:0/4034479593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8590162ec0 0x7f85901652e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8588009dc0 tx=0x7f858801eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68e0fd69f2b83277 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f858dd9b640 1 -- 172.21.15.67:0/4034479593 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880b4070 con 0x7f8590162ec0 2024-04-25T15:52:07.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f858dd9b640 1 -- 172.21.15.67:0/4034479593 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85880a7970 con 0x7f8590162ec0 2024-04-25T15:52:07.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f858dd9b640 1 -- 172.21.15.67:0/4034479593 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880a7d20 con 0x7f8590162ec0 2024-04-25T15:52:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f859515a640 1 -- 172.21.15.67:0/4034479593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8590162ec0 msgr2=0x7f85901652e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f859515a640 1 --2- 172.21.15.67:0/4034479593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8590162ec0 0x7f85901652e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8588009dc0 tx=0x7f858801eba0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f859515a640 1 -- 172.21.15.67:0/4034479593 shutdown_connections 2024-04-25T15:52:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f859515a640 1 --2- 172.21.15.67:0/4034479593 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8590162ec0 0x7f85901652e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f859515a640 1 -- 172.21.15.67:0/4034479593 >> 172.21.15.67:0/4034479593 conn(0x7f8590070db0 msgr2=0x7f85900ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f859515a640 1 -- 172.21.15.67:0/4034479593 shutdown_connections 2024-04-25T15:52:07.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.887+0000 7f859515a640 1 -- 172.21.15.67:0/4034479593 wait complete. 2024-04-25T15:52:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 Processor -- start 2024-04-25T15:52:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 -- start start 2024-04-25T15:52:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8590073bf0 0x7f8590153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85900caf60 0x7f85900c5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8590162ec0 0x7f859015c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f859015a3f0 con 0x7f8590162ec0 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8590159620 con 0x7f8590073bf0 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f858f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8590162ec0 0x7f859015c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f858f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8590162ec0 0x7f859015c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60566/0 (socket says 172.21.15.67:60566) 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f858f59e640 1 -- 172.21.15.67:0/3772295173 learned_addr learned my addr 172.21.15.67:0/3772295173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f859015a570 con 0x7f85900caf60 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.891+0000 7f858e59c640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85900caf60 0x7f85900c5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8590073bf0 0x7f8590153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858f59e640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85900caf60 msgr2=0x7f85900c5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858f59e640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85900caf60 0x7f85900c5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858f59e640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8590073bf0 msgr2=0x7f8590153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858f59e640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8590073bf0 0x7f8590153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858f59e640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8590154530 con 0x7f8590162ec0 2024-04-25T15:52:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8590073bf0 0x7f8590153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f858f59e640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8590162ec0 0x7f859015c190 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f85880251b0 tx=0x7f858801b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880b4040 con 0x7f8590162ec0 2024-04-25T15:52:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85880bd070 con 0x7f8590162ec0 2024-04-25T15:52:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8588003ba0 con 0x7f8590162ec0 2024-04-25T15:52:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85901f95a0 con 0x7f8590162ec0 2024-04-25T15:52:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85901f9b60 con 0x7f8590162ec0 2024-04-25T15:52:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.895+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f859006a480 con 0x7f8590162ec0 2024-04-25T15:52:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8588004040 con 0x7f8590162ec0 2024-04-25T15:52:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f85777fe640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f857c06f400 0x7f857c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5857+0+0 (secure 0 0 0) 0x7f8588129810 con 0x7f8590162ec0 2024-04-25T15:52:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v43) ==== 54+0+16 (secure 0 0 0) 0x7f8588018790 con 0x7f8590162ec0 2024-04-25T15:52:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f857c06f400 0x7f857c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f85901f9d60 0x7f85900c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85900c8a40 con 0x7f85901f9d60 2024-04-25T15:52:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f858e59c640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f85901f9d60 0x7f85900c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f858e59c640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f85901f9d60 0x7f85900c9f80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f857c06f400 0x7f857c071910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8580007fd0 tx=0x7f8580009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.899+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f85900c8a40 con 0x7f85901f9d60 2024-04-25T15:52:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.931+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f85900cad40 con 0x7f85901f9d60 2024-04-25T15:52:07.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.931+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f85900cad40 con 0x7f85901f9d60 2024-04-25T15:52:07.937 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:07.937 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:07.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.931+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8590154700 0x7f85901f6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.931+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8590158f60 con 0x7f8590154700 2024-04-25T15:52:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.931+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8590154700 0x7f85901f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.931+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8590154700 0x7f85901f6410 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.935+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8590158f60 con 0x7f8590154700 2024-04-25T15:52:07.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.955+0000 7f5c53fff640 1 -- 172.21.15.67:0/3744512243 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v44) ==== 97+0+0 (secure 0 0 0) 0x7f5c6c15ef20 con 0x7f5c6c0c96e0 2024-04-25T15:52:07.958 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c5806f430 msgr2=0x7f5c58071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c5806f430 0x7f5c58071940 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0015a0 tx=0x7f5c5c007970 comp rx=0 tx=0).stop 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 msgr2=0x7f5c6c15e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c15e4a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000b440 tx=0x7f5c6000b900 comp rx=0 tx=0).stop 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 shutdown_connections 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c5806f430 0x7f5c58071940 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c6c160800 0x7f5c6c15e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c6c0c96e0 0x7f5c6c15e4a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.959+0000 7f5c720ac640 1 --2- 172.21.15.67:0/3744512243 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c6c0c92d0 0x7f5c6c15f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.963+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 >> 172.21.15.67:0/3744512243 conn(0x7f5c6c070db0 msgr2=0x7f5c6c0ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.963+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 shutdown_connections 2024-04-25T15:52:07.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.963+0000 7f5c720ac640 1 -- 172.21.15.67:0/3744512243 wait complete. 2024-04-25T15:52:07.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.963+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f859015d5d0 con 0x7f8590154700 2024-04-25T15:52:07.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.967+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f859015d5d0 con 0x7f8590154700 2024-04-25T15:52:07.971 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.967+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8590155370 0x7f8590063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.967+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f85900c57d0 con 0x7f8590155370 2024-04-25T15:52:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.967+0000 7f858e59c640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8590155370 0x7f8590063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:07.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.967+0000 7f858e59c640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8590155370 0x7f8590063090 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:07.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.967+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f85900c57d0 con 0x7f8590155370 2024-04-25T15:52:07.976 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:07.972+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.995+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8590155dc0 con 0x7f8590155370 2024-04-25T15:52:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.999+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8590155dc0 con 0x7f8590155370 2024-04-25T15:52:08.005 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.999+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f85900cc730 0x7f85900ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.999+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8590063ea0 con 0x7f85900cc730 2024-04-25T15:52:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:07.999+0000 7f858f59e640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f85900cc730 0x7f85900ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.003+0000 7f858f59e640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f85900cc730 0x7f85900ccb40 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.003+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8590063ea0 con 0x7f85900cc730 2024-04-25T15:52:08.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.031+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f85900cd350 con 0x7f85900cc730 2024-04-25T15:52:08.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.031+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f85900cd350 con 0x7f85900cc730 2024-04-25T15:52:08.037 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.031+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f859002cef0 0x7f859002d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.031+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f85900cd560 con 0x7f859002cef0 2024-04-25T15:52:08.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.035+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f859002cef0 0x7f859002d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.035+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f859002cef0 0x7f859002d300 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:08.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.035+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f85900cd560 con 0x7f859002cef0 2024-04-25T15:52:08.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.051+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f85900cd560 con 0x7f859002cef0 2024-04-25T15:52:08.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.055+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f85900cd560 con 0x7f859002cef0 2024-04-25T15:52:08.059 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:08.059 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:08.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.055+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f859020a850 0x7f859020cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:08.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.055+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f859002dd50 con 0x7f859020a850 2024-04-25T15:52:08.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.055+0000 7f858e59c640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f859020a850 0x7f859020cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:08.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.055+0000 7f858e59c640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f859020a850 0x7f859020cc70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:08.061 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:08.061 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:08.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.059+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f859002dd50 con 0x7f859020a850 2024-04-25T15:52:08.076 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:08.076+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:08.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:08.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.083+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f859002dd50 con 0x7f859020a850 2024-04-25T15:52:08.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.087+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f859002dd50 con 0x7f859020a850 2024-04-25T15:52:08.093 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.091+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f859020d6c0 0x7f859020fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.091+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f859002dd50 con 0x7f859020d6c0 2024-04-25T15:52:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.091+0000 7f858f59e640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f859020d6c0 0x7f859020fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.091+0000 7f858f59e640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f859020d6c0 0x7f859020fae0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.091+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f859002dd50 con 0x7f859020d6c0 2024-04-25T15:52:08.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.107+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f859002dd50 con 0x7f859020d6c0 2024-04-25T15:52:08.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.111+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f859002dd50 con 0x7f859020d6c0 2024-04-25T15:52:08.115 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:08.116 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:08.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.111+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8590210530 0x7f8590212950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:08.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.111+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8590215920 con 0x7f8590210530 2024-04-25T15:52:08.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.111+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8590210530 0x7f8590212950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:08.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.111+0000 7f858ed9d640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8590210530 0x7f8590212950 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:08.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.115+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8590215920 con 0x7f8590210530 2024-04-25T15:52:08.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.127+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8590000960 con 0x7f8590210530 2024-04-25T15:52:08.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f85777fe640 1 -- 172.21.15.67:0/3772295173 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8590000960 con 0x7f8590210530 2024-04-25T15:52:08.137 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f85901f9d60 msgr2=0x7f85900c9f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f85901f9d60 0x7f85900c9f80 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8590154700 msgr2=0x7f85901f6410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8590154700 0x7f85901f6410 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8590155370 msgr2=0x7f8590063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8590155370 0x7f8590063090 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f858f59e640 1 -- 172.21.15.67:0/3772295173 reap_dead start 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f85900cc730 msgr2=0x7f85900ccb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f85900cc730 0x7f85900ccb40 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f859002cef0 msgr2=0x7f859002d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f859002cef0 0x7f859002d300 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f859020a850 msgr2=0x7f859020cc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f859020a850 0x7f859020cc70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f859020d6c0 msgr2=0x7f859020fae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f859020d6c0 0x7f859020fae0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8590210530 msgr2=0x7f8590212950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8590210530 0x7f8590212950 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f858f59e640 1 -- 172.21.15.67:0/3772295173 reap_dead start 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f857c06f400 msgr2=0x7f857c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.131+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f857c06f400 0x7f857c071910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8580007fd0 tx=0x7f8580009040 comp rx=0 tx=0).stop 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8590162ec0 msgr2=0x7f859015c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8590162ec0 0x7f859015c190 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f85880251b0 tx=0x7f858801b940 comp rx=0 tx=0).stop 2024-04-25T15:52:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 shutdown_connections 2024-04-25T15:52:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f857c06f400 0x7f857c071910 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 --2- 172.21.15.67:0/3772295173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8590162ec0 0x7f859015c190 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 >> 172.21.15.67:0/3772295173 conn(0x7f8590070db0 msgr2=0x7f8590157490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 shutdown_connections 2024-04-25T15:52:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:08.135+0000 7f859515a640 1 -- 172.21.15.67:0/3772295173 wait complete. 2024-04-25T15:52:08.206 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:08.203+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.012010305999993, 2024-04-25T15:52:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937687, 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212220+0000", 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212245+0000", 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212458+0000", 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021240899999999999 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212488+0000", 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0715e-05 2024-04-25T15:52:08.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212836+0000", 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000348096 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213194+0000", 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000357413 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213591+0000", 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039711599999999998 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224723+0000", 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011132177 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224892+0000", 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016872100000000001 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225031+0000", 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013966 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225148+0000", 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116168 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011964004999996, 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013374822, 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212257+0000", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212263+0000", 2024-04-25T15:52:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0900000000000001e-06 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212576+0000", 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031227099999999998 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212590+0000", 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4133e-05 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214544+0000", 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001953939 2024-04-25T15:52:08.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214793+0000", 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024941399999999999 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215181+0000", 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038823800000000001 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224739+0000", 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095574579999999996 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225301+0000", 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056223199999999997 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225459+0000", 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015786199999999999 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225631+0000", 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172242 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011953591999998, 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013916951, 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:08.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8299999999999996e-07 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212272+0000", 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6630000000000001e-06 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212665+0000", 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039338399999999999 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212687+0000", 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1882000000000002e-05 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215937+0000", 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032501119999999999 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032469999999999998 2024-04-25T15:52:08.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216646+0000", 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038413800000000001 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224753+0000", 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081070039999999993 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225911+0000", 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001158002 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226062+0000", 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015119600000000001 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226184+0000", 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121187 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:08.240 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011945623000003, 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01435588, 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6000000000000003e-07 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212279+0000", 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1169999999999997e-06 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212723+0000", 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044335799999999998 2024-04-25T15:52:08.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212733+0000", 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349e-05 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217516+0000", 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004782606 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217766+0000", 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025045699999999999 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039180399999999999 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224766+0000", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066076449999999997 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226308+0000", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00154228 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226483+0000", 2024-04-25T15:52:08.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174938 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226630+0000", 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014766599999999999 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011938642999993, 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014867514, 2024-04-25T15:52:08.245 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:08.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2699999999999999e-07 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212287+0000", 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6229999999999996e-06 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212756+0000", 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046907999999999997 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9560000000000003e-06 2024-04-25T15:52:08.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218829+0000", 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060638690000000004 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219182+0000", 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035291899999999999 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219507+0000", 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032545399999999998 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224779+0000", 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052717279999999998 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226860+0000", 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020809790000000002 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227036+0000", 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176062 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227149+0000", 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113217 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011931172999994, 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015281197, 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212290+0000", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3e-07 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212333+0000", 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3251000000000003e-05 2024-04-25T15:52:08.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212784+0000", 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045139399999999998 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212793+0000", 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7960000000000001e-06 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220253+0000", 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074595699999999996 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220482+0000", 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000229554 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220879+0000", 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039646400000000002 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225181+0000", 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00430205 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227277+0000", 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020961119999999998 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227449+0000", 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017166800000000001 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227570+0000", 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012170799999999999 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.254 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011881693000007, 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016239407000000001, 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212340+0000", 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1200000000000001e-06 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212346+0000", 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.584e-06 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212813+0000", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046677600000000002 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212821+0000", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221805+0000", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089835999999999996 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222154+0000", 2024-04-25T15:52:08.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034940299999999997 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222503+0000", 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034907900000000002 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227840+0000", 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367349999999996 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228310+0000", 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047011999999999998 2024-04-25T15:52:08.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228439+0000", 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012899999999999999 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228578+0000", 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138837 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011870482999996, 2024-04-25T15:52:08.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016443972000000001, 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212351+0000", 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212357+0000", 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000003e-06 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212848+0000", 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049076600000000001 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212863+0000", 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5498999999999998e-05 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223300+0000", 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010437093999999999 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223537+0000", 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000236665 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224027+0000", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048967900000000003 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228184+0000", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041571209999999997 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228599+0000", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000415392 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228676+0000", 2024-04-25T15:52:08.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7095999999999995e-05 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228794+0000", 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011746700000000001 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011859504, 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013451848000000001, 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.264 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212362+0000", 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212368+0000", 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6229999999999999e-06 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212925+0000", 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055690199999999996 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212960+0000", 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5045000000000001e-05 2024-04-25T15:52:08.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213123+0000", 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163222 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213447+0000", 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000323627 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213759+0000", 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031232800000000001 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216671+0000", 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002911145 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223654+0000", 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069831370000000004 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223829+0000", 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174857 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225813+0000", 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019840679999999999 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.011848341000004, 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014242332, 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212373+0000", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4e-07 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212384+0000", 2024-04-25T15:52:08.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1443e-05 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213026+0000", 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064212500000000001 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213045+0000", 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8621999999999999e-05 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214708+0000", 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016628109999999999 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214970+0000", 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026197699999999999 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215380+0000", 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000410789 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218599+0000", 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032185270000000001 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226296+0000", 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076965269999999999 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226473+0000", 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017755200000000001 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226614+0000", 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.273 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:4 2.10 2:0f65c9b8:::60a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.977045007000001, 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010394548999999999, 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247176+0000", 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2100000000000001e-06 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247209+0000", 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2945999999999999e-05 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247394+0000", 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018468799999999999 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247417+0000", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2733000000000001e-05 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249734+0000", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002317684 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250032+0000", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029727699999999998 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250453+0000", 2024-04-25T15:52:08.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042143500000000001 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255742+0000", 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052886809999999999 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257232+0000", 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014906260000000001 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257393+0000", 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016088699999999999 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257570+0000", 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017638199999999999 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.893076441000005, 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094210270000000002, 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.278 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6699999999999999e-07 2024-04-25T15:52:08.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331165+0000", 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331418+0000", 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271799999999997 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331428+0000", 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6460000000000005e-06 2024-04-25T15:52:08.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333229+0000", 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018009499999999999 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334396+0000", 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011673269999999999 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334597+0000", 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020135300000000001 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339905+0000", 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053077610000000002 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340210+0000", 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030499999999999999 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340385+0000", 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017450199999999999 2024-04-25T15:52:08.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340565+0000", 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018011700000000001 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.893051694999997, 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095762039999999996, 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.283 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331174+0000", 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-06 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331467+0000", 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029356 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331478+0000", 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336604+0000", 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051267170000000003 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336735+0000", 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013052699999999999 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336996+0000", 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260511 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339986+0000", 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002990487 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340579+0000", 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059299899999999998 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340673+0000", 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4241999999999997e-05 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340745+0000", 2024-04-25T15:52:08.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1484999999999997e-05 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:17 2.10 2:0e9da45e:::61b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.893044449000001, 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010814343000000001, 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:08.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331180+0000", 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6650000000000001e-06 2024-04-25T15:52:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331501+0000", 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032061600000000002 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331509+0000", 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8240000000000006e-06 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333383+0000", 2024-04-25T15:52:08.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018733020000000001 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333652+0000", 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026890399999999999 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333865+0000", 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021341 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338313+0000", 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044478269999999997 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341868+0000", 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035546729999999999 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341920+0000", 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2507999999999999e-05 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341990+0000", 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0110000000000005e-05 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.893034776999997, 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011086621, 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331186+0000", 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.51e-07 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331191+0000", 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3970000000000002e-06 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331546+0000", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035447799999999998 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331557+0000", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0791000000000001e-05 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334408+0000", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028516520000000001 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334546+0000", 2024-04-25T15:52:08.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013802700000000001 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334877+0000", 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000331081 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339910+0000", 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050323449999999997 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342080+0000", 2024-04-25T15:52:08.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021703899999999999 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342184+0000", 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103767 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342272+0000", 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8242e-05 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.297 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.893026757000001, 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010331643, 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331194+0000", 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331198+0000", 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1400000000000002e-06 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331608+0000", 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040962799999999999 2024-04-25T15:52:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331620+0000", 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337545+0000", 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005924726 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338432+0000", 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088720699999999999 2024-04-25T15:52:08.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338639+0000", 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020665000000000001 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341318+0000", 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026785860000000002 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341406+0000", 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8344000000000007e-05 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341458+0000", 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805999999999998e-05 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341525+0000", 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7336999999999996e-05 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:2 2.10 2:0daba267:::402.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:08.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.893020297999996, 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011245289, 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331215+0000", 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331647+0000", 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043194799999999998 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331656+0000", 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335316+0000", 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036607710000000002 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335504+0000", 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018793299999999999 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335699+0000", 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194544 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339945+0000", 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042457600000000003 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342333+0000", 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023885730000000002 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342383+0000", 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9314999999999998e-05 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342445+0000", 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2691e-05 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.893001537000004, 2024-04-25T15:52:08.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087706009999999994, 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331239+0000", 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9653e-05 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331678+0000", 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043868999999999999 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331701+0000", 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3019000000000001e-05 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333753+0000", 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020522050000000001 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334131+0000", 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000378484 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334455+0000", 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032319899999999998 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337268+0000", 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028133199999999998 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339816+0000", 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025479140000000001 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339907+0000", 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1150000000000007e-05 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339989+0000", 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2392000000000007e-05 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:27 2.10 2:091e33df:::502.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.892978495999998, 2024-04-25T15:52:08.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011447874, 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4199999999999996e-07 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331249+0000", 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2259999999999999e-06 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331730+0000", 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048009600000000001 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331739+0000", 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1840000000000005e-06 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336653+0000", 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049141510000000003 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336830+0000", 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017740099999999999 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:08.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337033+0000", 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020295999999999999 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340803+0000", 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037692480000000002 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342567+0000", 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017649250000000001 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342650+0000", 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2748000000000001e-05 2024-04-25T15:52:08.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342690+0000", 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9393e-05 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:31 2.10 2:0eee8b97:::202.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.892968639000003, 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012346048, 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:08.317 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:08.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5300000000000004e-07 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331255+0000", 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8380000000000002e-06 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331760+0000", 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050452600000000004 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331768+0000", 2024-04-25T15:52:08.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337613+0000", 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058448700000000003 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337836+0000", 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000223043 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.343598+0000", 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057612679999999999 2024-04-25T15:52:08.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.321 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.321 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.321 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:08.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:08.322 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:08.319+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:08.424 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:08.420+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:08.497 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:08.497 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:08.497 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:08.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:08.558 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:08.555+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:08.625 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:08.619+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:08.644 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:08.645 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:08.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:08.726 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:08.724+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:08.829 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:08.823+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.626563355000002, 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.841 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:52:08.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.626523710000001, 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.623616687999998, 2024-04-25T15:52:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.623463752000006, 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:52:08.852 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:08.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.621948107999998, 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.856 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:52:08.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.621805460999994, 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:08.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:52:08.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.621702369999994, 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.867 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.621549619999996, 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.619601394, 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:52:08.871 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:52:08.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.591573763, 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:52:08.874 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:52:08.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:08.877 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.589550872000004, 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:52:08.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.589380323, 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:52:08.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.884 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.587024826999993, 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:52:08.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:52:08.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:52:08.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.889 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.586990115000006, 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.586969306, 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:08.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.896 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.510082259000001, 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:52:08.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:52:08.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:52:08.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.510058177000005, 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:52:08.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.510048479999995, 2024-04-25T15:52:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:52:08.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 76.504153543000001, 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.910 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:52:08.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:52:08.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.914714885999999, 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:52:08.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:52:08.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:08.918 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:08.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:08.962 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:08.959+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:09.029 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:09.024+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:09.080 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:09.080 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:09.097 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:09.139 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:09.140+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:09.231 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:09.228+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:09.258 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:09.258 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:09.258 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T15:52:09.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:52:09.287 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:09.287 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:09.288 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:09.288 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:09.288 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:09.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:09.332 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:09.327+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:09.433 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:09.428+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.431+0000 7f6f4caae640 1 Processor -- start 2024-04-25T15:52:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.431+0000 7f6f4caae640 1 -- start start 2024-04-25T15:52:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f4caae640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f480b5d70 0x7f6f4806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f4caae640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f4806d780 con 0x7f6f4815fe00 2024-04-25T15:52:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f4caae640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f48160920 con 0x7f6f4806ca60 2024-04-25T15:52:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f4caae640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6f4815d510 con 0x7f6f480b5d70 2024-04-25T15:52:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f46d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6f4815fe00 0x7f6f48160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48726/0 (socket says 172.21.15.67:48726) 2024-04-25T15:52:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f45d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f480b5d70 0x7f6f4806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f46d9d640 1 -- 172.21.15.67:0/3509976118 learned_addr learned my addr 172.21.15.67:0/3509976118 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f45d9b640 1 -- 172.21.15.67:0/3509976118 >> v1:172.21.15.176:6789/0 conn(0x7f6f4806ca60 legacy=0x7f6f480b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f45d9b640 1 -- 172.21.15.67:0/3509976118 >> v1:172.21.15.67:6789/0 conn(0x7f6f4815fe00 legacy=0x7f6f48160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f4559a640 1 -- 172.21.15.67:0/3509976118 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 84711095 0 0) 0x7f6f4806d780 con 0x7f6f4815fe00 2024-04-25T15:52:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.435+0000 7f6f45d9b640 1 -- 172.21.15.67:0/3509976118 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f48160be0 con 0x7f6f480b5d70 2024-04-25T15:52:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f45d9b640 1 --2- 172.21.15.67:0/3509976118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f480b5d70 0x7f6f4806ce70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0097f0 tx=0x7f6f3c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426870cfa3ec1532 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:09.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4559a640 1 -- 172.21.15.67:0/3509976118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c0b0070 con 0x7f6f480b5d70 2024-04-25T15:52:09.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4559a640 1 -- 172.21.15.67:0/3509976118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f3c004230 con 0x7f6f480b5d70 2024-04-25T15:52:09.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4559a640 1 -- 172.21.15.67:0/3509976118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c003620 con 0x7f6f480b5d70 2024-04-25T15:52:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4caae640 1 -- 172.21.15.67:0/3509976118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f480b5d70 msgr2=0x7f6f4806ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4caae640 1 --2- 172.21.15.67:0/3509976118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f480b5d70 0x7f6f4806ce70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0097f0 tx=0x7f6f3c01c460 comp rx=0 tx=0).stop 2024-04-25T15:52:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4caae640 1 -- 172.21.15.67:0/3509976118 shutdown_connections 2024-04-25T15:52:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4caae640 1 --2- 172.21.15.67:0/3509976118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f480b5d70 0x7f6f4806ce70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4caae640 1 -- 172.21.15.67:0/3509976118 >> 172.21.15.67:0/3509976118 conn(0x7f6f4806e420 msgr2=0x7f6f4806e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4caae640 1 -- 172.21.15.67:0/3509976118 shutdown_connections 2024-04-25T15:52:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.439+0000 7f6f4caae640 1 -- 172.21.15.67:0/3509976118 wait complete. 2024-04-25T15:52:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 Processor -- start 2024-04-25T15:52:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 -- start start 2024-04-25T15:52:09.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4806ca60 0x7f6f4815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f480b5d70 0x7f6f4814c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f4815fe00 0x7f6f48149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f480b6e90 con 0x7f6f4806ca60 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4815c890 con 0x7f6f4815fe00 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6f480b7010 con 0x7f6f480b5d70 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4806ca60 0x7f6f4815df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4806ca60 0x7f6f4815df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60574/0 (socket says 172.21.15.67:60574) 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 -- 172.21.15.67:0/582925854 learned_addr learned my addr 172.21.15.67:0/582925854 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f45d9b640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f480b5d70 0x7f6f4814c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f46d9d640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f4815fe00 0x7f6f48149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 -- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f480b5d70 msgr2=0x7f6f4814c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f480b5d70 0x7f6f4814c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 -- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f4815fe00 msgr2=0x7f6f48149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f4815fe00 0x7f6f48149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 -- 172.21.15.67:0/582925854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4814f140 con 0x7f6f4806ca60 2024-04-25T15:52:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4659c640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4806ca60 0x7f6f4815df70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800b410 tx=0x7f6f3800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f36ffd640 1 -- 172.21.15.67:0/582925854 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38015020 con 0x7f6f4806ca60 2024-04-25T15:52:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f36ffd640 1 -- 172.21.15.67:0/582925854 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f38011850 con 0x7f6f4806ca60 2024-04-25T15:52:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f4814f3e0 con 0x7f6f4806ca60 2024-04-25T15:52:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.443+0000 7f6f36ffd640 1 -- 172.21.15.67:0/582925854 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38011c40 con 0x7f6f4806ca60 2024-04-25T15:52:09.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.447+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f4814e690 con 0x7f6f4806ca60 2024-04-25T15:52:09.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.447+0000 7f6f36ffd640 1 -- 172.21.15.67:0/582925854 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6f38018020 con 0x7f6f4806ca60 2024-04-25T15:52:09.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.447+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f4806a480 con 0x7f6f4806ca60 2024-04-25T15:52:09.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.447+0000 7f6f36ffd640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6f2806f430 0x7f6f28071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.447+0000 7f6f36ffd640 1 -- 172.21.15.67:0/582925854 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5857+0+0 (secure 0 0 0) 0x7f6f3808f2d0 con 0x7f6f4806ca60 2024-04-25T15:52:09.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.447+0000 7f6f45d9b640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6f2806f430 0x7f6f28071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.447+0000 7f6f45d9b640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6f2806f430 0x7f6f28071940 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0ad000 tx=0x7f6f3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:09.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.455+0000 7f6f36ffd640 1 -- 172.21.15.67:0/582925854 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6f3801f020 con 0x7f6f4806ca60 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.290559294999994, 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:52:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:52:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.287944498000002, 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.287445601000002, 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.286356354999995, 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.285735860000003, 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:52:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.283341633999996, 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:52:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.254221231000002, 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:52:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.249476497000003, 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:52:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.245948995999996, 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.245816515000001, 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.624877057000006, 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:52:09.542 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:52:09.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.613441925000004, 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:52:09.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.613044489000004, 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:52:09.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.612934748000001, 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:52:09.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:52:09.558 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.612434003000004, 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:52:09.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.612058016999995, 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.562 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:52:09.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 65.573459834000005, 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:52:09.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 65.573359574999998, 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:52:09.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.573 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 65.571254659999994, 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:09.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:52:09.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 65.570879083999998, 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:09.577 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:52:09.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:52:09.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:09.580 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:09.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:09.582 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:09.575+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:09.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.651+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f6f480cd380 con 0x7f6f4806ca60 2024-04-25T15:52:09.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.651+0000 7f6f36ffd640 1 -- 172.21.15.67:0/582925854 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v44) ==== 81+0+35017 (secure 0 0 0) 0x7f6f3805c1b0 con 0x7f6f4806ca60 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 44, 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:52:07.945558+0000", 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:51:46.224511+0000", 2024-04-25T15:52:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5806080, 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:52:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:52:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:09.667 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:09.668 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.5399999618530273, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.5399999618530273, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.75, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.75, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:52:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:52:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.3200000524520874, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.3200000524520874, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5, 2024-04-25T15:52:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:52:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.685 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.688 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:52:09.689 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.690 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:52:09.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:52:09.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:52:09.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.699 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.703 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.704 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 32, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.707 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:52:09.708 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:09.709 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.710 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:09.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.716 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.717 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2f", 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: 6, 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: 2 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.719 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:09.721 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:09.722 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6f2806f430 msgr2=0x7f6f28071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6f2806f430 0x7f6f28071940 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c0ad000 tx=0x7f6f3c008040 comp rx=0 tx=0).stop 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4806ca60 msgr2=0x7f6f4815df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4806ca60 0x7f6f4815df70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800b410 tx=0x7f6f3800b8d0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 shutdown_connections 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6f2806f430 0x7f6f28071940 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f4815fe00 0x7f6f48149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f480b5d70 0x7f6f4814c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 --2- 172.21.15.67:0/582925854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4806ca60 0x7f6f4815df70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 >> 172.21.15.67:0/582925854 conn(0x7f6f4806e420 msgr2=0x7f6f480d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 shutdown_connections 2024-04-25T15:52:09.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.655+0000 7f6f4caae640 1 -- 172.21.15.67:0/582925854 wait complete. 2024-04-25T15:52:09.725 INFO:tasks.thrashosds.thrasher:j is {'epoch': 44, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:52:07.945558+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:51:46.224511+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.2f', 'osds': [6, 2]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:52:09.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:52:09.726 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:09.719+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:09.778 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:09.778 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:09.778 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:09.778 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:09.795 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:09.839 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:09.836+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:09.925 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:09.919+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:09.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9cecad8640 1 Processor -- start 2024-04-25T15:52:09.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9cecad8640 1 -- start start 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9cecad8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9ce80c96e0 0x7f9ce8168190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9cecad8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ce806d8e0 con 0x7f9ce80c92d0 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9cecad8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ce8073c70 con 0x7f9ce8160800 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9cecad8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9ce806dbc0 con 0x7f9ce80c96e0 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9ce5d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9ce80c92d0 0x7f9ce8072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48732/0 (socket says 172.21.15.67:48732) 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9ce5d9b640 1 -- 172.21.15.67:0/502967137 learned_addr learned my addr 172.21.15.67:0/502967137 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9ce6d9d640 1 --2- 172.21.15.67:0/502967137 >> v2:172.21.15.176:3300/0 conn(0x7f9ce80c96e0 0x7f9ce8168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4097937996 0 0) 0x7f9ce806d8e0 con 0x7f9ce80c92d0 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.923+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cd0003530 con 0x7f9ce80c92d0 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1540891836 0 0) 0x7f9cd0003530 con 0x7f9ce80c92d0 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 >> v2:172.21.15.176:3300/0 conn(0x7f9ce80c96e0 msgr2=0x7f9ce8168190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 --2- 172.21.15.67:0/502967137 >> v2:172.21.15.176:3300/0 conn(0x7f9ce80c96e0 0x7f9ce8168190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 >> v1:172.21.15.176:6789/0 conn(0x7f9ce8160800 legacy=0x7f9ce806d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce8073f50 con 0x7f9ce80c92d0 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f9cdc001680 con 0x7f9ce80c92d0 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 >> v1:172.21.15.67:6789/0 conn(0x7f9ce80c92d0 legacy=0x7f9ce8072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd00039c0 0x7f9cd0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9cd0004810 0x7f9cd0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd0005280 0x7f9cd000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce806dbc0 con 0x7f9cd00039c0 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd000c1a0 con 0x7f9cd0005280 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd000c3a0 con 0x7f9cd0004810 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce659c640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd00039c0 0x7f9cd0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce5d9b640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9cd0004810 0x7f9cd0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce659c640 1 -- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9cd0004810 msgr2=0x7f9cd0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce659c640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9cd0004810 0x7f9cd0004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce659c640 1 -- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd0005280 msgr2=0x7f9cd000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:09.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce659c640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd0005280 0x7f9cd000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce6d9d640 1 -- 172.21.15.67:0/502967137 reap_dead start 2024-04-25T15:52:09.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce659c640 1 -- 172.21.15.67:0/502967137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce8073f50 con 0x7f9cd00039c0 2024-04-25T15:52:09.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce5d9b640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9cd0004810 0x7f9cd0004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:09.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.927+0000 7f9ce659c640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd00039c0 0x7f9cd0003e30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8001780 tx=0x7f9cd801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec725878dc63502 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:09.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd80013a0 con 0x7f9cd00039c0 2024-04-25T15:52:09.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd80052f0 con 0x7f9cd00039c0 2024-04-25T15:52:09.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9ce559a640 1 -- 172.21.15.67:0/502967137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8003840 con 0x7f9cd00039c0 2024-04-25T15:52:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 -- 172.21.15.67:0/502967137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd00039c0 msgr2=0x7f9cd0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd00039c0 0x7f9cd0003e30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8001780 tx=0x7f9cd801c440 comp rx=0 tx=0).stop 2024-04-25T15:52:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 -- 172.21.15.67:0/502967137 shutdown_connections 2024-04-25T15:52:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 --2- 172.21.15.67:0/502967137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cd00039c0 0x7f9cd0003e30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 -- 172.21.15.67:0/502967137 >> 172.21.15.67:0/502967137 conn(0x7f9ce8070db0 msgr2=0x7f9ce80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 -- 172.21.15.67:0/502967137 shutdown_connections 2024-04-25T15:52:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 -- 172.21.15.67:0/502967137 wait complete. 2024-04-25T15:52:09.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 Processor -- start 2024-04-25T15:52:09.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.931+0000 7f9cecad8640 1 -- start start 2024-04-25T15:52:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd00039c0 0x7f9ce81f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce81f9360 0x7f9ce815f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce8150300 0x7f9ce8150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8160d90 con 0x7f9ce81f9360 2024-04-25T15:52:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8160c10 con 0x7f9cd00039c0 2024-04-25T15:52:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ce8072dc0 con 0x7f9ce8150300 2024-04-25T15:52:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce81f9360 0x7f9ce815f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce81f9360 0x7f9ce815f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60586/0 (socket says 172.21.15.67:60586) 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 -- 172.21.15.67:0/2207449787 learned_addr learned my addr 172.21.15.67:0/2207449787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce6d9d640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce8150300 0x7f9ce8150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce659c640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd00039c0 0x7f9ce81f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 -- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce8150300 msgr2=0x7f9ce8150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce8150300 0x7f9ce8150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 -- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd00039c0 msgr2=0x7f9ce81f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd00039c0 0x7f9ce81f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 -- 172.21.15.67:0/2207449787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce815da00 con 0x7f9ce81f9360 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9ce5d9b640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce81f9360 0x7f9ce815f440 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc007fd0 tx=0x7f9cdc00dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cd6ffd640 1 -- 172.21.15.67:0/2207449787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc019070 con 0x7f9ce81f9360 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cd6ffd640 1 -- 172.21.15.67:0/2207449787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cdc0074e0 con 0x7f9ce81f9360 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce81f8e80 con 0x7f9ce81f9360 2024-04-25T15:52:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cd6ffd640 1 -- 172.21.15.67:0/2207449787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0049f0 con 0x7f9ce81f9360 2024-04-25T15:52:09.942 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:09.942 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:09.942 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:09.942 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:09.942 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.935+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce815d440 con 0x7f9ce81f9360 2024-04-25T15:52:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.939+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce806a480 con 0x7f9ce81f9360 2024-04-25T15:52:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.939+0000 7f9cd6ffd640 1 -- 172.21.15.67:0/2207449787 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9cdc01c070 con 0x7f9ce81f9360 2024-04-25T15:52:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.939+0000 7f9cd6ffd640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9cc806f400 0x7f9cc8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.939+0000 7f9cd6ffd640 1 -- 172.21.15.67:0/2207449787 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5857+0+0 (secure 0 0 0) 0x7f9cdc08ec50 con 0x7f9ce81f9360 2024-04-25T15:52:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.939+0000 7f9ce659c640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9cc806f400 0x7f9cc8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.939+0000 7f9ce659c640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9cc806f400 0x7f9cc8071910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8001750 tx=0x7f9cd80011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:09.947+0000 7f9cd6ffd640 1 -- 172.21.15.67:0/2207449787 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9cdc094020 con 0x7f9ce81f9360 2024-04-25T15:52:09.957 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:09.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:52:10.043 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:10.044+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.115+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ce806c820 con 0x7f9cc806f400 2024-04-25T15:52:10.126 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:10.120+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.135+0000 7f9cd6ffd640 1 -- 172.21.15.67:0/2207449787 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252051 (secure 0 0 0) 0x7f9ce806c820 con 0x7f9cc806f400 2024-04-25T15:52:10.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:10.144 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:52:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9cc806f400 msgr2=0x7f9cc8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9cc806f400 0x7f9cc8071910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8001750 tx=0x7f9cd80011a0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce81f9360 msgr2=0x7f9ce815f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce81f9360 0x7f9ce815f440 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc007fd0 tx=0x7f9cdc00dc80 comp rx=0 tx=0).stop 2024-04-25T15:52:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 shutdown_connections 2024-04-25T15:52:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9cc806f400 0x7f9cc8071910 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ce8150300 0x7f9ce8150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ce81f9360 0x7f9ce815f440 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 --2- 172.21.15.67:0/2207449787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9cd00039c0 0x7f9ce81f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 >> 172.21.15.67:0/2207449787 conn(0x7f9ce8070db0 msgr2=0x7f9ce8155c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:10.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 shutdown_connections 2024-04-25T15:52:10.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.139+0000 7f9cecad8640 1 -- 172.21.15.67:0/2207449787 wait complete. 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902729437000005, 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:52:10.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:52:10.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902698876000002, 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.163 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902691044999997, 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:52:10.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:10.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.169 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902682115999994, 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:52:10.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902675884000004, 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.175 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:52:10.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:52:10.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.181 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902660474000001, 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:52:10.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:52:10.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:52:10.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902647642000005, 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.187 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:52:10.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 87.902642428000007, 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.191 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:52:10.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:52:10.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.817032311000006, 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.195 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:52:10.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:52:10.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:10.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 72.264502317999998, 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:52:10.200 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:10.205 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 72.263109479999997, 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.225089354000005, 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:52:10.209 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:52:10.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:52:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.214 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.225066028000001, 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:52:10.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:52:10.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:52:10.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:52:10.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.225025488, 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.220 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.225 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.225016827999994, 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:52:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:10.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.230 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.224975154999996, 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:10.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:10.234 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.224965897999994, 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:52:10.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.220828471999994, 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.238 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:52:10.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:52:10.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.220773894999994, 2024-04-25T15:52:10.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:52:10.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.220740687000003, 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:52:10.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:52:10.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:10.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:52:10.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:52:10.251 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.251 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.251 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.251 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:10.251 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:10.251 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:10.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:10.253 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-04-25T15:52:09.833535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604666,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":319,"num_write_kb":1704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2232,"num_keys_recovered":12,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221392,"kb_used_data":5232,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753328,"statfs":{"total":773094113280,"available":772867407872,"internally_reserved":0,"allocated":5357568,"data_stored":2895652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.907538"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.864287+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:52:06.864287+0000","last_peered":"2024-04-25T15:52:06.864287+0000","last_clean":"2024-04-25T15:52:06.864287+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:52:06.864287+0000","last_undegraded":"2024-04-25T15:52:06.864287+0000","last_fullsized":"2024-04-25T15:52:06.864287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861497+0000","last_change":"2024-04-25T15:51:47.264956+0000","last_active":"2024-04-25T15:52:06.861497+0000","last_peered":"2024-04-25T15:52:06.861497+0000","last_clean":"2024-04-25T15:52:06.861497+0000","last_became_active":"2024-04-25T15:51:47.264571+0000","last_became_peered":"2024-04-25T15:51:47.264571+0000","last_unstale":"2024-04-25T15:52:06.861497+0000","last_undegraded":"2024-04-25T15:52:06.861497+0000","last_fullsized":"2024-04-25T15:52:06.861497+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:39.343444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.545853+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:52:06.545853+0000","last_peered":"2024-04-25T15:52:06.545853+0000","last_clean":"2024-04-25T15:52:06.545853+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:52:06.545853+0000","last_undegraded":"2024-04-25T15:52:06.545853+0000","last_fullsized":"2024-04-25T15:52:06.545853+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562205+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:52:05.562205+0000","last_peered":"2024-04-25T15:52:05.562205+0000","last_clean":"2024-04-25T15:52:05.562205+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:52:05.562205+0000","last_undegraded":"2024-04-25T15:52:05.562205+0000","last_fullsized":"2024-04-25T15:52:05.562205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863513+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:52:06.863513+0000","last_peered":"2024-04-25T15:52:06.863513+0000","last_clean":"2024-04-25T15:52:06.863513+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:52:06.863513+0000","last_undegraded":"2024-04-25T15:52:06.863513+0000","last_fullsized":"2024-04-25T15:52:06.863513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.561994+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:52:05.561994+0000","last_peered":"2024-04-25T15:52:05.561994+0000","last_clean":"2024-04-25T15:52:05.561994+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:52:05.561994+0000","last_undegraded":"2024-04-25T15:52:05.561994+0000","last_fullsized":"2024-04-25T15:52:05.561994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.862977+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:52:06.862977+0000","last_peered":"2024-04-25T15:52:06.862977+0000","last_clean":"2024-04-25T15:52:06.862977+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:52:06.862977+0000","last_undegraded":"2024-04-25T15:52:06.862977+0000","last_fullsized":"2024-04-25T15:52:06.862977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.461684+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:52:05.461684+0000","last_peered":"2024-04-25T15:52:05.461684+0000","last_clean":"2024-04-25T15:52:05.461684+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:52:05.461684+0000","last_undegraded":"2024-04-25T15:52:05.461684+0000","last_fullsized":"2024-04-25T15:52:05.461684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":52,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.545814+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:52:06.545814+0000","last_peered":"2024-04-25T15:52:06.545814+0000","last_clean":"2024-04-25T15:52:06.545814+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:52:06.545814+0000","last_undegraded":"2024-04-25T15:52:06.545814+0000","last_fullsized":"2024-04-25T15:52:06.545814+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.362403+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:52:06.362403+0000","last_peered":"2024-04-25T15:52:06.362403+0000","last_clean":"2024-04-25T15:52:06.362403+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:52:06.362403+0000","last_undegraded":"2024-04-25T15:52:06.362403+0000","last_fullsized":"2024-04-25T15:52:06.362403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.362042+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:52:06.362042+0000","last_peered":"2024-04-25T15:52:06.362042+0000","last_clean":"2024-04-25T15:52:06.362042+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:52:06.362042+0000","last_undegraded":"2024-04-25T15:52:06.362042+0000","last_fullsized":"2024-04-25T15:52:06.362042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.943957+0000","last_change":"2024-04-25T15:50:31.179032+0000","last_active":"2024-04-25T15:52:04.943957+0000","last_peered":"2024-04-25T15:52:04.943957+0000","last_clean":"2024-04-25T15:52:04.943957+0000","last_became_active":"2024-04-25T15:50:31.176691+0000","last_became_peered":"2024-04-25T15:50:31.176691+0000","last_unstale":"2024-04-25T15:52:04.943957+0000","last_undegraded":"2024-04-25T15:52:04.943957+0000","last_fullsized":"2024-04-25T15:52:04.943957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:57.798169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462574+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:52:05.462574+0000","last_peered":"2024-04-25T15:52:05.462574+0000","last_clean":"2024-04-25T15:52:05.462574+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:52:05.462574+0000","last_undegraded":"2024-04-25T15:52:05.462574+0000","last_fullsized":"2024-04-25T15:52:05.462574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":61,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861075+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:52:06.861075+0000","last_peered":"2024-04-25T15:52:06.861075+0000","last_clean":"2024-04-25T15:52:06.861075+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:52:06.861075+0000","last_undegraded":"2024-04-25T15:52:06.861075+0000","last_fullsized":"2024-04-25T15:52:06.861075+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.860991+0000","last_change":"2024-04-25T15:51:47.265508+0000","last_active":"2024-04-25T15:52:06.860991+0000","last_peered":"2024-04-25T15:52:06.860991+0000","last_clean":"2024-04-25T15:52:06.860991+0000","last_became_active":"2024-04-25T15:51:47.264990+0000","last_became_peered":"2024-04-25T15:51:47.264990+0000","last_unstale":"2024-04-25T15:52:06.860991+0000","last_undegraded":"2024-04-25T15:52:06.860991+0000","last_fullsized":"2024-04-25T15:52:06.860991+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:00.837180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.561894+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:52:05.561894+0000","last_peered":"2024-04-25T15:52:05.561894+0000","last_clean":"2024-04-25T15:52:05.561894+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:52:05.561894+0000","last_undegraded":"2024-04-25T15:52:05.561894+0000","last_fullsized":"2024-04-25T15:52:05.561894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.362433+0000","last_change":"2024-04-25T15:51:41.970074+0000","last_active":"2024-04-25T15:52:06.362433+0000","last_peered":"2024-04-25T15:52:06.362433+0000","last_clean":"2024-04-25T15:52:06.362433+0000","last_became_active":"2024-04-25T15:51:41.969256+0000","last_became_peered":"2024-04-25T15:51:41.969256+0000","last_unstale":"2024-04-25T15:52:06.362433+0000","last_undegraded":"2024-04-25T15:52:06.362433+0000","last_fullsized":"2024-04-25T15:52:06.362433+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:46.412980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.545922+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:52:06.545922+0000","last_peered":"2024-04-25T15:52:06.545922+0000","last_clean":"2024-04-25T15:52:06.545922+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:52:06.545922+0000","last_undegraded":"2024-04-25T15:52:06.545922+0000","last_fullsized":"2024-04-25T15:52:06.545922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.864544+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:52:06.864544+0000","last_peered":"2024-04-25T15:52:06.864544+0000","last_clean":"2024-04-25T15:52:06.864544+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:52:06.864544+0000","last_undegraded":"2024-04-25T15:52:06.864544+0000","last_fullsized":"2024-04-25T15:52:06.864544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.361604+0000","last_change":"2024-04-25T15:50:31.198641+0000","last_active":"2024-04-25T15:52:06.361604+0000","last_peered":"2024-04-25T15:52:06.361604+0000","last_clean":"2024-04-25T15:52:06.361604+0000","last_became_active":"2024-04-25T15:50:31.197928+0000","last_became_peered":"2024-04-25T15:50:31.197928+0000","last_unstale":"2024-04-25T15:52:06.361604+0000","last_undegraded":"2024-04-25T15:52:06.361604+0000","last_fullsized":"2024-04-25T15:52:06.361604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:06:45.515138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863416+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:52:06.863416+0000","last_peered":"2024-04-25T15:52:06.863416+0000","last_clean":"2024-04-25T15:52:06.863416+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:52:06.863416+0000","last_undegraded":"2024-04-25T15:52:06.863416+0000","last_fullsized":"2024-04-25T15:52:06.863416+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.545209+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:52:06.545209+0000","last_peered":"2024-04-25T15:52:06.545209+0000","last_clean":"2024-04-25T15:52:06.545209+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:52:06.545209+0000","last_undegraded":"2024-04-25T15:52:06.545209+0000","last_fullsized":"2024-04-25T15:52:06.545209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.461723+0000","last_change":"2024-04-25T15:50:31.203194+0000","last_active":"2024-04-25T15:52:05.461723+0000","last_peered":"2024-04-25T15:52:05.461723+0000","last_clean":"2024-04-25T15:52:05.461723+0000","last_became_active":"2024-04-25T15:50:31.202927+0000","last_became_peered":"2024-04-25T15:50:31.202927+0000","last_unstale":"2024-04-25T15:52:05.461723+0000","last_undegraded":"2024-04-25T15:52:05.461723+0000","last_fullsized":"2024-04-25T15:52:05.461723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:32:49.040238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562008+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:52:05.562008+0000","last_peered":"2024-04-25T15:52:05.562008+0000","last_clean":"2024-04-25T15:52:05.562008+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:52:05.562008+0000","last_undegraded":"2024-04-25T15:52:05.562008+0000","last_fullsized":"2024-04-25T15:52:05.562008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462294+0000","last_change":"2024-04-25T15:50:31.203908+0000","last_active":"2024-04-25T15:52:05.462294+0000","last_peered":"2024-04-25T15:52:05.462294+0000","last_clean":"2024-04-25T15:52:05.462294+0000","last_became_active":"2024-04-25T15:50:31.203519+0000","last_became_peered":"2024-04-25T15:50:31.203519+0000","last_unstale":"2024-04-25T15:52:05.462294+0000","last_undegraded":"2024-04-25T15:52:05.462294+0000","last_fullsized":"2024-04-25T15:52:05.462294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:38.912094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.362446+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:52:06.362446+0000","last_peered":"2024-04-25T15:52:06.362446+0000","last_clean":"2024-04-25T15:52:06.362446+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:52:06.362446+0000","last_undegraded":"2024-04-25T15:52:06.362446+0000","last_fullsized":"2024-04-25T15:52:06.362446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861511+0000","last_change":"2024-04-25T15:51:47.271333+0000","last_active":"2024-04-25T15:52:06.861511+0000","last_peered":"2024-04-25T15:52:06.861511+0000","last_clean":"2024-04-25T15:52:06.861511+0000","last_became_active":"2024-04-25T15:51:47.270971+0000","last_became_peered":"2024-04-25T15:51:47.270971+0000","last_unstale":"2024-04-25T15:52:06.861511+0000","last_undegraded":"2024-04-25T15:52:06.861511+0000","last_fullsized":"2024-04-25T15:52:06.861511+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:23.860435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462600+0000","last_change":"2024-04-25T15:50:31.203892+0000","last_active":"2024-04-25T15:52:05.462600+0000","last_peered":"2024-04-25T15:52:05.462600+0000","last_clean":"2024-04-25T15:52:05.462600+0000","last_became_active":"2024-04-25T15:50:31.203512+0000","last_became_peered":"2024-04-25T15:50:31.203512+0000","last_unstale":"2024-04-25T15:52:05.462600+0000","last_undegraded":"2024-04-25T15:52:05.462600+0000","last_fullsized":"2024-04-25T15:52:05.462600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:23.181753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462268+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:52:05.462268+0000","last_peered":"2024-04-25T15:52:05.462268+0000","last_clean":"2024-04-25T15:52:05.462268+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:52:05.462268+0000","last_undegraded":"2024-04-25T15:52:05.462268+0000","last_fullsized":"2024-04-25T15:52:05.462268+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562092+0000","last_change":"2024-04-25T15:50:31.195866+0000","last_active":"2024-04-25T15:52:05.562092+0000","last_peered":"2024-04-25T15:52:05.562092+0000","last_clean":"2024-04-25T15:52:05.562092+0000","last_became_active":"2024-04-25T15:50:31.188085+0000","last_became_peered":"2024-04-25T15:50:31.188085+0000","last_unstale":"2024-04-25T15:52:05.562092+0000","last_undegraded":"2024-04-25T15:52:05.562092+0000","last_fullsized":"2024-04-25T15:52:05.562092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:56.080127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.545237+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:52:06.545237+0000","last_peered":"2024-04-25T15:52:06.545237+0000","last_clean":"2024-04-25T15:52:06.545237+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:52:06.545237+0000","last_undegraded":"2024-04-25T15:52:06.545237+0000","last_fullsized":"2024-04-25T15:52:06.545237+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.361349+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:52:06.361349+0000","last_peered":"2024-04-25T15:52:06.361349+0000","last_clean":"2024-04-25T15:52:06.361349+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:52:06.361349+0000","last_undegraded":"2024-04-25T15:52:06.361349+0000","last_fullsized":"2024-04-25T15:52:06.361349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.943776+0000","last_change":"2024-04-25T15:51:47.268608+0000","last_active":"2024-04-25T15:52:04.943776+0000","last_peered":"2024-04-25T15:52:04.943776+0000","last_clean":"2024-04-25T15:52:04.943776+0000","last_became_active":"2024-04-25T15:51:47.267582+0000","last_became_peered":"2024-04-25T15:51:47.267582+0000","last_unstale":"2024-04-25T15:52:04.943776+0000","last_undegraded":"2024-04-25T15:52:04.943776+0000","last_fullsized":"2024-04-25T15:52:04.943776+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:08.075848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":66,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462886+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:52:05.462886+0000","last_peered":"2024-04-25T15:52:05.462886+0000","last_clean":"2024-04-25T15:52:05.462886+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:52:05.462886+0000","last_undegraded":"2024-04-25T15:52:05.462886+0000","last_fullsized":"2024-04-25T15:52:05.462886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.943585+0000","last_change":"2024-04-25T15:51:59.931929+0000","last_active":"2024-04-25T15:52:04.943585+0000","last_peered":"2024-04-25T15:52:04.943585+0000","last_clean":"2024-04-25T15:52:04.943585+0000","last_became_active":"2024-04-25T15:51:59.931459+0000","last_became_peered":"2024-04-25T15:51:59.931459+0000","last_unstale":"2024-04-25T15:52:04.943585+0000","last_undegraded":"2024-04-25T15:52:04.943585+0000","last_fullsized":"2024-04-25T15:52:04.943585+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:49.500585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":67,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.361029+0000","last_change":"2024-04-25T15:50:30.195949+0000","last_active":"2024-04-25T15:52:06.361029+0000","last_peered":"2024-04-25T15:52:06.361029+0000","last_clean":"2024-04-25T15:52:06.361029+0000","last_became_active":"2024-04-25T15:50:30.194911+0000","last_became_peered":"2024-04-25T15:50:30.194911+0000","last_unstale":"2024-04-25T15:52:06.361029+0000","last_undegraded":"2024-04-25T15:52:06.361029+0000","last_fullsized":"2024-04-25T15:52:06.361029+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:23:38.997355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":63,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.461907+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:52:05.461907+0000","last_peered":"2024-04-25T15:52:05.461907+0000","last_clean":"2024-04-25T15:52:05.461907+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:52:05.461907+0000","last_undegraded":"2024-04-25T15:52:05.461907+0000","last_fullsized":"2024-04-25T15:52:05.461907+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462010+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:52:05.462010+0000","last_peered":"2024-04-25T15:52:05.462010+0000","last_clean":"2024-04-25T15:52:05.462010+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:52:05.462010+0000","last_undegraded":"2024-04-25T15:52:05.462010+0000","last_fullsized":"2024-04-25T15:52:05.462010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":63,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562091+0000","last_change":"2024-04-25T15:50:30.202392+0000","last_active":"2024-04-25T15:52:05.562091+0000","last_peered":"2024-04-25T15:52:05.562091+0000","last_clean":"2024-04-25T15:52:05.562091+0000","last_became_active":"2024-04-25T15:50:30.201434+0000","last_became_peered":"2024-04-25T15:50:30.201434+0000","last_unstale":"2024-04-25T15:52:05.562091+0000","last_undegraded":"2024-04-25T15:52:05.562091+0000","last_fullsized":"2024-04-25T15:52:05.562091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:24:42.675958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861144+0000","last_change":"2024-04-25T15:51:47.271089+0000","last_active":"2024-04-25T15:52:06.861144+0000","last_peered":"2024-04-25T15:52:06.861144+0000","last_clean":"2024-04-25T15:52:06.861144+0000","last_became_active":"2024-04-25T15:51:47.270705+0000","last_became_peered":"2024-04-25T15:51:47.270705+0000","last_unstale":"2024-04-25T15:52:06.861144+0000","last_undegraded":"2024-04-25T15:52:06.861144+0000","last_fullsized":"2024-04-25T15:52:06.861144+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:08.380566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":65,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.362017+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:52:06.362017+0000","last_peered":"2024-04-25T15:52:06.362017+0000","last_clean":"2024-04-25T15:52:06.362017+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:52:06.362017+0000","last_undegraded":"2024-04-25T15:52:06.362017+0000","last_fullsized":"2024-04-25T15:52:06.362017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562415+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:52:05.562415+0000","last_peered":"2024-04-25T15:52:05.562415+0000","last_clean":"2024-04-25T15:52:05.562415+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:52:05.562415+0000","last_undegraded":"2024-04-25T15:52:05.562415+0000","last_fullsized":"2024-04-25T15:52:05.562415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562515+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:52:05.562515+0000","last_peered":"2024-04-25T15:52:05.562515+0000","last_clean":"2024-04-25T15:52:05.562515+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:52:05.562515+0000","last_undegraded":"2024-04-25T15:52:05.562515+0000","last_fullsized":"2024-04-25T15:52:05.562515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562587+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:52:05.562587+0000","last_peered":"2024-04-25T15:52:05.562587+0000","last_clean":"2024-04-25T15:52:05.562587+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:52:05.562587+0000","last_undegraded":"2024-04-25T15:52:05.562587+0000","last_fullsized":"2024-04-25T15:52:05.562587+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":34,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861383+0000","last_change":"2024-04-25T15:51:47.302272+0000","last_active":"2024-04-25T15:52:06.861383+0000","last_peered":"2024-04-25T15:52:06.861383+0000","last_clean":"2024-04-25T15:52:06.861383+0000","last_became_active":"2024-04-25T15:51:47.266158+0000","last_became_peered":"2024-04-25T15:51:47.266158+0000","last_unstale":"2024-04-25T15:52:06.861383+0000","last_undegraded":"2024-04-25T15:52:06.861383+0000","last_fullsized":"2024-04-25T15:52:06.861383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:28.386086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":672,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.944102+0000","last_change":"2024-04-25T15:50:31.193171+0000","last_active":"2024-04-25T15:52:04.944102+0000","last_peered":"2024-04-25T15:52:04.944102+0000","last_clean":"2024-04-25T15:52:04.944102+0000","last_became_active":"2024-04-25T15:50:31.192295+0000","last_became_peered":"2024-04-25T15:50:31.192295+0000","last_unstale":"2024-04-25T15:52:04.944102+0000","last_undegraded":"2024-04-25T15:52:04.944102+0000","last_fullsized":"2024-04-25T15:52:04.944102+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:45.146144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"24'14","reported_seq":74,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.361024+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:52:06.361024+0000","last_peered":"2024-04-25T15:52:06.361024+0000","last_clean":"2024-04-25T15:52:06.361024+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:52:06.361024+0000","last_undegraded":"2024-04-25T15:52:06.361024+0000","last_fullsized":"2024-04-25T15:52:06.361024+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863790+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:52:06.863790+0000","last_peered":"2024-04-25T15:52:06.863790+0000","last_clean":"2024-04-25T15:52:06.863790+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:52:06.863790+0000","last_undegraded":"2024-04-25T15:52:06.863790+0000","last_fullsized":"2024-04-25T15:52:06.863790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":35,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.944152+0000","last_change":"2024-04-25T15:51:59.929732+0000","last_active":"2024-04-25T15:52:04.944152+0000","last_peered":"2024-04-25T15:52:04.944152+0000","last_clean":"2024-04-25T15:52:04.944152+0000","last_became_active":"2024-04-25T15:51:59.929402+0000","last_became_peered":"2024-04-25T15:51:59.929402+0000","last_unstale":"2024-04-25T15:52:04.944152+0000","last_undegraded":"2024-04-25T15:52:04.944152+0000","last_fullsized":"2024-04-25T15:52:04.944152+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:22.568747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863424+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:52:06.863424+0000","last_peered":"2024-04-25T15:52:06.863424+0000","last_clean":"2024-04-25T15:52:06.863424+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:52:06.863424+0000","last_undegraded":"2024-04-25T15:52:06.863424+0000","last_fullsized":"2024-04-25T15:52:06.863424+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":65,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562239+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:52:05.562239+0000","last_peered":"2024-04-25T15:52:05.562239+0000","last_clean":"2024-04-25T15:52:05.562239+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:52:05.562239+0000","last_undegraded":"2024-04-25T15:52:05.562239+0000","last_fullsized":"2024-04-25T15:52:05.562239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863623+0000","last_change":"2024-04-25T15:50:31.210173+0000","last_active":"2024-04-25T15:52:06.863623+0000","last_peered":"2024-04-25T15:52:06.863623+0000","last_clean":"2024-04-25T15:52:06.863623+0000","last_became_active":"2024-04-25T15:50:31.209772+0000","last_became_peered":"2024-04-25T15:50:31.209772+0000","last_unstale":"2024-04-25T15:52:06.863623+0000","last_undegraded":"2024-04-25T15:52:06.863623+0000","last_fullsized":"2024-04-25T15:52:06.863623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:24:34.395083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":66,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.360994+0000","last_change":"2024-04-25T15:50:30.194000+0000","last_active":"2024-04-25T15:52:06.360994+0000","last_peered":"2024-04-25T15:52:06.360994+0000","last_clean":"2024-04-25T15:52:06.360994+0000","last_became_active":"2024-04-25T15:50:30.192871+0000","last_became_peered":"2024-04-25T15:50:30.192871+0000","last_unstale":"2024-04-25T15:52:06.360994+0000","last_undegraded":"2024-04-25T15:52:06.360994+0000","last_fullsized":"2024-04-25T15:52:06.360994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:28.322479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.361615+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:52:06.361615+0000","last_peered":"2024-04-25T15:52:06.361615+0000","last_clean":"2024-04-25T15:52:06.361615+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:52:06.361615+0000","last_undegraded":"2024-04-25T15:52:06.361615+0000","last_fullsized":"2024-04-25T15:52:06.361615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":123,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.545359+0000","last_change":"2024-04-25T15:51:47.287562+0000","last_active":"2024-04-25T15:52:06.545359+0000","last_peered":"2024-04-25T15:52:06.545359+0000","last_clean":"2024-04-25T15:52:06.545359+0000","last_became_active":"2024-04-25T15:51:47.264513+0000","last_became_peered":"2024-04-25T15:51:47.264513+0000","last_unstale":"2024-04-25T15:52:06.545359+0000","last_undegraded":"2024-04-25T15:52:06.545359+0000","last_fullsized":"2024-04-25T15:52:06.545359+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:51:11.555766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863333+0000","last_change":"2024-04-25T15:50:31.209394+0000","last_active":"2024-04-25T15:52:06.863333+0000","last_peered":"2024-04-25T15:52:06.863333+0000","last_clean":"2024-04-25T15:52:06.863333+0000","last_became_active":"2024-04-25T15:50:31.207822+0000","last_became_peered":"2024-04-25T15:50:31.207822+0000","last_unstale":"2024-04-25T15:52:06.863333+0000","last_undegraded":"2024-04-25T15:52:06.863333+0000","last_fullsized":"2024-04-25T15:52:06.863333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:23.992625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":62,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.944358+0000","last_change":"2024-04-25T15:50:30.184174+0000","last_active":"2024-04-25T15:52:04.944358+0000","last_peered":"2024-04-25T15:52:04.944358+0000","last_clean":"2024-04-25T15:52:04.944358+0000","last_became_active":"2024-04-25T15:50:30.183244+0000","last_became_peered":"2024-04-25T15:50:30.183244+0000","last_unstale":"2024-04-25T15:52:04.944358+0000","last_undegraded":"2024-04-25T15:52:04.944358+0000","last_fullsized":"2024-04-25T15:52:04.944358+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:03.744157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.864171+0000","last_change":"2024-04-25T15:50:31.206815+0000","last_active":"2024-04-25T15:52:06.864171+0000","last_peered":"2024-04-25T15:52:06.864171+0000","last_clean":"2024-04-25T15:52:06.864171+0000","last_became_active":"2024-04-25T15:50:31.197159+0000","last_became_peered":"2024-04-25T15:50:31.197159+0000","last_unstale":"2024-04-25T15:52:06.864171+0000","last_undegraded":"2024-04-25T15:52:06.864171+0000","last_fullsized":"2024-04-25T15:52:06.864171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:06:22.512733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":65,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.544998+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:52:06.544998+0000","last_peered":"2024-04-25T15:52:06.544998+0000","last_clean":"2024-04-25T15:52:06.544998+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:52:06.544998+0000","last_undegraded":"2024-04-25T15:52:06.544998+0000","last_fullsized":"2024-04-25T15:52:06.544998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.864365+0000","last_change":"2024-04-25T15:50:31.210542+0000","last_active":"2024-04-25T15:52:06.864365+0000","last_peered":"2024-04-25T15:52:06.864365+0000","last_clean":"2024-04-25T15:52:06.864365+0000","last_became_active":"2024-04-25T15:50:31.209856+0000","last_became_peered":"2024-04-25T15:50:31.209856+0000","last_unstale":"2024-04-25T15:52:06.864365+0000","last_undegraded":"2024-04-25T15:52:06.864365+0000","last_fullsized":"2024-04-25T15:52:06.864365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:15:49.726520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":28,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861386+0000","last_change":"2024-04-25T15:51:47.305459+0000","last_active":"2024-04-25T15:52:06.861386+0000","last_peered":"2024-04-25T15:52:06.861386+0000","last_clean":"2024-04-25T15:52:06.861386+0000","last_became_active":"2024-04-25T15:51:47.266603+0000","last_became_peered":"2024-04-25T15:51:47.266603+0000","last_unstale":"2024-04-25T15:52:06.861386+0000","last_undegraded":"2024-04-25T15:52:06.861386+0000","last_fullsized":"2024-04-25T15:52:06.861386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:12.124748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562346+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:52:05.562346+0000","last_peered":"2024-04-25T15:52:05.562346+0000","last_clean":"2024-04-25T15:52:05.562346+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:52:05.562346+0000","last_undegraded":"2024-04-25T15:52:05.562346+0000","last_fullsized":"2024-04-25T15:52:05.562346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":66,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.544811+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:52:06.544811+0000","last_peered":"2024-04-25T15:52:06.544811+0000","last_clean":"2024-04-25T15:52:06.544811+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:52:06.544811+0000","last_undegraded":"2024-04-25T15:52:06.544811+0000","last_fullsized":"2024-04-25T15:52:06.544811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863957+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:52:06.863957+0000","last_peered":"2024-04-25T15:52:06.863957+0000","last_clean":"2024-04-25T15:52:06.863957+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:52:06.863957+0000","last_undegraded":"2024-04-25T15:52:06.863957+0000","last_fullsized":"2024-04-25T15:52:06.863957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":156,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863098+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:52:06.863098+0000","last_peered":"2024-04-25T15:52:06.863098+0000","last_clean":"2024-04-25T15:52:06.863098+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:52:06.863098+0000","last_undegraded":"2024-04-25T15:52:06.863098+0000","last_fullsized":"2024-04-25T15:52:06.863098+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":66,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562533+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:52:05.562533+0000","last_peered":"2024-04-25T15:52:05.562533+0000","last_clean":"2024-04-25T15:52:05.562533+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:52:05.562533+0000","last_undegraded":"2024-04-25T15:52:05.562533+0000","last_fullsized":"2024-04-25T15:52:05.562533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462152+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:52:05.462152+0000","last_peered":"2024-04-25T15:52:05.462152+0000","last_clean":"2024-04-25T15:52:05.462152+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:52:05.462152+0000","last_undegraded":"2024-04-25T15:52:05.462152+0000","last_fullsized":"2024-04-25T15:52:05.462152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":82,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.863037+0000","last_change":"2024-04-25T15:51:47.292263+0000","last_active":"2024-04-25T15:52:06.863037+0000","last_peered":"2024-04-25T15:52:06.863037+0000","last_clean":"2024-04-25T15:52:06.863037+0000","last_became_active":"2024-04-25T15:51:47.264780+0000","last_became_peered":"2024-04-25T15:51:47.264780+0000","last_unstale":"2024-04-25T15:52:06.863037+0000","last_undegraded":"2024-04-25T15:52:06.863037+0000","last_fullsized":"2024-04-25T15:52:06.863037+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:40.269146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.943184+0000","last_change":"2024-04-25T15:51:59.931120+0000","last_active":"2024-04-25T15:52:04.943184+0000","last_peered":"2024-04-25T15:52:04.943184+0000","last_clean":"2024-04-25T15:52:04.943184+0000","last_became_active":"2024-04-25T15:51:59.930703+0000","last_became_peered":"2024-04-25T15:51:59.930703+0000","last_unstale":"2024-04-25T15:52:04.943184+0000","last_undegraded":"2024-04-25T15:52:04.943184+0000","last_fullsized":"2024-04-25T15:52:04.943184+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:24:44.112752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":66,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.544880+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:52:06.544880+0000","last_peered":"2024-04-25T15:52:06.544880+0000","last_clean":"2024-04-25T15:52:06.544880+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:52:06.544880+0000","last_undegraded":"2024-04-25T15:52:06.544880+0000","last_fullsized":"2024-04-25T15:52:06.544880+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.860811+0000","last_change":"2024-04-25T15:51:47.255396+0000","last_active":"2024-04-25T15:52:06.860811+0000","last_peered":"2024-04-25T15:52:06.860811+0000","last_clean":"2024-04-25T15:52:06.860811+0000","last_became_active":"2024-04-25T15:51:47.254596+0000","last_became_peered":"2024-04-25T15:51:47.254596+0000","last_unstale":"2024-04-25T15:52:06.860811+0000","last_undegraded":"2024-04-25T15:52:06.860811+0000","last_fullsized":"2024-04-25T15:52:06.860811+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:06:53.573154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562312+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:52:05.562312+0000","last_peered":"2024-04-25T15:52:05.562312+0000","last_clean":"2024-04-25T15:52:05.562312+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:52:05.562312+0000","last_undegraded":"2024-04-25T15:52:05.562312+0000","last_fullsized":"2024-04-25T15:52:05.562312+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.944476+0000","last_change":"2024-04-25T15:50:31.183927+0000","last_active":"2024-04-25T15:52:04.944476+0000","last_peered":"2024-04-25T15:52:04.944476+0000","last_clean":"2024-04-25T15:52:04.944476+0000","last_became_active":"2024-04-25T15:50:31.183046+0000","last_became_peered":"2024-04-25T15:50:31.183046+0000","last_unstale":"2024-04-25T15:52:04.944476+0000","last_undegraded":"2024-04-25T15:52:04.944476+0000","last_fullsized":"2024-04-25T15:52:04.944476+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:44:59.338508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":24,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.860861+0000","last_change":"2024-04-25T15:51:47.263905+0000","last_active":"2024-04-25T15:52:06.860861+0000","last_peered":"2024-04-25T15:52:06.860861+0000","last_clean":"2024-04-25T15:52:06.860861+0000","last_became_active":"2024-04-25T15:51:47.263546+0000","last_became_peered":"2024-04-25T15:51:47.263546+0000","last_unstale":"2024-04-25T15:52:06.860861+0000","last_undegraded":"2024-04-25T15:52:06.860861+0000","last_fullsized":"2024-04-25T15:52:06.860861+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:14.832346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562508+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:52:05.562508+0000","last_peered":"2024-04-25T15:52:05.562508+0000","last_clean":"2024-04-25T15:52:05.562508+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:52:05.562508+0000","last_undegraded":"2024-04-25T15:52:05.562508+0000","last_fullsized":"2024-04-25T15:52:05.562508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.862619+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:52:06.862619+0000","last_peered":"2024-04-25T15:52:06.862619+0000","last_clean":"2024-04-25T15:52:06.862619+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:52:06.862619+0000","last_undegraded":"2024-04-25T15:52:06.862619+0000","last_fullsized":"2024-04-25T15:52:06.862619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861345+0000","last_change":"2024-04-25T15:51:47.263756+0000","last_active":"2024-04-25T15:52:06.861345+0000","last_peered":"2024-04-25T15:52:06.861345+0000","last_clean":"2024-04-25T15:52:06.861345+0000","last_became_active":"2024-04-25T15:51:47.263130+0000","last_became_peered":"2024-04-25T15:51:47.263130+0000","last_unstale":"2024-04-25T15:52:06.861345+0000","last_undegraded":"2024-04-25T15:52:06.861345+0000","last_fullsized":"2024-04-25T15:52:06.861345+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:38.024802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'9","reported_seq":34,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861624+0000","last_change":"2024-04-25T15:51:47.314396+0000","last_active":"2024-04-25T15:52:06.861624+0000","last_peered":"2024-04-25T15:52:06.861624+0000","last_clean":"2024-04-25T15:52:06.861624+0000","last_became_active":"2024-04-25T15:51:47.274984+0000","last_became_peered":"2024-04-25T15:51:47.274984+0000","last_unstale":"2024-04-25T15:52:06.861624+0000","last_undegraded":"2024-04-25T15:52:06.861624+0000","last_fullsized":"2024-04-25T15:52:06.861624+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:53.936857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462422+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:52:05.462422+0000","last_peered":"2024-04-25T15:52:05.462422+0000","last_clean":"2024-04-25T15:52:05.462422+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:52:05.462422+0000","last_undegraded":"2024-04-25T15:52:05.462422+0000","last_fullsized":"2024-04-25T15:52:05.462422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":67,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.362529+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:52:06.362529+0000","last_peered":"2024-04-25T15:52:06.362529+0000","last_clean":"2024-04-25T15:52:06.362529+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:52:06.362529+0000","last_undegraded":"2024-04-25T15:52:06.362529+0000","last_fullsized":"2024-04-25T15:52:06.362529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":61,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861457+0000","last_change":"2024-04-25T15:51:59.936821+0000","last_active":"2024-04-25T15:52:06.861457+0000","last_peered":"2024-04-25T15:52:06.861457+0000","last_clean":"2024-04-25T15:52:06.861457+0000","last_became_active":"2024-04-25T15:51:59.936474+0000","last_became_peered":"2024-04-25T15:51:59.936474+0000","last_unstale":"2024-04-25T15:52:06.861457+0000","last_undegraded":"2024-04-25T15:52:06.861457+0000","last_fullsized":"2024-04-25T15:52:06.861457+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:05.147722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":62,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.862529+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:52:06.862529+0000","last_peered":"2024-04-25T15:52:06.862529+0000","last_clean":"2024-04-25T15:52:06.862529+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:52:06.862529+0000","last_undegraded":"2024-04-25T15:52:06.862529+0000","last_fullsized":"2024-04-25T15:52:06.862529+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:04.943659+0000","last_change":"2024-04-25T15:50:31.161552+0000","last_active":"2024-04-25T15:52:04.943659+0000","last_peered":"2024-04-25T15:52:04.943659+0000","last_clean":"2024-04-25T15:52:04.943659+0000","last_became_active":"2024-04-25T15:50:31.160480+0000","last_became_peered":"2024-04-25T15:50:31.160480+0000","last_unstale":"2024-04-25T15:52:04.943659+0000","last_undegraded":"2024-04-25T15:52:04.943659+0000","last_fullsized":"2024-04-25T15:52:04.943659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:49.375723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":34,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.860797+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:52:06.860797+0000","last_peered":"2024-04-25T15:52:06.860797+0000","last_clean":"2024-04-25T15:52:06.860797+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:52:06.860797+0000","last_undegraded":"2024-04-25T15:52:06.860797+0000","last_fullsized":"2024-04-25T15:52:06.860797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462249+0000","last_change":"2024-04-25T15:51:47.270024+0000","last_active":"2024-04-25T15:52:05.462249+0000","last_peered":"2024-04-25T15:52:05.462249+0000","last_clean":"2024-04-25T15:52:05.462249+0000","last_became_active":"2024-04-25T15:51:47.269733+0000","last_became_peered":"2024-04-25T15:51:47.269733+0000","last_unstale":"2024-04-25T15:52:05.462249+0000","last_undegraded":"2024-04-25T15:52:05.462249+0000","last_fullsized":"2024-04-25T15:52:05.462249+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:23.505183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":30,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.861217+0000","last_change":"2024-04-25T15:51:47.286937+0000","last_active":"2024-04-25T15:52:06.861217+0000","last_peered":"2024-04-25T15:52:06.861217+0000","last_clean":"2024-04-25T15:52:06.861217+0000","last_became_active":"2024-04-25T15:51:47.266040+0000","last_became_peered":"2024-04-25T15:51:47.266040+0000","last_unstale":"2024-04-25T15:52:06.861217+0000","last_undegraded":"2024-04-25T15:52:06.861217+0000","last_fullsized":"2024-04-25T15:52:06.861217+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:48.792187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":52,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.545396+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:52:06.545396+0000","last_peered":"2024-04-25T15:52:06.545396+0000","last_clean":"2024-04-25T15:52:06.545396+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:52:06.545396+0000","last_undegraded":"2024-04-25T15:52:06.545396+0000","last_fullsized":"2024-04-25T15:52:06.545396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":64,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.862626+0000","last_change":"2024-04-25T15:50:30.198888+0000","last_active":"2024-04-25T15:52:06.862626+0000","last_peered":"2024-04-25T15:52:06.862626+0000","last_clean":"2024-04-25T15:52:06.862626+0000","last_became_active":"2024-04-25T15:50:30.198135+0000","last_became_peered":"2024-04-25T15:50:30.198135+0000","last_unstale":"2024-04-25T15:52:06.862626+0000","last_undegraded":"2024-04-25T15:52:06.862626+0000","last_fullsized":"2024-04-25T15:52:06.862626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:12.585688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.361418+0000","last_change":"2024-04-25T15:50:31.198821+0000","last_active":"2024-04-25T15:52:06.361418+0000","last_peered":"2024-04-25T15:52:06.361418+0000","last_clean":"2024-04-25T15:52:06.361418+0000","last_became_active":"2024-04-25T15:50:31.197889+0000","last_became_peered":"2024-04-25T15:50:31.197889+0000","last_unstale":"2024-04-25T15:52:06.361418+0000","last_undegraded":"2024-04-25T15:52:06.361418+0000","last_fullsized":"2024-04-25T15:52:06.361418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:14.226403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562358+0000","last_change":"2024-04-25T15:50:31.164502+0000","last_active":"2024-04-25T15:52:05.562358+0000","last_peered":"2024-04-25T15:52:05.562358+0000","last_clean":"2024-04-25T15:52:05.562358+0000","last_became_active":"2024-04-25T15:50:31.163605+0000","last_became_peered":"2024-04-25T15:50:31.163605+0000","last_unstale":"2024-04-25T15:52:05.562358+0000","last_undegraded":"2024-04-25T15:52:05.562358+0000","last_fullsized":"2024-04-25T15:52:05.562358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:48.812233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":65,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.862550+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:52:06.862550+0000","last_peered":"2024-04-25T15:52:06.862550+0000","last_clean":"2024-04-25T15:52:06.862550+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:52:06.862550+0000","last_undegraded":"2024-04-25T15:52:06.862550+0000","last_fullsized":"2024-04-25T15:52:06.862550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562248+0000","last_change":"2024-04-25T15:50:31.170843+0000","last_active":"2024-04-25T15:52:05.562248+0000","last_peered":"2024-04-25T15:52:05.562248+0000","last_clean":"2024-04-25T15:52:05.562248+0000","last_became_active":"2024-04-25T15:50:31.170038+0000","last_became_peered":"2024-04-25T15:50:31.170038+0000","last_unstale":"2024-04-25T15:52:05.562248+0000","last_undegraded":"2024-04-25T15:52:05.562248+0000","last_fullsized":"2024-04-25T15:52:05.562248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:59.993330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":72,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.864259+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:52:06.864259+0000","last_peered":"2024-04-25T15:52:06.864259+0000","last_clean":"2024-04-25T15:52:06.864259+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:52:06.864259+0000","last_undegraded":"2024-04-25T15:52:06.864259+0000","last_fullsized":"2024-04-25T15:52:06.864259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.562518+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:52:05.562518+0000","last_peered":"2024-04-25T15:52:05.562518+0000","last_clean":"2024-04-25T15:52:05.562518+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:52:05.562518+0000","last_undegraded":"2024-04-25T15:52:05.562518+0000","last_fullsized":"2024-04-25T15:52:05.562518+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":62,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462166+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:52:05.462166+0000","last_peered":"2024-04-25T15:52:05.462166+0000","last_clean":"2024-04-25T15:52:05.462166+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:52:05.462166+0000","last_undegraded":"2024-04-25T15:52:05.462166+0000","last_fullsized":"2024-04-25T15:52:05.462166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":58,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:05.462055+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:52:05.462055+0000","last_peered":"2024-04-25T15:52:05.462055+0000","last_clean":"2024-04-25T15:52:05.462055+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:52:05.462055+0000","last_undegraded":"2024-04-25T15:52:05.462055+0000","last_fullsized":"2024-04-25T15:52:05.462055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":39,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-25T15:52:06.864581+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:52:06.864581+0000","last_peered":"2024-04-25T15:52:06.864581+0000","last_clean":"2024-04-25T15:52:06.864581+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:52:06.864581+0000","last_undegraded":"2024-04-25T15:52:06.864581+0000","last_fullsized":"2024-04-25T15:52:06.864581+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14298,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":218,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2232,"num_keys_recovered":12,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":28596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1118208,"data_stored":803699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":218574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":536576,"data_stored":215163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":512000,"data_stored":212561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":524288,"data_stored":218608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":212693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":516096,"data_stored":212322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343736,"statfs":{"total":96636764160,"available":96607985664,"internally_reserved":0,"allocated":1110016,"data_stored":802032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":7818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":7784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:52:10.257 INFO:tasks.thrashosds.thrasher:Setting 3.7 to [4, 6, 2, 1] 2024-04-25T15:52:10.257 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.7', '4', '6', '2', '1'] 2024-04-25T15:52:10.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.7 4 6 2 1 2024-04-25T15:52:10.258 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:10.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1d1a7640 1 Processor -- start 2024-04-25T15:52:10.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1d1a7640 1 -- start start 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1d1a7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9c18073bf0 0x7f9c1806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1d1a7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c1806dc90 0x7f9c18163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1d1a7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c180c77a0 con 0x7f9c180caf60 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1d1a7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9c180cab60 con 0x7f9c1806dc90 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1d1a7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9c18159620 con 0x7f9c18073bf0 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c16d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9c180caf60 0x7f9c1815d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48746/0 (socket says 172.21.15.67:48746) 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c16d9d640 1 -- 172.21.15.67:0/3444935738 learned_addr learned my addr 172.21.15.67:0/3444935738 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.163+0000 7f9c1759e640 1 --2- 172.21.15.67:0/3444935738 >> v2:172.21.15.67:3300/0 conn(0x7f9c1806dc90 0x7f9c18163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c1659c640 1 --2- 172.21.15.67:0/3444935738 >> v2:172.21.15.176:3300/0 conn(0x7f9c18073bf0 0x7f9c1806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c15d9b640 1 -- 172.21.15.67:0/3444935738 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242176585 0 0) 0x7f9c180c77a0 con 0x7f9c180caf60 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c15d9b640 1 -- 172.21.15.67:0/3444935738 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c00003530 con 0x7f9c180caf60 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c1759e640 1 -- 172.21.15.67:0/3444935738 >> v2:172.21.15.176:3300/0 conn(0x7f9c18073bf0 msgr2=0x7f9c1806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c1759e640 1 --2- 172.21.15.67:0/3444935738 >> v2:172.21.15.176:3300/0 conn(0x7f9c18073bf0 0x7f9c1806d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c1759e640 1 -- 172.21.15.67:0/3444935738 >> v1:172.21.15.67:6789/0 conn(0x7f9c180caf60 legacy=0x7f9c1815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c1759e640 1 -- 172.21.15.67:0/3444935738 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c180c7a60 con 0x7f9c1806dc90 2024-04-25T15:52:10.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c1759e640 1 --2- 172.21.15.67:0/3444935738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c18163ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9c180cf8d0 tx=0x7f9c1001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956560040f7231f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c15d9b640 1 -- 172.21.15.67:0/3444935738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c100b2070 con 0x7f9c1806dc90 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c15d9b640 1 -- 172.21.15.67:0/3444935738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c100a57f0 con 0x7f9c1806dc90 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.167+0000 7f9c15d9b640 1 -- 172.21.15.67:0/3444935738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c100a5bc0 con 0x7f9c1806dc90 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/3444935738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 msgr2=0x7f9c18163ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/3444935738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c18163ec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9c180cf8d0 tx=0x7f9c1001c930 comp rx=0 tx=0).stop 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/3444935738 shutdown_connections 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/3444935738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c18163ec0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/3444935738 >> v2:172.21.15.176:3300/0 conn(0x7f9c18073bf0 0x7f9c1806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/3444935738 >> 172.21.15.67:0/3444935738 conn(0x7f9c18070db0 msgr2=0x7f9c180ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/3444935738 shutdown_connections 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/3444935738 wait complete. 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 Processor -- start 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.171+0000 7f9c1d1a7640 1 -- start start 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c181f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c18073bf0 0x7f9c180cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c180caf60 0x7f9c1814d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1815a570 con 0x7f9c1806dc90 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1815a3f0 con 0x7f9c180caf60 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9c18164c10 con 0x7f9c18073bf0 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c181f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c181f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60606/0 (socket says 172.21.15.67:60606) 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 -- 172.21.15.67:0/446315815 learned_addr learned my addr 172.21.15.67:0/446315815 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1659c640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c18073bf0 0x7f9c180cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1759e640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c180caf60 0x7f9c1814d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 -- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c18073bf0 msgr2=0x7f9c180cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c18073bf0 0x7f9c180cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 -- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c180caf60 msgr2=0x7f9c1814d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c180caf60 0x7f9c1814d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 -- 172.21.15.67:0/446315815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c18158bb0 con 0x7f9c1806dc90 2024-04-25T15:52:10.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c16d9d640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c181f5a90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c3f0 tx=0x7f9c0800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9bff7fe640 1 -- 172.21.15.67:0/446315815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08017070 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9bff7fe640 1 -- 172.21.15.67:0/446315815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c08005cf0 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c18156210 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c18166d50 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9bff7fe640 1 -- 172.21.15.67:0/446315815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c080072f0 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.175+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bf80052a0 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.179+0000 7f9bff7fe640 1 -- 172.21.15.67:0/446315815 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9c0801a070 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.179+0000 7f9bff7fe640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c0406f430 0x7f9c04071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.179+0000 7f9bff7fe640 1 -- 172.21.15.67:0/446315815 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5857+0+0 (secure 0 0 0) 0x7f9c0808e9b0 con 0x7f9c1806dc90 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.179+0000 7f9c1659c640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c0406f430 0x7f9c04071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.179+0000 7f9c1659c640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c0406f430 0x7f9c04071940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9c1806ccc0 tx=0x7f9c0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.183+0000 7f9bff7fe640 1 -- 172.21.15.67:0/446315815 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c08021020 con 0x7f9c1806dc90 2024-04-25T15:52:10.267 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:10.247+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:10.353 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:10.347+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:10.451 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:10.451 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:10.454 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:10.451+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.459+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f9bf8005860 con 0x7f9c1806dc90 2024-04-25T15:52:10.467 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:10.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 Processor -- start 2024-04-25T15:52:10.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 -- start start 2024-04-25T15:52:10.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340733f0 0x7f10340ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340c92d0 0x7f10340c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1034160d90 0x7f10341611a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f103406d050 con 0x7f10340c92d0 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1034074720 con 0x7f1034160d90 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.487+0000 7f103bdcc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f10340c9c20 con 0x7f10340733f0 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340c92d0 0x7f10340c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10340c92d0 0x7f10340c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60622/0 (socket says 172.21.15.67:60622) 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 -- 172.21.15.67:0/1391686068 learned_addr learned my addr 172.21.15.67:0/1391686068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039b68640 1 --2- 172.21.15.67:0/1391686068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340733f0 0x7f10340ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f103a369640 1 --2- 172.21.15.67:0/1391686068 >> v2:172.21.15.176:3300/0 conn(0x7f1034160d90 0x7f10341611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 -- 172.21.15.67:0/1391686068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340733f0 msgr2=0x7f10340ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 --2- 172.21.15.67:0/1391686068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340733f0 0x7f10340ca780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 -- 172.21.15.67:0/1391686068 >> v2:172.21.15.176:3300/0 conn(0x7f1034160d90 msgr2=0x7f10341611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 --2- 172.21.15.67:0/1391686068 >> v2:172.21.15.176:3300/0 conn(0x7f1034160d90 0x7f10341611a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 -- 172.21.15.67:0/1391686068 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1034074990 con 0x7f10340c92d0 2024-04-25T15:52:10.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039b68640 1 --2- 172.21.15.67:0/1391686068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340733f0 0x7f10340ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:10.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1039367640 1 --2- 172.21.15.67:0/1391686068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340c92d0 0x7f10340c96e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1034073c60 tx=0x7f102401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304d0b9563338e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1038b66640 1 -- 172.21.15.67:0/1391686068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10240ab5f0 con 0x7f10340c92d0 2024-04-25T15:52:10.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1038b66640 1 -- 172.21.15.67:0/1391686068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10240b0070 con 0x7f10340c92d0 2024-04-25T15:52:10.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.491+0000 7f1038b66640 1 -- 172.21.15.67:0/1391686068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10240ab9f0 con 0x7f10340c92d0 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 -- 172.21.15.67:0/1391686068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340c92d0 msgr2=0x7f10340c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 --2- 172.21.15.67:0/1391686068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340c92d0 0x7f10340c96e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1034073c60 tx=0x7f102401c540 comp rx=0 tx=0).stop 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 -- 172.21.15.67:0/1391686068 shutdown_connections 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 --2- 172.21.15.67:0/1391686068 >> v2:172.21.15.176:3300/0 conn(0x7f1034160d90 0x7f10341611a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 --2- 172.21.15.67:0/1391686068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340c92d0 0x7f10340c96e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 --2- 172.21.15.67:0/1391686068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340733f0 0x7f10340ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 -- 172.21.15.67:0/1391686068 >> 172.21.15.67:0/1391686068 conn(0x7f1034070db0 msgr2=0x7f10340ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 -- 172.21.15.67:0/1391686068 shutdown_connections 2024-04-25T15:52:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 -- 172.21.15.67:0/1391686068 wait complete. 2024-04-25T15:52:10.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 Processor -- start 2024-04-25T15:52:10.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.495+0000 7f103bdcc640 1 -- start start 2024-04-25T15:52:10.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340733f0 0x7f103415fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340c92d0 0x7f1034154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1034160d90 0x7f10340cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1034161c70 con 0x7f10340733f0 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1034161af0 con 0x7f1034160d90 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1034164b10 con 0x7f10340c92d0 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340733f0 0x7f103415fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340733f0 0x7f103415fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60634/0 (socket says 172.21.15.67:60634) 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 -- 172.21.15.67:0/3710486259 learned_addr learned my addr 172.21.15.67:0/3710486259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039367640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340c92d0 0x7f1034154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103a369640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1034160d90 0x7f10340cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340c92d0 msgr2=0x7f1034154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f10340c92d0 0x7f1034154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1034160d90 msgr2=0x7f10340cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1034160d90 0x7f10340cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10340cead0 con 0x7f10340733f0 2024-04-25T15:52:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f8406b4b640 1 Processor -- start 2024-04-25T15:52:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f8406b4b640 1 -- start start 2024-04-25T15:52:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340733f0 0x7f103415fa90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f103000d800 tx=0x7f103000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030018070 con 0x7f10340733f0 2024-04-25T15:52:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1030004030 con 0x7f10340733f0 2024-04-25T15:52:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030004400 con 0x7f10340733f0 2024-04-25T15:52:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f103415ec00 con 0x7f10340733f0 2024-04-25T15:52:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10341fb170 con 0x7f10340733f0 2024-04-25T15:52:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.499+0000 7f103bdcc640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f103406a480 con 0x7f10340733f0 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f8406b4b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84000cb1a0 0x7f8400160500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f8406b4b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f8400156530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f8406b4b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f840006fe40 con 0x7f84000cad90 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f8406b4b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f840015fd90 con 0x7f84000cb1a0 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f8406b4b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f840006f550 con 0x7f8400160a40 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f8400156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f8400156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:38216/0 (socket says 172.21.15.67:38216) 2024-04-25T15:52:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 -- 172.21.15.67:0/3072405943 learned_addr learned my addr 172.21.15.67:0/3072405943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:10.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f83fffff640 1 --2- 172.21.15.67:0/3072405943 >> v2:172.21.15.67:3300/0 conn(0x7f84000cb1a0 0x7f8400160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f10300096a0 con 0x7f10340733f0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f102a7fc640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f101806f430 0x7f1018071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5857+0+0 (secure 0 0 0) 0x7f103008f540 con 0x7f10340733f0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f1039367640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f101806f430 0x7f1018071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v44) ==== 54+0+16 (secure 0 0 0) 0x7f1030091bc0 con 0x7f10340733f0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f103bdcc640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1034154ac0 0x7f103415ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f103bdcc640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1034074120 con 0x7f1034154ac0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f103a369640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1034154ac0 0x7f103415ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f103a369640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1034154ac0 0x7f103415ffd0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f1039367640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f101806f430 0x7f1018071940 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f10240af000 tx=0x7f10240a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 -- 172.21.15.67:0/3072405943 >> v2:172.21.15.67:3300/0 conn(0x7f84000cb1a0 msgr2=0x7f8400160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:10.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 --2- 172.21.15.67:0/3072405943 >> v2:172.21.15.67:3300/0 conn(0x7f84000cb1a0 0x7f8400160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 -- 172.21.15.67:0/3072405943 >> v1:172.21.15.176:6789/0 conn(0x7f84000cad90 legacy=0x7f840006ee40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:52:10.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 -- 172.21.15.67:0/3072405943 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840006c7c0 con 0x7f8400160a40 2024-04-25T15:52:10.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.503+0000 7f84050e8640 1 --2- 172.21.15.67:0/3072405943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f8400156530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009e50 tx=0x7f83f801c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ddcd69003ccd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1034074120 con 0x7f1034154ac0 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f83ff7fe640 1 -- 172.21.15.67:0/3072405943 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80a5580 con 0x7f8400160a40 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f83ff7fe640 1 -- 172.21.15.67:0/3072405943 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f80a5720 con 0x7f8400160a40 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f83ff7fe640 1 -- 172.21.15.67:0/3072405943 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80a5af0 con 0x7f8400160a40 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 -- 172.21.15.67:0/3072405943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 msgr2=0x7f8400156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 --2- 172.21.15.67:0/3072405943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f8400156530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009e50 tx=0x7f83f801c810 comp rx=0 tx=0).stop 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 -- 172.21.15.67:0/3072405943 shutdown_connections 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 --2- 172.21.15.67:0/3072405943 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f8400156530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 --2- 172.21.15.67:0/3072405943 >> v2:172.21.15.67:3300/0 conn(0x7f84000cb1a0 0x7f8400160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 -- 172.21.15.67:0/3072405943 >> 172.21.15.67:0/3072405943 conn(0x7f84000cebf0 msgr2=0x7f84000cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 -- 172.21.15.67:0/3072405943 shutdown_connections 2024-04-25T15:52:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.507+0000 7f8406b4b640 1 -- 172.21.15.67:0/3072405943 wait complete. 2024-04-25T15:52:10.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 Processor -- start 2024-04-25T15:52:10.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 -- start start 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84000cad90 0x7f84001f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84000cb1a0 0x7f8400144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f840015eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8400156e80 con 0x7f84000cad90 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f840015ffe0 con 0x7f84000cb1a0 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f8406b4b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f840015c7a0 con 0x7f8400160a40 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f840015eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f840015eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:38232/0 (socket says 172.21.15.67:38232) 2024-04-25T15:52:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 -- 172.21.15.67:0/1972241837 learned_addr learned my addr 172.21.15.67:0/1972241837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f83fffff640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84000cb1a0 0x7f8400144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84048e7640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84000cad90 0x7f84001f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 -- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84000cb1a0 msgr2=0x7f8400144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84000cb1a0 0x7f8400144810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 -- 172.21.15.67:0/1972241837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84000cad90 msgr2=0x7f84001f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84000cad90 0x7f84001f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84050e8640 1 -- 172.21.15.67:0/1972241837 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840014bad0 con 0x7f8400160a40 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.511+0000 7f84048e7640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84000cad90 0x7f84001f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f84050e8640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f840015eab0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009e50 tx=0x7f83f80195b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f83fd7fa640 1 -- 172.21.15.67:0/1972241837 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80a5580 con 0x7f8400160a40 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f83fd7fa640 1 -- 172.21.15.67:0/1972241837 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f80be070 con 0x7f8400160a40 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f83fd7fa640 1 -- 172.21.15.67:0/1972241837 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80b88d0 con 0x7f8400160a40 2024-04-25T15:52:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84001f6780 con 0x7f8400160a40 2024-04-25T15:52:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001488b0 con 0x7f8400160a40 2024-04-25T15:52:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f840006a480 con 0x7f8400160a40 2024-04-25T15:52:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f83fd7fa640 1 -- 172.21.15.67:0/1972241837 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f83f80b8a70 con 0x7f8400160a40 2024-04-25T15:52:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f83fd7fa640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f83e406f430 0x7f83e4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.515+0000 7f83fd7fa640 1 -- 172.21.15.67:0/1972241837 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 5857+0+0 (secure 0 0 0) 0x7f83f8129b00 con 0x7f8400160a40 2024-04-25T15:52:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.519+0000 7f84048e7640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f83e406f430 0x7f83e4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.519+0000 7f84048e7640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f83e406f430 0x7f83e4071940 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83ec006fd0 tx=0x7f83ec009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.523+0000 7f83fd7fa640 1 -- 172.21.15.67:0/1972241837 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f83f80c7020 con 0x7f8400160a40 2024-04-25T15:52:10.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.527+0000 7f103bdcc640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10341fb600 con 0x7f1034154ac0 2024-04-25T15:52:10.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.531+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10341fb600 con 0x7f1034154ac0 2024-04-25T15:52:10.536 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.531+0000 7f103bdcc640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f10340cc730 0x7f10340ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.531+0000 7f103bdcc640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1034073920 con 0x7f10340cc730 2024-04-25T15:52:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.531+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f10340cc730 0x7f10340ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.535+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f10340cc730 0x7f10340ccb40 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.535+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1034073920 con 0x7f10340cc730 2024-04-25T15:52:10.555 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:10.552+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.563+0000 7f103bdcc640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1034069c70 con 0x7f10340cc730 2024-04-25T15:52:10.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.563+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1034069c70 con 0x7f10340cc730 2024-04-25T15:52:10.569 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:10.570 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:10.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.563+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1010001660 0x7f1010003b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.563+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1010006c10 con 0x7f1010001660 2024-04-25T15:52:10.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.563+0000 7f1039367640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1010001660 0x7f1010003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.567+0000 7f1039367640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1010001660 0x7f1010003b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.567+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1010006c10 con 0x7f1010001660 2024-04-25T15:52:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.591+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1010005c30 con 0x7f1010001660 2024-04-25T15:52:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.595+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1010005c30 con 0x7f1010001660 2024-04-25T15:52:10.600 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.595+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f10100041f0 0x7f1010004600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.595+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1010004bc0 con 0x7f10100041f0 2024-04-25T15:52:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.595+0000 7f103a369640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f10100041f0 0x7f1010004600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.595+0000 7f103a369640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f10100041f0 0x7f1010004600 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.599+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1010004bc0 con 0x7f10100041f0 2024-04-25T15:52:10.611 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:10.611 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:10.611 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:10.611 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:10.611 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.623+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1010005130 con 0x7f10100041f0 2024-04-25T15:52:10.629 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.627+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1010005130 con 0x7f10100041f0 2024-04-25T15:52:10.630 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.627+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1010008e30 0x7f101000b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.627+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f101000e1b0 con 0x7f1010008e30 2024-04-25T15:52:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.627+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1010008e30 0x7f101000b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.627+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1010008e30 0x7f101000b250 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.627+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f101000e1b0 con 0x7f1010008e30 2024-04-25T15:52:10.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.643+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f101000ced0 con 0x7f1010008e30 2024-04-25T15:52:10.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.643+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f101000ced0 con 0x7f1010008e30 2024-04-25T15:52:10.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:10.648 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:10.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.643+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f101000c060 0x7f101000c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.643+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f101000de40 con 0x7f101000c060 2024-04-25T15:52:10.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.643+0000 7f1039367640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f101000c060 0x7f101000c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.655+0000 7f1039367640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f101000c060 0x7f101000c4b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.655+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f101000de40 con 0x7f101000c060 2024-04-25T15:52:10.671 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:10.668+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:10.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.683+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f101000b7f0 con 0x7f101000c060 2024-04-25T15:52:10.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.683+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f101000b7f0 con 0x7f101000c060 2024-04-25T15:52:10.689 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.683+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f101000d120 0x7f101000d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.683+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1010014a60 con 0x7f101000d120 2024-04-25T15:52:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.687+0000 7f103a369640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f101000d120 0x7f101000d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.687+0000 7f103a369640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f101000d120 0x7f101000d590 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.687+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1010014a60 con 0x7f101000d120 2024-04-25T15:52:10.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.715+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10100138e0 con 0x7f101000d120 2024-04-25T15:52:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.715+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10100138e0 con 0x7f101000d120 2024-04-25T15:52:10.721 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:10.722 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:10.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.715+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1010013f60 0x7f1010014370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:10.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.715+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1010017e00 con 0x7f1010013f60 2024-04-25T15:52:10.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.719+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1010013f60 0x7f1010014370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:10.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.719+0000 7f1039b68640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1010013f60 0x7f1010014370 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:10.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.719+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1010017e00 con 0x7f1010013f60 2024-04-25T15:52:10.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.731+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1010010110 con 0x7f1010013f60 2024-04-25T15:52:10.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f102a7fc640 1 -- 172.21.15.67:0/3710486259 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1010010110 con 0x7f1010013f60 2024-04-25T15:52:10.739 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:10.739 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:10.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1034154ac0 msgr2=0x7f103415ffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1034154ac0 0x7f103415ffd0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f10340cc730 msgr2=0x7f10340ccb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f10340cc730 0x7f10340ccb40 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1010001660 msgr2=0x7f1010003b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1010001660 0x7f1010003b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f103a369640 1 -- 172.21.15.67:0/3710486259 reap_dead start 2024-04-25T15:52:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f10100041f0 msgr2=0x7f1010004600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f10100041f0 0x7f1010004600 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1010008e30 msgr2=0x7f101000b250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1010008e30 0x7f101000b250 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f101000c060 msgr2=0x7f101000c4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f101000c060 0x7f101000c4b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f101000d120 msgr2=0x7f101000d590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f101000d120 0x7f101000d590 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1010013f60 msgr2=0x7f1010014370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1010013f60 0x7f1010014370 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f101806f430 msgr2=0x7f1018071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f101806f430 0x7f1018071940 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f10240af000 tx=0x7f10240a7000 comp rx=0 tx=0).stop 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f103a369640 1 -- 172.21.15.67:0/3710486259 reap_dead start 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f103a369640 1 -- 172.21.15.67:0/3710486259 reap_dead start 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340733f0 msgr2=0x7f103415fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340733f0 0x7f103415fa90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f103000d800 tx=0x7f103000dcc0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 shutdown_connections 2024-04-25T15:52:10.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 --2- 172.21.15.67:0/3710486259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10340733f0 0x7f103415fa90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 >> 172.21.15.67:0/3710486259 conn(0x7f1034070db0 msgr2=0x7f1034162100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:10.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 shutdown_connections 2024-04-25T15:52:10.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.735+0000 7f1017fff640 1 -- 172.21.15.67:0/3710486259 wait complete. 2024-04-25T15:52:10.757 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:10.756+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.799+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.7", "id": [4, 6, 2, 1]} v 0) -- 0x7f840006d9f0 con 0x7f8400160a40 2024-04-25T15:52:10.823 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:10.823 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:10.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:10.883 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:10.884+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:10.959 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:10.960+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.963+0000 7f9bff7fe640 1 -- 172.21.15.67:0/446315815 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v45) ==== 91+0+0 (secure 0 0 0) 0x7f9c0805ba00 con 0x7f9c1806dc90 2024-04-25T15:52:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c0406f430 msgr2=0x7f9c04071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c0406f430 0x7f9c04071940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9c1806ccc0 tx=0x7f9c0c008040 comp rx=0 tx=0).stop 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 msgr2=0x7f9c181f5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c181f5a90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c3f0 tx=0x7f9c0800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 shutdown_connections 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9c0406f430 0x7f9c04071940 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c180caf60 0x7f9c1814d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c18073bf0 0x7f9c180cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 --2- 172.21.15.67:0/446315815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c1806dc90 0x7f9c181f5a90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 >> 172.21.15.67:0/446315815 conn(0x7f9c18070db0 msgr2=0x7f9c1815d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 shutdown_connections 2024-04-25T15:52:10.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.967+0000 7f9c1d1a7640 1 -- 172.21.15.67:0/446315815 wait complete. 2024-04-25T15:52:10.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f83fd7fa640 1 -- 172.21.15.67:0/1972241837 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.7", "id": [4, 6, 2, 1]}]=0 set 3.7 pg_upmap_items mapping to [4->6,2->1] v45) ==== 146+0+0 (secure 0 0 0) 0x7f840014bad0 con 0x7f8400160a40 2024-04-25T15:52:10.978 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.7 pg_upmap_items mapping to [4->6,2->1] 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f83e406f430 msgr2=0x7f83e4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f83e406f430 0x7f83e4071940 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83ec006fd0 tx=0x7f83ec009060 comp rx=0 tx=0).stop 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 msgr2=0x7f840015eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f840015eab0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009e50 tx=0x7f83f80195b0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 shutdown_connections 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f83e406f430 0x7f83e4071940 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8400160a40 0x7f840015eab0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84000cb1a0 0x7f8400144810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 --2- 172.21.15.67:0/1972241837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84000cad90 0x7f84001f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 >> 172.21.15.67:0/1972241837 conn(0x7f84000cebf0 msgr2=0x7f840014dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 shutdown_connections 2024-04-25T15:52:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:10.975+0000 7f8406b4b640 1 -- 172.21.15.67:0/1972241837 wait complete. 2024-04-25T15:52:11.033 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:11.033 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:11.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:11.091 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:11.092+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:11.161 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:11.156+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:11.236 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:11.236 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:11.254 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:11.261 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:11.255+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:11.362 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:11.359+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:11.442 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:11.443 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:11.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:11.506 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:11.503+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:11.563 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:11.564+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:11.658 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:11.658 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:11.664 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:11.659+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:11.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:11.765 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:11.764+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:11.856 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:11.856 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:11.865 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:11.859+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:11.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:11.966 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:11.963+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:12.067 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:12.060+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:12.090 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:12.090 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:12.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:12.194 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:12.191+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:12.268 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:12.263+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:12.322 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:12.322 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:12.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:12.386 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:12.383+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:12.470 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:12.468+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:12.520 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:12.520 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:12.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:12.570 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:12.568+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:12.668 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:12.668 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:12.670 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:12.667+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:12.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:12.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:12.771 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:12.772+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:12.872 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:12.872+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:12.891 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:12.891 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:12.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:12.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.907+0000 7f5c49176640 1 Processor -- start 2024-04-25T15:52:12.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.907+0000 7f5c49176640 1 -- start start 2024-04-25T15:52:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c49176640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440736d0 0x7f5c4406c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c49176640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c440caa60 con 0x7f5c4406cb80 2024-04-25T15:52:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c49176640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4415a5f0 con 0x7f5c4406d460 2024-04-25T15:52:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c49176640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c44068590 con 0x7f5c440736d0 2024-04-25T15:52:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c4359e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c4406cb80 0x7f5c4415aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46862/0 (socket says 172.21.15.67:46862) 2024-04-25T15:52:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c4359e640 1 -- 172.21.15.67:0/3385190528 learned_addr learned my addr 172.21.15.67:0/3385190528 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:12.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c4259c640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440736d0 0x7f5c4406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265231868 0 0) 0x7f5c440caa60 con 0x7f5c4406cb80 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c2c003530 con 0x7f5c4406cb80 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3941842050 0 0) 0x7f5c2c003530 con 0x7f5c4406cb80 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440736d0 msgr2=0x7f5c4406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440736d0 0x7f5c4406c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 >> v1:172.21.15.176:6789/0 conn(0x7f5c4406d460 legacy=0x7f5c441547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c440ca100 con 0x7f5c4406cb80 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f5c3c001a00 con 0x7f5c4406cb80 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 >> v1:172.21.15.67:6789/0 conn(0x7f5c4406cb80 legacy=0x7f5c4415aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0039f0 0x7f5c2c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c2c004810 0x7f5c2c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440736d0 0x7f5c2c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c44068590 con 0x7f5c2c0039f0 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c00bb60 con 0x7f5c2c004810 2024-04-25T15:52:12.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c2c00bd60 con 0x7f5c440736d0 2024-04-25T15:52:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0039f0 0x7f5c2c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c42d9d640 1 -- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440736d0 msgr2=0x7f5c2c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440736d0 0x7f5c2c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c42d9d640 1 -- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c2c004810 msgr2=0x7f5c2c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c2c004810 0x7f5c2c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.911+0000 7f5c42d9d640 1 -- 172.21.15.67:0/3385190528 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c003530 con 0x7f5c2c0039f0 2024-04-25T15:52:12.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0039f0 0x7f5c2c003e80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5c34001750 tx=0x7f5c3401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dbe37a02f7e7c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:12.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c340013a0 con 0x7f5c2c0039f0 2024-04-25T15:52:12.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c340052f0 con 0x7f5c2c0039f0 2024-04-25T15:52:12.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c41d9b640 1 -- 172.21.15.67:0/3385190528 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34003840 con 0x7f5c2c0039f0 2024-04-25T15:52:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c49176640 1 -- 172.21.15.67:0/3385190528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0039f0 msgr2=0x7f5c2c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:12.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c49176640 1 --2- 172.21.15.67:0/3385190528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c2c0039f0 0x7f5c2c003e80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5c34001750 tx=0x7f5c3401c320 comp rx=0 tx=0).stop 2024-04-25T15:52:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c4359e640 1 -- 172.21.15.67:0/3385190528 reap_dead start 2024-04-25T15:52:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c49176640 1 -- 172.21.15.67:0/3385190528 shutdown_connections 2024-04-25T15:52:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c49176640 1 -- 172.21.15.67:0/3385190528 >> 172.21.15.67:0/3385190528 conn(0x7f5c440ce1c0 msgr2=0x7f5c440ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c49176640 1 -- 172.21.15.67:0/3385190528 shutdown_connections 2024-04-25T15:52:12.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.915+0000 7f5c49176640 1 -- 172.21.15.67:0/3385190528 wait complete. 2024-04-25T15:52:12.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 Processor -- start 2024-04-25T15:52:12.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 -- start start 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c440acda0 0x7f5c441f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440ad1b0 0x7f5c44157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c441f6ce0 0x7f5c440c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c44166910 con 0x7f5c441f6ce0 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c44166790 con 0x7f5c440acda0 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c44166a90 con 0x7f5c440ad1b0 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c441f6ce0 0x7f5c440c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c441f6ce0 0x7f5c440c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54374/0 (socket says 172.21.15.67:54374) 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 -- 172.21.15.67:0/2865169863 learned_addr learned my addr 172.21.15.67:0/2865169863 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c440acda0 0x7f5c441f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440ad1b0 0x7f5c44157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440ad1b0 msgr2=0x7f5c44157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440ad1b0 0x7f5c44157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c440acda0 msgr2=0x7f5c441f67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c440acda0 0x7f5c441f67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c441f3c20 con 0x7f5c441f6ce0 2024-04-25T15:52:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c440ad1b0 0x7f5c44157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:12.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c4359e640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c441f6ce0 0x7f5c440c55c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5c44154f10 tx=0x7f5c3c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00a5e0 con 0x7f5c441f6ce0 2024-04-25T15:52:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c441f3ec0 con 0x7f5c441f6ce0 2024-04-25T15:52:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c3c00a780 con 0x7f5c441f6ce0 2024-04-25T15:52:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.919+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00ab30 con 0x7f5c441f6ce0 2024-04-25T15:52:12.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.923+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c44156270 con 0x7f5c441f6ce0 2024-04-25T15:52:12.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.923+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5c3c00acd0 con 0x7f5c441f6ce0 2024-04-25T15:52:12.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.923+0000 7f5c2b7fe640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3006f400 0x7f5c30071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.923+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5886+0+0 (secure 0 0 0) 0x7f5c3c08eb60 con 0x7f5c441f6ce0 2024-04-25T15:52:12.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.923+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5c4406a480 con 0x7f5c441f6ce0 2024-04-25T15:52:12.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v46) ==== 54+0+16 (secure 0 0 0) 0x7f5c3c01a020 con 0x7f5c441f6ce0 2024-04-25T15:52:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3006f400 0x7f5c30071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c44159450 0x7f5c440c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c44153b90 con 0x7f5c44159450 2024-04-25T15:52:12.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c44159450 0x7f5c440c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c44159450 0x7f5c440c5b00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:12.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3006f400 0x7f5c30071910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c340ac000 tx=0x7f5c340011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:12.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.927+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c38009b40 con 0x7f5c44159450 2024-04-25T15:52:12.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.943+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c44157f40 con 0x7f5c44159450 2024-04-25T15:52:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.943+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c44157f40 con 0x7f5c44159450 2024-04-25T15:52:12.948 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.943+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c44156950 0x7f5c441f7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.943+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5c441599d0 con 0x7f5c44156950 2024-04-25T15:52:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.943+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c44156950 0x7f5c441f7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.947+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c44156950 0x7f5c441f7190 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.947+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c441599d0 con 0x7f5c44156950 2024-04-25T15:52:12.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.959+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c441f7920 con 0x7f5c44156950 2024-04-25T15:52:12.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.963+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c441f7920 con 0x7f5c44156950 2024-04-25T15:52:12.967 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.963+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c440635a0 0x7f5c440639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.963+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f5c441f7ef0 con 0x7f5c440635a0 2024-04-25T15:52:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.963+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c440635a0 0x7f5c440639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.963+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c440635a0 0x7f5c440639b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.963+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c441f7ef0 con 0x7f5c440635a0 2024-04-25T15:52:12.972 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:12.967+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:12.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.979+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c44162f60 con 0x7f5c440635a0 2024-04-25T15:52:12.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.979+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c44162f60 con 0x7f5c440635a0 2024-04-25T15:52:12.984 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:12.984 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:12.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.979+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c44163a40 0x7f5c4402c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:12.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.979+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5c44163e50 con 0x7f5c44163a40 2024-04-25T15:52:12.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.979+0000 7f5c4359e640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c44163a40 0x7f5c4402c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:12.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.979+0000 7f5c4359e640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c44163a40 0x7f5c4402c9e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:12.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:12.979+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c44163e50 con 0x7f5c44163a40 2024-04-25T15:52:12.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:52:13.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.007+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c4402d1f0 con 0x7f5c44163a40 2024-04-25T15:52:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.007+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c4402d1f0 con 0x7f5c44163a40 2024-04-25T15:52:13.013 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:13.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:13.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.011+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c4402d910 0x7f5c44208510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.011+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5c4402de30 con 0x7f5c4402d910 2024-04-25T15:52:13.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.011+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c4402d910 0x7f5c44208510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.011+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c4402d910 0x7f5c44208510 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:13.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.011+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c4402de30 con 0x7f5c4402d910 2024-04-25T15:52:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.027+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c44074fc0 con 0x7f5c4402d910 2024-04-25T15:52:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.031+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c44074fc0 con 0x7f5c4402d910 2024-04-25T15:52:13.035 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.031+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c44208f60 0x7f5c4420b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.031+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f5c4420e2b0 con 0x7f5c44208f60 2024-04-25T15:52:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.031+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c44208f60 0x7f5c4420b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.031+0000 7f5c4259c640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c44208f60 0x7f5c4420b380 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:13.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.035+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c4420e2b0 con 0x7f5c44208f60 2024-04-25T15:52:13.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.055+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c4420b8c0 con 0x7f5c44208f60 2024-04-25T15:52:13.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.059+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c4420b8c0 con 0x7f5c44208f60 2024-04-25T15:52:13.065 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:13.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:13.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.059+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c4420bfe0 0x7f5c4420c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.059+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5c44211700 con 0x7f5c4420bfe0 2024-04-25T15:52:13.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.063+0000 7f5c4359e640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c4420bfe0 0x7f5c4420c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.063+0000 7f5c4359e640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c4420bfe0 0x7f5c4420c3f0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:13.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.063+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c44211700 con 0x7f5c4420bfe0 2024-04-25T15:52:13.074 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:13.072+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:13.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.083+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c44210580 con 0x7f5c4420bfe0 2024-04-25T15:52:13.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.091+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c44210580 con 0x7f5c4420bfe0 2024-04-25T15:52:13.094 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.091+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c44210c00 0x7f5c44211050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.091+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5c44214aa0 con 0x7f5c44210c00 2024-04-25T15:52:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.091+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c44210c00 0x7f5c44211050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.091+0000 7f5c42d9d640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c44210c00 0x7f5c44211050 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.095+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c44214aa0 con 0x7f5c44210c00 2024-04-25T15:52:13.109 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:13.109 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:13.109 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:13.109 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:13.109 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.119+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f5c4420c980 con 0x7f5c44210c00 2024-04-25T15:52:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c2b7fe640 1 -- 172.21.15.67:0/2865169863 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5c4420c980 con 0x7f5c44210c00 2024-04-25T15:52:13.129 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c44159450 msgr2=0x7f5c440c5b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5c44159450 0x7f5c440c5b00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c44156950 msgr2=0x7f5c441f7190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f5c44156950 0x7f5c441f7190 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c440635a0 msgr2=0x7f5c440639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f5c440635a0 0x7f5c440639b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c4359e640 1 -- 172.21.15.67:0/2865169863 reap_dead start 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c44163a40 msgr2=0x7f5c4402c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5c44163a40 0x7f5c4402c9e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.123+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c4402d910 msgr2=0x7f5c44208510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f5c4402d910 0x7f5c44208510 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c44208f60 msgr2=0x7f5c4420b380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f5c44208f60 0x7f5c4420b380 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c4420bfe0 msgr2=0x7f5c4420c3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5c4420bfe0 0x7f5c4420c3f0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c44210c00 msgr2=0x7f5c44211050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5c44210c00 0x7f5c44211050 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c4359e640 1 -- 172.21.15.67:0/2865169863 reap_dead start 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3006f400 msgr2=0x7f5c30071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3006f400 0x7f5c30071910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c340ac000 tx=0x7f5c340011a0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c441f6ce0 msgr2=0x7f5c440c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c441f6ce0 0x7f5c440c55c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5c44154f10 tx=0x7f5c3c00ccd0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 shutdown_connections 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3006f400 0x7f5c30071910 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 --2- 172.21.15.67:0/2865169863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c441f6ce0 0x7f5c440c55c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 >> 172.21.15.67:0/2865169863 conn(0x7f5c440ce1c0 msgr2=0x7f5c44165770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 shutdown_connections 2024-04-25T15:52:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.127+0000 7f5c49176640 1 -- 172.21.15.67:0/2865169863 wait complete. 2024-04-25T15:52:13.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.179+0000 7fefd1709640 1 Processor -- start 2024-04-25T15:52:13.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefd1709640 1 -- start start 2024-04-25T15:52:13.187 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:13.183+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefd1709640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fefcc162ec0 0x7fefcc1652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefcaffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fefcc0caf60 0x7fefcc15d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46874/0 (socket says 172.21.15.67:46874) 2024-04-25T15:52:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefd1709640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefcc0c77a0 con 0x7fefcc0caf60 2024-04-25T15:52:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefcaffd640 1 -- 172.21.15.67:0/3480349398 learned_addr learned my addr 172.21.15.67:0/3480349398 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:13.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefcb7fe640 1 --2- 172.21.15.67:0/3480349398 >> v2:172.21.15.67:3300/0 conn(0x7fefcc162ec0 0x7fefcc1652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefd1709640 1 -- 172.21.15.67:0/3480349398 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefcc0c7a60 con 0x7fefcc073bf0 2024-04-25T15:52:13.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefd1709640 1 -- 172.21.15.67:0/3480349398 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fefcc0cab60 con 0x7fefcc162ec0 2024-04-25T15:52:13.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefc9ffb640 1 -- 172.21.15.67:0/3480349398 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3821348204 0 0) 0x7fefcc0c77a0 con 0x7fefcc0caf60 2024-04-25T15:52:13.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefc9ffb640 1 -- 172.21.15.67:0/3480349398 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefb4003530 con 0x7fefcc0caf60 2024-04-25T15:52:13.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefcb7fe640 1 -- 172.21.15.67:0/3480349398 >> v1:172.21.15.176:6789/0 conn(0x7fefcc073bf0 legacy=0x7fefcc06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefcb7fe640 1 -- 172.21.15.67:0/3480349398 >> v1:172.21.15.67:6789/0 conn(0x7fefcc0caf60 legacy=0x7fefcc15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.183+0000 7fefcb7fe640 1 -- 172.21.15.67:0/3480349398 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc0c7d40 con 0x7fefcc162ec0 2024-04-25T15:52:13.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefcb7fe640 1 --2- 172.21.15.67:0/3480349398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc162ec0 0x7fefcc1652e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009d90 tx=0x7fefc401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e85b42c7df1772 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:13.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefc9ffb640 1 -- 172.21.15.67:0/3480349398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40b2070 con 0x7fefcc162ec0 2024-04-25T15:52:13.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefc9ffb640 1 -- 172.21.15.67:0/3480349398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc40a5730 con 0x7fefcc162ec0 2024-04-25T15:52:13.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefc9ffb640 1 -- 172.21.15.67:0/3480349398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40a5b00 con 0x7fefcc162ec0 2024-04-25T15:52:13.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefd1709640 1 -- 172.21.15.67:0/3480349398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc162ec0 msgr2=0x7fefcc1652e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefd1709640 1 --2- 172.21.15.67:0/3480349398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc162ec0 0x7fefcc1652e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009d90 tx=0x7fefc401c970 comp rx=0 tx=0).stop 2024-04-25T15:52:13.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefd1709640 1 -- 172.21.15.67:0/3480349398 shutdown_connections 2024-04-25T15:52:13.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefd1709640 1 --2- 172.21.15.67:0/3480349398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc162ec0 0x7fefcc1652e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefd1709640 1 -- 172.21.15.67:0/3480349398 >> 172.21.15.67:0/3480349398 conn(0x7fefcc070db0 msgr2=0x7fefcc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:13.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefd1709640 1 -- 172.21.15.67:0/3480349398 shutdown_connections 2024-04-25T15:52:13.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.187+0000 7fefd1709640 1 -- 172.21.15.67:0/3480349398 wait complete. 2024-04-25T15:52:13.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 Processor -- start 2024-04-25T15:52:13.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 -- start start 2024-04-25T15:52:13.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefcc073bf0 0x7fefcc1f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc0caf60 0x7fefcc15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefcc162ec0 0x7fefcc141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc15a3f0 con 0x7fefcc0caf60 2024-04-25T15:52:13.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc159620 con 0x7fefcc073bf0 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.191+0000 7fefd1709640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fefcc15a570 con 0x7fefcc162ec0 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc0caf60 0x7fefcc15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc0caf60 0x7fefcc15bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54398/0 (socket says 172.21.15.67:54398) 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 -- 172.21.15.67:0/2490686034 learned_addr learned my addr 172.21.15.67:0/2490686034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefcb7fe640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefcc162ec0 0x7fefcc141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefcaffd640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefcc073bf0 0x7fefcc1f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 -- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefcc162ec0 msgr2=0x7fefcc141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefcc162ec0 0x7fefcc141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 -- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefcc073bf0 msgr2=0x7fefcc1f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefcc073bf0 0x7fefcc1f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:13.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 -- 172.21.15.67:0/2490686034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc1f68d0 con 0x7fefcc0caf60 2024-04-25T15:52:13.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefca7fc640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc0caf60 0x7fefcc15bce0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c3c0 tx=0x7fefc000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:13.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefb37fe640 1 -- 172.21.15.67:0/2490686034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0017070 con 0x7fefcc0caf60 2024-04-25T15:52:13.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefb37fe640 1 -- 172.21.15.67:0/2490686034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc00052d0 con 0x7fefcc0caf60 2024-04-25T15:52:13.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefb37fe640 1 -- 172.21.15.67:0/2490686034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0003820 con 0x7fefcc0caf60 2024-04-25T15:52:13.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefcc146290 con 0x7fefcc0caf60 2024-04-25T15:52:13.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefcc146850 con 0x7fefcc0caf60 2024-04-25T15:52:13.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.195+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefcc06a480 con 0x7fefcc0caf60 2024-04-25T15:52:13.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.199+0000 7fefb37fe640 1 -- 172.21.15.67:0/2490686034 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fefc001a070 con 0x7fefcc0caf60 2024-04-25T15:52:13.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.199+0000 7fefb37fe640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fefb806f4d0 0x7fefb80719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:13.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.199+0000 7fefb37fe640 1 -- 172.21.15.67:0/2490686034 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5886+0+0 (secure 0 0 0) 0x7fefc008e560 con 0x7fefcc0caf60 2024-04-25T15:52:13.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.199+0000 7fefcaffd640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fefb806f4d0 0x7fefb80719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:13.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.199+0000 7fefcaffd640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fefb806f4d0 0x7fefb80719e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0011f0 tx=0x7fefbc007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.207+0000 7fefb37fe640 1 -- 172.21.15.67:0/2490686034 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fefc0005480 con 0x7fefcc0caf60 2024-04-25T15:52:13.275 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:13.272+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:13.375 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:13.376+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:13.378 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163740657000005, 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937687, 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212220+0000", 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212245+0000", 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212458+0000", 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021240899999999999 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212488+0000", 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0715e-05 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212836+0000", 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000348096 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213194+0000", 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000357413 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213591+0000", 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039711599999999998 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224723+0000", 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011132177 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224892+0000", 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016872100000000001 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225031+0000", 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013966 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225148+0000", 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116168 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:13.383 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163694355999993, 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013374822, 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212257+0000", 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212263+0000", 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0900000000000001e-06 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212576+0000", 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031227099999999998 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212590+0000", 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4133e-05 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214544+0000", 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001953939 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214793+0000", 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024941399999999999 2024-04-25T15:52:13.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215181+0000", 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038823800000000001 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224739+0000", 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095574579999999996 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225301+0000", 2024-04-25T15:52:13.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056223199999999997 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225459+0000", 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015786199999999999 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225631+0000", 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172242 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163683942999995, 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013916951, 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8299999999999996e-07 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212272+0000", 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6630000000000001e-06 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212665+0000", 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039338399999999999 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212687+0000", 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1882000000000002e-05 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215937+0000", 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032501119999999999 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032469999999999998 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216646+0000", 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038413800000000001 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224753+0000", 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081070039999999993 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225911+0000", 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001158002 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226062+0000", 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015119600000000001 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226184+0000", 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121187 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163675974, 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01435588, 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:13.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6000000000000003e-07 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212279+0000", 2024-04-25T15:52:13.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1169999999999997e-06 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212723+0000", 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044335799999999998 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212733+0000", 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349e-05 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217516+0000", 2024-04-25T15:52:13.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004782606 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217766+0000", 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025045699999999999 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039180399999999999 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224766+0000", 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066076449999999997 2024-04-25T15:52:13.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226308+0000", 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00154228 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226483+0000", 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174938 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226630+0000", 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014766599999999999 2024-04-25T15:52:13.398 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163668994000005, 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014867514, 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2699999999999999e-07 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212287+0000", 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6229999999999996e-06 2024-04-25T15:52:13.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212756+0000", 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046907999999999997 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9560000000000003e-06 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218829+0000", 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060638690000000004 2024-04-25T15:52:13.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219182+0000", 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035291899999999999 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219507+0000", 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032545399999999998 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224779+0000", 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052717279999999998 2024-04-25T15:52:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226860+0000", 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020809790000000002 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227036+0000", 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176062 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227149+0000", 2024-04-25T15:52:13.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113217 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163661524000005, 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015281197, 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212290+0000", 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3e-07 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212333+0000", 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3251000000000003e-05 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212784+0000", 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045139399999999998 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212793+0000", 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7960000000000001e-06 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220253+0000", 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074595699999999996 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220482+0000", 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000229554 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220879+0000", 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039646400000000002 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225181+0000", 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00430205 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227277+0000", 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020961119999999998 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227449+0000", 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017166800000000001 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227570+0000", 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012170799999999999 2024-04-25T15:52:13.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163612044000004, 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016239407000000001, 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:13.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212340+0000", 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1200000000000001e-06 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212346+0000", 2024-04-25T15:52:13.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.584e-06 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212813+0000", 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046677600000000002 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212821+0000", 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221805+0000", 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089835999999999996 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222154+0000", 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034940299999999997 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222503+0000", 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034907900000000002 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227840+0000", 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367349999999996 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228310+0000", 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047011999999999998 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228439+0000", 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012899999999999999 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228578+0000", 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138837 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163600833999993, 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016443972000000001, 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212351+0000", 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212357+0000", 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000003e-06 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212848+0000", 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049076600000000001 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212863+0000", 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5498999999999998e-05 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223300+0000", 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010437093999999999 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223537+0000", 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000236665 2024-04-25T15:52:13.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224027+0000", 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048967900000000003 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228184+0000", 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041571209999999997 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228599+0000", 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000415392 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228676+0000", 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7095999999999995e-05 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228794+0000", 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011746700000000001 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.418 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163589854999998, 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013451848000000001, 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212362+0000", 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212368+0000", 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6229999999999999e-06 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212925+0000", 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055690199999999996 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212960+0000", 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5045000000000001e-05 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213123+0000", 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163222 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213447+0000", 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000323627 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:13.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213759+0000", 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031232800000000001 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216671+0000", 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002911145 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223654+0000", 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069831370000000004 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223829+0000", 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174857 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225813+0000", 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019840679999999999 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.163578692000002, 2024-04-25T15:52:13.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014242332, 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212373+0000", 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4e-07 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212384+0000", 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1443e-05 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213026+0000", 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064212500000000001 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213045+0000", 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8621999999999999e-05 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214708+0000", 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016628109999999999 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214970+0000", 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026197699999999999 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215380+0000", 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000410789 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218599+0000", 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032185270000000001 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226296+0000", 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076965269999999999 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226473+0000", 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017755200000000001 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226614+0000", 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:4 2.10 2:0f65c9b8:::60a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:13.428 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.128775357999999, 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010394548999999999, 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:13.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247176+0000", 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2100000000000001e-06 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247209+0000", 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2945999999999999e-05 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247394+0000", 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018468799999999999 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247417+0000", 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2733000000000001e-05 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249734+0000", 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002317684 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250032+0000", 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029727699999999998 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250453+0000", 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042143500000000001 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255742+0000", 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052886809999999999 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257232+0000", 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014906260000000001 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257393+0000", 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016088699999999999 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257570+0000", 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017638199999999999 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.433 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044806792000003, 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094210270000000002, 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6699999999999999e-07 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331165+0000", 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331418+0000", 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271799999999997 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331428+0000", 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6460000000000005e-06 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333229+0000", 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018009499999999999 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334396+0000", 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011673269999999999 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334597+0000", 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020135300000000001 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339905+0000", 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053077610000000002 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340210+0000", 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030499999999999999 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340385+0000", 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017450199999999999 2024-04-25T15:52:13.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340565+0000", 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018011700000000001 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044782045999995, 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095762039999999996, 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.439 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:13.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331174+0000", 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-06 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331467+0000", 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029356 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331478+0000", 2024-04-25T15:52:13.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336604+0000", 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051267170000000003 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336735+0000", 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013052699999999999 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336996+0000", 2024-04-25T15:52:13.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260511 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339986+0000", 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002990487 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340579+0000", 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059299899999999998 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340673+0000", 2024-04-25T15:52:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4241999999999997e-05 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340745+0000", 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1484999999999997e-05 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:17 2.10 2:0e9da45e:::61b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044774799999999, 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010814343000000001, 2024-04-25T15:52:13.444 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331180+0000", 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6650000000000001e-06 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331501+0000", 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032061600000000002 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331509+0000", 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8240000000000006e-06 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333383+0000", 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018733020000000001 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333652+0000", 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026890399999999999 2024-04-25T15:52:13.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333865+0000", 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021341 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338313+0000", 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044478269999999997 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341868+0000", 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035546729999999999 2024-04-25T15:52:13.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341920+0000", 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2507999999999999e-05 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341990+0000", 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0110000000000005e-05 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.449 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044765127999995, 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011086621, 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331186+0000", 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.51e-07 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331191+0000", 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3970000000000002e-06 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331546+0000", 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035447799999999998 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331557+0000", 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0791000000000001e-05 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334408+0000", 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028516520000000001 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334546+0000", 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013802700000000001 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334877+0000", 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000331081 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339910+0000", 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050323449999999997 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342080+0000", 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021703899999999999 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342184+0000", 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103767 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342272+0000", 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8242e-05 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044757107999999, 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010331643, 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331194+0000", 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:13.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331198+0000", 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1400000000000002e-06 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331608+0000", 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040962799999999999 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331620+0000", 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337545+0000", 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005924726 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338432+0000", 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088720699999999999 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338639+0000", 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020665000000000001 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341318+0000", 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026785860000000002 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341406+0000", 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8344000000000007e-05 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341458+0000", 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805999999999998e-05 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341525+0000", 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7336999999999996e-05 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:2 2.10 2:0daba267:::402.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044750648999994, 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011245289, 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.460 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331215+0000", 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331647+0000", 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043194799999999998 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331656+0000", 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335316+0000", 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036607710000000002 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335504+0000", 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018793299999999999 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335699+0000", 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194544 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339945+0000", 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042457600000000003 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342333+0000", 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023885730000000002 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342383+0000", 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9314999999999998e-05 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342445+0000", 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2691e-05 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044731888000001, 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087706009999999994, 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:13.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331239+0000", 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9653e-05 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331678+0000", 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043868999999999999 2024-04-25T15:52:13.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331701+0000", 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3019000000000001e-05 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333753+0000", 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020522050000000001 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334131+0000", 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000378484 2024-04-25T15:52:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334455+0000", 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032319899999999998 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337268+0000", 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028133199999999998 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339816+0000", 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025479140000000001 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339907+0000", 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1150000000000007e-05 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339989+0000", 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2392000000000007e-05 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.470 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:27 2.10 2:091e33df:::502.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044708846999995, 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011447874, 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:13.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4199999999999996e-07 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331249+0000", 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2259999999999999e-06 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331730+0000", 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048009600000000001 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331739+0000", 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1840000000000005e-06 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336653+0000", 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049141510000000003 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336830+0000", 2024-04-25T15:52:13.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017740099999999999 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337033+0000", 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020295999999999999 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340803+0000", 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037692480000000002 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:13.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342567+0000", 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017649250000000001 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342650+0000", 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2748000000000001e-05 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342690+0000", 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9393e-05 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:31 2.10 2:0eee8b97:::202.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.044698990000001, 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012346048, 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:13.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5300000000000004e-07 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331255+0000", 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8380000000000002e-06 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:13.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331760+0000", 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050452600000000004 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331768+0000", 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337613+0000", 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058448700000000003 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:13.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337836+0000", 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000223043 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.343598+0000", 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057612679999999999 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:13.479 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:13.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:13.481 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:13.475+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:13.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.491+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fefcc1f5fb0 con 0x7fefcc0caf60 2024-04-25T15:52:13.581 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:13.575+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:13.667 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:13.667 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:13.667 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:13.667 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:13.682 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:13.680+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:13.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:13.784 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:13.784+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:13.860 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:13.861 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:13.861 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:13.861 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:13.861 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:13.876 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:13.884 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:13.884+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:13.985 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:13.984+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:13.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.991+0000 7fefb37fe640 1 -- 172.21.15.67:0/2490686034 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v47) ==== 101+0+0 (secure 0 0 0) 0x7fefc0063400 con 0x7fefcc0caf60 2024-04-25T15:52:13.997 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fefb806f4d0 msgr2=0x7fefb80719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fefb806f4d0 0x7fefb80719e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0011f0 tx=0x7fefbc007d40 comp rx=0 tx=0).stop 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc0caf60 msgr2=0x7fefcc15bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc0caf60 0x7fefcc15bce0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fefc000c3c0 tx=0x7fefc000c880 comp rx=0 tx=0).stop 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 shutdown_connections 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fefb806f4d0 0x7fefb80719e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefcc162ec0 0x7fefcc141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:14.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc0caf60 0x7fefcc15bce0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 --2- 172.21.15.67:0/2490686034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefcc073bf0 0x7fefcc1f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 >> 172.21.15.67:0/2490686034 conn(0x7fefcc070db0 msgr2=0x7fefcc15d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 shutdown_connections 2024-04-25T15:52:14.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:13.995+0000 7fefd1709640 1 -- 172.21.15.67:0/2490686034 wait complete. 2024-04-25T15:52:14.015 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.800465965000001, 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:52:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:52:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.80042632, 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:52:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.797519297999997, 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:52:14.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.026 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.797366362000005, 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:52:14.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.030 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.795850717999997, 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:14.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:52:14.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:52:14.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.795708071000007, 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.795604979999993, 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:52:14.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.795452229999995, 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:14.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:14.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:52:14.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.793504003999999, 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:52:14.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.765476372999998, 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:52:14.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.763453482000003, 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:52:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.057 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.763282932999999, 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:14.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:52:14.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:52:14.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.760927437000007, 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:14.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.760892725000005, 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.760871915999999, 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.074 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.683984869, 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:52:14.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:52:14.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.683960787000004, 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:52:14.079 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:52:14.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:52:14.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:14.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:52:14.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.683951089999994, 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:14.084 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.678056153, 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:14.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:52:14.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:52:14.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.091 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.088617495999998, 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:14.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:52:14.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:52:14.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:14.096 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:14.098 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:14.099 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:14.096+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:14.198 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:14.195+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:14.259 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:14.259 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:14.259 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:14.260 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:14.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:14.324 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:14.324+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:14.400 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:14.395+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:14.425 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:14.425 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:14.425 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:14.425 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:14.425 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:14.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:14.501 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:14.495+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:14.601 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:14.595+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:14.628 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.417261370000006, 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:52:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.414646572999999, 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.414147675999999, 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.413058430000007, 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.412437935, 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.410043709000007, 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:52:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 92.380923306, 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 92.376178572000001, 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:52:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 92.372651071000007, 2024-04-25T15:52:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:52:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 92.372518589999999, 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.751579132000003, 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:52:14.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:52:14.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.740144000000001, 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:52:14.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:14.676 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.739746564000001, 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:52:14.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.739636822999998, 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.680 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.739136078000001, 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:52:14.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:52:14.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:52:14.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.738760092000007, 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:52:14.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:52:14.691 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 70.700161909000002, 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 70.700061649999995, 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:14.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:52:14.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 70.697956735000005, 2024-04-25T15:52:14.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:52:14.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:14.702 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 70.697581158999995, 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:14.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.705 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:14.706 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:14.706 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:14.707 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:14.707 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:14.708+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:14.808 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:14.803+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:14.903 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:14.903 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:14.903 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:14.904 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:14.909 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:14.903+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:14.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:15.009 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:15.008+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:15.071 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:15.071 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:15.071 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:15.071 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:15.071 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:15.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:15.131 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:15.132+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:15.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:15.211 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:15.207+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:15.286 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:15.286 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:15.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:11 2.12 2:48cd7578:::60e.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036091111000005, 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062900839999999996, 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247626+0000", 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247629+0000", 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679999999999999e-06 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247650+0000", 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1163999999999999e-05 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247754+0000", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103331 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247772+0000", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247939+0000", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167601 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248238+0000", 2024-04-25T15:52:15.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029904200000000001 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248542+0000", 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030408799999999999 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251351+0000", 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028086159999999999 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253820+0000", 2024-04-25T15:52:15.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024694930000000001 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253869+0000", 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8483000000000003e-05 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253916+0000", 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7620999999999997e-05 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:12 2.12 2:48cd7578:::60e.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.291 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036060550000002, 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006670764, 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247657+0000", 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247658+0000", 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247662+0000", 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4059999999999998e-06 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247829+0000", 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166818 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247842+0000", 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2774e-05 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249197+0000", 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001355387 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249380+0000", 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018307800000000001 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:15.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249695+0000", 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031538099999999999 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253973+0000", 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042777869999999999 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254182+0000", 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020837699999999999 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254262+0000", 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0482000000000006e-05 2024-04-25T15:52:15.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254328+0000", 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5679999999999995e-05 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:13 2.16 2:6f2c300e:::60f.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036052718999997, 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011906903999999999, 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.296 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247665+0000", 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3000000000000001e-07 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247672+0000", 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4529999999999999e-06 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247928+0000", 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025586599999999999 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247943+0000", 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5121e-05 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248076+0000", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133578 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248269+0000", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192308 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248428+0000", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015971599999999999 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250775+0000", 2024-04-25T15:52:15.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023469960000000001 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259432+0000", 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086568540000000003 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259510+0000", 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7967999999999996e-05 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259572+0000", 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1613999999999997e-05 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.300 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:14 2.16 2:6f2c300e:::60f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036043789999994, 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012082196, 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:15.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247674+0000", 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4099999999999999e-07 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:15.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.693e-06 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032722699999999998 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248017+0000", 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2284e-05 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249539+0000", 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015215000000000001 2024-04-25T15:52:15.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249662+0000", 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123363 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249877+0000", 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021425799999999999 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254077+0000", 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042000609999999997 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259640+0000", 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055630929999999999 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259714+0000", 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4750000000000001e-05 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259756+0000", 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1526000000000002e-05 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:25 2.3 2:c112adae:::101.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036037558000004, 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087246289999999994, 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247680+0000", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3599999999999999e-07 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2179e-05 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249535+0000", 2024-04-25T15:52:15.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018422060000000001 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249553+0000", 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8043999999999999e-05 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249685+0000", 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013254400000000001 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249862+0000", 2024-04-25T15:52:15.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176269 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250114+0000", 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025255200000000002 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254289+0000", 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041748009999999997 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256305+0000", 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020161770000000001 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256354+0000", 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8470999999999997e-05 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256405+0000", 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0949999999999998e-05 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:26 2.3 2:c27f454c:::mds1_sessionmap:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.310 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036022148000001, 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089015670000000009, 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247695+0000", 2024-04-25T15:52:15.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247696+0000", 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5700000000000002e-07 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247706+0000", 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0056e-05 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249613+0000", 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019074020000000001 2024-04-25T15:52:15.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249633+0000", 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9190000000000001e-05 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250796+0000", 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116325 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136298 2024-04-25T15:52:15.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251069+0000", 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013733300000000001 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256487+0000", 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054173759999999998 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256506+0000", 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9409e-05 2024-04-25T15:52:15.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256553+0000", 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7134e-05 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256597+0000", 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3562000000000001e-05 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:30 2.12 2:4bb29eec:::201.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036009316000005, 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088320120000000002, 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247708+0000", 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247709+0000", 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1399999999999997e-07 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247711+0000", 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7630000000000001e-06 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249720+0000", 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020086219999999998 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249736+0000", 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6154000000000002e-05 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250351+0000", 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061498499999999997 2024-04-25T15:52:15.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250634+0000", 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028290500000000002 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256540+0000", 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059060689999999999 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:32 2.18 2:1d00e16c:::201.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.319 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 93.036004102000007, 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089443290000000009, 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247713+0000", 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247714+0000", 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9799999999999999e-07 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247716+0000", 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5000000000000002e-06 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249769+0000", 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020523719999999998 2024-04-25T15:52:15.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249779+0000", 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0786e-05 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7941999999999993e-05 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250316+0000", 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045888899999999998 2024-04-25T15:52:15.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256658+0000", 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063414420000000001 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:25 2.3 2:c216c0d1:::mds2_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.950393985000005, 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064886459999999998, 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333324+0000", 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333325+0000", 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5990000000000001e-06 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333332+0000", 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9920000000000001e-06 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333382+0000", 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0179000000000002e-05 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333391+0000", 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0629999999999995e-06 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333453+0000", 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2012000000000002e-05 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333580+0000", 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012628800000000001 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:15.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.333708+0000", 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128665 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.335443+0000", 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001734452 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339712+0000", 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042694209999999998 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339766+0000", 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3915000000000001e-05 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:52.339812+0000", 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6060000000000003e-05 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:10 2.3 2:c04be1b2:::621.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.397863991999998, 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062158869999999998, 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885854+0000", 2024-04-25T15:52:15.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0999999999999999e-07 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885857+0000", 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885945+0000", 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7701999999999995e-05 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885959+0000", 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4283e-05 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887626+0000", 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016669300000000001 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887704+0000", 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7354999999999994e-05 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887826+0000", 2024-04-25T15:52:15.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122631 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890156+0000", 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023302829999999998 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.891936+0000", 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017792000000000001 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892004+0000", 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8311000000000004e-05 2024-04-25T15:52:15.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.892069+0000", 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5412000000000003e-05 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:1 2.17 e20/17 2:eb104683:::203.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 77.396471153999997, 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007093271, 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.333 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887246+0000", 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:15.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.375e-06 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887259+0000", 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1243000000000001e-05 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887274+0000", 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.47e-05 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887286+0000", 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.271e-05 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889237+0000", 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019509340000000001 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889268+0000", 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.078e-05 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894287+0000", 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050192930000000002 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894340+0000", 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2235999999999999e-05 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:7 2.16 2:6b7c230a:::62a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.358451028000005, 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011902502000000001, 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925266+0000", 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925269+0000", 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.227e-06 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925283+0000", 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4493e-05 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925447+0000", 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016378200000000001 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925479+0000", 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1640000000000002e-05 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925592+0000", 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113027 2024-04-25T15:52:15.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925782+0000", 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018986400000000001 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926009+0000", 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022701199999999999 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931318+0000", 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053092959999999998 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936893+0000", 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005575417 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937053+0000", 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015959200000000001 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116152 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:8 2.16 2:6b7c230a:::62a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.358427702, 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012248436, 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925290+0000", 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925291+0000", 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2999999999999999e-07 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925296+0000", 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.904e-06 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925551+0000", 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025534100000000002 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925570+0000", 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8885999999999998e-05 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926948+0000", 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00137795 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927023+0000", 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5689e-05 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928481+0000", 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001457114 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936451+0000", 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079706059999999999 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937283+0000", 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083168900000000004 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937430+0000", 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146988 2024-04-25T15:52:15.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937538+0000", 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108439 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:12 2.2 2:447438e6:::62c.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.358387162, 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011021593, 2024-04-25T15:52:15.347 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925330+0000", 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925331+0000", 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8299999999999996e-07 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925337+0000", 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6339999999999998e-06 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925682+0000", 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034491699999999998 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925699+0000", 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7799999999999999e-05 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926755+0000", 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010558410000000001 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926830+0000", 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4956999999999994e-05 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927099+0000", 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026851700000000002 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931552+0000", 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004453093 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936104+0000", 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004551617 2024-04-25T15:52:15.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936272+0000", 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000168493 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936352+0000", 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9940999999999999e-05 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.352 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:24 2.3 2:c4549ada:::mds4_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.358378501999994, 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011223132, 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925339+0000", 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925340+0000", 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0699999999999997e-07 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925365+0000", 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.499e-05 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925740+0000", 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000374893 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925757+0000", 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7028999999999999e-05 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926158+0000", 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040133399999999999 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926506+0000", 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034792999999999998 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926842+0000", 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033583400000000002 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931093+0000", 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042515060000000004 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936277+0000", 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051840339999999997 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936440+0000", 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162829 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936562+0000", 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012214600000000001 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.357 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:28 2.2 2:4182a243:::204.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.358336828999995, 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011144482000000001, 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925381+0000", 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925382+0000", 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0330000000000001e-06 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925386+0000", 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7099999999999998e-06 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925945+0000", 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055874899999999999 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925962+0000", 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6920000000000001e-05 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928191+0000", 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022284729999999999 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928504+0000", 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031300399999999997 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936525+0000", 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080215930000000005 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.361 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:30 2.18 2:1f0916d5:::204.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.358327571999993, 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011691037, 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:15.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925390+0000", 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925391+0000", 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0599999999999999e-07 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.925396+0000", 2024-04-25T15:52:15.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4099999999999999e-06 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926002+0000", 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060585800000000003 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926019+0000", 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.696e-05 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927121+0000", 2024-04-25T15:52:15.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001101433 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927456+0000", 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033506699999999998 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937081+0000", 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096254029999999994 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:3 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:15.365 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.354190145999993, 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067676780000000001, 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929527+0000", 2024-04-25T15:52:15.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929528+0000", 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.78e-07 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929535+0000", 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3380000000000001e-06 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929545+0000", 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0387999999999999e-05 2024-04-25T15:52:15.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929561+0000", 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5520999999999999e-05 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929862+0000", 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000301527 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929885+0000", 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2861999999999999e-05 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936164+0000", 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062793880000000003 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936295+0000", 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013077600000000001 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:4 2.1c e20/17 2:3d0fce1b:::628.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.354135568999993, 2024-04-25T15:52:15.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068164100000000002, 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929582+0000", 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.01e-07 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929588+0000", 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5110000000000003e-06 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929595+0000", 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2150000000000004e-06 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929601+0000", 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2899999999999999e-06 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930046+0000", 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044498999999999998 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930065+0000", 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9131999999999999e-05 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936302+0000", 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062362549999999996 2024-04-25T15:52:15.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936398+0000", 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6615999999999998e-05 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:23 2.1c e20/17 2:399c8d31:::104.00000000.inode:head v 20'5, mlcod=20'1)", 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.354102361000002, 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069967939999999998, 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:15.373 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929615+0000", 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:15.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929616+0000", 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929620+0000", 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4320000000000001e-06 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929627+0000", 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6390000000000003e-06 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929632+0000", 2024-04-25T15:52:15.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3480000000000003e-06 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930243+0000", 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061054700000000002 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930261+0000", 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8873e-05 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936519+0000", 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062580090000000001 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:15.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936612+0000", 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2572999999999996e-05 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:15.377 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:15.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:15.379 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:15.376+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.339+0000 7faf484f3640 1 Processor -- start 2024-04-25T15:52:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.339+0000 7faf484f3640 1 -- start start 2024-04-25T15:52:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf484f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf4006c640 0x7faf4015b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf484f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf40074780 0x7faf40164f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf484f3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf400c92d0 con 0x7faf40074370 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf484f3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faf40074b90 con 0x7faf40074780 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf484f3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faf4015b560 con 0x7faf4006c640 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf46a90640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf40074780 0x7faf40164f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf46a90640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf40074780 0x7faf40164f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54406/0 (socket says 172.21.15.67:54406) 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf46a90640 1 -- 172.21.15.67:0/3978242714 learned_addr learned my addr 172.21.15.67:0/3978242714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf4628f640 1 --2- 172.21.15.67:0/3978242714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf4006c640 0x7faf4015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf46a90640 1 -- 172.21.15.67:0/3978242714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf4006c640 msgr2=0x7faf4015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf46a90640 1 --2- 172.21.15.67:0/3978242714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf4006c640 0x7faf4015b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf46a90640 1 -- 172.21.15.67:0/3978242714 >> v1:172.21.15.176:6789/0 conn(0x7faf40074370 legacy=0x7faf400c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.343+0000 7faf46a90640 1 -- 172.21.15.67:0/3978242714 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4006e210 con 0x7faf40074780 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf46a90640 1 --2- 172.21.15.67:0/3978242714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf40164f50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faf38009e50 tx=0x7faf3801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd99dc2aca27193a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf4528d640 1 -- 172.21.15.67:0/3978242714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380b4070 con 0x7faf40074780 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf4528d640 1 -- 172.21.15.67:0/3978242714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf380af400 con 0x7faf40074780 2024-04-25T15:52:15.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf4528d640 1 -- 172.21.15.67:0/3978242714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380af7d0 con 0x7faf40074780 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 -- 172.21.15.67:0/3978242714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 msgr2=0x7faf40164f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 --2- 172.21.15.67:0/3978242714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf40164f50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faf38009e50 tx=0x7faf3801f210 comp rx=0 tx=0).stop 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 -- 172.21.15.67:0/3978242714 shutdown_connections 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 --2- 172.21.15.67:0/3978242714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf40164f50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 --2- 172.21.15.67:0/3978242714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf4006c640 0x7faf4015b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 -- 172.21.15.67:0/3978242714 >> 172.21.15.67:0/3978242714 conn(0x7faf40070db0 msgr2=0x7faf400ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 -- 172.21.15.67:0/3978242714 shutdown_connections 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.347+0000 7faf484f3640 1 -- 172.21.15.67:0/3978242714 wait complete. 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.351+0000 7faf484f3640 1 Processor -- start 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.351+0000 7faf484f3640 1 -- start start 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf4006c640 0x7faf401432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf40074370 0x7faf40156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf401423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faf40073890 con 0x7faf40074780 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf4628f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf4006c640 0x7faf401432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faf4006e7b0 con 0x7faf4006c640 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf46a90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf401423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf4628f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf4006c640 0x7faf401432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:43566/0 (socket says 172.21.15.67:43566) 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf4628f640 1 -- 172.21.15.67:0/843480407 learned_addr learned my addr 172.21.15.67:0/843480407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faf40073b70 con 0x7faf40074370 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf45a8e640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf40074370 0x7faf40156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf46a90640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf40074370 msgr2=0x7faf40156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf46a90640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf40074370 0x7faf40156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf46a90640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf4006c640 msgr2=0x7faf401432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf46a90640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf4006c640 0x7faf401432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf46a90640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf40143820 con 0x7faf40074780 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf46a90640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf401423c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faf380ae000 tx=0x7faf3801bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380be030 con 0x7faf40074780 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf380afd70 con 0x7faf40074780 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf40155e80 con 0x7faf40074780 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380b4040 con 0x7faf40074780 2024-04-25T15:52:15.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.355+0000 7faf484f3640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf40149800 con 0x7faf40074780 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7faf1c005280 con 0x7faf40074780 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7faf380c1070 con 0x7faf40074780 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf36ffd640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf2406f400 0x7faf24071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5886+0+0 (secure 0 0 0) 0x7faf38131060 con 0x7faf40074780 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf2406f400 0x7faf24071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v47) ==== 54+0+16 (secure 0 0 0) 0x7faf38018d70 con 0x7faf40074780 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf1c0045f0 0x7faf1c004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf1c005560 con 0x7faf1c0045f0 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf45a8e640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf1c0045f0 0x7faf1c004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf2406f400 0x7faf24071910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf3c007a50 tx=0x7faf3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.359+0000 7faf45a8e640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf1c0045f0 0x7faf1c004a00 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.363+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c005560 con 0x7faf1c0045f0 2024-04-25T15:52:15.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.391+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c0013e0 con 0x7faf1c0045f0 2024-04-25T15:52:15.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.391+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c0013e0 con 0x7faf1c0045f0 2024-04-25T15:52:15.397 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:15.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:15.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.391+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faf1c001ea0 0x7faf1c0029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.391+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7faf1c002330 con 0x7faf1c001ea0 2024-04-25T15:52:15.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.391+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faf1c001ea0 0x7faf1c0029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.395+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faf1c001ea0 0x7faf1c0029a0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.395+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c002330 con 0x7faf1c001ea0 2024-04-25T15:52:15.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.423+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c003210 con 0x7faf1c001ea0 2024-04-25T15:52:15.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.423+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c003210 con 0x7faf1c001ea0 2024-04-25T15:52:15.429 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.423+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faf1c009c90 0x7faf1c00c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.423+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7faf1c00ee80 con 0x7faf1c009c90 2024-04-25T15:52:15.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.423+0000 7faf45a8e640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faf1c009c90 0x7faf1c00c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.423+0000 7faf45a8e640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faf1c009c90 0x7faf1c00c0b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.427+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c00ee80 con 0x7faf1c009c90 2024-04-25T15:52:15.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.451+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c00c8c0 con 0x7faf1c009c90 2024-04-25T15:52:15.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.455+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c00c8c0 con 0x7faf1c009c90 2024-04-25T15:52:15.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:15.461 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:15.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.455+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faf1c00c840 0x7faf1c00cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.455+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7faf1c00d190 con 0x7faf1c00c840 2024-04-25T15:52:15.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.455+0000 7faf46a90640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faf1c00c840 0x7faf1c00cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.455+0000 7faf46a90640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faf1c00c840 0x7faf1c00cc50 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.459+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c00d190 con 0x7faf1c00c840 2024-04-25T15:52:15.479 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:15.475+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.483+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c00d6a0 con 0x7faf1c00c840 2024-04-25T15:52:15.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.487+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c00d6a0 con 0x7faf1c00c840 2024-04-25T15:52:15.491 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:15.492 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:15.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.487+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faf1c00ddc0 0x7faf1c0130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.487+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7faf1c015f00 con 0x7faf1c00ddc0 2024-04-25T15:52:15.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.487+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faf1c00ddc0 0x7faf1c0130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.487+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faf1c00ddc0 0x7faf1c0130b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.491+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c015f00 con 0x7faf1c00ddc0 2024-04-25T15:52:15.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.515+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c013670 con 0x7faf1c00ddc0 2024-04-25T15:52:15.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.519+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c013670 con 0x7faf1c00ddc0 2024-04-25T15:52:15.525 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.519+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faf1c0154d0 0x7faf1c0158e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.519+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7faf1c0192a0 con 0x7faf1c0154d0 2024-04-25T15:52:15.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.523+0000 7faf45a8e640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faf1c0154d0 0x7faf1c0158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.523+0000 7faf45a8e640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faf1c0154d0 0x7faf1c0158e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.523+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c0192a0 con 0x7faf1c0154d0 2024-04-25T15:52:15.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.551+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c018120 con 0x7faf1c0154d0 2024-04-25T15:52:15.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.555+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c018120 con 0x7faf1c0154d0 2024-04-25T15:52:15.560 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:15.560 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:15.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.555+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faf1c0187a0 0x7faf1c018bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.555+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7faf1c01c610 con 0x7faf1c0187a0 2024-04-25T15:52:15.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.555+0000 7faf46a90640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faf1c0187a0 0x7faf1c018bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.555+0000 7faf46a90640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faf1c0187a0 0x7faf1c018bb0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.559+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c01c610 con 0x7faf1c0187a0 2024-04-25T15:52:15.580 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:15.575+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.583+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c013820 con 0x7faf1c0187a0 2024-04-25T15:52:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.587+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c013820 con 0x7faf1c0187a0 2024-04-25T15:52:15.593 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.591+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faf1c013f40 0x7faf1c014350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.591+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7faf1c014b00 con 0x7faf1c013f40 2024-04-25T15:52:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.591+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faf1c013f40 0x7faf1c014350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.591+0000 7faf4628f640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faf1c013f40 0x7faf1c014350 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.591+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf1c014b00 con 0x7faf1c013f40 2024-04-25T15:52:15.597 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:15.597 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:15.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:15.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.619+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faf1c014e10 con 0x7faf1c013f40 2024-04-25T15:52:15.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf36ffd640 1 -- 172.21.15.67:0/843480407 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faf1c014e10 con 0x7faf1c013f40 2024-04-25T15:52:15.627 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:15.628 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf1c0045f0 msgr2=0x7faf1c004a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf1c0045f0 0x7faf1c004a00 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faf1c001ea0 msgr2=0x7faf1c0029a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faf1c001ea0 0x7faf1c0029a0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faf1c009c90 msgr2=0x7faf1c00c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faf1c009c90 0x7faf1c00c0b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faf1c00c840 msgr2=0x7faf1c00cc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faf1c00c840 0x7faf1c00cc50 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faf1c00ddc0 msgr2=0x7faf1c0130b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faf1c00ddc0 0x7faf1c0130b0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faf1c0154d0 msgr2=0x7faf1c0158e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faf1c0154d0 0x7faf1c0158e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faf1c0187a0 msgr2=0x7faf1c018bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faf1c0187a0 0x7faf1c018bb0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf46a90640 1 -- 172.21.15.67:0/843480407 reap_dead start 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faf1c013f40 msgr2=0x7faf1c014350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faf1c013f40 0x7faf1c014350 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf46a90640 1 -- 172.21.15.67:0/843480407 reap_dead start 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf46a90640 1 -- 172.21.15.67:0/843480407 reap_dead start 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf46a90640 1 -- 172.21.15.67:0/843480407 reap_dead start 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf2406f400 msgr2=0x7faf24071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf2406f400 0x7faf24071910 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf3c007a50 tx=0x7faf3c008040 comp rx=0 tx=0).stop 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 msgr2=0x7faf401423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf401423c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faf380ae000 tx=0x7faf3801bfb0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 shutdown_connections 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf2406f400 0x7faf24071910 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 --2- 172.21.15.67:0/843480407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf40074780 0x7faf401423c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 >> 172.21.15.67:0/843480407 conn(0x7faf40070db0 msgr2=0x7faf400cdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 shutdown_connections 2024-04-25T15:52:15.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:15.623+0000 7faf34ff9640 1 -- 172.21.15.67:0/843480407 wait complete. 2024-04-25T15:52:15.681 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:15.680+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:15.768 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:15.768 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:15.782 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:15.780+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:15.785 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:15.882 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:15.880+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:15.977 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:15.978 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:15.983 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:15.979+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:15.995 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:16.001 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:16.001 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:16.001 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-25T15:52:16.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-25T15:52:16.084 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:16.079+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:16.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:52:16.185 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:16.183+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:16.191 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:16.191 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.195+0000 7f821d175640 1 Processor -- start 2024-04-25T15:52:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.195+0000 7f821d175640 1 -- start start 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821d175640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82181609d0 0x7f821815c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821d175640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8218071b70 0x7f8218071f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821d175640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821815d1f0 con 0x7f82180caff0 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821d175640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8218160540 con 0x7f82181609d0 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821d175640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f82180cb400 con 0x7f8218071b70 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f8216d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82180caff0 0x7f82180ca5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46886/0 (socket says 172.21.15.67:46886) 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f8216d9d640 1 -- 172.21.15.67:0/1371054541 learned_addr learned my addr 172.21.15.67:0/1371054541 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:16.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821659c640 1 --2- 172.21.15.67:0/1371054541 >> v2:172.21.15.67:3300/0 conn(0x7f82181609d0 0x7f821815c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821759e640 1 --2- 172.21.15.67:0/1371054541 >> v2:172.21.15.176:3300/0 conn(0x7f8218071b70 0x7f8218071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821659c640 1 -- 172.21.15.67:0/1371054541 >> v2:172.21.15.176:3300/0 conn(0x7f8218071b70 msgr2=0x7f8218071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821659c640 1 --2- 172.21.15.67:0/1371054541 >> v2:172.21.15.176:3300/0 conn(0x7f8218071b70 0x7f8218071f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821659c640 1 -- 172.21.15.67:0/1371054541 >> v1:172.21.15.67:6789/0 conn(0x7f82180caff0 legacy=0x7f82180ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f8215d9b640 1 -- 172.21.15.67:0/1371054541 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2623736321 0 0) 0x7f821815d1f0 con 0x7f82180caff0 2024-04-25T15:52:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.199+0000 7f821659c640 1 -- 172.21.15.67:0/1371054541 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821815d4d0 con 0x7f82181609d0 2024-04-25T15:52:16.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.203+0000 7f821659c640 1 --2- 172.21.15.67:0/1371054541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82181609d0 0x7f821815c880 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8218157160 tx=0x7f820c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b95935ae3a7cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.203+0000 7f8215d9b640 1 -- 172.21.15.67:0/1371054541 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c0b1070 con 0x7f82181609d0 2024-04-25T15:52:16.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.203+0000 7f8215d9b640 1 -- 172.21.15.67:0/1371054541 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f820c0ac5f0 con 0x7f82181609d0 2024-04-25T15:52:16.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.203+0000 7f8215d9b640 1 -- 172.21.15.67:0/1371054541 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c0ac9c0 con 0x7f82181609d0 2024-04-25T15:52:16.209 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.203+0000 7f821d175640 1 -- 172.21.15.67:0/1371054541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82181609d0 msgr2=0x7f821815c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.203+0000 7f821d175640 1 --2- 172.21.15.67:0/1371054541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82181609d0 0x7f821815c880 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8218157160 tx=0x7f820c01c480 comp rx=0 tx=0).stop 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 -- 172.21.15.67:0/1371054541 shutdown_connections 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 --2- 172.21.15.67:0/1371054541 >> v2:172.21.15.176:3300/0 conn(0x7f8218071b70 0x7f8218071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 --2- 172.21.15.67:0/1371054541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82181609d0 0x7f821815c880 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 -- 172.21.15.67:0/1371054541 >> 172.21.15.67:0/1371054541 conn(0x7f82180700b0 msgr2=0x7f82180704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 -- 172.21.15.67:0/1371054541 shutdown_connections 2024-04-25T15:52:16.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 -- 172.21.15.67:0/1371054541 wait complete. 2024-04-25T15:52:16.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 Processor -- start 2024-04-25T15:52:16.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.207+0000 7f821d175640 1 -- start start 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8218071b70 0x7f8218159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82180caff0 0x7f8218155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82181609d0 0x7f8218155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f821815cdc0 con 0x7f8218071b70 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8218160de0 con 0x7f82181609d0 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f8216d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8218071b70 0x7f8218159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f821815cf40 con 0x7f82180caff0 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f8216d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8218071b70 0x7f8218159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54442/0 (socket says 172.21.15.67:54442) 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f8216d9d640 1 -- 172.21.15.67:0/472504971 learned_addr learned my addr 172.21.15.67:0/472504971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:16.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821659c640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82180caff0 0x7f8218155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821759e640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82181609d0 0x7f8218155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821759e640 1 -- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82180caff0 msgr2=0x7f8218155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821759e640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82180caff0 0x7f8218155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821759e640 1 -- 172.21.15.67:0/472504971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8218071b70 msgr2=0x7f8218159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821759e640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8218071b70 0x7f8218159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821759e640 1 -- 172.21.15.67:0/472504971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82181564e0 con 0x7f82181609d0 2024-04-25T15:52:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821759e640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82181609d0 0x7f8218155fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8210007cf0 tx=0x7f821000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821000edd0 con 0x7f82181609d0 2024-04-25T15:52:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82100039a0 con 0x7f82181609d0 2024-04-25T15:52:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821815cb90 con 0x7f82181609d0 2024-04-25T15:52:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8218157fd0 con 0x7f82181609d0 2024-04-25T15:52:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.211+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210003d90 con 0x7f82181609d0 2024-04-25T15:52:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f821806a480 con 0x7f82181609d0 2024-04-25T15:52:16.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f821001b020 con 0x7f82181609d0 2024-04-25T15:52:16.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f81ff7fe640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f820406f400 0x7f8204071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5886+0+0 (secure 0 0 0) 0x7f8210090170 con 0x7f82181609d0 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f81ff7fe640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f82040778b0 0x7f8204079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f8216d9d640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f820406f400 0x7f8204071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f820407a470 con 0x7f82040778b0 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f8210090520 con 0x7f82181609d0 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f821659c640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f82040778b0 0x7f8204079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f8216d9d640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f820406f400 0x7f8204071910 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8208000f70 tx=0x7f8208007eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.215+0000 7f821659c640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f82040778b0 0x7f8204079da0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.219+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f820407a470 con 0x7f82040778b0 2024-04-25T15:52:16.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f8218166fc0 con 0x7f82040778b0 2024-04-25T15:52:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f81ff7fe640 1 -- 172.21.15.67:0/472504971 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f8218166fc0 con 0x7f82040778b0 2024-04-25T15:52:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f82040778b0 msgr2=0x7f8204079da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f82040778b0 0x7f8204079da0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f820406f400 msgr2=0x7f8204071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f820406f400 0x7f8204071910 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8208000f70 tx=0x7f8208007eb0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82181609d0 msgr2=0x7f8218155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 --2- 172.21.15.67:0/472504971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82181609d0 0x7f8218155fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8210007cf0 tx=0x7f821000e4b0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821759e640 1 -- 172.21.15.67:0/472504971 reap_dead start 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 shutdown_connections 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 >> 172.21.15.67:0/472504971 conn(0x7f82180700b0 msgr2=0x7f82180ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 shutdown_connections 2024-04-25T15:52:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.251+0000 7f821d175640 1 -- 172.21.15.67:0/472504971 wait complete. 2024-04-25T15:52:16.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-25T15:52:16.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd09065640 1 Processor -- start 2024-04-25T15:52:16.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd09065640 1 -- start start 2024-04-25T15:52:16.299 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:16.295+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd09065640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd0415b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd09065640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd040c92d0 con 0x7fdd04074370 2024-04-25T15:52:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd09065640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd0415b560 con 0x7fdd04074780 2024-04-25T15:52:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd09065640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd04074b90 con 0x7fdd0406c640 2024-04-25T15:52:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd0259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdd04074370 0x7fdd040c7fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46900/0 (socket says 172.21.15.67:46900) 2024-04-25T15:52:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd0259c640 1 -- 172.21.15.67:0/1325151092 learned_addr learned my addr 172.21.15.67:0/1325151092 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:16.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd02d9d640 1 --2- 172.21.15.67:0/1325151092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd0415b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd01d9b640 1 -- 172.21.15.67:0/1325151092 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983240635 0 0) 0x7fdd040c92d0 con 0x7fdd04074370 2024-04-25T15:52:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd01d9b640 1 -- 172.21.15.67:0/1325151092 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcec003530 con 0x7fdd04074370 2024-04-25T15:52:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd02d9d640 1 -- 172.21.15.67:0/1325151092 >> v1:172.21.15.176:6789/0 conn(0x7fdd04074780 legacy=0x7fdd04164f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd02d9d640 1 -- 172.21.15.67:0/1325151092 >> v1:172.21.15.67:6789/0 conn(0x7fdd04074370 legacy=0x7fdd040c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.295+0000 7fdd02d9d640 1 -- 172.21.15.67:0/1325151092 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0406e210 con 0x7fdd0406c640 2024-04-25T15:52:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.299+0000 7fdd02d9d640 1 --2- 172.21.15.67:0/1325151092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd0415b020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40097f0 tx=0x7fdcf401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae39e51bfcfc4dff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.299+0000 7fdd01d9b640 1 -- 172.21.15.67:0/1325151092 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf40b5070 con 0x7fdd0406c640 2024-04-25T15:52:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.299+0000 7fdd01d9b640 1 -- 172.21.15.67:0/1325151092 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf40a89d0 con 0x7fdd0406c640 2024-04-25T15:52:16.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.299+0000 7fdd01d9b640 1 -- 172.21.15.67:0/1325151092 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf40a8da0 con 0x7fdd0406c640 2024-04-25T15:52:16.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 -- 172.21.15.67:0/1325151092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 msgr2=0x7fdd0415b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 --2- 172.21.15.67:0/1325151092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd0415b020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40097f0 tx=0x7fdcf401ecd0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 -- 172.21.15.67:0/1325151092 shutdown_connections 2024-04-25T15:52:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 --2- 172.21.15.67:0/1325151092 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd0415b020 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 -- 172.21.15.67:0/1325151092 >> 172.21.15.67:0/1325151092 conn(0x7fdd04070db0 msgr2=0x7fdd040ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 -- 172.21.15.67:0/1325151092 shutdown_connections 2024-04-25T15:52:16.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 -- 172.21.15.67:0/1325151092 wait complete. 2024-04-25T15:52:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 Processor -- start 2024-04-25T15:52:16.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.303+0000 7fdd09065640 1 -- start start 2024-04-25T15:52:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd09065640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd04157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd09065640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd04074370 0x7fdd041505e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd04074370 0x7fdd041505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd04074370 0x7fdd041505e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54454/0 (socket says 172.21.15.67:54454) 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0259c640 1 -- 172.21.15.67:0/154951253 learned_addr learned my addr 172.21.15.67:0/154951253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd09065640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd04074780 0x7fdd040c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd02d9d640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd04157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd04073890 con 0x7fdd04074370 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0406e7b0 con 0x7fdd04074780 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd04073b70 con 0x7fdd0406c640 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0359e640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd04074780 0x7fdd040c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0359e640 1 -- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 msgr2=0x7fdd04157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0359e640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd04157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0359e640 1 -- 172.21.15.67:0/154951253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd04074370 msgr2=0x7fdd041505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0359e640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd04074370 0x7fdd041505e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0359e640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd041579c0 con 0x7fdd04074780 2024-04-25T15:52:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0259c640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd04074370 0x7fdd041505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:16.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd02d9d640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd04157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdd0359e640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd04074780 0x7fdd040c7150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00c730 tx=0x7fdcfc00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdceb7fe640 1 -- 172.21.15.67:0/154951253 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc016020 con 0x7fdd04074780 2024-04-25T15:52:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdceb7fe640 1 -- 172.21.15.67:0/154951253 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcfc006d60 con 0x7fdd04074780 2024-04-25T15:52:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.307+0000 7fdceb7fe640 1 -- 172.21.15.67:0/154951253 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc003bc0 con 0x7fdd04074780 2024-04-25T15:52:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.311+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd040c5040 con 0x7fdd04074780 2024-04-25T15:52:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.311+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd041580c0 con 0x7fdd04074780 2024-04-25T15:52:16.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.311+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd040bb850 con 0x7fdd04074780 2024-04-25T15:52:16.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.311+0000 7fdceb7fe640 1 -- 172.21.15.67:0/154951253 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdcfc0050d0 con 0x7fdd04074780 2024-04-25T15:52:16.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.311+0000 7fdceb7fe640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcf006f400 0x7fdcf0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.311+0000 7fdceb7fe640 1 -- 172.21.15.67:0/154951253 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5886+0+0 (secure 0 0 0) 0x7fdcfc08f040 con 0x7fdd04074780 2024-04-25T15:52:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.315+0000 7fdd02d9d640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcf006f400 0x7fdcf0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.315+0000 7fdd02d9d640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcf006f400 0x7fdcf0071910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40b2000 tx=0x7fdcf4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.319+0000 7fdceb7fe640 1 -- 172.21.15.67:0/154951253 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdcfc020020 con 0x7fdd04074780 2024-04-25T15:52:16.387 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:16.383+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:16.400 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:16.400 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:16.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.475+0000 7fe935eea640 1 Processor -- start 2024-04-25T15:52:16.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.475+0000 7fe935eea640 1 -- start start 2024-04-25T15:52:16.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.475+0000 7fe935eea640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe930156330 0x7fe93006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9300c5100 0x7fe930163630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46904/0 (socket says 172.21.15.67:46904) 2024-04-25T15:52:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92f7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe930156330 0x7fe93006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92ffff640 1 -- 172.21.15.67:0/814888436 learned_addr learned my addr 172.21.15.67:0/814888436 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:16.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe93006fa10 con 0x7fe9300c5100 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9300ca8f0 con 0x7fe93006cb80 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2626130461 0 0) 0x7fe93006fa10 con 0x7fe9300c5100 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe9300ca6c0 con 0x7fe930156330 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe918004640 con 0x7fe9300c5100 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 126093790 0 0) 0x7fe9300ca8f0 con 0x7fe93006cb80 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe93006fa10 con 0x7fe93006cb80 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 898368212 0 0) 0x7fe918004640 con 0x7fe9300c5100 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92f7fe640 1 -- 172.21.15.67:0/814888436 >> v1:172.21.15.176:6789/0 conn(0x7fe93006cb80 legacy=0x7fe9300c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92f7fe640 1 -- 172.21.15.67:0/814888436 >> v1:172.21.15.67:6789/0 conn(0x7fe9300c5100 legacy=0x7fe930163630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.479+0000 7fe92f7fe640 1 -- 172.21.15.67:0/814888436 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9300c3aa0 con 0x7fe930156330 2024-04-25T15:52:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe92f7fe640 1 --2- 172.21.15.67:0/814888436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe930156330 0x7fe93006c640 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe9200097c0 tx=0x7fe92001ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3448c6c65aee1f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200b5070 con 0x7fe930156330 2024-04-25T15:52:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9200a8630 con 0x7fe930156330 2024-04-25T15:52:16.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe92e7fc640 1 -- 172.21.15.67:0/814888436 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928001a00 con 0x7fe930156330 2024-04-25T15:52:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe930156330 msgr2=0x7fe93006c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe935eea640 1 --2- 172.21.15.67:0/814888436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe930156330 0x7fe93006c640 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe9200097c0 tx=0x7fe92001ebb0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 shutdown_connections 2024-04-25T15:52:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe935eea640 1 --2- 172.21.15.67:0/814888436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe930156330 0x7fe93006c640 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 >> 172.21.15.67:0/814888436 conn(0x7fe9300ce1c0 msgr2=0x7fe9300ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 shutdown_connections 2024-04-25T15:52:16.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.483+0000 7fe935eea640 1 -- 172.21.15.67:0/814888436 wait complete. 2024-04-25T15:52:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 Processor -- start 2024-04-25T15:52:16.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 -- start start 2024-04-25T15:52:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe93006cb80 0x7fe930158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9300c5100 0x7fe9300c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe930156330 0x7fe930150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe92f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe93006cb80 0x7fe930158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe92f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe93006cb80 0x7fe930158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:58642/0 (socket says 172.21.15.67:58642) 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe92effd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9300c5100 0x7fe9300c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe92effd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9300c5100 0x7fe9300c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:43616/0 (socket says 172.21.15.67:43616) 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe930156330 0x7fe930150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe930156330 0x7fe930150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54470/0 (socket says 172.21.15.67:54470) 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe92f7fe640 1 -- 172.21.15.67:0/1033329183 learned_addr learned my addr 172.21.15.67:0/1033329183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9300c43d0 con 0x7fe930156330 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe9300c4060 con 0x7fe9300c5100 2024-04-25T15:52:16.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.487+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe930156740 con 0x7fe93006cb80 2024-04-25T15:52:16.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe92effd640 1 -- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe93006cb80 msgr2=0x7fe930158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe92effd640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe93006cb80 0x7fe930158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe92effd640 1 -- 172.21.15.67:0/1033329183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe930156330 msgr2=0x7fe930150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe92effd640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe930156330 0x7fe930150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe92effd640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe930154cb0 con 0x7fe9300c5100 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe92f7fe640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe93006cb80 0x7fe930158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe92effd640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9300c5100 0x7fe9300c7df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe924007fc0 tx=0x7fe92400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924015e30 con 0x7fe9300c5100 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9240050d0 con 0x7fe9300c5100 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9300c5570 con 0x7fe9300c5100 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe924003600 con 0x7fe9300c5100 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe930157980 con 0x7fe9300c5100 2024-04-25T15:52:16.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.491+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe930158600 con 0x7fe9300c5100 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe9240037a0 con 0x7fe9300c5100 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe917fff640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe91c06f450 0x7fe91c071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5886+0+0 (secure 0 0 0) 0x7fe924096ca0 con 0x7fe9300c5100 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe92f7fe640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe91c06f450 0x7fe91c071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe917fff640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe91c077980 0x7fe91c079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe91c07a540 con 0x7fe91c077980 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fe924097180 con 0x7fe9300c5100 2024-04-25T15:52:16.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe92ffff640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe91c077980 0x7fe91c079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe92f7fe640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe91c06f450 0x7fe91c071960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe92001c7e0 tx=0x7fe920008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe92ffff640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe91c077980 0x7fe91c079e70 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.495+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe91c07a540 con 0x7fe91c077980 2024-04-25T15:52:16.507 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:16.507+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.527+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fe9301eedb0 con 0x7fe91c077980 2024-04-25T15:52:16.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.527+0000 7fe917fff640 1 -- 172.21.15.67:0/1033329183 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fe9301eedb0 con 0x7fe91c077980 2024-04-25T15:52:16.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe91c077980 msgr2=0x7fe91c079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe91c077980 0x7fe91c079e70 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe91c06f450 msgr2=0x7fe91c071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe91c06f450 0x7fe91c071960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe92001c7e0 tx=0x7fe920008040 comp rx=0 tx=0).stop 2024-04-25T15:52:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9300c5100 msgr2=0x7fe9300c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 --2- 172.21.15.67:0/1033329183 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9300c5100 0x7fe9300c7df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe924007fc0 tx=0x7fe92400c800 comp rx=0 tx=0).stop 2024-04-25T15:52:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe92ffff640 1 -- 172.21.15.67:0/1033329183 reap_dead start 2024-04-25T15:52:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 shutdown_connections 2024-04-25T15:52:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 >> 172.21.15.67:0/1033329183 conn(0x7fe9300ce1c0 msgr2=0x7fe9300c88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 shutdown_connections 2024-04-25T15:52:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.531+0000 7fe935eea640 1 -- 172.21.15.67:0/1033329183 wait complete. 2024-04-25T15:52:16.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-25T15:52:16.602 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:16.599+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:16.609 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:16.609 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:16.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:16.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.659+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fdd040c4dd0 con 0x7fdd04074780 2024-04-25T15:52:16.688 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:16.683+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.795+0000 7f21b7b6d640 1 Processor -- start 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.795+0000 7f21b7b6d640 1 -- start start 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b7b6d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21b00caf60 0x7f21b015d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b7b6d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b015d790 0x7f21b006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b7b6d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f21b0073bf0 0x7f21b006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b7b6d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f21b00cab60 con 0x7f21b00caf60 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b7b6d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f21b0159620 con 0x7f21b0073bf0 2024-04-25T15:52:16.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b7b6d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f21b00c77a0 con 0x7f21b015d790 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21b00caf60 0x7f21b015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21b00caf60 0x7f21b015d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54474/0 (socket says 172.21.15.67:54474) 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 -- 172.21.15.67:0/2882286711 learned_addr learned my addr 172.21.15.67:0/2882286711 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b610a640 1 --2- 172.21.15.67:0/2882286711 >> v2:172.21.15.176:3300/0 conn(0x7f21b0073bf0 0x7f21b006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5108640 1 --2- 172.21.15.67:0/2882286711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b015d790 0x7f21b006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 -- 172.21.15.67:0/2882286711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b015d790 msgr2=0x7f21b006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 --2- 172.21.15.67:0/2882286711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b015d790 0x7f21b006d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 -- 172.21.15.67:0/2882286711 >> v2:172.21.15.176:3300/0 conn(0x7f21b0073bf0 msgr2=0x7f21b006dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 --2- 172.21.15.67:0/2882286711 >> v2:172.21.15.176:3300/0 conn(0x7f21b0073bf0 0x7f21b006dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b5909640 1 -- 172.21.15.67:0/2882286711 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b00c7a60 con 0x7f21b00caf60 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.799+0000 7f21b610a640 1 --2- 172.21.15.67:0/2882286711 >> v2:172.21.15.176:3300/0 conn(0x7f21b0073bf0 0x7f21b006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b5108640 1 --2- 172.21.15.67:0/2882286711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b015d790 0x7f21b006d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b5909640 1 --2- 172.21.15.67:0/2882286711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b015d250 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f21b006fd00 tx=0x7f21ac01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a37dfe5ebcb580 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b4907640 1 -- 172.21.15.67:0/2882286711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac0b0070 con 0x7f21b00caf60 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b4907640 1 -- 172.21.15.67:0/2882286711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21ac0ab600 con 0x7f21b00caf60 2024-04-25T15:52:16.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b4907640 1 -- 172.21.15.67:0/2882286711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac0ab9d0 con 0x7f21b00caf60 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 -- 172.21.15.67:0/2882286711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 msgr2=0x7f21b015d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 --2- 172.21.15.67:0/2882286711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b015d250 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f21b006fd00 tx=0x7f21ac01c540 comp rx=0 tx=0).stop 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 -- 172.21.15.67:0/2882286711 shutdown_connections 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 --2- 172.21.15.67:0/2882286711 >> v2:172.21.15.176:3300/0 conn(0x7f21b0073bf0 0x7f21b006dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 --2- 172.21.15.67:0/2882286711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b015d790 0x7f21b006d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 --2- 172.21.15.67:0/2882286711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b015d250 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 -- 172.21.15.67:0/2882286711 >> 172.21.15.67:0/2882286711 conn(0x7f21b0070db0 msgr2=0x7f21b00ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 -- 172.21.15.67:0/2882286711 shutdown_connections 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.803+0000 7f21b7b6d640 1 -- 172.21.15.67:0/2882286711 wait complete. 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 Processor -- start 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 -- start start 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b0073bf0 0x7f21b01f9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b0153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21b015d790 0x7f21b014e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0165ed0 con 0x7f21b00caf60 2024-04-25T15:52:16.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f21b015a600 con 0x7f21b015d790 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b5108640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b0153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b7b6d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f21b00733d0 con 0x7f21b0073bf0 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b5108640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b0153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54484/0 (socket says 172.21.15.67:54484) 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b5108640 1 -- 172.21.15.67:0/4082788036 learned_addr learned my addr 172.21.15.67:0/4082788036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b5909640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b0073bf0 0x7f21b01f9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.807+0000 7f21b610a640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21b015d790 0x7f21b014e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b5108640 1 -- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b0073bf0 msgr2=0x7f21b01f9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b5108640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f21b0073bf0 0x7f21b01f9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b5108640 1 -- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21b015d790 msgr2=0x7f21b014e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b5108640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21b015d790 0x7f21b014e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b5108640 1 -- 172.21.15.67:0/4082788036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b01f5aa0 con 0x7f21b00caf60 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b5108640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b0153490 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f21a000c820 tx=0x7f21a000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0015070 con 0x7f21b00caf60 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a0004220 con 0x7f21b00caf60 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b00ce580 con 0x7f21b00caf60 2024-04-25T15:52:16.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b01f6040 con 0x7f21b00caf60 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a000ce10 con 0x7f21b00caf60 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f21b006a480 con 0x7f21b00caf60 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.811+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f21a0007830 con 0x7f21b00caf60 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21a67fc640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f219406f500 0x7f2194071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5886+0+0 (secure 0 0 0) 0x7f21a008f610 con 0x7f21b00caf60 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21a67fc640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2194077a30 0x7f2194079f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21a0014e50 con 0x7f2194077a30 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f21a008fa00 con 0x7f21b00caf60 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21b610a640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2194077a30 0x7f2194079f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21b5909640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f219406f500 0x7f2194071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:16.826 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:16.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21b610a640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2194077a30 0x7f2194079f20 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21b5909640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f219406f500 0x7f2194071a10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0ad000 tx=0x7f21ac0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:16.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.815+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f21a0014e50 con 0x7f2194077a30 2024-04-25T15:52:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f21b00c8a40 con 0x7f2194077a30 2024-04-25T15:52:16.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21a67fc640 1 -- 172.21.15.67:0/4082788036 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f21b00c8a40 con 0x7f2194077a30 2024-04-25T15:52:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2194077a30 msgr2=0x7f2194079f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2194077a30 0x7f2194079f20 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f219406f500 msgr2=0x7f2194071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f219406f500 0x7f2194071a10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f21ac0ad000 tx=0x7f21ac0a6000 comp rx=0 tx=0).stop 2024-04-25T15:52:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 msgr2=0x7f21b0153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 --2- 172.21.15.67:0/4082788036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21b00caf60 0x7f21b0153490 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f21a000c820 tx=0x7f21a000cce0 comp rx=0 tx=0).stop 2024-04-25T15:52:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b610a640 1 -- 172.21.15.67:0/4082788036 reap_dead start 2024-04-25T15:52:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 shutdown_connections 2024-04-25T15:52:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 >> 172.21.15.67:0/4082788036 conn(0x7f21b0070db0 msgr2=0x7f21b0158d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 shutdown_connections 2024-04-25T15:52:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:16.847+0000 7f21b7b6d640 1 -- 172.21.15.67:0/4082788036 wait complete. 2024-04-25T15:52:16.857 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:16.858 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:16.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-25T15:52:16.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:16.927 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:16.923+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:17.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.011+0000 7fdceb7fe640 1 -- 172.21.15.67:0/154951253 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v48) ==== 87+0+0 (secure 0 0 0) 0x7fdcfc05b9b0 con 0x7fdd04074780 2024-04-25T15:52:17.018 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:52:17.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcf006f400 msgr2=0x7fdcf0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcf006f400 0x7fdcf0071910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40b2000 tx=0x7fdcf4008040 comp rx=0 tx=0).stop 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd04074780 msgr2=0x7fdd040c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd04074780 0x7fdd040c7150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00c730 tx=0x7fdcfc00cbf0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 shutdown_connections 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcf006f400 0x7fdcf0071910 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd04074780 0x7fdd040c7150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd04074370 0x7fdd041505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 --2- 172.21.15.67:0/154951253 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd0406c640 0x7fdd04157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 >> 172.21.15.67:0/154951253 conn(0x7fdd04070db0 msgr2=0x7fdd04073110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 shutdown_connections 2024-04-25T15:52:17.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.015+0000 7fdd09065640 1 -- 172.21.15.67:0/154951253 wait complete. 2024-04-25T15:52:17.021 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:17.019+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:17.122 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:17.119+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:17.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.119+0000 7f1373156640 1 Processor -- start 2024-04-25T15:52:17.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.119+0000 7f1373156640 1 -- start start 2024-04-25T15:52:17.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1373156640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f136c160800 0x7f136c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1373156640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f136c0c96e0 0x7f136c167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1373156640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136c06d8e0 con 0x7f136c0c92d0 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1373156640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f136c06dbc0 con 0x7f136c160800 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1373156640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f136c073c70 con 0x7f136c0c96e0 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1370ef2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f136c160800 0x7f136c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1370ef2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f136c160800 0x7f136c06d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54488/0 (socket says 172.21.15.67:54488) 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1370ef2640 1 -- 172.21.15.67:0/31788928 learned_addr learned my addr 172.21.15.67:0/31788928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:17.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f13716f3640 1 --2- 172.21.15.67:0/31788928 >> v2:172.21.15.176:3300/0 conn(0x7f136c0c96e0 0x7f136c167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1370ef2640 1 -- 172.21.15.67:0/31788928 >> v2:172.21.15.176:3300/0 conn(0x7f136c0c96e0 msgr2=0x7f136c167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1370ef2640 1 --2- 172.21.15.67:0/31788928 >> v2:172.21.15.176:3300/0 conn(0x7f136c0c96e0 0x7f136c167180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1370ef2640 1 -- 172.21.15.67:0/31788928 >> v1:172.21.15.176:6789/0 conn(0x7f136c0c92d0 legacy=0x7f136c072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:17.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.123+0000 7f1370ef2640 1 -- 172.21.15.67:0/31788928 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c073f50 con 0x7f136c160800 2024-04-25T15:52:17.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1370ef2640 1 --2- 172.21.15.67:0/31788928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c160800 0x7f136c06d030 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f136c06dd40 tx=0x7f136001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec5adbb6d0c8219 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f136b7fe640 1 -- 172.21.15.67:0/31788928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13600b0070 con 0x7f136c160800 2024-04-25T15:52:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f136b7fe640 1 -- 172.21.15.67:0/31788928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13600ab600 con 0x7f136c160800 2024-04-25T15:52:17.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f136b7fe640 1 -- 172.21.15.67:0/31788928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13600ab9d0 con 0x7f136c160800 2024-04-25T15:52:17.131 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:17.131 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 -- 172.21.15.67:0/31788928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c160800 msgr2=0x7f136c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 --2- 172.21.15.67:0/31788928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c160800 0x7f136c06d030 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f136c06dd40 tx=0x7f136001c450 comp rx=0 tx=0).stop 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 -- 172.21.15.67:0/31788928 shutdown_connections 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 --2- 172.21.15.67:0/31788928 >> v2:172.21.15.176:3300/0 conn(0x7f136c0c96e0 0x7f136c167180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 --2- 172.21.15.67:0/31788928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c160800 0x7f136c06d030 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 -- 172.21.15.67:0/31788928 >> 172.21.15.67:0/31788928 conn(0x7f136c070db0 msgr2=0x7f136c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 -- 172.21.15.67:0/31788928 shutdown_connections 2024-04-25T15:52:17.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.127+0000 7f1373156640 1 -- 172.21.15.67:0/31788928 wait complete. 2024-04-25T15:52:17.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 Processor -- start 2024-04-25T15:52:17.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 -- start start 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f136c0c92d0 0x7f136c0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c0c96e0 0x7f136c0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f136c160800 0x7f136c14ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f136c160c10 con 0x7f136c0c96e0 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f136c06d570 con 0x7f136c160800 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f1373156640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f136c160d90 con 0x7f136c0c92d0 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f136bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c0c96e0 0x7f136c0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f136bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c0c96e0 0x7f136c0c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54502/0 (socket says 172.21.15.67:54502) 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.131+0000 7f136bfff640 1 -- 172.21.15.67:0/1395956242 learned_addr learned my addr 172.21.15.67:0/1395956242 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f1370ef2640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f136c0c92d0 0x7f136c0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f13716f3640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f136c160800 0x7f136c14ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f136bfff640 1 -- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f136c0c92d0 msgr2=0x7f136c0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f136bfff640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f136c0c92d0 0x7f136c0ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f136bfff640 1 -- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f136c160800 msgr2=0x7f136c14ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f136bfff640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f136c160800 0x7f136c14ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f136bfff640 1 -- 172.21.15.67:0/1395956242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c0cd0c0 con 0x7f136c0c96e0 2024-04-25T15:52:17.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f136bfff640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c0c96e0 0x7f136c0c70d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f135800b470 tx=0x7f135800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358015030 con 0x7f136c0c96e0 2024-04-25T15:52:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1358004020 con 0x7f136c0c96e0 2024-04-25T15:52:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358004410 con 0x7f136c0c96e0 2024-04-25T15:52:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c0c79d0 con 0x7f136c0c96e0 2024-04-25T15:52:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c15ee40 con 0x7f136c0c96e0 2024-04-25T15:52:17.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.135+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f136c0cc730 con 0x7f136c0c96e0 2024-04-25T15:52:17.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1358018070 con 0x7f136c0c96e0 2024-04-25T15:52:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13697fa640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f135006f430 0x7f1350071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5886+0+0 (secure 0 0 0) 0x7f135808e0d0 con 0x7f136c0c96e0 2024-04-25T15:52:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13697fa640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f13500778e0 0x7f1350079dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f135007a4a0 con 0x7f13500778e0 2024-04-25T15:52:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f1358093060 con 0x7f136c0c96e0 2024-04-25T15:52:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f1370ef2640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f135006f430 0x7f1350071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13716f3640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f13500778e0 0x7f1350079dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f1370ef2640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f135006f430 0x7f1350071940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f136c0cc4f0 tx=0x7f1360008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.139+0000 7f13716f3640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f13500778e0 0x7f1350079dd0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.143+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f135007a4a0 con 0x7f13500778e0 2024-04-25T15:52:17.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.175+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f136c06a480 con 0x7f13500778e0 2024-04-25T15:52:17.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f13697fa640 1 -- 172.21.15.67:0/1395956242 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f136c06a480 con 0x7f13500778e0 2024-04-25T15:52:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f13500778e0 msgr2=0x7f1350079dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f13500778e0 0x7f1350079dd0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f135006f430 msgr2=0x7f1350071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f135006f430 0x7f1350071940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f136c0cc4f0 tx=0x7f1360008040 comp rx=0 tx=0).stop 2024-04-25T15:52:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c0c96e0 msgr2=0x7f136c0c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 --2- 172.21.15.67:0/1395956242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136c0c96e0 0x7f136c0c70d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f135800b470 tx=0x7f135800b930 comp rx=0 tx=0).stop 2024-04-25T15:52:17.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f13716f3640 1 -- 172.21.15.67:0/1395956242 reap_dead start 2024-04-25T15:52:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 shutdown_connections 2024-04-25T15:52:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 >> 172.21.15.67:0/1395956242 conn(0x7f136c070db0 msgr2=0x7f136c151080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 shutdown_connections 2024-04-25T15:52:17.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.179+0000 7f1373156640 1 -- 172.21.15.67:0/1395956242 wait complete. 2024-04-25T15:52:17.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-25T15:52:17.254 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:17.251+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:17.324 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:17.319+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:17.362 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:17.362 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:17.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:17.430 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:17.427+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.447+0000 7fa4ea2b7640 1 Processor -- start 2024-04-25T15:52:17.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.447+0000 7fa4ea2b7640 1 -- start start 2024-04-25T15:52:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4ea2b7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e4168190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4ea2b7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e406d8e0 con 0x7fa4e40c92d0 2024-04-25T15:52:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4ea2b7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e4073c70 con 0x7fa4e4160800 2024-04-25T15:52:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4ea2b7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e406dbc0 con 0x7fa4e40c96e0 2024-04-25T15:52:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4dbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa4e40c92d0 0x7fa4e4072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46920/0 (socket says 172.21.15.67:46920) 2024-04-25T15:52:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4dbfff640 1 -- 172.21.15.67:0/2280241653 learned_addr learned my addr 172.21.15.67:0/2280241653 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e8854640 1 --2- 172.21.15.67:0/2280241653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e4168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e37fe640 1 -- 172.21.15.67:0/2280241653 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2141991325 0 0) 0x7fa4e406d8e0 con 0x7fa4e40c92d0 2024-04-25T15:52:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e37fe640 1 -- 172.21.15.67:0/2280241653 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4d0003530 con 0x7fa4e40c92d0 2024-04-25T15:52:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e37fe640 1 -- 172.21.15.67:0/2280241653 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800708770 0 0) 0x7fa4e4073c70 con 0x7fa4e4160800 2024-04-25T15:52:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e37fe640 1 -- 172.21.15.67:0/2280241653 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4e406d8e0 con 0x7fa4e4160800 2024-04-25T15:52:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e8854640 1 -- 172.21.15.67:0/2280241653 >> v1:172.21.15.176:6789/0 conn(0x7fa4e4160800 legacy=0x7fa4e406d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e8854640 1 -- 172.21.15.67:0/2280241653 >> v1:172.21.15.67:6789/0 conn(0x7fa4e40c92d0 legacy=0x7fa4e4072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.451+0000 7fa4e8854640 1 -- 172.21.15.67:0/2280241653 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e4073f50 con 0x7fa4e40c96e0 2024-04-25T15:52:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4e8854640 1 --2- 172.21.15.67:0/2280241653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e4168190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009dc0 tx=0x7fa4dc01c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b19d8e28f74f66c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4e37fe640 1 -- 172.21.15.67:0/2280241653 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0b2070 con 0x7fa4e40c96e0 2024-04-25T15:52:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4e37fe640 1 -- 172.21.15.67:0/2280241653 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc0a58d0 con 0x7fa4e40c96e0 2024-04-25T15:52:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4e37fe640 1 -- 172.21.15.67:0/2280241653 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0a5ca0 con 0x7fa4e40c96e0 2024-04-25T15:52:17.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/2280241653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 msgr2=0x7fa4e4168190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4ea2b7640 1 --2- 172.21.15.67:0/2280241653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e4168190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009dc0 tx=0x7fa4dc01c920 comp rx=0 tx=0).stop 2024-04-25T15:52:17.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/2280241653 shutdown_connections 2024-04-25T15:52:17.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4ea2b7640 1 --2- 172.21.15.67:0/2280241653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e4168190 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/2280241653 >> 172.21.15.67:0/2280241653 conn(0x7fa4e4070db0 msgr2=0x7fa4e40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/2280241653 shutdown_connections 2024-04-25T15:52:17.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.455+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/2280241653 wait complete. 2024-04-25T15:52:17.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.459+0000 7fa4ea2b7640 1 Processor -- start 2024-04-25T15:52:17.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.459+0000 7fa4ea2b7640 1 -- start start 2024-04-25T15:52:17.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4e40c92d0 0x7fa4e4150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e40c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4e4160800 0x7fa4e414a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e4160d90 con 0x7fa4e4160800 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e4160c10 con 0x7fa4e40c92d0 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4dbfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e40c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4e40c92d0 0x7fa4e4150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4e4160800 0x7fa4e414a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4e40c92d0 0x7fa4e4150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:43672/0 (socket says 172.21.15.67:43672) 2024-04-25T15:52:17.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4e4160800 0x7fa4e414a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54510/0 (socket says 172.21.15.67:54510) 2024-04-25T15:52:17.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e3fff640 1 -- 172.21.15.67:0/1210150678 learned_addr learned my addr 172.21.15.67:0/1210150678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:17.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/1210150678 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e40734d0 con 0x7fa4e40c96e0 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 -- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 msgr2=0x7fa4e40c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4e40c96e0 0x7fa4e40c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 -- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4e40c92d0 msgr2=0x7fa4e4150c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4e40c92d0 0x7fa4e4150c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 -- 172.21.15.67:0/1210150678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e40c71a0 con 0x7fa4e4160800 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e8854640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4e4160800 0x7fa4e414a280 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40cb180 tx=0x7fa4dc0196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0bd020 con 0x7fa4e4160800 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc0b8470 con 0x7fa4e4160800 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/1210150678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e40c7460 con 0x7fa4e4160800 2024-04-25T15:52:17.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.463+0000 7fa4ea2b7640 1 -- 172.21.15.67:0/1210150678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e414eb40 con 0x7fa4e4160800 2024-04-25T15:52:17.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.467+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa4c8001040 con 0x7fa4e4160800 2024-04-25T15:52:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.467+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0b2040 con 0x7fa4e4160800 2024-04-25T15:52:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.467+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa4dc0c0020 con 0x7fa4e4160800 2024-04-25T15:52:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e17fa640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa4c406f430 0x7fa4c4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5886+0+0 (secure 0 0 0) 0x7fa4dc12a060 con 0x7fa4e4160800 2024-04-25T15:52:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e17fa640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fa4c4077960 0x7fa4c4079e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4dc0c1e70 con 0x7fa4c4077960 2024-04-25T15:52:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa4dc12f050 con 0x7fa4e4160800 2024-04-25T15:52:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e3fff640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa4c406f430 0x7fa4c4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4dbfff640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fa4c4077960 0x7fa4c4079e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4dbfff640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fa4c4077960 0x7fa4c4079e50 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e3fff640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa4c406f430 0x7fa4c4071940 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40011f0 tx=0x7fa4d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.471+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa4dc0c1e70 con 0x7fa4c4077960 2024-04-25T15:52:17.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.503+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fa4c8002de0 con 0x7fa4c4077960 2024-04-25T15:52:17.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4e17fa640 1 -- 172.21.15.67:0/1210150678 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fa4c8002de0 con 0x7fa4c4077960 2024-04-25T15:52:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fa4c4077960 msgr2=0x7fa4c4079e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fa4c4077960 0x7fa4c4079e50 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa4c406f430 msgr2=0x7fa4c4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa4c406f430 0x7fa4c4071940 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40011f0 tx=0x7fa4d4009040 comp rx=0 tx=0).stop 2024-04-25T15:52:17.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4e4160800 msgr2=0x7fa4e414a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 --2- 172.21.15.67:0/1210150678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4e4160800 0x7fa4e414a280 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40cb180 tx=0x7fa4dc0196c0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4e8854640 1 -- 172.21.15.67:0/1210150678 reap_dead start 2024-04-25T15:52:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 shutdown_connections 2024-04-25T15:52:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 >> 172.21.15.67:0/1210150678 conn(0x7fa4e4070db0 msgr2=0x7fa4e4151de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 shutdown_connections 2024-04-25T15:52:17.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.507+0000 7fa4da7fc640 1 -- 172.21.15.67:0/1210150678 wait complete. 2024-04-25T15:52:17.525 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:17.523+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:17.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-25T15:52:17.626 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:17.623+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:17.638 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:17.639 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:17.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:17.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:17.727 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:17.727+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:17.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb568dab640 1 Processor -- start 2024-04-25T15:52:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb568dab640 1 -- start start 2024-04-25T15:52:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb568dab640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb5640c4e30 0x7fb564164ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb568dab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb564071f00 0x7fb5640d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb568dab640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5640c54e0 con 0x7fb564071af0 2024-04-25T15:52:17.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb568dab640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb5640c5970 con 0x7fb564071f00 2024-04-25T15:52:17.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb568dab640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb564164300 con 0x7fb5640c4e30 2024-04-25T15:52:17.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb561d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb564071af0 0x7fb5640cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46932/0 (socket says 172.21.15.67:46932) 2024-04-25T15:52:17.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb561d9b640 1 -- 172.21.15.67:0/4133080850 learned_addr learned my addr 172.21.15.67:0/4133080850 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:17.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.779+0000 7fb562d9d640 1 --2- 172.21.15.67:0/4133080850 >> v2:172.21.15.67:3300/0 conn(0x7fb564071f00 0x7fb5640d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb56159a640 1 -- 172.21.15.67:0/4133080850 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2991632491 0 0) 0x7fb5640c54e0 con 0x7fb564071af0 2024-04-25T15:52:17.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb56159a640 1 -- 172.21.15.67:0/4133080850 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb54c003530 con 0x7fb564071af0 2024-04-25T15:52:17.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb562d9d640 1 -- 172.21.15.67:0/4133080850 >> v2:172.21.15.176:3300/0 conn(0x7fb5640c4e30 msgr2=0x7fb564164ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:17.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb562d9d640 1 --2- 172.21.15.67:0/4133080850 >> v2:172.21.15.176:3300/0 conn(0x7fb5640c4e30 0x7fb564164ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb562d9d640 1 -- 172.21.15.67:0/4133080850 >> v1:172.21.15.67:6789/0 conn(0x7fb564071af0 legacy=0x7fb5640cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb562d9d640 1 -- 172.21.15.67:0/4133080850 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5640c5240 con 0x7fb564071f00 2024-04-25T15:52:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb562d9d640 1 --2- 172.21.15.67:0/4133080850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071f00 0x7fb5640d01e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb5640726c0 tx=0x7fb55c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c64a97dd7d52942 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb56159a640 1 -- 172.21.15.67:0/4133080850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0b2070 con 0x7fb564071f00 2024-04-25T15:52:17.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb56159a640 1 -- 172.21.15.67:0/4133080850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb55c0a5760 con 0x7fb564071f00 2024-04-25T15:52:17.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.783+0000 7fb56159a640 1 -- 172.21.15.67:0/4133080850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0a5b30 con 0x7fb564071f00 2024-04-25T15:52:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 -- 172.21.15.67:0/4133080850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071f00 msgr2=0x7fb5640d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 --2- 172.21.15.67:0/4133080850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071f00 0x7fb5640d01e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb5640726c0 tx=0x7fb55c01c8a0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 -- 172.21.15.67:0/4133080850 shutdown_connections 2024-04-25T15:52:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 --2- 172.21.15.67:0/4133080850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071f00 0x7fb5640d01e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 --2- 172.21.15.67:0/4133080850 >> v2:172.21.15.176:3300/0 conn(0x7fb5640c4e30 0x7fb564164ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 -- 172.21.15.67:0/4133080850 >> 172.21.15.67:0/4133080850 conn(0x7fb5640700b0 msgr2=0x7fb5640704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 -- 172.21.15.67:0/4133080850 shutdown_connections 2024-04-25T15:52:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.787+0000 7fb568dab640 1 -- 172.21.15.67:0/4133080850 wait complete. 2024-04-25T15:52:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb568dab640 1 Processor -- start 2024-04-25T15:52:17.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb568dab640 1 -- start start 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb568dab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071af0 0x7fb5640ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb568dab640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb564071f00 0x7fb5641425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb568dab640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5640c4e30 0x7fb56413eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb56259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071af0 0x7fb5640ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb56259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071af0 0x7fb5640ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54536/0 (socket says 172.21.15.67:54536) 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb56259c640 1 -- 172.21.15.67:0/697011382 learned_addr learned my addr 172.21.15.67:0/697011382 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb568dab640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb5640cc2a0 con 0x7fb564071af0 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb561d9b640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb564071f00 0x7fb5641425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb562d9d640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5640c4e30 0x7fb56413eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.791+0000 7fb568dab640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb564072310 con 0x7fb564071f00 2024-04-25T15:52:17.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb568dab640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb5640caac0 con 0x7fb5640c4e30 2024-04-25T15:52:17.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb56259c640 1 -- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5640c4e30 msgr2=0x7fb56413eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb56259c640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb5640c4e30 0x7fb56413eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb56259c640 1 -- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb564071f00 msgr2=0x7fb5641425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb56259c640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb564071f00 0x7fb5641425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb56259c640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5640cec40 con 0x7fb564071af0 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb56259c640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071af0 0x7fb5640ce700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb55800b440 tx=0x7fb55800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558015020 con 0x7fb564071af0 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5580118b0 con 0x7fb564071af0 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558011ca0 con 0x7fb564071af0 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb568dab640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56414d6b0 con 0x7fb564071af0 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb568dab640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5640cd940 con 0x7fb564071af0 2024-04-25T15:52:17.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.795+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb53c001040 con 0x7fb564071af0 2024-04-25T15:52:17.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb558018020 con 0x7fb564071af0 2024-04-25T15:52:17.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb552ffd640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb54406f400 0x7fb544071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5886+0+0 (secure 0 0 0) 0x7fb55808f4e0 con 0x7fb564071af0 2024-04-25T15:52:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb552ffd640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb5440778b0 0x7fb544079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb54407a470 con 0x7fb5440778b0 2024-04-25T15:52:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fb558096050 con 0x7fb564071af0 2024-04-25T15:52:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb561d9b640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb54406f400 0x7fb544071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb562d9d640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb5440778b0 0x7fb544079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb561d9b640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb54406f400 0x7fb544071910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb554007fd0 tx=0x7fb554009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.799+0000 7fb562d9d640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb5440778b0 0x7fb544079da0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.803+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb54407a470 con 0x7fb5440778b0 2024-04-25T15:52:17.829 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:17.827+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fb53c002de0 con 0x7fb5440778b0 2024-04-25T15:52:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb552ffd640 1 -- 172.21.15.67:0/697011382 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fb53c002de0 con 0x7fb5440778b0 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb5440778b0 msgr2=0x7fb544079da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb5440778b0 0x7fb544079da0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb54406f400 msgr2=0x7fb544071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb54406f400 0x7fb544071910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb554007fd0 tx=0x7fb554009040 comp rx=0 tx=0).stop 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071af0 msgr2=0x7fb5640ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 --2- 172.21.15.67:0/697011382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb564071af0 0x7fb5640ce700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb55800b440 tx=0x7fb55800b900 comp rx=0 tx=0).stop 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb562d9d640 1 -- 172.21.15.67:0/697011382 reap_dead start 2024-04-25T15:52:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 shutdown_connections 2024-04-25T15:52:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 >> 172.21.15.67:0/697011382 conn(0x7fb5640700b0 msgr2=0x7fb5640ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 shutdown_connections 2024-04-25T15:52:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.835+0000 7fb550ff9640 1 -- 172.21.15.67:0/697011382 wait complete. 2024-04-25T15:52:17.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.855+0000 7fdc7f656640 1 Processor -- start 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.855+0000 7fdc7f656640 1 -- start start 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7f656640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdc78164440 0x7fdc780cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7f656640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc7815b050 0x7fdc781616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7f656640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc78164a10 con 0x7fdc78161c00 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7f656640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdc78161230 con 0x7fdc7815b050 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7f656640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdc78164cf0 con 0x7fdc78164440 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7dbf3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdc78161c00 0x7fdc780cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46936/0 (socket says 172.21.15.67:46936) 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7dbf3640 1 -- 172.21.15.67:0/1682925385 learned_addr learned my addr 172.21.15.67:0/1682925385 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2523897621 0 0) 0x7fdc78164a10 con 0x7fdc78161c00 2024-04-25T15:52:17.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc64003530 con 0x7fdc78161c00 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 880796261 0 0) 0x7fdc64003530 con 0x7fdc78161c00 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 >> v2:172.21.15.176:3300/0 conn(0x7fdc78164440 msgr2=0x7fdc780cbfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 --2- 172.21.15.67:0/1682925385 >> v2:172.21.15.176:3300/0 conn(0x7fdc78164440 0x7fdc780cbfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 >> v2:172.21.15.67:3300/0 conn(0x7fdc7815b050 msgr2=0x7fdc781616c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 --2- 172.21.15.67:0/1682925385 >> v2:172.21.15.67:3300/0 conn(0x7fdc7815b050 0x7fdc781616c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc78164fd0 con 0x7fdc78161c00 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fdc70001a00 con 0x7fdc78161c00 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 >> v1:172.21.15.67:6789/0 conn(0x7fdc78161c00 legacy=0x7fdc780cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc640039c0 0x7fdc64003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc64004810 0x7fdc64004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc64005360 0x7fdc6400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78161230 con 0x7fdc640039c0 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78164cf0 con 0x7fdc64004810 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6400c360 con 0x7fdc64005360 2024-04-25T15:52:17.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc640039c0 0x7fdc64003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc64005360 0x7fdc6400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7d3f2640 1 -- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc64005360 msgr2=0x7fdc6400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc64005360 0x7fdc6400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7d3f2640 1 -- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc64004810 msgr2=0x7fdc64004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.859+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc64004810 0x7fdc64004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc64004810 0x7fdc64004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7dbf3640 1 -- 172.21.15.67:0/1682925385 reap_dead start 2024-04-25T15:52:17.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7d3f2640 1 -- 172.21.15.67:0/1682925385 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc78164fd0 con 0x7fdc640039c0 2024-04-25T15:52:17.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-04-25T15:52:17.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc64004810 0x7fdc64004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:17.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.863+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc640039c0 0x7fdc64003e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009800 tx=0x7fdc7401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a213e9890a91bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc740b0070 con 0x7fdc640039c0 2024-04-25T15:52:17.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc740ab600 con 0x7fdc640039c0 2024-04-25T15:52:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc6ffff640 1 -- 172.21.15.67:0/1682925385 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc740093d0 con 0x7fdc640039c0 2024-04-25T15:52:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc7f656640 1 -- 172.21.15.67:0/1682925385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc640039c0 msgr2=0x7fdc64003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc7f656640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc640039c0 0x7fdc64003e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009800 tx=0x7fdc7401c320 comp rx=0 tx=0).stop 2024-04-25T15:52:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc7f656640 1 -- 172.21.15.67:0/1682925385 shutdown_connections 2024-04-25T15:52:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc7f656640 1 --2- 172.21.15.67:0/1682925385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc640039c0 0x7fdc64003e30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc7f656640 1 -- 172.21.15.67:0/1682925385 >> 172.21.15.67:0/1682925385 conn(0x7fdc78068c70 msgr2=0x7fdc78071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc7f656640 1 -- 172.21.15.67:0/1682925385 shutdown_connections 2024-04-25T15:52:17.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.867+0000 7fdc7f656640 1 -- 172.21.15.67:0/1682925385 wait complete. 2024-04-25T15:52:17.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 Processor -- start 2024-04-25T15:52:17.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 -- start start 2024-04-25T15:52:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc640039c0 0x7fdc78156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc780cad10 0x7fdc7813a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc78143390 0x7fdc78159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78160e50 con 0x7fdc780cad10 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7806d750 con 0x7fdc640039c0 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7f656640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7815b460 con 0x7fdc78143390 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7d3f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc640039c0 0x7fdc78156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7d3f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc640039c0 0x7fdc78156d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:43722/0 (socket says 172.21.15.67:43722) 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7d3f2640 1 -- 172.21.15.67:0/16149114 learned_addr learned my addr 172.21.15.67:0/16149114 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc78143390 0x7fdc78159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.871+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc780cad10 0x7fdc7813a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7d3f2640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc78143390 msgr2=0x7fdc78159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc78143390 0x7fdc78159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7d3f2640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc780cad10 msgr2=0x7fdc7813a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc780cad10 0x7fdc7813a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7d3f2640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc781425d0 con 0x7fdc640039c0 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc78143390 0x7fdc78159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc640039c0 0x7fdc78156d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdc7401c150 tx=0x7fdc64005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc740b0040 con 0x7fdc640039c0 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc740ba070 con 0x7fdc640039c0 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc740093d0 con 0x7fdc640039c0 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc781428b0 con 0x7fdc640039c0 2024-04-25T15:52:17.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc78143ae0 con 0x7fdc640039c0 2024-04-25T15:52:17.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.875+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdc7806a480 con 0x7fdc640039c0 2024-04-25T15:52:17.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdc740b3070 con 0x7fdc640039c0 2024-04-25T15:52:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc6dffb640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc5c06f430 0x7fdc5c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5886+0+0 (secure 0 0 0) 0x7fdc741296e0 con 0x7fdc640039c0 2024-04-25T15:52:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v48) ==== 54+0+16 (secure 0 0 0) 0x7fdc74016030 con 0x7fdc640039c0 2024-04-25T15:52:17.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc5c06f430 0x7fdc5c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc5b7fe640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fdc54001660 0x7fdc54003b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc54006c10 con 0x7fdc54001660 2024-04-25T15:52:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fdc54001660 0x7fdc54003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc5c06f430 0x7fdc5c071940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc780c74b0 tx=0x7fdc68008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.879+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fdc54001660 0x7fdc54003b70 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.883+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc54006c10 con 0x7fdc54001660 2024-04-25T15:52:17.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.911+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc54005c60 con 0x7fdc54001660 2024-04-25T15:52:17.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.915+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc54005c60 con 0x7fdc54001660 2024-04-25T15:52:17.918 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:17.918 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:17.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.915+0000 7fdc5b7fe640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fdc540041f0 0x7fdc54004600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.915+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdc54004bf0 con 0x7fdc540041f0 2024-04-25T15:52:17.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.915+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fdc540041f0 0x7fdc54004600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.915+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fdc540041f0 0x7fdc54004600 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.915+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc54004bf0 con 0x7fdc540041f0 2024-04-25T15:52:17.923 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:17.923 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:17.929 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:17.923+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:17.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:17.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.943+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc54005160 con 0x7fdc540041f0 2024-04-25T15:52:17.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.947+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc54005160 con 0x7fdc540041f0 2024-04-25T15:52:17.950 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:17.951 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:17.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.947+0000 7fdc5b7fe640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fdc540093a0 0x7fdc5400b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.947+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdc5400e6d0 con 0x7fdc540093a0 2024-04-25T15:52:17.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.947+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fdc540093a0 0x7fdc5400b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.947+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fdc540093a0 0x7fdc5400b7c0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.947+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc5400e6d0 con 0x7fdc540093a0 2024-04-25T15:52:17.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.975+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc5400c030 con 0x7fdc540093a0 2024-04-25T15:52:17.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.979+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc5400c030 con 0x7fdc540093a0 2024-04-25T15:52:17.983 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.979+0000 7fdc5b7fe640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fdc5400bfb0 0x7fdc5400c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.979+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fdc5400bfb0 0x7fdc5400c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.979+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fdc5400bfb0 0x7fdc5400c3c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.979+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdc5400c980 con 0x7fdc5400bfb0 2024-04-25T15:52:17.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:17.979+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc5400c980 con 0x7fdc5400bfb0 2024-04-25T15:52:18.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.007+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc5400ce60 con 0x7fdc5400bfb0 2024-04-25T15:52:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.011+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc5400ce60 con 0x7fdc5400bfb0 2024-04-25T15:52:18.016 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.011+0000 7fdc5b7fe640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fdc5400d280 0x7fdc54012930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.011+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdc540157b0 con 0x7fdc5400d280 2024-04-25T15:52:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.011+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fdc5400d280 0x7fdc54012930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.015+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fdc5400d280 0x7fdc54012930 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.015+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc540157b0 con 0x7fdc5400d280 2024-04-25T15:52:18.030 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:18.027+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:18.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.043+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc54013140 con 0x7fdc5400d280 2024-04-25T15:52:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.047+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc54013140 con 0x7fdc5400d280 2024-04-25T15:52:18.051 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:18.052 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:18.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.047+0000 7fdc5b7fe640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fdc54013860 0x7fdc54013c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.047+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdc54018c00 con 0x7fdc54013860 2024-04-25T15:52:18.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.047+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fdc54013860 0x7fdc54013c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.047+0000 7fdc7cbf1640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fdc54013860 0x7fdc54013c70 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.051+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc54018c00 con 0x7fdc54013860 2024-04-25T15:52:18.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.071+0000 7f0239a7c640 1 Processor -- start 2024-04-25T15:52:18.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.071+0000 7f0239a7c640 1 -- start start 2024-04-25T15:52:18.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f0239a7c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02340caf60 0x7f023415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f0239a7c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02340c77a0 con 0x7f0234073bf0 2024-04-25T15:52:18.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f0239a7c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02340c7a60 con 0x7f023415d790 2024-04-25T15:52:18.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f0239a7c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f02340cab60 con 0x7f02340caf60 2024-04-25T15:52:18.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f0233fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0234073bf0 0x7f0234163ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46952/0 (socket says 172.21.15.67:46952) 2024-04-25T15:52:18.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f0233fff640 1 -- 172.21.15.67:0/1448791347 learned_addr learned my addr 172.21.15.67:0/1448791347 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:18.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f02337fe640 1 --2- 172.21.15.67:0/1448791347 >> v2:172.21.15.67:3300/0 conn(0x7f02340caf60 0x7f023415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f02327fc640 1 -- 172.21.15.67:0/1448791347 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2601519258 0 0) 0x7f02340c77a0 con 0x7f0234073bf0 2024-04-25T15:52:18.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f02337fe640 1 -- 172.21.15.67:0/1448791347 >> v1:172.21.15.176:6789/0 conn(0x7f023415d790 legacy=0x7f023406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f02337fe640 1 -- 172.21.15.67:0/1448791347 >> v1:172.21.15.67:6789/0 conn(0x7f0234073bf0 legacy=0x7f0234163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.075+0000 7f02337fe640 1 -- 172.21.15.67:0/1448791347 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02340c7d40 con 0x7f02340caf60 2024-04-25T15:52:18.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7fdc5b7fe640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc54017a00 con 0x7fdc54013860 2024-04-25T15:52:18.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f02337fe640 1 --2- 172.21.15.67:0/1448791347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f023415d250 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f02280097c0 tx=0x7f022801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f3d416a805dc81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f02327fc640 1 -- 172.21.15.67:0/1448791347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02280b0070 con 0x7f02340caf60 2024-04-25T15:52:18.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f02327fc640 1 -- 172.21.15.67:0/1448791347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02280ab600 con 0x7f02340caf60 2024-04-25T15:52:18.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f02327fc640 1 -- 172.21.15.67:0/1448791347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02280ab9d0 con 0x7f02340caf60 2024-04-25T15:52:18.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f0239a7c640 1 -- 172.21.15.67:0/1448791347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 msgr2=0x7f023415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f0239a7c640 1 --2- 172.21.15.67:0/1448791347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f023415d250 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f02280097c0 tx=0x7f022801c4a0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f0239a7c640 1 -- 172.21.15.67:0/1448791347 shutdown_connections 2024-04-25T15:52:18.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f0239a7c640 1 --2- 172.21.15.67:0/1448791347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f023415d250 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f0239a7c640 1 -- 172.21.15.67:0/1448791347 >> 172.21.15.67:0/1448791347 conn(0x7f0234070db0 msgr2=0x7f02340ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:18.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f0239a7c640 1 -- 172.21.15.67:0/1448791347 shutdown_connections 2024-04-25T15:52:18.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.079+0000 7f0239a7c640 1 -- 172.21.15.67:0/1448791347 wait complete. 2024-04-25T15:52:18.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc54017a00 con 0x7fdc54013860 2024-04-25T15:52:18.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7f0239a7c640 1 Processor -- start 2024-04-25T15:52:18.087 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:18.087 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7f0239a7c640 1 -- start start 2024-04-25T15:52:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fdc781554e0 0x7fdc781558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fdc78143d80 con 0x7fdc781554e0 2024-04-25T15:52:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fdc781554e0 0x7fdc781558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7fdc7dbf3640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fdc781554e0 0x7fdc781558f0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7f0239a7c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0234073bf0 0x7f0234154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7f0239a7c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f02340c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f02340c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7f0232ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f02340c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54582/0 (socket says 172.21.15.67:54582) 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.083+0000 7f0232ffd640 1 -- 172.21.15.67:0/68935398 learned_addr learned my addr 172.21.15.67:0/68935398 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0239a7c640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f023415d790 0x7f0234147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f023406c640 con 0x7f02340caf60 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0234159620 con 0x7f0234073bf0 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f023406c7c0 con 0x7f023415d790 2024-04-25T15:52:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f02337fe640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0234073bf0 0x7f0234154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0233fff640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f023415d790 0x7f0234147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0232ffd640 1 -- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f023415d790 msgr2=0x7f0234147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0232ffd640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f023415d790 0x7f0234147570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0232ffd640 1 -- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0234073bf0 msgr2=0x7f0234154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0232ffd640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0234073bf0 0x7f0234154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0232ffd640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02340c5700 con 0x7f02340caf60 2024-04-25T15:52:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc78143d80 con 0x7fdc781554e0 2024-04-25T15:52:18.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.087+0000 7f0232ffd640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f02340c51c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f022400c390 tx=0x7f022400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0224017070 con 0x7f02340caf60 2024-04-25T15:52:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02340ce600 con 0x7f02340caf60 2024-04-25T15:52:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023414d6b0 con 0x7f02340caf60 2024-04-25T15:52:18.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02240052d0 con 0x7f02340caf60 2024-04-25T15:52:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0224003820 con 0x7f02340caf60 2024-04-25T15:52:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0214001040 con 0x7f02340caf60 2024-04-25T15:52:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f022401a070 con 0x7f02340caf60 2024-04-25T15:52:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f021bfff640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f022006f450 0x7f0220071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.095+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5886+0+0 (secure 0 0 0) 0x7f022408e510 con 0x7f02340caf60 2024-04-25T15:52:18.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f021bfff640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0220077980 0x7f0220079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f02337fe640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f022006f450 0x7f0220071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f022007a540 con 0x7f0220077980 2024-04-25T15:52:18.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f0224095050 con 0x7f02340caf60 2024-04-25T15:52:18.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f0233fff640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0220077980 0x7f0220079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f02337fe640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f022006f450 0x7f0220071960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f02280ad000 tx=0x7f02280a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f0233fff640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0220077980 0x7f0220079e70 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.099+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f022007a540 con 0x7f0220077980 2024-04-25T15:52:18.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.111+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc78143d80 con 0x7fdc781554e0 2024-04-25T15:52:18.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.119+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc78143d80 con 0x7fdc781554e0 2024-04-25T15:52:18.122 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.119+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fdc78203a20 0x7fdc78203e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.119+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fdc78143d80 con 0x7fdc78203a20 2024-04-25T15:52:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.119+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fdc78203a20 0x7fdc78203e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.119+0000 7fdc7d3f2640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fdc78203a20 0x7fdc78203e30 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.119+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc78143d80 con 0x7fdc78203a20 2024-04-25T15:52:18.131 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:18.127+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:18.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.131+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f0214002de0 con 0x7f0220077980 2024-04-25T15:52:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f021bfff640 1 -- 172.21.15.67:0/68935398 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f0214002de0 con 0x7f0220077980 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0220077980 msgr2=0x7f0220079e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0220077980 0x7f0220079e70 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f022006f450 msgr2=0x7f0220071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f022006f450 0x7f0220071960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f02280ad000 tx=0x7f02280a6000 comp rx=0 tx=0).stop 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 msgr2=0x7f02340c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 --2- 172.21.15.67:0/68935398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02340caf60 0x7f02340c51c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f022400c390 tx=0x7f022400c850 comp rx=0 tx=0).stop 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0233fff640 1 -- 172.21.15.67:0/68935398 reap_dead start 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 shutdown_connections 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 >> 172.21.15.67:0/68935398 conn(0x7f0234070db0 msgr2=0x7f02340cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 shutdown_connections 2024-04-25T15:52:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.135+0000 7f0239a7c640 1 -- 172.21.15.67:0/68935398 wait complete. 2024-04-25T15:52:18.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.147+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fdc78143d80 con 0x7fdc78203a20 2024-04-25T15:52:18.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.151+0000 7fdc6dffb640 1 -- 172.21.15.67:0/16149114 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdc78143d80 con 0x7fdc78203a20 2024-04-25T15:52:18.157 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.151+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fdc54001660 msgr2=0x7fdc54003b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.151+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fdc54001660 0x7fdc54003b70 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.151+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fdc540041f0 msgr2=0x7fdc54004600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fdc540041f0 0x7fdc54004600 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fdc540093a0 msgr2=0x7fdc5400b7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fdc540093a0 0x7fdc5400b7c0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7dbf3640 1 -- 172.21.15.67:0/16149114 reap_dead start 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fdc5400bfb0 msgr2=0x7fdc5400c3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fdc5400bfb0 0x7fdc5400c3c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fdc5400d280 msgr2=0x7fdc54012930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fdc5400d280 0x7fdc54012930 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fdc54013860 msgr2=0x7fdc54013c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fdc54013860 0x7fdc54013c70 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fdc781554e0 msgr2=0x7fdc781558f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fdc781554e0 0x7fdc781558f0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fdc78203a20 msgr2=0x7fdc78203e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fdc78203a20 0x7fdc78203e30 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7dbf3640 1 -- 172.21.15.67:0/16149114 reap_dead start 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc5c06f430 msgr2=0x7fdc5c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc5c06f430 0x7fdc5c071940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdc780c74b0 tx=0x7fdc68008040 comp rx=0 tx=0).stop 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc640039c0 msgr2=0x7fdc78156d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc640039c0 0x7fdc78156d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdc7401c150 tx=0x7fdc64005840 comp rx=0 tx=0).stop 2024-04-25T15:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 shutdown_connections 2024-04-25T15:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc5c06f430 0x7fdc5c071940 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 --2- 172.21.15.67:0/16149114 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc640039c0 0x7fdc78156d90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 >> 172.21.15.67:0/16149114 conn(0x7fdc78068c70 msgr2=0x7fdc780ca600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 shutdown_connections 2024-04-25T15:52:18.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.155+0000 7fdc7f656640 1 -- 172.21.15.67:0/16149114 wait complete. 2024-04-25T15:52:18.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-25T15:52:18.169 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:18.169 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:18.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:18.169 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:18.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:18.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.955529869, 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013243108, 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0946e-05 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211617+0000", 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211761+0000", 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014392699999999999 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211793+0000", 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1568e-05 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212109+0000", 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031626100000000003 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212422+0000", 2024-04-25T15:52:18.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031341099999999998 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213414+0000", 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099192799999999999 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223830+0000", 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010415654999999999 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224524+0000", 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069455899999999998 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224662+0000", 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013742600000000001 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224828+0000", 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016641799999999999 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:18.174 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.95548856400001, 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013508035, 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:18.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211628+0000", 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3400000000000001e-06 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211641+0000", 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2866e-05 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027786899999999998 2024-04-25T15:52:18.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211973+0000", 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4401000000000003e-05 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214895+0000", 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029218109999999999 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215130+0000", 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023500499999999999 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215556+0000", 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042636900000000001 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224481+0000", 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089248869999999994 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224861+0000", 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038038699999999999 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225017+0000", 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155296 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225135+0000", 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117804 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.95547118499999, 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013948439999999999, 2024-04-25T15:52:18.179 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211645+0000", 2024-04-25T15:52:18.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1699999999999997e-07 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211680+0000", 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5374999999999999e-05 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212061+0000", 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038129700000000001 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212135+0000", 2024-04-25T15:52:18.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4103000000000005e-05 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218486+0000", 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063501709999999999 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218861+0000", 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037518799999999999 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219238+0000", 2024-04-25T15:52:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037732500000000002 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225340+0000", 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061016430000000003 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225414+0000", 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.436e-05 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225489+0000", 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4882999999999994e-05 2024-04-25T15:52:18.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225592+0000", 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103378 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.95542946, 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013711255, 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.184 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211687+0000", 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-06 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211694+0000", 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6329999999999996e-06 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212201+0000", 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050685200000000002 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212239+0000", 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7537999999999999e-05 2024-04-25T15:52:18.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004023726 2024-04-25T15:52:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216562+0000", 2024-04-25T15:52:18.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029964199999999997 2024-04-25T15:52:18.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033712999999999999 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223552+0000", 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066527330000000001 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225162+0000", 2024-04-25T15:52:18.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016101869999999999 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225266+0000", 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010402400000000001 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225397+0000", 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130684 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.191 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:6 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.951156133, 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198140000000006, 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:18.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215961+0000", 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5469999999999999e-06 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215983+0000", 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2014999999999998e-05 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216017+0000", 2024-04-25T15:52:18.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3114999999999997e-05 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216043+0000", 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6755999999999999e-05 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216409+0000", 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036599799999999997 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216466+0000", 2024-04-25T15:52:18.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6947000000000002e-05 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224337+0000", 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078708230000000007 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224479+0000", 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014161299999999999 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:7 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:18.195 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.951005927, 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198700000000006, 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216110+0000", 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3570000000000001e-06 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216125+0000", 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4772e-05 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216151+0000", 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5496000000000001e-05 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216172+0000", 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1095e-05 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217019+0000", 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084760800000000002 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217071+0000", 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1894000000000002e-05 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224489+0000", 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074175710000000004 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224629+0000", 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014007700000000001 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:8 2.10 e20/17 2:09401a81:::601.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.949321031, 2024-04-25T15:52:18.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069206529999999997, 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217796+0000", 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9069999999999999e-06 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217816+0000", 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0296e-05 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217847+0000", 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0877999999999998e-05 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217872+0000", 2024-04-25T15:52:18.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4749e-05 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218099+0000", 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022737199999999999 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8900000000000002e-05 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224635+0000", 2024-04-25T15:52:18.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064771480000000003 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224715+0000", 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9402999999999999e-05 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:9 2.10 e20/17 2:09401a81:::601.00000000:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.948206153, 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058821409999999996, 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218911+0000", 2024-04-25T15:52:18.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218930+0000", 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8916e-05 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218960+0000", 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0065000000000001e-05 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218984+0000", 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4528999999999998e-05 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219206+0000", 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022124300000000001 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219263+0000", 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7003000000000003e-05 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224719+0000", 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054565830000000001 2024-04-25T15:52:18.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224791+0000", 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1928999999999995e-05 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:11 2.10 e20/17 2:0bd6d154:::602.00000000:head v 20'6, mlcod=0'0)", 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.945828521, 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050276770000000004, 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.207 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221288+0000", 2024-04-25T15:52:18.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221306+0000", 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719000000000001e-05 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221336+0000", 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0026e-05 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221365+0000", 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8725e-05 2024-04-25T15:52:18.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222395+0000", 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001030078 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222455+0000", 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9803999999999999e-05 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226163+0000", 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037085629999999998 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226314+0000", 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150829 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:19 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.919476822999997, 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010474059000000001, 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247641+0000", 2024-04-25T15:52:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.129e-06 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247667+0000", 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6089e-05 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247827+0000", 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000159977 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247854+0000", 2024-04-25T15:52:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6971999999999999e-05 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248122+0000", 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026745499999999999 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248469+0000", 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034735899999999999 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248784+0000", 2024-04-25T15:52:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031528900000000001 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251768+0000", 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002983383 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257863+0000", 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006095339 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258009+0000", 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000145921 2024-04-25T15:52:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258112+0000", 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103146 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:20 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.919438052000004, 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010923117, 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247688+0000", 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3659999999999998e-06 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247937+0000", 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024977700000000002 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247958+0000", 2024-04-25T15:52:18.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249612+0000", 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016540089999999999 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249860+0000", 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024840600000000001 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:18.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250352+0000", 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049215499999999998 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255543+0000", 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051905060000000001 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258273+0000", 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027298460000000002 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258467+0000", 2024-04-25T15:52:18.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019409900000000001 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258600+0000", 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013348099999999999 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:23 2.15 2:aa0f8bf4:::101.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.919422999000005, 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082242619999999995, 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.221 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247728+0000", 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5049000000000003e-05 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248028+0000", 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029943400000000001 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248050+0000", 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2722e-05 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248203+0000", 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015264 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248563+0000", 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036047500000000002 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248940+0000", 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037679400000000001 2024-04-25T15:52:18.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252184+0000", 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003243821 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255610+0000", 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034255370000000002 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255793+0000", 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018366899999999999 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255916+0000", 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122998 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.919381064000007, 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.226 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:52:18.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:52:18.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:4 2.10 e20/17 2:0f65c9b8:::60a.00000000:head v 20'10, mlcod=20'7)", 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.916023291000002, 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054254350000000002, 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251094+0000", 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.136e-06 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251114+0000", 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0003000000000001e-05 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251147+0000", 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2947999999999997e-05 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251171+0000", 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3845999999999998e-05 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251347+0000", 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176813 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251408+0000", 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0470000000000002e-05 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256382+0000", 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049743510000000001 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256517+0000", 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013486800000000001 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:3 2.0 e20/17 2:03625d61:::614.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.836474022999994, 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060303220000000003, 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330643+0000", 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9850000000000001e-06 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330658+0000", 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5386000000000001e-05 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330686+0000", 2024-04-25T15:52:18.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7971e-05 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330702+0000", 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5268999999999999e-05 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330841+0000", 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013921800000000001 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330881+0000", 2024-04-25T15:52:18.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0052000000000002e-05 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336596+0000", 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057151320000000004 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336671+0000", 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5308999999999993e-05 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:4 2.0 e20/17 2:03625d61:::614.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.833202532000001, 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068526890000000003, 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333914+0000", 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3859999999999999e-06 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333929+0000", 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5508000000000001e-05 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333945+0000", 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5968999999999999e-05 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333961+0000", 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5160999999999999e-05 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334268+0000", 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030782800000000001 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334291+0000", 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2459e-05 2024-04-25T15:52:18.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340696+0000", 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064053749999999996 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340765+0000", 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9003000000000003e-05 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.242 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:6 2.9 2:941d59bc:::61f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 80.280152232000006, 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054913510000000002, 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886964+0000", 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9999999999999996e-07 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886973+0000", 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1400000000000006e-06 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887182+0000", 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020973699999999999 2024-04-25T15:52:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887195+0000", 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2502e-05 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888149+0000", 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095407200000000004 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888285+0000", 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013629399999999999 2024-04-25T15:52:18.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888481+0000", 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000195343 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890469+0000", 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019881650000000001 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892371+0000", 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019022620000000001 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892426+0000", 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4404999999999998e-05 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892454+0000", 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8631000000000001e-05 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.247 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 80.280071387000007, 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052764769999999999, 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:18.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8400000000000004e-07 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887048+0000", 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7390000000000001e-06 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887454+0000", 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040556499999999998 2024-04-25T15:52:18.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887460+0000", 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8639999999999998e-06 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888968+0000", 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015077599999999999 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889063+0000", 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4658000000000001e-05 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889206+0000", 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014365600000000001 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890596+0000", 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390023 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892161+0000", 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015643219999999999 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892259+0000", 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7955000000000007e-05 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892320+0000", 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1351000000000003e-05 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:24 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 80.280064991000003, 2024-04-25T15:52:18.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055188479999999998, 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887051+0000", 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1699999999999998e-07 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887063+0000", 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2575000000000001e-05 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887475+0000", 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041185899999999999 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887481+0000", 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0970000000000002e-06 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887550+0000", 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9258000000000007e-05 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887693+0000", 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014290400000000001 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:18.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887898+0000", 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020492000000000001 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889987+0000", 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020884929999999999 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892508+0000", 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002521644 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892542+0000", 2024-04-25T15:52:18.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3654e-05 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892569+0000", 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6927000000000002e-05 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 74.239898148999998, 2024-04-25T15:52:18.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010008022, 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927218+0000", 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927222+0000", 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6519999999999997e-06 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927339+0000", 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116705 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927350+0000", 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0995000000000001e-05 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928080+0000", 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072958300000000001 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928207+0000", 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012715 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928433+0000", 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022608599999999999 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070747400000000004 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937109+0000", 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079685439999999993 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9806999999999998e-05 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937225+0000", 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6313000000000001e-05 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:18.262 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:18.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:18.265 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:18.259+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:18.364 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:18.363+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f4532edd640 1 Processor -- start 2024-04-25T15:52:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f4532edd640 1 -- start start 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f4532edd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f452c071af0 0x7f452c071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f4532edd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f452c169620 0x7f452c16ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f4532edd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f452c1651d0 con 0x7f452c0c4e30 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f4532edd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f452c0c54e0 con 0x7f452c071af0 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f4532edd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f452c0c5970 con 0x7f452c169620 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f452bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f452c071af0 0x7f452c071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f452bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f452c071af0 0x7f452c071f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54586/0 (socket says 172.21.15.67:54586) 2024-04-25T15:52:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f452bfff640 1 -- 172.21.15.67:0/3900933743 learned_addr learned my addr 172.21.15.67:0/3900933743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:18.466 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f453147a640 1 --2- 172.21.15.67:0/3900933743 >> v2:172.21.15.176:3300/0 conn(0x7f452c169620 0x7f452c16ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f452bfff640 1 -- 172.21.15.67:0/3900933743 >> v2:172.21.15.176:3300/0 conn(0x7f452c169620 msgr2=0x7f452c16ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f452bfff640 1 --2- 172.21.15.67:0/3900933743 >> v2:172.21.15.176:3300/0 conn(0x7f452c169620 0x7f452c16ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f452bfff640 1 -- 172.21.15.67:0/3900933743 >> v1:172.21.15.67:6789/0 conn(0x7f452c0c4e30 legacy=0x7f452c164ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.459+0000 7f452bfff640 1 -- 172.21.15.67:0/3900933743 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c164300 con 0x7f452c071af0 2024-04-25T15:52:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.463+0000 7f452bfff640 1 --2- 172.21.15.67:0/3900933743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c071af0 0x7f452c071f00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f452c0b6160 tx=0x7f451801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5b70581cec323c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.463+0000 7f452b7fe640 1 -- 172.21.15.67:0/3900933743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45180b0070 con 0x7f452c071af0 2024-04-25T15:52:18.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.463+0000 7f452b7fe640 1 -- 172.21.15.67:0/3900933743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45180ab600 con 0x7f452c071af0 2024-04-25T15:52:18.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.463+0000 7f452b7fe640 1 -- 172.21.15.67:0/3900933743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45180ab9b0 con 0x7f452c071af0 2024-04-25T15:52:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 -- 172.21.15.67:0/3900933743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c071af0 msgr2=0x7f452c071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 --2- 172.21.15.67:0/3900933743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c071af0 0x7f452c071f00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f452c0b6160 tx=0x7f451801c4b0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 -- 172.21.15.67:0/3900933743 shutdown_connections 2024-04-25T15:52:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 --2- 172.21.15.67:0/3900933743 >> v2:172.21.15.176:3300/0 conn(0x7f452c169620 0x7f452c16ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 --2- 172.21.15.67:0/3900933743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c071af0 0x7f452c071f00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 -- 172.21.15.67:0/3900933743 >> 172.21.15.67:0/3900933743 conn(0x7f452c0700b0 msgr2=0x7f452c0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 -- 172.21.15.67:0/3900933743 shutdown_connections 2024-04-25T15:52:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 -- 172.21.15.67:0/3900933743 wait complete. 2024-04-25T15:52:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 Processor -- start 2024-04-25T15:52:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.467+0000 7f4532edd640 1 -- start start 2024-04-25T15:52:18.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f4532edd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f452c071af0 0x7f452c06e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f4532edd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f452c0c4e30 0x7f452c0c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f4532edd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c169620 0x7f452c0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f4532edd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f452c0cc2a0 con 0x7f452c169620 2024-04-25T15:52:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c169620 0x7f452c0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c169620 0x7f452c0c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54590/0 (socket says 172.21.15.67:54590) 2024-04-25T15:52:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 -- 172.21.15.67:0/1446820030 learned_addr learned my addr 172.21.15.67:0/1446820030 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f4532edd640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f452c0c5240 con 0x7f452c071af0 2024-04-25T15:52:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f4532edd640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f452c0caac0 con 0x7f452c0c4e30 2024-04-25T15:52:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f4530c79640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f452c071af0 0x7f452c06e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f452bfff640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f452c0c4e30 0x7f452c0c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 -- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f452c0c4e30 msgr2=0x7f452c0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f452c0c4e30 0x7f452c0c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 -- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f452c071af0 msgr2=0x7f452c06e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f452c071af0 0x7f452c06e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c0c8200 con 0x7f452c169620 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f453147a640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c169620 0x7f452c0c7cc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4524007e90 tx=0x7f452400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4524014420 con 0x7f452c169620 2024-04-25T15:52:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.471+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4524019070 con 0x7f452c169620 2024-04-25T15:52:18.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.475+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4524014800 con 0x7f452c169620 2024-04-25T15:52:18.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.475+0000 7f4532edd640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c1557a0 con 0x7f452c169620 2024-04-25T15:52:18.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.475+0000 7f4532edd640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c155d60 con 0x7f452c169620 2024-04-25T15:52:18.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.475+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4504001040 con 0x7f452c169620 2024-04-25T15:52:18.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.475+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f45240149a0 con 0x7f452c169620 2024-04-25T15:52:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f45297fa640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f451006f400 0x7f4510071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5886+0+0 (secure 0 0 0) 0x7f452408fa50 con 0x7f452c169620 2024-04-25T15:52:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f45297fa640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f45100778b0 0x7f4510079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4524019070 con 0x7f45100778b0 2024-04-25T15:52:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f45240942a0 con 0x7f452c169620 2024-04-25T15:52:18.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f452bfff640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f45100778b0 0x7f4510079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f4530c79640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f451006f400 0x7f4510071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:18.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f452bfff640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f45100778b0 0x7f4510079da0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4524019070 con 0x7f45100778b0 2024-04-25T15:52:18.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.479+0000 7f4530c79640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f451006f400 0x7f4510071910 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4520001770 tx=0x7f4520009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:18.512 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:18.512 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:18.512 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:18.512 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:18.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f4504002de0 con 0x7f45100778b0 2024-04-25T15:52:18.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f45297fa640 1 -- 172.21.15.67:0/1446820030 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f4504002de0 con 0x7f45100778b0 2024-04-25T15:52:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f45100778b0 msgr2=0x7f4510079da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f4516ffd640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f45100778b0 0x7f4510079da0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f451006f400 msgr2=0x7f4510071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f4516ffd640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f451006f400 0x7f4510071910 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4520001770 tx=0x7f4520009040 comp rx=0 tx=0).stop 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c169620 msgr2=0x7f452c0c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f4516ffd640 1 --2- 172.21.15.67:0/1446820030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f452c169620 0x7f452c0c7cc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4524007e90 tx=0x7f452400e4b0 comp rx=0 tx=0).stop 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.511+0000 7f453147a640 1 -- 172.21.15.67:0/1446820030 reap_dead start 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.515+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 shutdown_connections 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.515+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 >> 172.21.15.67:0/1446820030 conn(0x7f452c0700b0 msgr2=0x7f452c153990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.515+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 shutdown_connections 2024-04-25T15:52:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:18.515+0000 7f4516ffd640 1 -- 172.21.15.67:0/1446820030 wait complete. 2024-04-25T15:52:18.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:18.566 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:18.563+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:18.667 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:18.663+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:18.767 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:18.767+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:18.793 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:18.793 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:18.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:18.868 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:18.863+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:18.970 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:18.967+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:19.023 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:19.023 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:19.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:19.023 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.809203444, 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937687, 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212220+0000", 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212245+0000", 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212458+0000", 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021240899999999999 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212488+0000", 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0715e-05 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212836+0000", 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000348096 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213194+0000", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000357413 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213591+0000", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039711599999999998 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224723+0000", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011132177 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224892+0000", 2024-04-25T15:52:19.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016872100000000001 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225031+0000", 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013966 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225148+0000", 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116168 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.028 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.80915714299999, 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013374822, 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:19.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212257+0000", 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212263+0000", 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0900000000000001e-06 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212576+0000", 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031227099999999998 2024-04-25T15:52:19.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212590+0000", 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4133e-05 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214544+0000", 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001953939 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214793+0000", 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024941399999999999 2024-04-25T15:52:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215181+0000", 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038823800000000001 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224739+0000", 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095574579999999996 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225301+0000", 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056223199999999997 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225459+0000", 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015786199999999999 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225631+0000", 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172242 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:19.033 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.80914672999999, 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013916951, 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8299999999999996e-07 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212272+0000", 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6630000000000001e-06 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212665+0000", 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039338399999999999 2024-04-25T15:52:19.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212687+0000", 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1882000000000002e-05 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215937+0000", 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032501119999999999 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032469999999999998 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216646+0000", 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038413800000000001 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224753+0000", 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081070039999999993 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225911+0000", 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001158002 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226062+0000", 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015119600000000001 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226184+0000", 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121187 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.809138761, 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01435588, 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6000000000000003e-07 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212279+0000", 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1169999999999997e-06 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212723+0000", 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044335799999999998 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212733+0000", 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349e-05 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217516+0000", 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004782606 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217766+0000", 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025045699999999999 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039180399999999999 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224766+0000", 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066076449999999997 2024-04-25T15:52:19.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226308+0000", 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00154228 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226483+0000", 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174938 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226630+0000", 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014766599999999999 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.043 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.809131781, 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014867514, 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:19.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2699999999999999e-07 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212287+0000", 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6229999999999996e-06 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212756+0000", 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046907999999999997 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9560000000000003e-06 2024-04-25T15:52:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218829+0000", 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060638690000000004 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219182+0000", 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035291899999999999 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219507+0000", 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032545399999999998 2024-04-25T15:52:19.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224779+0000", 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052717279999999998 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226860+0000", 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020809790000000002 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227036+0000", 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176062 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227149+0000", 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113217 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.80912431100001, 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015281197, 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.049 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212290+0000", 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3e-07 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212333+0000", 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3251000000000003e-05 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212784+0000", 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045139399999999998 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212793+0000", 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7960000000000001e-06 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220253+0000", 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074595699999999996 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220482+0000", 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000229554 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220879+0000", 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039646400000000002 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225181+0000", 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00430205 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227277+0000", 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020961119999999998 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227449+0000", 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017166800000000001 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227570+0000", 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012170799999999999 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.809074831, 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016239407000000001, 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212340+0000", 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1200000000000001e-06 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212346+0000", 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.584e-06 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212813+0000", 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046677600000000002 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212821+0000", 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221805+0000", 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089835999999999996 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222154+0000", 2024-04-25T15:52:19.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034940299999999997 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222503+0000", 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034907900000000002 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227840+0000", 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367349999999996 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228310+0000", 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047011999999999998 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228439+0000", 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012899999999999999 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228578+0000", 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138837 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.80906362100001, 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016443972000000001, 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212351+0000", 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212357+0000", 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000003e-06 2024-04-25T15:52:19.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212848+0000", 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049076600000000001 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212863+0000", 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5498999999999998e-05 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223300+0000", 2024-04-25T15:52:19.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010437093999999999 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223537+0000", 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000236665 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224027+0000", 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048967900000000003 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228184+0000", 2024-04-25T15:52:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041571209999999997 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228599+0000", 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000415392 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228676+0000", 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7095999999999995e-05 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228794+0000", 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011746700000000001 2024-04-25T15:52:19.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.809052642, 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013451848000000001, 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212362+0000", 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212368+0000", 2024-04-25T15:52:19.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6229999999999999e-06 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212925+0000", 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055690199999999996 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212960+0000", 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5045000000000001e-05 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213123+0000", 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163222 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213447+0000", 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000323627 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213759+0000", 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031232800000000001 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216671+0000", 2024-04-25T15:52:19.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002911145 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223654+0000", 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069831370000000004 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223829+0000", 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174857 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225813+0000", 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019840679999999999 2024-04-25T15:52:19.069 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.809041479, 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014242332, 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:19.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212373+0000", 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4e-07 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212384+0000", 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1443e-05 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213026+0000", 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064212500000000001 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213045+0000", 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8621999999999999e-05 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214708+0000", 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016628109999999999 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214970+0000", 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026197699999999999 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215380+0000", 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000410789 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218599+0000", 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032185270000000001 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226296+0000", 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076965269999999999 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226473+0000", 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017755200000000001 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226614+0000", 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:4 2.10 2:0f65c9b8:::60a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.774238144999998, 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010394548999999999, 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247176+0000", 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2100000000000001e-06 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247209+0000", 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2945999999999999e-05 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247394+0000", 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018468799999999999 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247417+0000", 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2733000000000001e-05 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249734+0000", 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002317684 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250032+0000", 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029727699999999998 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250453+0000", 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042143500000000001 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255742+0000", 2024-04-25T15:52:19.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052886809999999999 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257232+0000", 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014906260000000001 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257393+0000", 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016088699999999999 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257570+0000", 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017638199999999999 2024-04-25T15:52:19.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690269579000002, 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094210270000000002, 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6699999999999999e-07 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331165+0000", 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331418+0000", 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271799999999997 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331428+0000", 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6460000000000005e-06 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333229+0000", 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018009499999999999 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334396+0000", 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011673269999999999 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334597+0000", 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020135300000000001 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339905+0000", 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053077610000000002 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340210+0000", 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030499999999999999 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340385+0000", 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017450199999999999 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340565+0000", 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018011700000000001 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690244832999994, 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095762039999999996, 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331174+0000", 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-06 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331467+0000", 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029356 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331478+0000", 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336604+0000", 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051267170000000003 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336735+0000", 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013052699999999999 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336996+0000", 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260511 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339986+0000", 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002990487 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340579+0000", 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059299899999999998 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340673+0000", 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4241999999999997e-05 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340745+0000", 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1484999999999997e-05 2024-04-25T15:52:19.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:17 2.10 2:0e9da45e:::61b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690237586999999, 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010814343000000001, 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331180+0000", 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6650000000000001e-06 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331501+0000", 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032061600000000002 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331509+0000", 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8240000000000006e-06 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333383+0000", 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018733020000000001 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333652+0000", 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026890399999999999 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333865+0000", 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021341 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338313+0000", 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044478269999999997 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341868+0000", 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035546729999999999 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341920+0000", 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2507999999999999e-05 2024-04-25T15:52:19.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341990+0000", 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0110000000000005e-05 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690227914999994, 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011086621, 2024-04-25T15:52:19.095 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331186+0000", 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.51e-07 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331191+0000", 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3970000000000002e-06 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331546+0000", 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035447799999999998 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331557+0000", 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0791000000000001e-05 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334408+0000", 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028516520000000001 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334546+0000", 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013802700000000001 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334877+0000", 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000331081 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339910+0000", 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050323449999999997 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342080+0000", 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021703899999999999 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342184+0000", 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103767 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342272+0000", 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8242e-05 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690219894999998, 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010331643, 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331194+0000", 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331198+0000", 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1400000000000002e-06 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331608+0000", 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040962799999999999 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331620+0000", 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337545+0000", 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005924726 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338432+0000", 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088720699999999999 2024-04-25T15:52:19.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338639+0000", 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020665000000000001 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341318+0000", 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026785860000000002 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341406+0000", 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8344000000000007e-05 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341458+0000", 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805999999999998e-05 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341525+0000", 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7336999999999996e-05 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:2 2.10 2:0daba267:::402.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.105 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690213435999993, 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011245289, 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331215+0000", 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331647+0000", 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043194799999999998 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331656+0000", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335316+0000", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036607710000000002 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335504+0000", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018793299999999999 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335699+0000", 2024-04-25T15:52:19.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194544 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339945+0000", 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042457600000000003 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342333+0000", 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023885730000000002 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342383+0000", 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9314999999999998e-05 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342445+0000", 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2691e-05 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690194675000001, 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087706009999999994, 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331239+0000", 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9653e-05 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331678+0000", 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043868999999999999 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331701+0000", 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3019000000000001e-05 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333753+0000", 2024-04-25T15:52:19.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020522050000000001 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334131+0000", 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000378484 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334455+0000", 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032319899999999998 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337268+0000", 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028133199999999998 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339816+0000", 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025479140000000001 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339907+0000", 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1150000000000007e-05 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339989+0000", 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2392000000000007e-05 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.114 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:27 2.10 2:091e33df:::502.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690171633999995, 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011447874, 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:19.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4199999999999996e-07 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331249+0000", 2024-04-25T15:52:19.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2259999999999999e-06 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331730+0000", 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048009600000000001 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331739+0000", 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1840000000000005e-06 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336653+0000", 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049141510000000003 2024-04-25T15:52:19.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336830+0000", 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017740099999999999 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337033+0000", 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020295999999999999 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340803+0000", 2024-04-25T15:52:19.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037692480000000002 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342567+0000", 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017649250000000001 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342650+0000", 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2748000000000001e-05 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342690+0000", 2024-04-25T15:52:19.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9393e-05 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:31 2.10 2:0eee8b97:::202.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.690161777, 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012346048, 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5300000000000004e-07 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331255+0000", 2024-04-25T15:52:19.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8380000000000002e-06 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331760+0000", 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050452600000000004 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331768+0000", 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337613+0000", 2024-04-25T15:52:19.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058448700000000003 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337836+0000", 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000223043 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.343598+0000", 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057612679999999999 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.123 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.124 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:19.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:52:19.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:19.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:19.119+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:19.226 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:19.223+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:19.327 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:19.327+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.363+0000 7f0d02d60640 1 Processor -- start 2024-04-25T15:52:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.363+0000 7f0d02d60640 1 -- start start 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.363+0000 7f0d02d60640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0cfc068830 0x7f0cfc06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.363+0000 7f0d02d60640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc068c40 con 0x7f0cfc06cb80 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.363+0000 7f0d02d60640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cfc160990 con 0x7f0cfc15dd00 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.363+0000 7f0d02d60640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0cfc15e110 con 0x7f0cfc068830 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0d012fd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0cfc06cb80 0x7f0cfc167190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46978/0 (socket says 172.21.15.67:46978) 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0d012fd640 1 -- 172.21.15.67:0/3868571765 learned_addr learned my addr 172.21.15.67:0/3868571765 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/3868571765 >> v2:172.21.15.176:3300/0 conn(0x7f0cfc068830 0x7f0cfc06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2478333337 0 0) 0x7f0cf40014a0 con 0x7f0cfc06cb80 2024-04-25T15:52:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cec003530 con 0x7f0cfc06cb80 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1999215439 0 0) 0x7f0cec003530 con 0x7f0cfc06cb80 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 >> v2:172.21.15.176:3300/0 conn(0x7f0cfc068830 msgr2=0x7f0cfc06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 --2- 172.21.15.67:0/3868571765 >> v2:172.21.15.176:3300/0 conn(0x7f0cfc068830 0x7f0cfc06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 >> v1:172.21.15.176:6789/0 conn(0x7f0cfc15dd00 legacy=0x7f0cfc1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc160c70 con 0x7f0cfc06cb80 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f0cf4001bd0 con 0x7f0cfc06cb80 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 >> v1:172.21.15.67:6789/0 conn(0x7f0cfc06cb80 legacy=0x7f0cfc167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cec0039c0 0x7f0cec003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cec004810 0x7f0cec004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec0051d0 0x7f0cec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc15e110 con 0x7f0cec004810 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec00c170 con 0x7f0cec0051d0 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0cec00c370 con 0x7f0cec0039c0 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cec004810 0x7f0cec004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0d00afc640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cec0039c0 0x7f0cec003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0d012fd640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec0051d0 0x7f0cec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfbfff640 1 -- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cec0039c0 msgr2=0x7f0cec003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cec0039c0 0x7f0cec003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfbfff640 1 -- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec0051d0 msgr2=0x7f0cec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec0051d0 0x7f0cec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.367+0000 7f0cfbfff640 1 -- 172.21.15.67:0/3868571765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc160c70 con 0x7f0cec004810 2024-04-25T15:52:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d012fd640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec0051d0 0x7f0cec00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d012fd640 1 -- 172.21.15.67:0/3868571765 reap_dead start 2024-04-25T15:52:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cec004810 0x7f0cec004c20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc06fdb0 tx=0x7f0ce801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e04f97b591b02b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:19.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8009690 con 0x7f0cec004810 2024-04-25T15:52:19.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce80059a0 con 0x7f0cec004810 2024-04-25T15:52:19.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0cfb7fe640 1 -- 172.21.15.67:0/3868571765 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8005d90 con 0x7f0cec004810 2024-04-25T15:52:19.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d02d60640 1 -- 172.21.15.67:0/3868571765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cec004810 msgr2=0x7f0cec004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:19.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d02d60640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cec004810 0x7f0cec004c20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc06fdb0 tx=0x7f0ce801ca30 comp rx=0 tx=0).stop 2024-04-25T15:52:19.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d02d60640 1 -- 172.21.15.67:0/3868571765 shutdown_connections 2024-04-25T15:52:19.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d02d60640 1 --2- 172.21.15.67:0/3868571765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cec004810 0x7f0cec004c20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:19.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d02d60640 1 -- 172.21.15.67:0/3868571765 >> 172.21.15.67:0/3868571765 conn(0x7f0cfc0ce1c0 msgr2=0x7f0cfc0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:19.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d02d60640 1 -- 172.21.15.67:0/3868571765 shutdown_connections 2024-04-25T15:52:19.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.371+0000 7f0d02d60640 1 -- 172.21.15.67:0/3868571765 wait complete. 2024-04-25T15:52:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:19.377 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:19.377 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:19.377 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:19.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 Processor -- start 2024-04-25T15:52:19.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 -- start start 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec004810 0x7f0cfc0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cfc074230 0x7f0cfc1f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cfc15cd80 0x7f0cfc1e9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc0c4240 con 0x7f0cfc074230 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0cfc0c3ed0 con 0x7f0cec004810 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0d02d60640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0cfc074900 con 0x7f0cfc15cd80 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.375+0000 7f0cfbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cfc074230 0x7f0cfc1f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cfc074230 0x7f0cfc1f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54600/0 (socket says 172.21.15.67:54600) 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 -- 172.21.15.67:0/1943498646 learned_addr learned my addr 172.21.15.67:0/1943498646 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0d012fd640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cfc15cd80 0x7f0cfc1e9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0d00afc640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec004810 0x7f0cfc0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 -- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cfc15cd80 msgr2=0x7f0cfc1e9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:19.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cfc15cd80 0x7f0cfc1e9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:19.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 -- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec004810 msgr2=0x7f0cfc0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:19.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec004810 0x7f0cfc0cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:19.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 -- 172.21.15.67:0/1943498646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cfc0cd980 con 0x7f0cfc074230 2024-04-25T15:52:19.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cfbfff640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cfc074230 0x7f0cfc1f6510 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80af040 tx=0x7f0ce80197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:19.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cf97fa640 1 -- 172.21.15.67:0/1943498646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80a5a30 con 0x7f0cfc074230 2024-04-25T15:52:19.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cf97fa640 1 -- 172.21.15.67:0/1943498646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce80bc070 con 0x7f0cfc074230 2024-04-25T15:52:19.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0cf97fa640 1 -- 172.21.15.67:0/1943498646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80a5e30 con 0x7f0cfc074230 2024-04-25T15:52:19.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0d02d60640 1 -- 172.21.15.67:0/1943498646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cfc0cdc60 con 0x7f0cfc074230 2024-04-25T15:52:19.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0d02d60640 1 -- 172.21.15.67:0/1943498646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cfc15d190 con 0x7f0cfc074230 2024-04-25T15:52:19.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.379+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cd40052a0 con 0x7f0cfc074230 2024-04-25T15:52:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.383+0000 7f0cf97fa640 1 -- 172.21.15.67:0/1943498646 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0ce8006030 con 0x7f0cfc074230 2024-04-25T15:52:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.383+0000 7f0cf97fa640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0ce006f400 0x7f0ce0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.383+0000 7f0d00afc640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0ce006f400 0x7f0ce0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:19.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.391+0000 7f0cf97fa640 1 -- 172.21.15.67:0/1943498646 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5886+0+0 (secure 0 0 0) 0x7f0ce8127e40 con 0x7f0cfc074230 2024-04-25T15:52:19.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.391+0000 7f0d00afc640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0ce006f400 0x7f0ce0071910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0000c10 tx=0x7f0cf0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:19.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.395+0000 7f0cf97fa640 1 -- 172.21.15.67:0/1943498646 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0ce8016740 con 0x7f0cfc074230 2024-04-25T15:52:19.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:19.428 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:19.427+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:19.528 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:19.527+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:19.604 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:19.604 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:19.604 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:19.604 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:19.604 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:19.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:19.629 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:19.623+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:19.730 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:19.727+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:19.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:19.727+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f0cd4005590 con 0x7f0cfc074230 2024-04-25T15:52:19.831 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:19.827+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:19.832 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:19.832 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:19.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:19.832 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.61718321399999, 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.61714356900001, 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:52:19.840 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.614236547, 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:52:19.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.614083611, 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:52:19.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:52:19.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.612567967, 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:19.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:52:19.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.61242532, 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:52:19.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.612322229, 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:52:19.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:52:19.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:19.861 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.612169479, 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:52:19.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:52:19.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.61022125300001, 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:19.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:19.869 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.582193622000005, 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.580170730999995, 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:52:19.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:52:19.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.580000182000006, 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:52:19.877 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:52:19.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:52:19.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.577644685999999, 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:52:19.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:52:19.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.577609973999998, 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.577589165000006, 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:52:19.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:52:19.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.500702118000007, 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:52:19.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.500678035999996, 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:52:19.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:52:19.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.500668339000001, 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:52:19.905 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 87.494773402000007, 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:52:19.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:52:19.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 75.905334745000005, 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:19.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:19.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:19.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:19.918 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:19.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:19.962 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:19.959+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:20.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.015+0000 7f0cf97fa640 1 -- 172.21.15.67:0/1943498646 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v49) ==== 97+0+0 (secure 0 0 0) 0x7f0ce80f6780 con 0x7f0cfc074230 2024-04-25T15:52:20.020 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0ce006f400 msgr2=0x7f0ce0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0ce006f400 0x7f0ce0071910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0000c10 tx=0x7f0cf0009040 comp rx=0 tx=0).stop 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cfc074230 msgr2=0x7f0cfc1f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cfc074230 0x7f0cfc1f6510 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0ce80af040 tx=0x7f0ce80197d0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 shutdown_connections 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0ce006f400 0x7f0ce0071910 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0cfc15cd80 0x7f0cfc1e9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cfc074230 0x7f0cfc1f6510 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 --2- 172.21.15.67:0/1943498646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0cec004810 0x7f0cfc0cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 >> 172.21.15.67:0/1943498646 conn(0x7f0cfc0ce1c0 msgr2=0x7f0cfc1559d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:20.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 shutdown_connections 2024-04-25T15:52:20.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.019+0000 7f0ce6ffd640 1 -- 172.21.15.67:0/1943498646 wait complete. 2024-04-25T15:52:20.033 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:20.031+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:20.077 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:20.078 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:20.078 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:20.078 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:20.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:20.139 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:20.139+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:20.235 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:20.235+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:20.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:20.288 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:20.288 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:20.288 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:20.288 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:20.288 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:20.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:20.336 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:20.331+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:20.439 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:20.435+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.451+0000 7f89cfeaa640 1 Processor -- start 2024-04-25T15:52:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.451+0000 7f89cfeaa640 1 -- start start 2024-04-25T15:52:20.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cfeaa640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f89c8160800 0x7f89c806d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cfeaa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89c80c92d0 0x7f89c8072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cfeaa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c806d8e0 con 0x7f89c80c96e0 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cfeaa640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89c806dbc0 con 0x7f89c80c92d0 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cfeaa640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f89c8073c70 con 0x7f89c8160800 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cd445640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89c80c92d0 0x7f89c8072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cd445640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89c80c92d0 0x7f89c8072dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54608/0 (socket says 172.21.15.67:54608) 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cdc46640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f89c8160800 0x7f89c806d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cdc46640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f89c8160800 0x7f89c806d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:43784/0 (socket says 172.21.15.67:43784) 2024-04-25T15:52:20.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cd445640 1 -- 172.21.15.67:0/2526071578 learned_addr learned my addr 172.21.15.67:0/2526071578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cd445640 1 -- 172.21.15.67:0/2526071578 >> v2:172.21.15.176:3300/0 conn(0x7f89c8160800 msgr2=0x7f89c806d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89ccc44640 1 -- 172.21.15.67:0/2526071578 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1583825839 0 0) 0x7f89c806d8e0 con 0x7f89c80c96e0 2024-04-25T15:52:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cd445640 1 --2- 172.21.15.67:0/2526071578 >> v2:172.21.15.176:3300/0 conn(0x7f89c8160800 0x7f89c806d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cd445640 1 -- 172.21.15.67:0/2526071578 >> v1:172.21.15.176:6789/0 conn(0x7f89c80c96e0 legacy=0x7f89c815c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.455+0000 7f89cd445640 1 -- 172.21.15.67:0/2526071578 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c8073f50 con 0x7f89c80c92d0 2024-04-25T15:52:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cd445640 1 --2- 172.21.15.67:0/2526071578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c8072dc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f89c806dd40 tx=0x7f89b801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a6995e4aa730f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89ccc44640 1 -- 172.21.15.67:0/2526071578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b80b0070 con 0x7f89c80c92d0 2024-04-25T15:52:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89ccc44640 1 -- 172.21.15.67:0/2526071578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b80ab600 con 0x7f89c80c92d0 2024-04-25T15:52:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89ccc44640 1 -- 172.21.15.67:0/2526071578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b80ab9d0 con 0x7f89c80c92d0 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 -- 172.21.15.67:0/2526071578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 msgr2=0x7f89c8072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/2526071578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c8072dc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f89c806dd40 tx=0x7f89b801c490 comp rx=0 tx=0).stop 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 -- 172.21.15.67:0/2526071578 shutdown_connections 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/2526071578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c8072dc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/2526071578 >> v2:172.21.15.176:3300/0 conn(0x7f89c8160800 0x7f89c806d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 -- 172.21.15.67:0/2526071578 >> 172.21.15.67:0/2526071578 conn(0x7f89c8070db0 msgr2=0x7f89c80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 -- 172.21.15.67:0/2526071578 shutdown_connections 2024-04-25T15:52:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.459+0000 7f89cfeaa640 1 -- 172.21.15.67:0/2526071578 wait complete. 2024-04-25T15:52:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.463+0000 7f89cfeaa640 1 Processor -- start 2024-04-25T15:52:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.463+0000 7f89cfeaa640 1 -- start start 2024-04-25T15:52:20.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c81fa490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89c80c96e0 0x7f89c815a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89c8160800 0x7f89c815ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c81fa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c81fa490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54616/0 (socket says 172.21.15.67:54616) 2024-04-25T15:52:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 -- 172.21.15.67:0/629575108 learned_addr learned my addr 172.21.15.67:0/629575108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89c8160c10 con 0x7f89c80c92d0 2024-04-25T15:52:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89c806d570 con 0x7f89c80c96e0 2024-04-25T15:52:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89c8160d90 con 0x7f89c8160800 2024-04-25T15:52:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89c80c96e0 0x7f89c815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89c8160800 0x7f89c815ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89c8160800 msgr2=0x7f89c815ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89c8160800 0x7f89c815ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89c80c96e0 msgr2=0x7f89c815a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89c80c96e0 0x7f89c815a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c81fa9d0 con 0x7f89c80c92d0 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89c80c96e0 0x7f89c815a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89c8160800 0x7f89c815ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c81fa490 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f89c400e8a0 tx=0x7f89c400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4017070 con 0x7f89c80c92d0 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c400c940 con 0x7f89c80c92d0 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c815ae90 con 0x7f89c80c92d0 2024-04-25T15:52:20.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c8158c80 con 0x7f89c80c92d0 2024-04-25T15:52:20.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c400ccf0 con 0x7f89c80c92d0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.467+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f89c806a480 con 0x7f89c80c92d0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f89c401a070 con 0x7f89c80c92d0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89be7fc640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89ac06f400 0x7f89ac071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5886+0+0 (secure 0 0 0) 0x7f89c408f230 con 0x7f89c80c92d0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v49) ==== 54+0+16 (secure 0 0 0) 0x7f89c40918b0 con 0x7f89c80c92d0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89ac06f400 0x7f89ac071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89c8159a90 0x7f89c8150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89c80c8c50 con 0x7f89c8159a90 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89c8159a90 0x7f89c8150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89ac06f400 0x7f89ac071910 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89b80ad000 tx=0x7f89b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.471+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89c8159a90 0x7f89c8150190 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.475+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c80c8c50 con 0x7f89c8159a90 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:2 2.f 2:f1f69726:::400.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 104.284770481, 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015880386999999999, 2024-04-25T15:52:20.497 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209702+0000", 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209710+0000", 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2470000000000008e-06 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209729+0000", 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8309999999999999e-05 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209789+0000", 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0250000000000001e-05 2024-04-25T15:52:20.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.209816+0000", 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7676999999999999e-05 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210080+0000", 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026343200000000002 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210428+0000", 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034807300000000001 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210738+0000", 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031002100000000001 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212649+0000", 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019114309999999999 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225447+0000", 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012797484 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225515+0000", 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8001000000000002e-05 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225582+0000", 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7460999999999997e-05 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:29 2.b e20/17 2:d0630e4c:::mds0_sessionmap:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 104.282155684, 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048060749999999999, 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212317+0000", 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212318+0000", 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6589999999999999e-06 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212332+0000", 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212352+0000", 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9731000000000001e-05 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7439999999999999e-05 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214540+0000", 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00217022 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214601+0000", 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0857999999999999e-05 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217045+0000", 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024446279999999999 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217123+0000", 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7533999999999993e-05 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.506 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:27 2.13 e20/17 2:cd5a64a3:::100.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 104.281656787, 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057155019999999999, 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212816+0000", 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212817+0000", 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.356e-06 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212859+0000", 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212876+0000", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7311000000000001e-05 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212891+0000", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4993e-05 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215051+0000", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021596110000000001 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215092+0000", 2024-04-25T15:52:20.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0550000000000003e-05 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218451+0000", 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033598109999999999 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218531+0000", 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9825000000000002e-05 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.b epoch 20/17 rep_tid 77 entries 20'2 (0'0) error 2:d5c7a900:::200.00000003:head by mds.0.4:34 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 104.280567541, 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046633910000000002, 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213905+0000", 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213906+0000", 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3060000000000001e-06 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213924+0000", 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7842999999999998e-05 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213960+0000", 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6397999999999999e-05 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213982+0000", 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1600999999999999e-05 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215622+0000", 2024-04-25T15:52:20.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016401969999999999 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218568+0000", 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029460459999999999 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:28 2.13 e20/17 2:c91d4a1d:::mds0_inotable:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 104.279947046, 2024-04-25T15:52:20.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040950229999999997, 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214525+0000", 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214527+0000", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9910000000000001e-06 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214541+0000", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3614e-05 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214599+0000", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8322999999999997e-05 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.214620+0000", 2024-04-25T15:52:20.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613999999999999e-05 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215828+0000", 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001207882 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215853+0000", 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5482000000000001e-05 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218571+0000", 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027175799999999998 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218620+0000", 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9536999999999997e-05 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.0.4:31 2.d e20/17 2:b50e409b:::mds_snaptable:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:20.518 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 104.27755282, 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033168020000000002, 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216920+0000", 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216921+0000", 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3480000000000001e-06 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216952+0000", 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1142000000000001e-05 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216969+0000", 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6719999999999999e-05 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.216984+0000", 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217108+0000", 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012454799999999999 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217145+0000", 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6619000000000003e-05 2024-04-25T15:52:20.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220149+0000", 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030046230000000001 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220236+0000", 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7038000000000003e-05 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:7 2.b e20/17 2:d70a01c6:::60c.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.248432417000004, 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024592329999999999, 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246040+0000", 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246042+0000", 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.057e-06 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246058+0000", 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5492e-05 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246084+0000", 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6410999999999998e-05 2024-04-25T15:52:20.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246099+0000", 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4516e-05 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246482+0000", 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038343900000000001 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.246511+0000", 2024-04-25T15:52:20.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9019000000000001e-05 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248433+0000", 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019216389999999999 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248499+0000", 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6660000000000002e-05 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 5 entries 20'1 (0'0) error 2:1d00e16c:::201.00000005:head by mds.1.10:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.243687683000005, 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002768763, 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250785+0000", 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250786+0000", 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.850000000000001e-07 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250801+0000", 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4785e-05 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250830+0000", 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9445e-05 2024-04-25T15:52:20.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250851+0000", 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1055000000000001e-05 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251002+0000", 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151015 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253554+0000", 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025514779999999998 2024-04-25T15:52:20.529 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:17 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.240160181999997, 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049951730000000003, 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254312+0000", 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254314+0000", 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.764e-06 2024-04-25T15:52:20.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254327+0000", 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2584e-05 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254386+0000", 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8925000000000002e-05 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254405+0000", 2024-04-25T15:52:20.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9525999999999999e-05 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254490+0000", 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4938999999999994e-05 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254507+0000", 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7224e-05 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259250+0000", 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047425180000000003 2024-04-25T15:52:20.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259307+0000", 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7692999999999998e-05 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:18 2.13 e20/17 2:ca4dd7f5:::611.00000000:head v 20'4, mlcod=20'1)", 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.240027701000002, 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048987550000000003, 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.534 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254445+0000", 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2600000000000002e-07 2024-04-25T15:52:20.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254454+0000", 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1380000000000007e-06 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254465+0000", 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1562999999999999e-05 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254475+0000", 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254767+0000", 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000291594 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254785+0000", 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8193999999999999e-05 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259312+0000", 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045267140000000003 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259344+0000", 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1912000000000002e-05 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.2.12:35 2.f 2:f426ad4f:::mds2_openfiles.0:head [omap-clear,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.619088242999993, 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022540479999999998, 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.538 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875384+0000", 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875386+0000", 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0509999999999999e-06 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875394+0000", 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9070000000000007e-06 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875414+0000", 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875428+0000", 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005000000000001e-05 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875561+0000", 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013300000000000001 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.875832+0000", 2024-04-25T15:52:20.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027068099999999999 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.876097+0000", 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026443099999999998 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877430+0000", 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013337780000000001 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877472+0000", 2024-04-25T15:52:20.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1390000000000002e-05 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877559+0000", 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7571999999999994e-05 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.877638+0000", 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8912999999999995e-05 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.543 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:23 2.18 e20/17 2:1c1db442:::103.00000000.inode:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.607653111000005, 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024440640000000001, 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886819+0000", 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886821+0000", 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2869999999999999e-06 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886835+0000", 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4518000000000001e-05 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886856+0000", 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1288999999999999e-05 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886870+0000", 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.377e-05 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886996+0000", 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012557399999999999 2024-04-25T15:52:20.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887029+0000", 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3099000000000003e-05 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889205+0000", 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021762790000000001 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889263+0000", 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8248000000000002e-05 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.547 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:3 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'4, mlcod=20'2)", 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.607255675000005, 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022679520000000002, 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887217+0000", 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887218+0000", 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887226+0000", 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0979999999999994e-06 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887238+0000", 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2700999999999999e-05 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887248+0000", 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4890000000000001e-06 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887995+0000", 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074715299999999997 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888026+0000", 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1421000000000003e-05 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889439+0000", 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001412649 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889485+0000", 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5624000000000002e-05 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:4 2.1a e20/17 2:5fdc322d:::61e.00000000:head v 20'5, mlcod=20'2)", 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.607145934000002, 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028547149999999999, 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887327+0000", 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.58e-07 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887332+0000", 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.78e-06 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887338+0000", 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6760000000000001e-06 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887344+0000", 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.964e-06 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888314+0000", 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097006200000000005 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888358+0000", 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3328999999999998e-05 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890124+0000", 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001765822 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890181+0000", 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7723999999999998e-05 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.555 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:13 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'6, mlcod=20'2)", 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.606645189000005, 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023987079999999998, 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887827+0000", 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887828+0000", 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0500000000000002e-07 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887850+0000", 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1892e-05 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887861+0000", 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1032e-05 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.887871+0000", 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121e-05 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888632+0000", 2024-04-25T15:52:20.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076099000000000002 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888658+0000", 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5491999999999999e-05 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890185+0000", 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001527238 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890226+0000", 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1137999999999998e-05 2024-04-25T15:52:20.559 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.3.14:14 2.1a e20/17 2:594e21d0:::623.00000000:head v 20'7, mlcod=20'2)", 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 82.606269202999997, 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020676359999999999, 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888203+0000", 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888204+0000", 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888213+0000", 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7169999999999997e-06 2024-04-25T15:52:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888227+0000", 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4508000000000001e-05 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888239+0000", 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2086000000000001e-05 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888936+0000", 2024-04-25T15:52:20.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069693700000000001 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.888963+0000", 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6607000000000001e-05 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890228+0000", 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012651050000000001 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.890271+0000", 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:52:20.563 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:13 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'8, mlcod=20'6)", 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.567671020000006, 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043440889999999998, 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926801+0000", 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926803+0000", 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.46e-06 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926819+0000", 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6010000000000001e-05 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926847+0000", 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.783e-05 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926864+0000", 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7096e-05 2024-04-25T15:52:20.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928172+0000", 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013085219999999999 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928233+0000", 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0492999999999998e-05 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931031+0000", 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027977119999999999 2024-04-25T15:52:20.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931146+0000", 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114966 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:14 2.13 e20/17 2:c81eaac8:::62d.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.567570760999999, 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043293189999999999, 2024-04-25T15:52:20.568 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:20.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926902+0000", 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8599999999999998e-07 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926910+0000", 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9379999999999992e-06 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926923+0000", 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3008e-05 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926934+0000", 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0304e-05 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928811+0000", 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001877587 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:20.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928865+0000", 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3640000000000001e-05 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931152+0000", 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022875669999999999 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931231+0000", 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8589000000000003e-05 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.d epoch 20/17 rep_tid 24 entries 20'3 (0'0) error 2:b2057508:::204.00000002:head by mds.4.16:29 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.565465845999995, 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00202038, 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929007+0000", 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929008+0000", 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7600000000000006e-07 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929020+0000", 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2476999999999999e-05 2024-04-25T15:52:20.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929040+0000", 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0145000000000001e-05 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929052+0000", 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2255e-05 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7516999999999996e-05 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.931027+0000", 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018870099999999999 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "pg_update_log_missing(2.18 epoch 20/17 rep_tid 19 entries 20'3 (0'0) error 2:1f0916d5:::204.00000003:head by mds.4.16:30 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 20'1)", 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.565090269999999, 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068184179999999997, 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-04-25T15:52:20.576 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929382+0000", 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929383+0000", 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:52:20.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929394+0000", 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1041e-05 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929416+0000", 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1988999999999999e-05 2024-04-25T15:52:20.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929437+0000", 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0749000000000001e-05 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929882+0000", 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044569400000000001 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:20.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936201+0000", 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063181019999999999 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:20.580 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.499+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c8073920 con 0x7f89c8159a90 2024-04-25T15:52:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.503+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c8073920 con 0x7f89c8159a90 2024-04-25T15:52:20.581 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:20.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:20.583 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:20.579+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.503+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89c815b060 0x7f89c81571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.503+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f89c8154fe0 con 0x7f89c815b060 2024-04-25T15:52:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.503+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89c815b060 0x7f89c81571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.503+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89c815b060 0x7f89c81571f0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.507+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c8154fe0 con 0x7f89c815b060 2024-04-25T15:52:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.531+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c8154d00 con 0x7f89c815b060 2024-04-25T15:52:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.535+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c8154d00 con 0x7f89c815b060 2024-04-25T15:52:20.584 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.535+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89c8157c40 0x7f89c8063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.535+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f89c8158050 con 0x7f89c8157c40 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.535+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89c8157c40 0x7f89c8063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.535+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89c8157c40 0x7f89c8063090 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.539+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c8158050 con 0x7f89c8157c40 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.563+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c8076050 con 0x7f89c8157c40 2024-04-25T15:52:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.567+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c8076050 con 0x7f89c8157c40 2024-04-25T15:52:20.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.567+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89c8063ae0 0x7f89c8151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.567+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f89c815f4c0 con 0x7f89c8063ae0 2024-04-25T15:52:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.567+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89c8063ae0 0x7f89c8151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.567+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89c8063ae0 0x7f89c8151b20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.567+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c815f4c0 con 0x7f89c8063ae0 2024-04-25T15:52:20.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.595+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c8152060 con 0x7f89c8063ae0 2024-04-25T15:52:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.599+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c8152060 con 0x7f89c8063ae0 2024-04-25T15:52:20.602 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.599+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89c8152780 0x7f89c802c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.599+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f89c820c560 con 0x7f89c8152780 2024-04-25T15:52:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.599+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89c8152780 0x7f89c802c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.599+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89c8152780 0x7f89c802c9e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.599+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c820c560 con 0x7f89c8152780 2024-04-25T15:52:20.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.627+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c802cf20 con 0x7f89c8152780 2024-04-25T15:52:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.631+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c802cf20 con 0x7f89c8152780 2024-04-25T15:52:20.637 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.631+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89c802d640 0x7f89c802da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.631+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f89c820f6f0 con 0x7f89c802d640 2024-04-25T15:52:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.631+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89c802d640 0x7f89c802da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.635+0000 7f89cd445640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89c802d640 0x7f89c802da50 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.635+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c820f6f0 con 0x7f89c802d640 2024-04-25T15:52:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.659+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c820ce60 con 0x7f89c802d640 2024-04-25T15:52:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.663+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c820ce60 con 0x7f89c802d640 2024-04-25T15:52:20.668 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.663+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89c820d580 0x7f89c820f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.663+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f89c8212880 con 0x7f89c820d580 2024-04-25T15:52:20.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.663+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89c820d580 0x7f89c820f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.667+0000 7f89ce447640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89c820d580 0x7f89c820f9a0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.667+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c8212880 con 0x7f89c820d580 2024-04-25T15:52:20.682 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:20.679+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.691+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c80750f0 con 0x7f89c820d580 2024-04-25T15:52:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.695+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c80750f0 con 0x7f89c820d580 2024-04-25T15:52:20.701 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.695+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89c82103f0 0x7f89c8210800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.695+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f89c8215c50 con 0x7f89c82103f0 2024-04-25T15:52:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.695+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89c82103f0 0x7f89c8210800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.699+0000 7f89cdc46640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89c82103f0 0x7f89c8210800 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.699+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89c8215c50 con 0x7f89c82103f0 2024-04-25T15:52:20.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.723+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89c8210e30 con 0x7f89c82103f0 2024-04-25T15:52:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89be7fc640 1 -- 172.21.15.67:0/629575108 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89c8210e30 con 0x7f89c82103f0 2024-04-25T15:52:20.734 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89c8159a90 msgr2=0x7f89c8150190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89c8159a90 0x7f89c8150190 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89c815b060 msgr2=0x7f89c81571f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89c815b060 0x7f89c81571f0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89c8157c40 msgr2=0x7f89c8063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89c8157c40 0x7f89c8063090 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89c8063ae0 msgr2=0x7f89c8151b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89c8063ae0 0x7f89c8151b20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89c8152780 msgr2=0x7f89c802c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89c8152780 0x7f89c802c9e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89c802d640 msgr2=0x7f89c802da50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89ce447640 1 -- 172.21.15.67:0/629575108 reap_dead start 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89c802d640 0x7f89c802da50 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89ce447640 1 -- 172.21.15.67:0/629575108 reap_dead start 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89c820d580 msgr2=0x7f89c820f9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89c820d580 0x7f89c820f9a0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89c82103f0 msgr2=0x7f89c8210800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89c82103f0 0x7f89c8210800 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89ce447640 1 -- 172.21.15.67:0/629575108 reap_dead start 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89ac06f400 msgr2=0x7f89ac071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89ac06f400 0x7f89ac071910 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89b80ad000 tx=0x7f89b8008040 comp rx=0 tx=0).stop 2024-04-25T15:52:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 msgr2=0x7f89c81fa490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c81fa490 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f89c400e8a0 tx=0x7f89c400eca0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 shutdown_connections 2024-04-25T15:52:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f89ac06f400 0x7f89ac071910 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 --2- 172.21.15.67:0/629575108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c80c92d0 0x7f89c81fa490 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 >> 172.21.15.67:0/629575108 conn(0x7f89c8070db0 msgr2=0x7f89c80c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 shutdown_connections 2024-04-25T15:52:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:20.731+0000 7f89cfeaa640 1 -- 172.21.15.67:0/629575108 wait complete. 2024-04-25T15:52:20.776 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:20.777 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:20.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:20.802 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:20.799+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:20.884 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:20.883+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:20.943 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:20.943 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:20.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:20.984 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:20.979+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:21.086 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:21.083+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:21.155 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:21.155 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:21.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:21.186 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:21.183+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:21.287 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:21.283+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:21.317 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:21.317 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:21.334 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:21.388 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:21.387+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:21.486 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:21.486 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:21.489 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:21.487+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:21.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:21.590 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:21.587+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:21.691 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:21.687+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:21.695 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:21.695 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:21.712 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:21.791 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:21.787+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:21.891 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:21.887+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:21.907 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:21.907 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:21.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:21.992 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:21.991+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:22.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:52:22.068 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:22.068 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:22.086 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:22.098 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:22.095+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:22.194 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:22.191+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3282e12640 1 Processor -- start 2024-04-25T15:52:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3282e12640 1 -- start start 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3282e12640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c0c49f0 0x7f327c0c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3282e12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f327c161620 0x7f327c163a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3282e12640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f327c06fa10 con 0x7f327c156330 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3282e12640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f327c0ca6c0 con 0x7f327c161620 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3282e12640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f327c0c43d0 con 0x7f327c0c49f0 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3280bae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f327c156330 0x7f327c06c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46980/0 (socket says 172.21.15.67:46980) 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3280bae640 1 -- 172.21.15.67:0/1687077174 learned_addr learned my addr 172.21.15.67:0/1687077174 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f32813af640 1 --2- 172.21.15.67:0/1687077174 >> v2:172.21.15.67:3300/0 conn(0x7f327c161620 0x7f327c163a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f3273fff640 1 --2- 172.21.15.67:0/1687077174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c0c49f0 0x7f327c0c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f32737fe640 1 -- 172.21.15.67:0/1687077174 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185053821 0 0) 0x7f327c06fa10 con 0x7f327c156330 2024-04-25T15:52:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f32737fe640 1 -- 172.21.15.67:0/1687077174 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3264003530 con 0x7f327c156330 2024-04-25T15:52:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f32813af640 1 -- 172.21.15.67:0/1687077174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c0c49f0 msgr2=0x7f327c0c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f32813af640 1 --2- 172.21.15.67:0/1687077174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c0c49f0 0x7f327c0c4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f32813af640 1 -- 172.21.15.67:0/1687077174 >> v1:172.21.15.67:6789/0 conn(0x7f327c156330 legacy=0x7f327c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.247+0000 7f32813af640 1 -- 172.21.15.67:0/1687077174 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c0ca8f0 con 0x7f327c161620 2024-04-25T15:52:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.251+0000 7f32813af640 1 --2- 172.21.15.67:0/1687077174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c161620 0x7f327c163a40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3274009dc0 tx=0x7f3274021370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8828756a0fc985d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.251+0000 7f32737fe640 1 -- 172.21.15.67:0/1687077174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32740b6070 con 0x7f327c161620 2024-04-25T15:52:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.251+0000 7f32737fe640 1 -- 172.21.15.67:0/1687077174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32740b1600 con 0x7f327c161620 2024-04-25T15:52:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.251+0000 7f32737fe640 1 -- 172.21.15.67:0/1687077174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32740b19f0 con 0x7f327c161620 2024-04-25T15:52:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 -- 172.21.15.67:0/1687077174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c161620 msgr2=0x7f327c163a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 --2- 172.21.15.67:0/1687077174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c161620 0x7f327c163a40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3274009dc0 tx=0x7f3274021370 comp rx=0 tx=0).stop 2024-04-25T15:52:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 -- 172.21.15.67:0/1687077174 shutdown_connections 2024-04-25T15:52:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 --2- 172.21.15.67:0/1687077174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c161620 0x7f327c163a40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 --2- 172.21.15.67:0/1687077174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c0c49f0 0x7f327c0c4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 -- 172.21.15.67:0/1687077174 >> 172.21.15.67:0/1687077174 conn(0x7f327c0ce1c0 msgr2=0x7f327c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 -- 172.21.15.67:0/1687077174 shutdown_connections 2024-04-25T15:52:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 -- 172.21.15.67:0/1687077174 wait complete. 2024-04-25T15:52:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 Processor -- start 2024-04-25T15:52:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.255+0000 7f3282e12640 1 -- start start 2024-04-25T15:52:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f327c0c49f0 0x7f327c1ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c156330 0x7f327c0b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c161620 0x7f327c14db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f327c0c4060 con 0x7f327c156330 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f327c0c3d80 con 0x7f327c0c49f0 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f327c156740 con 0x7f327c161620 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f32813af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c161620 0x7f327c14db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f32813af640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c161620 0x7f327c14db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:58794/0 (socket says 172.21.15.67:58794) 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3273fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c156330 0x7f327c0b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f32813af640 1 -- 172.21.15.67:0/154839826 learned_addr learned my addr 172.21.15.67:0/154839826 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3280bae640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f327c0c49f0 0x7f327c1ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3273fff640 1 -- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c161620 msgr2=0x7f327c14db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3273fff640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c161620 0x7f327c14db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3273fff640 1 -- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f327c0c49f0 msgr2=0x7f327c1ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3273fff640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f327c0c49f0 0x7f327c1ee9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3273fff640 1 -- 172.21.15.67:0/154839826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c14a680 con 0x7f327c156330 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3273fff640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c156330 0x7f327c0b1c00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f326c00e6b0 tx=0x7f326c00eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f32717fa640 1 -- 172.21.15.67:0/154839826 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c016070 con 0x7f327c156330 2024-04-25T15:52:22.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f32717fa640 1 -- 172.21.15.67:0/154839826 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f326c00ca30 con 0x7f327c156330 2024-04-25T15:52:22.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f32717fa640 1 -- 172.21.15.67:0/154839826 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c00b040 con 0x7f327c156330 2024-04-25T15:52:22.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327c14a940 con 0x7f327c156330 2024-04-25T15:52:22.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.259+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327c0cc300 con 0x7f327c156330 2024-04-25T15:52:22.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.263+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f327c06a480 con 0x7f327c156330 2024-04-25T15:52:22.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.263+0000 7f32717fa640 1 -- 172.21.15.67:0/154839826 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f326c019070 con 0x7f327c156330 2024-04-25T15:52:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.263+0000 7f32717fa640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f326806f400 0x7f3268071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.263+0000 7f32717fa640 1 -- 172.21.15.67:0/154839826 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5886+0+0 (secure 0 0 0) 0x7f326c08f340 con 0x7f327c156330 2024-04-25T15:52:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.267+0000 7f3280bae640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f326806f400 0x7f3268071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.267+0000 7f3280bae640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f326806f400 0x7f3268071910 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3278007fd0 tx=0x7f3278009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:22.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.271+0000 7f32717fa640 1 -- 172.21.15.67:0/154839826 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f326c05ba90 con 0x7f327c156330 2024-04-25T15:52:22.278 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:22.278 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:22.295 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:22.291+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:22.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:22.395 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:22.391+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:22.496 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:22.495+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:22.501 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:22.501 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:22.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:22.597 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:22.595+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.611+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f327c156920 con 0x7f327c156330 2024-04-25T15:52:22.699 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:22.695+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:22.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:22.772 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:22.772 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:22.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:22.838 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:22.835+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:22.900 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:22.899+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.947+0000 7feaa0f6f640 1 Processor -- start 2024-04-25T15:52:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7feaa0f6f640 1 -- start start 2024-04-25T15:52:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7feaa0f6f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fea9c0736d0 0x7fea9c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7feaa0f6f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea9c0caa60 con 0x7fea9c06cb80 2024-04-25T15:52:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7feaa0f6f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea9c15a5f0 con 0x7fea9c06d460 2024-04-25T15:52:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7feaa0f6f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fea9c068590 con 0x7fea9c0736d0 2024-04-25T15:52:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea9c06cb80 0x7fea9c15aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56070/0 (socket says 172.21.15.67:56070) 2024-04-25T15:52:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9ad9d640 1 -- 172.21.15.67:0/3940926494 learned_addr learned my addr 172.21.15.67:0/3940926494 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea99d9b640 1 --2- 172.21.15.67:0/3940926494 >> v2:172.21.15.176:3300/0 conn(0x7fea9c0736d0 0x7fea9c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497197565 0 0) 0x7fea9c0caa60 con 0x7fea9c06cb80 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea84003530 con 0x7fea9c06cb80 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1791382438 0 0) 0x7fea84003530 con 0x7fea9c06cb80 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 >> v2:172.21.15.176:3300/0 conn(0x7fea9c0736d0 msgr2=0x7fea9c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 --2- 172.21.15.67:0/3940926494 >> v2:172.21.15.176:3300/0 conn(0x7fea9c0736d0 0x7fea9c06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 >> v1:172.21.15.176:6789/0 conn(0x7fea9c06d460 legacy=0x7fea9c1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c0ca100 con 0x7fea9c06cb80 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fea94001a00 con 0x7fea9c06cb80 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 >> v1:172.21.15.67:6789/0 conn(0x7fea9c06cb80 legacy=0x7fea9c15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fea840039c0 0x7fea84003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea84004810 0x7fea84004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea840051d0 0x7fea8400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c068590 con 0x7fea840051d0 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fea8400c170 con 0x7fea84004810 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fea8400c370 con 0x7fea840039c0 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea840051d0 0x7fea8400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.951+0000 7fea9ad9d640 1 -- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fea840039c0 msgr2=0x7fea84003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea99d9b640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea84004810 0x7fea84004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fea840039c0 0x7fea84003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9ad9d640 1 -- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea84004810 msgr2=0x7fea84004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea84004810 0x7fea84004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9ad9d640 1 -- 172.21.15.67:0/3940926494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea84003530 con 0x7fea840051d0 2024-04-25T15:52:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea99d9b640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea84004810 0x7fea84004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9ad9d640 1 -- 172.21.15.67:0/3940926494 reap_dead start 2024-04-25T15:52:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea840051d0 0x7fea8400b9d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fea9400a830 tx=0x7fea9401d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f33977b8b6fb682 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9400a480 con 0x7fea840051d0 2024-04-25T15:52:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea940ad630 con 0x7fea840051d0 2024-04-25T15:52:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.955+0000 7fea9959a640 1 -- 172.21.15.67:0/3940926494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea940ada20 con 0x7fea840051d0 2024-04-25T15:52:22.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 -- 172.21.15.67:0/3940926494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea840051d0 msgr2=0x7fea8400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea840051d0 0x7fea8400b9d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fea9400a830 tx=0x7fea9401d2a0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 -- 172.21.15.67:0/3940926494 shutdown_connections 2024-04-25T15:52:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/3940926494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea840051d0 0x7fea8400b9d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 -- 172.21.15.67:0/3940926494 >> 172.21.15.67:0/3940926494 conn(0x7fea9c0ce1c0 msgr2=0x7fea9c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 -- 172.21.15.67:0/3940926494 shutdown_connections 2024-04-25T15:52:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 -- 172.21.15.67:0/3940926494 wait complete. 2024-04-25T15:52:22.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 Processor -- start 2024-04-25T15:52:22.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.959+0000 7feaa0f6f640 1 -- start start 2024-04-25T15:52:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7feaa0f6f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fea840051d0 0x7fea9c1f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7feaa0f6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea9c0c5e00 0x7fea9c1e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7feaa0f6f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea9c153090 0x7fea9c14ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7feaa0f6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c0ca340 con 0x7fea9c0c5e00 2024-04-25T15:52:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7feaa0f6f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c068ac0 con 0x7fea9c153090 2024-04-25T15:52:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7feaa0f6f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c06c640 con 0x7fea840051d0 2024-04-25T15:52:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea9c0c5e00 0x7fea9c1e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea9c0c5e00 0x7fea9c1e6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39984/0 (socket says 172.21.15.67:39984) 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 -- 172.21.15.67:0/2627213013 learned_addr learned my addr 172.21.15.67:0/2627213013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fea840051d0 0x7fea9c1f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea9c153090 0x7fea9c14ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fea840051d0 msgr2=0x7fea9c1f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fea840051d0 0x7fea9c1f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea9c153090 msgr2=0x7fea9c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fea9c153090 0x7fea9c14ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c1f47e0 con 0x7fea9c0c5e00 2024-04-25T15:52:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.963+0000 7fea99d9b640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea9c0c5e00 0x7fea9c1e6fa0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fea900108b0 tx=0x7fea90010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9000f040 con 0x7fea9c0c5e00 2024-04-25T15:52:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea90004220 con 0x7fea9c0c5e00 2024-04-25T15:52:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9c1f0c50 con 0x7fea9c0c5e00 2024-04-25T15:52:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9c1f3c60 con 0x7fea9c0c5e00 2024-04-25T15:52:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea900059a0 con 0x7fea9c0c5e00 2024-04-25T15:52:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fea90005b40 con 0x7fea9c0c5e00 2024-04-25T15:52:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7fea8affd640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fea7c06f400 0x7fea7c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.967+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5886+0+0 (secure 0 0 0) 0x7fea90093170 con 0x7fea9c0c5e00 2024-04-25T15:52:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7fea88ff9640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fea9c06a480 con 0x7fea9c0c5e00 2024-04-25T15:52:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fea7c06f400 0x7fea7c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v49) ==== 54+0+16 (secure 0 0 0) 0x7fea90018070 con 0x7fea9c0c5e00 2024-04-25T15:52:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fea74001660 0x7fea74003b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea74006c10 con 0x7fea74001660 2024-04-25T15:52:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fea74001660 0x7fea74003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fea7c06f400 0x7fea7c071910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fea9c074190 tx=0x7fea8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fea74001660 0x7fea74003b70 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.971+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea74006c10 con 0x7fea74001660 2024-04-25T15:52:23.001 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:22.995+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:22.999+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea74005c60 con 0x7fea74001660 2024-04-25T15:52:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.003+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea74005c60 con 0x7fea74001660 2024-04-25T15:52:23.008 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.003+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fea740041f0 0x7fea74004600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.003+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fea74004bf0 con 0x7fea740041f0 2024-04-25T15:52:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.003+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fea740041f0 0x7fea74004600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.003+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fea740041f0 0x7fea74004600 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.007+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea74004bf0 con 0x7fea740041f0 2024-04-25T15:52:23.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.019+0000 7f32717fa640 1 -- 172.21.15.67:0/154839826 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v50) ==== 91+0+0 (secure 0 0 0) 0x7f326c064160 con 0x7f327c156330 2024-04-25T15:52:23.025 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:52:23.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.023+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f326806f400 msgr2=0x7f3268071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.023+0000 7f3282e12640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f326806f400 0x7f3268071910 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3278007fd0 tx=0x7f3278009040 comp rx=0 tx=0).stop 2024-04-25T15:52:23.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.023+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c156330 msgr2=0x7f327c0b1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.023+0000 7f3282e12640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c156330 0x7f327c0b1c00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f326c00e6b0 tx=0x7f326c00eb70 comp rx=0 tx=0).stop 2024-04-25T15:52:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 shutdown_connections 2024-04-25T15:52:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f326806f400 0x7f3268071910 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f327c161620 0x7f327c14db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c156330 0x7f327c0b1c00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 --2- 172.21.15.67:0/154839826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f327c0c49f0 0x7f327c1ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 >> 172.21.15.67:0/154839826 conn(0x7f327c0ce1c0 msgr2=0x7f327c154630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 shutdown_connections 2024-04-25T15:52:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.027+0000 7f3282e12640 1 -- 172.21.15.67:0/154839826 wait complete. 2024-04-25T15:52:23.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.035+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea740051c0 con 0x7fea740041f0 2024-04-25T15:52:23.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.039+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea740051c0 con 0x7fea740041f0 2024-04-25T15:52:23.043 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.039+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fea740093a0 0x7fea7400b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.039+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fea7400e770 con 0x7fea740093a0 2024-04-25T15:52:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.039+0000 7fea99d9b640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fea740093a0 0x7fea7400b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.039+0000 7fea99d9b640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fea740093a0 0x7fea7400b7c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.043+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea7400e770 con 0x7fea740093a0 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:23.047 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.04226174, 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:23.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:23.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'48, mlcod=15'42)", 2024-04-25T15:52:23.051 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.040632522, 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012230665999999999, 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:23.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004239+0000", 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004250+0000", 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1518999999999999e-05 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004274+0000", 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004292+0000", 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7458000000000001e-05 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005373+0000", 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010819359999999999 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005450+0000", 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6200999999999998e-05 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016441+0000", 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991445000000001 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016468+0000", 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6801999999999999e-05 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:71 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'49, mlcod=15'42)", 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:23.055 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.039842083, 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011466877, 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005029+0000", 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999999999999999e-06 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005040+0000", 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1199000000000001e-05 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005063+0000", 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2342000000000001e-05 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005081+0000", 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7711999999999999e-05 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006374+0000", 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001293492 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006429+0000", 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5315999999999997e-05 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016469+0000", 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010039984 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5232000000000002e-05 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:72 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'50, mlcod=15'44)", 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:23.059 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.038565206, 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010219404, 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006306+0000", 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.48e-06 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006368+0000", 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1532000000000007e-05 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006394+0000", 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6917999999999999e-05 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006423+0000", 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8087000000000001e-05 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007111+0000", 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068813299999999995 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007172+0000", 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0918000000000002e-05 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016500+0000", 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093286540000000005 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016524+0000", 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3682000000000001e-05 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.063 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:73 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'51, mlcod=15'44)", 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.037333273, 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090122410000000007, 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007538+0000", 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-06 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007559+0000", 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0302e-05 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007589+0000", 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0088000000000001e-05 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007643+0000", 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4419000000000002e-05 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010532+0000", 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028890249999999999 2024-04-25T15:52:23.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010603+0000", 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1144999999999997e-05 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016525+0000", 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059219279999999999 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016549+0000", 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3444000000000001e-05 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:76 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'44)", 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.034279836, 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010878550000000001, 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:23.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010591+0000", 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7899999999999997e-07 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010599+0000", 2024-04-25T15:52:23.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6000000000000007e-06 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010611+0000", 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2194999999999999e-05 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010620+0000", 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8529999999999998e-06 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016013+0000", 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053922229999999998 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016045+0000", 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1933000000000001e-05 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021402+0000", 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053573559999999998 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021468+0000", 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6611000000000003e-05 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.833416662, 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010088613, 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.072 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211464+0000", 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0740000000000001e-05 2024-04-25T15:52:23.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211486+0000", 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1722e-05 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211654+0000", 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016800800000000001 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211678+0000", 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4283000000000001e-05 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212032+0000", 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035369900000000002 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212515+0000", 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048357899999999999 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024946400000000002 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217815+0000", 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050506989999999996 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221231+0000", 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003415567 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221412+0000", 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000180681 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221542+0000", 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130171 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.833374097, 2024-04-25T15:52:23.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:23.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:27 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.833360438, 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010202413, 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:23.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211510+0000", 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211538+0000", 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7475e-05 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211876+0000", 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033851299999999998 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211898+0000", 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1648999999999999e-05 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212060+0000", 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016183899999999999 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212463+0000", 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040314700000000002 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212983+0000", 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052057999999999998 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220727+0000", 2024-04-25T15:52:23.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077435919999999997 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221245+0000", 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051843400000000002 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221590+0000", 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034479999999999998 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221712+0000", 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121477 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.087 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.833326712, 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:23.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.83331463, 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:23.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:23.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:23.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.833294458, 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:23.099 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.100 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.100 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:23.100 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:23.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:23.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:23.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:23.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:23.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:23.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:23.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:23.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:23.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:23.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:23.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:23.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:23.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:23.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:23.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:23.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.108 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.108 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:23.108 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:23.108 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.83305611999999, 2024-04-25T15:52:23.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:17 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.795322204, 2024-04-25T15:52:23.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010948628, 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.113 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249551+0000", 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2329999999999998e-06 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249572+0000", 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0979e-05 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249640+0000", 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8326999999999998e-05 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249668+0000", 2024-04-25T15:52:23.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8368999999999999e-05 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250082+0000", 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041399200000000002 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250425+0000", 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034239200000000001 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253136+0000", 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027107210000000001 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256784+0000", 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036481980000000001 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260324+0000", 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035401909999999998 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260418+0000", 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3692999999999999e-05 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260496+0000", 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8533000000000002e-05 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:18 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.79528798299999, 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011222823999999999, 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249583+0000", 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1799999999999999e-06 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249591+0000", 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0669999999999992e-06 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249819+0000", 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022795000000000001 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249865+0000", 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5561e-05 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254040+0000", 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041751720000000004 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254349+0000", 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030965399999999997 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254663+0000", 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031403600000000001 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258978+0000", 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043149959999999998 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260612+0000", 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00163383 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260715+0000", 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010311599999999999 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260805+0000", 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9262e-05 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:7 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.714011710999998, 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086917899999999996, 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.122 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330859+0000", 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7000000000000005e-07 2024-04-25T15:52:23.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330864+0000", 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1120000000000004e-06 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331053+0000", 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189261 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331064+0000", 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1260000000000001e-05 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.124 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331313+0000", 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024867500000000001 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331526+0000", 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000212721 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331732+0000", 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000206839 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335642+0000", 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039095639999999999 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339325+0000", 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036827790000000002 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339449+0000", 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123924 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339550+0000", 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101085 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:8 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.71400362, 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010026810000000001, 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330867+0000", 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8899999999999999e-07 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330871+0000", 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2089999999999999e-06 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331105+0000", 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000234381 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331119+0000", 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4165000000000001e-05 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332225+0000", 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011058140000000001 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332348+0000", 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122872 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332557+0000", 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000208364 2024-04-25T15:52:23.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336449+0000", 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038921329999999999 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340746+0000", 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042971779999999996 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340840+0000", 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4109000000000003e-05 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340893+0000", 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2995999999999999e-05 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:32 2.1f 2:fe09a0b8:::202.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.713983857000002, 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090313700000000004, 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.132 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330887+0000", 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1799999999999995e-07 2024-04-25T15:52:23.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330890+0000", 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.427e-06 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331211+0000", 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000320639 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331220+0000", 2024-04-25T15:52:23.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4669999999999993e-06 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335755+0000", 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349500000000003 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336042+0000", 2024-04-25T15:52:23.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028675400000000001 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339917+0000", 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038755149999999999 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:1 2.1f 2:fb684bcd:::204.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.119253783999994, 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012576726, 2024-04-25T15:52:23.136 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000002e-07 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925629+0000", 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3002e-05 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925764+0000", 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013477399999999999 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925779+0000", 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228000000000001e-05 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925847+0000", 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7980999999999998e-05 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925951+0000", 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010333299999999999 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926085+0000", 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013462200000000001 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929258+0000", 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003172607 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938074+0000", 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088162910000000004 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938129+0000", 2024-04-25T15:52:23.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5234e-05 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938193+0000", 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3367000000000005e-05 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:31 2.1f 2:ff4f7703:::204.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.119239046999994, 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012700365, 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5499999999999999e-07 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925633+0000", 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.796e-06 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925809+0000", 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017651200000000001 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925824+0000", 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5071e-05 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926689+0000", 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086453799999999998 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926825+0000", 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013552799999999999 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938331+0000", 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011506565 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:23.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.067+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea7400d760 con 0x7fea740093a0 2024-04-25T15:52:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.071+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea7400d760 con 0x7fea740093a0 2024-04-25T15:52:23.148 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.071+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fea7400bd50 0x7fea7400c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.071+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fea7400c6a0 con 0x7fea7400bd50 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.071+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fea7400bd50 0x7fea7400c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.071+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fea7400bd50 0x7fea7400c160 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.071+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea7400c6a0 con 0x7fea7400bd50 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.099+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea7400cbb0 con 0x7fea7400bd50 2024-04-25T15:52:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.103+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea7400cbb0 con 0x7fea7400bd50 2024-04-25T15:52:23.150 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.103+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fea74010930 0x7fea74012d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.103+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fea74015b20 con 0x7fea74010930 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.103+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fea74010930 0x7fea74012d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.103+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fea74010930 0x7fea74012d50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.107+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea74015b20 con 0x7fea74010930 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.131+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea74013560 con 0x7fea74010930 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.135+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea74013560 con 0x7fea74010930 2024-04-25T15:52:23.151 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:23.152 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:23.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.135+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fea74013c80 0x7fea74014090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.135+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fea74018fc0 con 0x7fea74013c80 2024-04-25T15:52:23.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.135+0000 7fea99d9b640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fea74013c80 0x7fea74014090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.139+0000 7fea99d9b640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fea74013c80 0x7fea74014090 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.139+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea74018fc0 con 0x7fea74013c80 2024-04-25T15:52:23.153 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:23.143+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:23.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.167+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea74018fc0 con 0x7fea74013c80 2024-04-25T15:52:23.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.171+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea74018fc0 con 0x7fea74013c80 2024-04-25T15:52:23.174 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.171+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fea740182e0 0x7fea740186f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.171+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fea7401c360 con 0x7fea740182e0 2024-04-25T15:52:23.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.171+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fea740182e0 0x7fea740186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.171+0000 7fea9ad9d640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fea740182e0 0x7fea740186f0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.175+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea7401c360 con 0x7fea740182e0 2024-04-25T15:52:23.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.199+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea740145d0 con 0x7fea740182e0 2024-04-25T15:52:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.203+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea740145d0 con 0x7fea740182e0 2024-04-25T15:52:23.210 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:23.210 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:23.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.207+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fea74014ae0 0x7fea74014ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.207+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fea74015500 con 0x7fea74014ae0 2024-04-25T15:52:23.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.207+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fea74014ae0 0x7fea74014ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.207+0000 7fea9a59c640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fea74014ae0 0x7fea74014ef0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.207+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea74015500 con 0x7fea74014ae0 2024-04-25T15:52:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.235+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fea740008e0 con 0x7fea74014ae0 2024-04-25T15:52:23.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7fea8affd640 1 -- 172.21.15.67:0/2627213013 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fea740008e0 con 0x7fea74014ae0 2024-04-25T15:52:23.244 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fea74001660 msgr2=0x7fea74003b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fea74001660 0x7fea74003b70 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fea740041f0 msgr2=0x7fea74004600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fea740041f0 0x7fea74004600 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fea740093a0 msgr2=0x7fea7400b7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fea740093a0 0x7fea7400b7c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fea7400bd50 msgr2=0x7fea7400c160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fea7400bd50 0x7fea7400c160 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7fea9ad9d640 1 -- 172.21.15.67:0/2627213013 reap_dead start 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fea74010930 msgr2=0x7fea74012d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fea74010930 0x7fea74012d50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fea74013c80 msgr2=0x7fea74014090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fea74013c80 0x7fea74014090 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fea740182e0 msgr2=0x7fea740186f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fea740182e0 0x7fea740186f0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fea74014ae0 msgr2=0x7fea74014ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fea74014ae0 0x7fea74014ef0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7fea9ad9d640 1 -- 172.21.15.67:0/2627213013 reap_dead start 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fea7c06f400 msgr2=0x7fea7c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fea7c06f400 0x7fea7c071910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fea9c074190 tx=0x7fea8c009040 comp rx=0 tx=0).stop 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea9c0c5e00 msgr2=0x7fea9c1e6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea9c0c5e00 0x7fea9c1e6fa0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fea900108b0 tx=0x7fea90010d70 comp rx=0 tx=0).stop 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.239+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 shutdown_connections 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.243+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fea7c06f400 0x7fea7c071910 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.243+0000 7feaa0f6f640 1 --2- 172.21.15.67:0/2627213013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea9c0c5e00 0x7fea9c1e6fa0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.243+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 >> 172.21.15.67:0/2627213013 conn(0x7fea9c0ce1c0 msgr2=0x7fea9c158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.243+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 shutdown_connections 2024-04-25T15:52:23.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.243+0000 7feaa0f6f640 1 -- 172.21.15.67:0/2627213013 wait complete. 2024-04-25T15:52:23.249 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:23.243+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:23.351 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:23.351+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:23.378 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:23.378 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:23.378 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:23.378 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:23.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:23.452 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:23.451+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:23.546 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:23.547 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:23.547 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T15:52:23.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:52:23.594 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:23.591+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:23.610 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:23.610 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:23.610 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:23.610 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:23.610 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:23.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:23.654 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:23.651+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:23.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f399a88b640 1 Processor -- start 2024-04-25T15:52:23.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f399a88b640 1 -- start start 2024-04-25T15:52:23.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f399a88b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39940caf60 0x7f399415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f399a88b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3994073bf0 0x7f3994163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f399a88b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39940c77a0 con 0x7f399415d790 2024-04-25T15:52:23.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f399a88b640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f39940cab60 con 0x7f3994073bf0 2024-04-25T15:52:23.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f399a88b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3994159620 con 0x7f39940caf60 2024-04-25T15:52:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3993fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39940caf60 0x7f399415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3998e28640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3994073bf0 0x7f3994163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3993fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39940caf60 0x7f399415d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:47340/0 (socket says 172.21.15.67:47340) 2024-04-25T15:52:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3993fff640 1 -- 172.21.15.67:0/1880246312 learned_addr learned my addr 172.21.15.67:0/1880246312 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:23.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3993fff640 1 -- 172.21.15.67:0/1880246312 >> v2:172.21.15.176:3300/0 conn(0x7f3994073bf0 msgr2=0x7f3994163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3993fff640 1 --2- 172.21.15.67:0/1880246312 >> v2:172.21.15.176:3300/0 conn(0x7f3994073bf0 0x7f3994163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3993fff640 1 -- 172.21.15.67:0/1880246312 >> v1:172.21.15.176:6789/0 conn(0x7f399415d790 legacy=0x7f399406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.743+0000 7f3993fff640 1 -- 172.21.15.67:0/1880246312 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39940c7a60 con 0x7f39940caf60 2024-04-25T15:52:23.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.747+0000 7f3993fff640 1 --2- 172.21.15.67:0/1880246312 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39940caf60 0x7f399415d250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3984009870 tx=0x7f398401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90aa4449f22bfb53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.747+0000 7f3992ffd640 1 -- 172.21.15.67:0/1880246312 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39840b2070 con 0x7f39940caf60 2024-04-25T15:52:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.747+0000 7f3992ffd640 1 -- 172.21.15.67:0/1880246312 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39840a9070 con 0x7f39940caf60 2024-04-25T15:52:23.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.747+0000 7f3992ffd640 1 -- 172.21.15.67:0/1880246312 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39840ad880 con 0x7f39940caf60 2024-04-25T15:52:23.754 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:23.751+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:23.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 -- 172.21.15.67:0/1880246312 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39940caf60 msgr2=0x7f399415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 --2- 172.21.15.67:0/1880246312 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39940caf60 0x7f399415d250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3984009870 tx=0x7f398401c560 comp rx=0 tx=0).stop 2024-04-25T15:52:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 -- 172.21.15.67:0/1880246312 shutdown_connections 2024-04-25T15:52:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 --2- 172.21.15.67:0/1880246312 >> v2:172.21.15.176:3300/0 conn(0x7f3994073bf0 0x7f3994163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 --2- 172.21.15.67:0/1880246312 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39940caf60 0x7f399415d250 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 -- 172.21.15.67:0/1880246312 >> 172.21.15.67:0/1880246312 conn(0x7f3994070db0 msgr2=0x7f39940ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 -- 172.21.15.67:0/1880246312 shutdown_connections 2024-04-25T15:52:23.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 -- 172.21.15.67:0/1880246312 wait complete. 2024-04-25T15:52:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 Processor -- start 2024-04-25T15:52:23.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.751+0000 7f399a88b640 1 -- start start 2024-04-25T15:52:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f399a88b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3994073bf0 0x7f399414f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f399a88b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39940caf60 0x7f3994149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f399a88b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399415d790 0x7f399414cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f399a88b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f399415a570 con 0x7f399415d790 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f399a88b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f399415a3f0 con 0x7f39940caf60 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f399a88b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3994166c20 con 0x7f3994073bf0 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399415d790 0x7f399414cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399415d790 0x7f399414cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39996/0 (socket says 172.21.15.67:39996) 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 -- 172.21.15.67:0/157033514 learned_addr learned my addr 172.21.15.67:0/157033514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3993fff640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3994073bf0 0x7f399414f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f39937fe640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39940caf60 0x7f3994149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 -- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3994073bf0 msgr2=0x7f399414f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3994073bf0 0x7f399414f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 -- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39940caf60 msgr2=0x7f3994149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39940caf60 0x7f3994149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:23.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 -- 172.21.15.67:0/157033514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399414fce0 con 0x7f399415d790 2024-04-25T15:52:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3998e28640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399415d790 0x7f399414cfb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f398c00efc0 tx=0x7f398c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3990ff9640 1 -- 172.21.15.67:0/157033514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c018020 con 0x7f399415d790 2024-04-25T15:52:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3990ff9640 1 -- 172.21.15.67:0/157033514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398c014480 con 0x7f399415d790 2024-04-25T15:52:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f399a88b640 1 -- 172.21.15.67:0/157033514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399414e700 con 0x7f399415d790 2024-04-25T15:52:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.755+0000 7f3990ff9640 1 -- 172.21.15.67:0/157033514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c009e60 con 0x7f399415d790 2024-04-25T15:52:23.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.759+0000 7f399a88b640 1 -- 172.21.15.67:0/157033514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f399414ecc0 con 0x7f399415d790 2024-04-25T15:52:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.759+0000 7f3990ff9640 1 -- 172.21.15.67:0/157033514 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f398c01b020 con 0x7f399415d790 2024-04-25T15:52:23.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.759+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f399406a480 con 0x7f399415d790 2024-04-25T15:52:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.759+0000 7f3990ff9640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398006f430 0x7f3980071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.759+0000 7f3990ff9640 1 -- 172.21.15.67:0/157033514 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5886+0+0 (secure 0 0 0) 0x7f398c08f1b0 con 0x7f399415d790 2024-04-25T15:52:23.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.759+0000 7f3993fff640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398006f430 0x7f3980071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:23.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.763+0000 7f3993fff640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398006f430 0x7f3980071940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f39840af000 tx=0x7f3984008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:23.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:23.771+0000 7f3990ff9640 1 -- 172.21.15.67:0/157033514 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f398c010040 con 0x7f399415d790 2024-04-25T15:52:23.855 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:23.855+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.672547223, 2024-04-25T15:52:23.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013243108, 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0946e-05 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211617+0000", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211761+0000", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014392699999999999 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211793+0000", 2024-04-25T15:52:23.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1568e-05 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212109+0000", 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031626100000000003 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212422+0000", 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031341099999999998 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213414+0000", 2024-04-25T15:52:23.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099192799999999999 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223830+0000", 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010415654999999999 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224524+0000", 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069455899999999998 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224662+0000", 2024-04-25T15:52:23.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013742600000000001 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224828+0000", 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016641799999999999 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.672505918, 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013508035, 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211628+0000", 2024-04-25T15:52:23.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3400000000000001e-06 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211641+0000", 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2866e-05 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027786899999999998 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211973+0000", 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4401000000000003e-05 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214895+0000", 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029218109999999999 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215130+0000", 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023500499999999999 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215556+0000", 2024-04-25T15:52:23.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042636900000000001 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224481+0000", 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089248869999999994 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224861+0000", 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038038699999999999 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225017+0000", 2024-04-25T15:52:23.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155296 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225135+0000", 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117804 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.672488539, 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013948439999999999, 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.896 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211645+0000", 2024-04-25T15:52:23.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1699999999999997e-07 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211680+0000", 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5374999999999999e-05 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212061+0000", 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038129700000000001 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212135+0000", 2024-04-25T15:52:23.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4103000000000005e-05 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218486+0000", 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063501709999999999 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218861+0000", 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037518799999999999 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219238+0000", 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037732500000000002 2024-04-25T15:52:23.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225340+0000", 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061016430000000003 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225414+0000", 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.436e-05 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225489+0000", 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4882999999999994e-05 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225592+0000", 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103378 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.672446814, 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013711255, 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211687+0000", 2024-04-25T15:52:23.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-06 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211694+0000", 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6329999999999996e-06 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212201+0000", 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050685200000000002 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212239+0000", 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7537999999999999e-05 2024-04-25T15:52:23.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004023726 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216562+0000", 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029964199999999997 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033712999999999999 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223552+0000", 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066527330000000001 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225162+0000", 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016101869999999999 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225266+0000", 2024-04-25T15:52:23.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010402400000000001 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225397+0000", 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130684 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:6 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:23.906 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.668173487, 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198140000000006, 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:23.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215961+0000", 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5469999999999999e-06 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215983+0000", 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2014999999999998e-05 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216017+0000", 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3114999999999997e-05 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216043+0000", 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6755999999999999e-05 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216409+0000", 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036599799999999997 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216466+0000", 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6947000000000002e-05 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224337+0000", 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078708230000000007 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224479+0000", 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014161299999999999 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:7 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.668023281, 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198700000000006, 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216110+0000", 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3570000000000001e-06 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216125+0000", 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4772e-05 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216151+0000", 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5496000000000001e-05 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216172+0000", 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1095e-05 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217019+0000", 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084760800000000002 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217071+0000", 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1894000000000002e-05 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224489+0000", 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074175710000000004 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224629+0000", 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014007700000000001 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:8 2.10 e20/17 2:09401a81:::601.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.666338385, 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069206529999999997, 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217796+0000", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9069999999999999e-06 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217816+0000", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0296e-05 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217847+0000", 2024-04-25T15:52:23.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0877999999999998e-05 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217872+0000", 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4749e-05 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218099+0000", 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022737199999999999 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8900000000000002e-05 2024-04-25T15:52:23.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224635+0000", 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064771480000000003 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224715+0000", 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9402999999999999e-05 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:9 2.10 e20/17 2:09401a81:::601.00000000:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:23.918 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.66522350699999, 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058821409999999996, 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218911+0000", 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218930+0000", 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8916e-05 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218960+0000", 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0065000000000001e-05 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218984+0000", 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4528999999999998e-05 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219206+0000", 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022124300000000001 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219263+0000", 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7003000000000003e-05 2024-04-25T15:52:23.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224719+0000", 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054565830000000001 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224791+0000", 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1928999999999995e-05 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.922 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:11 2.10 e20/17 2:0bd6d154:::602.00000000:head v 20'6, mlcod=0'0)", 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.662845875, 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050276770000000004, 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221288+0000", 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221306+0000", 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719000000000001e-05 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221336+0000", 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0026e-05 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221365+0000", 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8725e-05 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222395+0000", 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001030078 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222455+0000", 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9803999999999999e-05 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226163+0000", 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037085629999999998 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226314+0000", 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150829 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:19 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.926 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.636494177, 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010474059000000001, 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247641+0000", 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.129e-06 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247667+0000", 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6089e-05 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247827+0000", 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000159977 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247854+0000", 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6971999999999999e-05 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248122+0000", 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026745499999999999 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248469+0000", 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034735899999999999 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248784+0000", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031528900000000001 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251768+0000", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002983383 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257863+0000", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006095339 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258009+0000", 2024-04-25T15:52:23.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000145921 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258112+0000", 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103146 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:20 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.636455406, 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010923117, 2024-04-25T15:52:23.931 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:23.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247688+0000", 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3659999999999998e-06 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247937+0000", 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024977700000000002 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247958+0000", 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-04-25T15:52:23.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249612+0000", 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016540089999999999 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249860+0000", 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024840600000000001 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250352+0000", 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049215499999999998 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255543+0000", 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051905060000000001 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258273+0000", 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027298460000000002 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258467+0000", 2024-04-25T15:52:23.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019409900000000001 2024-04-25T15:52:23.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258600+0000", 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013348099999999999 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:23 2.15 2:aa0f8bf4:::101.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:23.937 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.636440353, 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082242619999999995, 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247728+0000", 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5049000000000003e-05 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248028+0000", 2024-04-25T15:52:23.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029943400000000001 2024-04-25T15:52:23.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248050+0000", 2024-04-25T15:52:23.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2722e-05 2024-04-25T15:52:23.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248203+0000", 2024-04-25T15:52:23.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015264 2024-04-25T15:52:23.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248563+0000", 2024-04-25T15:52:23.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036047500000000002 2024-04-25T15:52:23.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248940+0000", 2024-04-25T15:52:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037679400000000001 2024-04-25T15:52:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252184+0000", 2024-04-25T15:52:23.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003243821 2024-04-25T15:52:23.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255610+0000", 2024-04-25T15:52:23.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034255370000000002 2024-04-25T15:52:23.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255793+0000", 2024-04-25T15:52:23.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018366899999999999 2024-04-25T15:52:23.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255916+0000", 2024-04-25T15:52:23.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122998 2024-04-25T15:52:23.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.947 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.947 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.947 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.947 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:23.948 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.636398418, 2024-04-25T15:52:23.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:52:23.948 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.948 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.948 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.948 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:23.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:23.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:52:23.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:52:23.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:52:23.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:52:23.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:52:23.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:52:23.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:52:23.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:52:23.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:52:23.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:52:23.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:52:23.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:52:23.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:52:23.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:52:23.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:52:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:52:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:52:23.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:52:23.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:52:23.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:52:23.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:52:23.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:52:23.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.960 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.960 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.960 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:4 2.10 e20/17 2:0f65c9b8:::60a.00000000:head v 20'10, mlcod=20'7)", 2024-04-25T15:52:23.960 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:23.961 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.63304064499999, 2024-04-25T15:52:23.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054254350000000002, 2024-04-25T15:52:23.961 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.961 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.961 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.961 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:23.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:23.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251094+0000", 2024-04-25T15:52:23.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.136e-06 2024-04-25T15:52:23.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251114+0000", 2024-04-25T15:52:23.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0003000000000001e-05 2024-04-25T15:52:23.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251147+0000", 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2947999999999997e-05 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251171+0000", 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3845999999999998e-05 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251347+0000", 2024-04-25T15:52:23.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176813 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251408+0000", 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0470000000000002e-05 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256382+0000", 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049743510000000001 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256517+0000", 2024-04-25T15:52:23.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013486800000000001 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:3 2.0 e20/17 2:03625d61:::614.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.553491377, 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060303220000000003, 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330643+0000", 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9850000000000001e-06 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330658+0000", 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5386000000000001e-05 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330686+0000", 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7971e-05 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330702+0000", 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5268999999999999e-05 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330841+0000", 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013921800000000001 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330881+0000", 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0052000000000002e-05 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336596+0000", 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057151320000000004 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336671+0000", 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5308999999999993e-05 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:4 2.0 e20/17 2:03625d61:::614.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.550219885999994, 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068526890000000003, 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.972 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333914+0000", 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3859999999999999e-06 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333929+0000", 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5508000000000001e-05 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333945+0000", 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5968999999999999e-05 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333961+0000", 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5160999999999999e-05 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334268+0000", 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030782800000000001 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334291+0000", 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2459e-05 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340696+0000", 2024-04-25T15:52:23.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064053749999999996 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340765+0000", 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9003000000000003e-05 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:6 2.9 2:941d59bc:::61f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.997169585999998, 2024-04-25T15:52:23.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054913510000000002, 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886964+0000", 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9999999999999996e-07 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886973+0000", 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1400000000000006e-06 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887182+0000", 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020973699999999999 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887195+0000", 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2502e-05 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888149+0000", 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095407200000000004 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888285+0000", 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013629399999999999 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888481+0000", 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000195343 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890469+0000", 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019881650000000001 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892371+0000", 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019022620000000001 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892426+0000", 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4404999999999998e-05 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892454+0000", 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8631000000000001e-05 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.997088740999999, 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052764769999999999, 2024-04-25T15:52:23.981 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8400000000000004e-07 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887048+0000", 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7390000000000001e-06 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887454+0000", 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040556499999999998 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887460+0000", 2024-04-25T15:52:23.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8639999999999998e-06 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888968+0000", 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015077599999999999 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889063+0000", 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4658000000000001e-05 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889206+0000", 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014365600000000001 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890596+0000", 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390023 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892161+0000", 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015643219999999999 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892259+0000", 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7955000000000007e-05 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892320+0000", 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1351000000000003e-05 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:24 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.997082344999995, 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055188479999999998, 2024-04-25T15:52:23.986 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887051+0000", 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1699999999999998e-07 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887063+0000", 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2575000000000001e-05 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887475+0000", 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041185899999999999 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887481+0000", 2024-04-25T15:52:23.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0970000000000002e-06 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887550+0000", 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9258000000000007e-05 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887693+0000", 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014290400000000001 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887898+0000", 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020492000000000001 2024-04-25T15:52:23.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889987+0000", 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020884929999999999 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892508+0000", 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002521644 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892542+0000", 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3654e-05 2024-04-25T15:52:23.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892569+0000", 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6927000000000002e-05 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 79.956915503000005, 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010008022, 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:23.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927218+0000", 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:23.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927222+0000", 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6519999999999997e-06 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927339+0000", 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116705 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927350+0000", 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0995000000000001e-05 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928080+0000", 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072958300000000001 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928207+0000", 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012715 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928433+0000", 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022608599999999999 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070747400000000004 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937109+0000", 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079685439999999993 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9806999999999998e-05 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937225+0000", 2024-04-25T15:52:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6313000000000001e-05 2024-04-25T15:52:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:23.996 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:23.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:23.997 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:23.991+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:24.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.091+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f39940c8a40 con 0x7f399415d790 2024-04-25T15:52:24.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.091+0000 7f3990ff9640 1 -- 172.21.15.67:0/157033514 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v50) ==== 81+0+35174 (secure 0 0 0) 0x7f398c05b9b0 con 0x7f399415d790 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 50, 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:52:23.023612+0000", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:51:46.224511+0000", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5804032, 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:52:24.099 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:52:24.100 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:52:24.101 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:24.102 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:24.103 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:24.104 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:52:24.105 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:24.106 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:24.107 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:24.108 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:24.109 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:52:24.110 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.5399999618530273, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.5399999618530273, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.75, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.75, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:24.111 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:24.112 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.113 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:24.114 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:24.115 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:52:24.116 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.3200000524520874, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.3200000524520874, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:52:24.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.118 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.119 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:52:24.120 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.120 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.120 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:52:24.120 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.120 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.122 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.123 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:52:24.123 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.123 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.123 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.124 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.125 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.126 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 45, 2024-04-25T15:52:24.126 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.126 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.126 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.126 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.127 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:52:24.128 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.131 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:52:24.134 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:52:24.135 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.136 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.140 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.141 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:52:24.142 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.143 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:52:24.145 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.146 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.147 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:24.148 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.148 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.148 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.148 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:24.148 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.149 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.149 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.149 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.149 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.149 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:52:24.150 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.150 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.150 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.151 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:52:24.152 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.154 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 32, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.156 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:52:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.160 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:52:24.161 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:52:24.162 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.163 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:52:24.164 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.166 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.168 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.169 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:24.170 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2f", 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: 6, 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: 2 2024-04-25T15:52:24.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:24.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.173 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:52:24.174 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:52:24.175 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:52:24.176 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:52:24.176 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:52:24.176 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:52:24.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.176 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:24.176 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398006f430 msgr2=0x7f3980071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398006f430 0x7f3980071940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f39840af000 tx=0x7f3984008040 comp rx=0 tx=0).stop 2024-04-25T15:52:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399415d790 msgr2=0x7f399414cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399415d790 0x7f399414cfb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f398c00efc0 tx=0x7f398c00c490 comp rx=0 tx=0).stop 2024-04-25T15:52:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 shutdown_connections 2024-04-25T15:52:24.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f398006f430 0x7f3980071940 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f399415d790 0x7f399414cfb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39940caf60 0x7f3994149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 --2- 172.21.15.67:0/157033514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3994073bf0 0x7f399414f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 >> 172.21.15.67:0/157033514 conn(0x7f3994070db0 msgr2=0x7f39940ca250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 shutdown_connections 2024-04-25T15:52:24.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.095+0000 7f397a7fc640 1 -- 172.21.15.67:0/157033514 wait complete. 2024-04-25T15:52:24.181 INFO:tasks.thrashosds.thrasher:j is {'epoch': 50, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:52:23.023612+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:51:46.224511+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.2f', 'osds': [6, 2]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:52:24.181 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.2f 2024-04-25T15:52:24.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 3.2f 2024-04-25T15:52:24.200 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:24.199+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:24.247 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:24.247 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:24.247 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:24.248 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:24.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:24.318 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:24.315+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:24.401 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:24.395+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdbace4c640 1 Processor -- start 2024-04-25T15:52:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdbace4c640 1 -- start start 2024-04-25T15:52:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdbace4c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8071af0 0x7fdba8071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdbace4c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba81651d0 con 0x7fdba80c4e30 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdbace4c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba80c54e0 con 0x7fdba8169620 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdbace4c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdba80c5970 con 0x7fdba8071af0 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdba80c4e30 0x7fdba8164ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56074/0 (socket says 172.21.15.67:56074) 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba659c640 1 -- 172.21.15.67:0/3685192561 learned_addr learned my addr 172.21.15.67:0/3685192561 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba5d9b640 1 --2- 172.21.15.67:0/3685192561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8071af0 0x7fdba8071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba559a640 1 -- 172.21.15.67:0/3685192561 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188729345 0 0) 0x7fdba81651d0 con 0x7fdba80c4e30 2024-04-25T15:52:24.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba559a640 1 -- 172.21.15.67:0/3685192561 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb90003530 con 0x7fdba80c4e30 2024-04-25T15:52:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba559a640 1 -- 172.21.15.67:0/3685192561 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288457879 0 0) 0x7fdba80c54e0 con 0x7fdba8169620 2024-04-25T15:52:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba5d9b640 1 -- 172.21.15.67:0/3685192561 >> v1:172.21.15.176:6789/0 conn(0x7fdba8169620 legacy=0x7fdba816ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba5d9b640 1 -- 172.21.15.67:0/3685192561 >> v1:172.21.15.67:6789/0 conn(0x7fdba80c4e30 legacy=0x7fdba8164ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.419+0000 7fdba5d9b640 1 -- 172.21.15.67:0/3685192561 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba8164300 con 0x7fdba8071af0 2024-04-25T15:52:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.423+0000 7fdba5d9b640 1 --2- 172.21.15.67:0/3685192561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8071af0 0x7fdba8071f00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c0097f0 tx=0x7fdb9c01c2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc3b4a46005247b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.423+0000 7fdba559a640 1 -- 172.21.15.67:0/3685192561 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c0b0070 con 0x7fdba8071af0 2024-04-25T15:52:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.423+0000 7fdba559a640 1 -- 172.21.15.67:0/3685192561 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c0ab400 con 0x7fdba8071af0 2024-04-25T15:52:24.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.423+0000 7fdba559a640 1 -- 172.21.15.67:0/3685192561 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98001680 con 0x7fdba8071af0 2024-04-25T15:52:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.427+0000 7fdbace4c640 1 -- 172.21.15.67:0/3685192561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8071af0 msgr2=0x7fdba8071f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.427+0000 7fdbace4c640 1 --2- 172.21.15.67:0/3685192561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8071af0 0x7fdba8071f00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c0097f0 tx=0x7fdb9c01c2b0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.427+0000 7fdbace4c640 1 -- 172.21.15.67:0/3685192561 shutdown_connections 2024-04-25T15:52:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.427+0000 7fdbace4c640 1 --2- 172.21.15.67:0/3685192561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8071af0 0x7fdba8071f00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.427+0000 7fdbace4c640 1 -- 172.21.15.67:0/3685192561 >> 172.21.15.67:0/3685192561 conn(0x7fdba80700b0 msgr2=0x7fdba80704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.427+0000 7fdbace4c640 1 -- 172.21.15.67:0/3685192561 shutdown_connections 2024-04-25T15:52:24.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.427+0000 7fdbace4c640 1 -- 172.21.15.67:0/3685192561 wait complete. 2024-04-25T15:52:24.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 Processor -- start 2024-04-25T15:52:24.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 -- start start 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdba80c4e30 0x7fdba8163c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8169620 0x7fdba81fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdba8154490 0x7fdba8158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdba80cc2a0 con 0x7fdba8154490 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdba80c5240 con 0x7fdba80c4e30 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdbace4c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdba80caac0 con 0x7fdba8169620 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdba8154490 0x7fdba8158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdba8154490 0x7fdba8158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40002/0 (socket says 172.21.15.67:40002) 2024-04-25T15:52:24.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 -- 172.21.15.67:0/4019042940 learned_addr learned my addr 172.21.15.67:0/4019042940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:24.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba5d9b640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8169620 0x7fdba81fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:24.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba659c640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdba80c4e30 0x7fdba8163c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:24.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 -- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8169620 msgr2=0x7fdba81fe4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8169620 0x7fdba81fe4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 -- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdba80c4e30 msgr2=0x7fdba8163c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdba80c4e30 0x7fdba8163c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.431+0000 7fdba6d9d640 1 -- 172.21.15.67:0/4019042940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba81fe9f0 con 0x7fdba8154490 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdba6d9d640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdba8154490 0x7fdba8158090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c7b0 tx=0x7fdba000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdb96ffd640 1 -- 172.21.15.67:0/4019042940 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0017070 con 0x7fdba8154490 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdb96ffd640 1 -- 172.21.15.67:0/4019042940 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba0003db0 con 0x7fdba8154490 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdb96ffd640 1 -- 172.21.15.67:0/4019042940 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba000ce60 con 0x7fdba8154490 2024-04-25T15:52:24.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdbace4c640 1 -- 172.21.15.67:0/4019042940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba81629f0 con 0x7fdba8154490 2024-04-25T15:52:24.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdbace4c640 1 -- 172.21.15.67:0/4019042940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba81fdcb0 con 0x7fdba8154490 2024-04-25T15:52:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdb96ffd640 1 -- 172.21.15.67:0/4019042940 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdba001a070 con 0x7fdba8154490 2024-04-25T15:52:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb800052a0 con 0x7fdba8154490 2024-04-25T15:52:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.435+0000 7fdb96ffd640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb8806f450 0x7fdb88071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.439+0000 7fdba659c640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb8806f450 0x7fdb88071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.439+0000 7fdba659c640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb8806f450 0x7fdb88071960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb98000f70 tx=0x7fdb980047f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.447+0000 7fdb96ffd640 1 -- 172.21.15.67:0/4019042940 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5886+0+0 (secure 0 0 0) 0x7fdba008cae0 con 0x7fdba8154490 2024-04-25T15:52:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.447+0000 7fdb96ffd640 1 -- 172.21.15.67:0/4019042940 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdba008ee20 con 0x7fdba8154490 2024-04-25T15:52:24.502 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:24.499+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:24.522 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:24.522 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:24.522 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:24.522 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:24.522 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:24.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:24.603 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:24.599+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:24.703 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:24.703+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:24.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:24.787+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd rm-pg-upmap", "pgid": "3.2f"} v 0) -- 0x7fdb80005860 con 0x7fdba8154490 2024-04-25T15:52:24.797 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:24.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.58336661200001, 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937687, 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212220+0000", 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212245+0000", 2024-04-25T15:52:24.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212458+0000", 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021240899999999999 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212488+0000", 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0715e-05 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212836+0000", 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000348096 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213194+0000", 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000357413 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213591+0000", 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039711599999999998 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224723+0000", 2024-04-25T15:52:24.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011132177 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224892+0000", 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016872100000000001 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225031+0000", 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013966 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225148+0000", 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116168 2024-04-25T15:52:24.802 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.58332031099999, 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013374822, 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212257+0000", 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212263+0000", 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0900000000000001e-06 2024-04-25T15:52:24.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212576+0000", 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031227099999999998 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212590+0000", 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4133e-05 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214544+0000", 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001953939 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214793+0000", 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024941399999999999 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215181+0000", 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038823800000000001 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224739+0000", 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095574579999999996 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225301+0000", 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056223199999999997 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225459+0000", 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015786199999999999 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225631+0000", 2024-04-25T15:52:24.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172242 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.583309898, 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013916951, 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8299999999999996e-07 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212272+0000", 2024-04-25T15:52:24.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6630000000000001e-06 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212665+0000", 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039338399999999999 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212687+0000", 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1882000000000002e-05 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215937+0000", 2024-04-25T15:52:24.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032501119999999999 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032469999999999998 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216646+0000", 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038413800000000001 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224753+0000", 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081070039999999993 2024-04-25T15:52:24.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225911+0000", 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001158002 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226062+0000", 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015119600000000001 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226184+0000", 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121187 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.583301929, 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01435588, 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:24.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6000000000000003e-07 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212279+0000", 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1169999999999997e-06 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212723+0000", 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044335799999999998 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212733+0000", 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349e-05 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217516+0000", 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004782606 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217766+0000", 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025045699999999999 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039180399999999999 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224766+0000", 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066076449999999997 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226308+0000", 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00154228 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226483+0000", 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174938 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226630+0000", 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014766599999999999 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.58329494900001, 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014867514, 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2699999999999999e-07 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212287+0000", 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6229999999999996e-06 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212756+0000", 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046907999999999997 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9560000000000003e-06 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218829+0000", 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060638690000000004 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219182+0000", 2024-04-25T15:52:24.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035291899999999999 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219507+0000", 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032545399999999998 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224779+0000", 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052717279999999998 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226860+0000", 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020809790000000002 2024-04-25T15:52:24.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227036+0000", 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176062 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227149+0000", 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113217 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.822 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.58328747900001, 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015281197, 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212290+0000", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3e-07 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212333+0000", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3251000000000003e-05 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212784+0000", 2024-04-25T15:52:24.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045139399999999998 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212793+0000", 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7960000000000001e-06 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220253+0000", 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074595699999999996 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220482+0000", 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000229554 2024-04-25T15:52:24.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220879+0000", 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039646400000000002 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225181+0000", 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00430205 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227277+0000", 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020961119999999998 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227449+0000", 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017166800000000001 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227570+0000", 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012170799999999999 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.827 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.583237999, 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016239407000000001, 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212340+0000", 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1200000000000001e-06 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212346+0000", 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.584e-06 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212813+0000", 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046677600000000002 2024-04-25T15:52:24.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212821+0000", 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221805+0000", 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089835999999999996 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222154+0000", 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034940299999999997 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222503+0000", 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034907900000000002 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227840+0000", 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367349999999996 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228310+0000", 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047011999999999998 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228439+0000", 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012899999999999999 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228578+0000", 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138837 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.58322678899999, 2024-04-25T15:52:24.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016443972000000001, 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212351+0000", 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212357+0000", 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000003e-06 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212848+0000", 2024-04-25T15:52:24.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049076600000000001 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212863+0000", 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5498999999999998e-05 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223300+0000", 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010437093999999999 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223537+0000", 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000236665 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224027+0000", 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048967900000000003 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228184+0000", 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041571209999999997 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228599+0000", 2024-04-25T15:52:24.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000415392 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228676+0000", 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7095999999999995e-05 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228794+0000", 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011746700000000001 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.58321581, 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013451848000000001, 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:24.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212362+0000", 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212368+0000", 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6229999999999999e-06 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212925+0000", 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055690199999999996 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212960+0000", 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5045000000000001e-05 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213123+0000", 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163222 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213447+0000", 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000323627 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213759+0000", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031232800000000001 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216671+0000", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002911145 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223654+0000", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069831370000000004 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223829+0000", 2024-04-25T15:52:24.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174857 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225813+0000", 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019840679999999999 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 108.583204647, 2024-04-25T15:52:24.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014242332, 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212373+0000", 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4e-07 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212384+0000", 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1443e-05 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213026+0000", 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064212500000000001 2024-04-25T15:52:24.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213045+0000", 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8621999999999999e-05 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214708+0000", 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016628109999999999 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214970+0000", 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026197699999999999 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215380+0000", 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000410789 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218599+0000", 2024-04-25T15:52:24.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032185270000000001 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226296+0000", 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076965269999999999 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226473+0000", 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017755200000000001 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226614+0000", 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:4 2.10 2:0f65c9b8:::60a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.548401313, 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010394548999999999, 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247176+0000", 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2100000000000001e-06 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247209+0000", 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2945999999999999e-05 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247394+0000", 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018468799999999999 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247417+0000", 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2733000000000001e-05 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249734+0000", 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002317684 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250032+0000", 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029727699999999998 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250453+0000", 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042143500000000001 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255742+0000", 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052886809999999999 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257232+0000", 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014906260000000001 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257393+0000", 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016088699999999999 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257570+0000", 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017638199999999999 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464432747000004, 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094210270000000002, 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6699999999999999e-07 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331165+0000", 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331418+0000", 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271799999999997 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331428+0000", 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6460000000000005e-06 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333229+0000", 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018009499999999999 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334396+0000", 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011673269999999999 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334597+0000", 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020135300000000001 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339905+0000", 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053077610000000002 2024-04-25T15:52:24.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340210+0000", 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030499999999999999 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340385+0000", 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017450199999999999 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340565+0000", 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018011700000000001 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.857 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464408000999995, 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095762039999999996, 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:24.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331174+0000", 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-06 2024-04-25T15:52:24.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331467+0000", 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029356 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331478+0000", 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336604+0000", 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051267170000000003 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336735+0000", 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013052699999999999 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336996+0000", 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260511 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339986+0000", 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002990487 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340579+0000", 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059299899999999998 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340673+0000", 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4241999999999997e-05 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340745+0000", 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1484999999999997e-05 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:17 2.10 2:0e9da45e:::61b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464400755, 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010814343000000001, 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331180+0000", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6650000000000001e-06 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331501+0000", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032061600000000002 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331509+0000", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8240000000000006e-06 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333383+0000", 2024-04-25T15:52:24.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018733020000000001 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333652+0000", 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026890399999999999 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333865+0000", 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021341 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338313+0000", 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044478269999999997 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341868+0000", 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035546729999999999 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341920+0000", 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2507999999999999e-05 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341990+0000", 2024-04-25T15:52:24.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0110000000000005e-05 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464391082999995, 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011086621, 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:24.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331186+0000", 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.51e-07 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331191+0000", 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3970000000000002e-06 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331546+0000", 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035447799999999998 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331557+0000", 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0791000000000001e-05 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334408+0000", 2024-04-25T15:52:24.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028516520000000001 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334546+0000", 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013802700000000001 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334877+0000", 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000331081 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339910+0000", 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050323449999999997 2024-04-25T15:52:24.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342080+0000", 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021703899999999999 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342184+0000", 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103767 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342272+0000", 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8242e-05 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464383063, 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010331643, 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331194+0000", 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331198+0000", 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1400000000000002e-06 2024-04-25T15:52:24.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331608+0000", 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040962799999999999 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331620+0000", 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337545+0000", 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005924726 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338432+0000", 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088720699999999999 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338639+0000", 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020665000000000001 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341318+0000", 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026785860000000002 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341406+0000", 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8344000000000007e-05 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341458+0000", 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805999999999998e-05 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341525+0000", 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7336999999999996e-05 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:2 2.10 2:0daba267:::402.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464376603999995, 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011245289, 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331215+0000", 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:24.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331647+0000", 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043194799999999998 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331656+0000", 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335316+0000", 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036607710000000002 2024-04-25T15:52:24.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335504+0000", 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018793299999999999 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335699+0000", 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194544 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339945+0000", 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042457600000000003 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342333+0000", 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023885730000000002 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342383+0000", 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9314999999999998e-05 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342445+0000", 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2691e-05 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464357843000002, 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087706009999999994, 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331239+0000", 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9653e-05 2024-04-25T15:52:24.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331678+0000", 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043868999999999999 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331701+0000", 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3019000000000001e-05 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333753+0000", 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020522050000000001 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334131+0000", 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000378484 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334455+0000", 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032319899999999998 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337268+0000", 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028133199999999998 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339816+0000", 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025479140000000001 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339907+0000", 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1150000000000007e-05 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339989+0000", 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2392000000000007e-05 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:27 2.10 2:091e33df:::502.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464334801999996, 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011447874, 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4199999999999996e-07 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331249+0000", 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2259999999999999e-06 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331730+0000", 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048009600000000001 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331739+0000", 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1840000000000005e-06 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336653+0000", 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049141510000000003 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336830+0000", 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017740099999999999 2024-04-25T15:52:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337033+0000", 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020295999999999999 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340803+0000", 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037692480000000002 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342567+0000", 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017649250000000001 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342650+0000", 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2748000000000001e-05 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342690+0000", 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9393e-05 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.892 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:31 2.10 2:0eee8b97:::202.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 92.464324945000001, 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012346048, 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:24.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5300000000000004e-07 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331255+0000", 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8380000000000002e-06 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331760+0000", 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050452600000000004 2024-04-25T15:52:24.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331768+0000", 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337613+0000", 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058448700000000003 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337836+0000", 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000223043 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:24.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.343598+0000", 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057612679999999999 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:24.896 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:24.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:24.898 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:24.895+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:24.998 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:24.995+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:25.044 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:25.044 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:25.045 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:25.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.051+0000 7fdb96ffd640 1 -- 172.21.15.67:0/4019042940 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap", "pgid": "3.2f"}]=0 clear 3.2f pg_upmap mapping v51) ==== 106+0+0 (secure 0 0 0) 0x7fdba005ae60 con 0x7fdba8154490 2024-04-25T15:52:25.054 INFO:teuthology.orchestra.run.smithi067.stderr:clear 3.2f pg_upmap mapping 2024-04-25T15:52:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb8806f450 msgr2=0x7fdb88071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb8806f450 0x7fdb88071960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb98000f70 tx=0x7fdb980047f0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdba8154490 msgr2=0x7fdba8158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdba8154490 0x7fdba8158090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c7b0 tx=0x7fdba000cc70 comp rx=0 tx=0).stop 2024-04-25T15:52:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 shutdown_connections 2024-04-25T15:52:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdb8806f450 0x7fdb88071960 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdba8154490 0x7fdba8158090 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdba8169620 0x7fdba81fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 --2- 172.21.15.67:0/4019042940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdba80c4e30 0x7fdba8163c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 >> 172.21.15.67:0/4019042940 conn(0x7fdba80700b0 msgr2=0x7fdba806de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 shutdown_connections 2024-04-25T15:52:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.055+0000 7fdb94ff9640 1 -- 172.21.15.67:0/4019042940 wait complete. 2024-04-25T15:52:25.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:25.130 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:25.127+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:25.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:52:25.199 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:25.199+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:25.205 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:25.205 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:25.205 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:25.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:25.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:25.301 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:25.299+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:25.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.343+0000 7f5128e0c640 1 Processor -- start 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.343+0000 7f5128e0c640 1 -- start start 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5128e0c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f51240c35a0 0x7f51240c9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5128e0c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f512406c640 0x7f512406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5128e0c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5124068bf0 con 0x7f5124161740 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5128e0c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5124068630 con 0x7f512406c640 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5128e0c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5124068910 con 0x7f51240c35a0 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5122d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f512406c640 0x7f512406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5122d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f512406c640 0x7f512406ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40006/0 (socket says 172.21.15.67:40006) 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5122d9d640 1 -- 172.21.15.67:0/216600125 learned_addr learned my addr 172.21.15.67:0/216600125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f512259c640 1 --2- 172.21.15.67:0/216600125 >> v2:172.21.15.176:3300/0 conn(0x7f51240c35a0 0x7f51240c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5122d9d640 1 -- 172.21.15.67:0/216600125 >> v2:172.21.15.176:3300/0 conn(0x7f51240c35a0 msgr2=0x7f51240c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5122d9d640 1 --2- 172.21.15.67:0/216600125 >> v2:172.21.15.176:3300/0 conn(0x7f51240c35a0 0x7f51240c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5122d9d640 1 -- 172.21.15.67:0/216600125 >> v1:172.21.15.176:6789/0 conn(0x7f5124161740 legacy=0x7f512415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.347+0000 7f5122d9d640 1 -- 172.21.15.67:0/216600125 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51240c8c60 con 0x7f512406c640 2024-04-25T15:52:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5122d9d640 1 --2- 172.21.15.67:0/216600125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f512406ca50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f51240cbd80 tx=0x7f511c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be391d1fb66c9b90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f512159a640 1 -- 172.21.15.67:0/216600125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0a56f0 con 0x7f512406c640 2024-04-25T15:52:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f512159a640 1 -- 172.21.15.67:0/216600125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511c0a5890 con 0x7f512406c640 2024-04-25T15:52:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f512159a640 1 -- 172.21.15.67:0/216600125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0a5c60 con 0x7f512406c640 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 -- 172.21.15.67:0/216600125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 msgr2=0x7f512406ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 --2- 172.21.15.67:0/216600125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f512406ca50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f51240cbd80 tx=0x7f511c01c940 comp rx=0 tx=0).stop 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 -- 172.21.15.67:0/216600125 shutdown_connections 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 --2- 172.21.15.67:0/216600125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f512406ca50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 --2- 172.21.15.67:0/216600125 >> v2:172.21.15.176:3300/0 conn(0x7f51240c35a0 0x7f51240c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 -- 172.21.15.67:0/216600125 >> 172.21.15.67:0/216600125 conn(0x7f51240ce1c0 msgr2=0x7f51240ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 -- 172.21.15.67:0/216600125 shutdown_connections 2024-04-25T15:52:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.351+0000 7f5128e0c640 1 -- 172.21.15.67:0/216600125 wait complete. 2024-04-25T15:52:25.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.355+0000 7f5128e0c640 1 Processor -- start 2024-04-25T15:52:25.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.355+0000 7f5128e0c640 1 -- start start 2024-04-25T15:52:25.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5128e0c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f5124143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5128e0c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51240c35a0 0x7f5124144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5128e0c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5124161740 0x7f5124144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f5124143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f5124143990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40014/0 (socket says 172.21.15.67:40014) 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 -- 172.21.15.67:0/1464704643 learned_addr learned my addr 172.21.15.67:0/1464704643 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f51240caaf0 con 0x7f512406c640 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f51240ca970 con 0x7f5124161740 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f512416bd90 con 0x7f51240c35a0 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5122d9d640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5124161740 0x7f5124144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f5121d9b640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51240c35a0 0x7f5124144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 -- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51240c35a0 msgr2=0x7f5124144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51240c35a0 0x7f5124144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 -- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5124161740 msgr2=0x7f5124144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5124161740 0x7f5124144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51240c8190 con 0x7f512406c640 2024-04-25T15:52:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.359+0000 7f512259c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f5124143990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f511400e8a0 tx=0x7f511400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5112ffd640 1 -- 172.21.15.67:0/1464704643 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5114017070 con 0x7f512406c640 2024-04-25T15:52:25.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51240c8470 con 0x7f512406c640 2024-04-25T15:52:25.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5124143470 con 0x7f512406c640 2024-04-25T15:52:25.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5112ffd640 1 -- 172.21.15.67:0/1464704643 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f511400c940 con 0x7f512406c640 2024-04-25T15:52:25.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5124142fc0 con 0x7f512406c640 2024-04-25T15:52:25.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5112ffd640 1 -- 172.21.15.67:0/1464704643 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511400ccf0 con 0x7f512406c640 2024-04-25T15:52:25.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5112ffd640 1 -- 172.21.15.67:0/1464704643 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f511401a070 con 0x7f512406c640 2024-04-25T15:52:25.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.363+0000 7f5112ffd640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510406f450 0x7f5104071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.367+0000 7f5121d9b640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510406f450 0x7f5104071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.367+0000 7f5121d9b640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510406f450 0x7f5104071960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f512415aa70 tx=0x7f5118007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.375+0000 7f5112ffd640 1 -- 172.21.15.67:0/1464704643 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5857+0+0 (secure 0 0 0) 0x7f511408cf30 con 0x7f512406c640 2024-04-25T15:52:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.375+0000 7f5112ffd640 1 -- 172.21.15.67:0/1464704643 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5114098080 con 0x7f512406c640 2024-04-25T15:52:25.401 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:25.395+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:33 2.1b 2:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.206909086, 2024-04-25T15:52:25.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058242010000000002, 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213732+0000", 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213743+0000", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1005999999999999e-05 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213761+0000", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8196e-05 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213870+0000", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010906800000000001 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213922+0000", 2024-04-25T15:52:25.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1790999999999999e-05 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214226+0000", 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030378099999999998 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214618+0000", 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000391894 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219556+0000", 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049384650000000004 2024-04-25T15:52:25.425 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:37 2.1e 2:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.20686944099999, 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012953823, 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:25.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213772+0000", 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9999999999999996e-07 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213785+0000", 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2069000000000001e-05 2024-04-25T15:52:25.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214018+0000", 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023360199999999999 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214040+0000", 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1965999999999999e-05 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214206+0000", 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016610400000000001 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214608+0000", 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040218699999999998 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226725+0000", 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012116995 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:22 2.e e20/17 2:717a0223:::608.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.20396241900001, 2024-04-25T15:52:25.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063622239999999997, 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216679+0000", 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216681+0000", 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2460000000000002e-06 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216700+0000", 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8825999999999999e-05 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216741+0000", 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1641000000000003e-05 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216767+0000", 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5896000000000001e-05 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217114+0000", 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034709600000000003 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217174+0000", 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9221000000000001e-05 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222907+0000", 2024-04-25T15:52:25.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057338509999999999 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223041+0000", 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133447 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:23 2.e e20/17 2:717a0223:::608.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.203809483, 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006306613, 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.433 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216832+0000", 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216833+0000", 2024-04-25T15:52:25.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.35e-06 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216850+0000", 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7316000000000002e-05 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216878+0000", 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7957999999999999e-05 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0375999999999999e-05 2024-04-25T15:52:25.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217888+0000", 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00098981999999999998 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217945+0000", 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6403999999999997e-05 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223049+0000", 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051039700000000002 2024-04-25T15:52:25.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223138+0000", 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9419000000000006e-05 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:18 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.20229383900001, 2024-04-25T15:52:25.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046950000000000004, 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218347+0000", 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218350+0000", 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5770000000000001e-06 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218372+0000", 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1759e-05 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218405+0000", 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3244999999999998e-05 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218430+0000", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5079000000000002e-05 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218591+0000", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016135800000000001 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218642+0000", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1134999999999997e-05 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222894+0000", 2024-04-25T15:52:25.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042512510000000002 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223042+0000", 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000148596 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:19 2.14 e20/17 2:2f55791f:::606.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.202151192, 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005754767, 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.441 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218490+0000", 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218491+0000", 2024-04-25T15:52:25.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2330000000000001e-06 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218505+0000", 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4399000000000001e-05 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218529+0000", 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3676000000000002e-05 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218549+0000", 2024-04-25T15:52:25.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0109e-05 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219378+0000", 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082895500000000003 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219435+0000", 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6294e-05 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224140+0000", 2024-04-25T15:52:25.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047056529999999997 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224245+0000", 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104448 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:1 2.14 e20/17 2:292cf221:::200.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.202048101, 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057550209999999999, 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.445 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218593+0000", 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218594+0000", 2024-04-25T15:52:25.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.173e-06 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218656+0000", 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1704000000000004e-05 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218681+0000", 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5072e-05 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218701+0000", 2024-04-25T15:52:25.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0075999999999999e-05 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220043+0000", 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013420730000000001 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220100+0000", 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7219999999999998e-05 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224251+0000", 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041508509999999997 2024-04-25T15:52:25.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224348+0000", 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6852e-05 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:3 2.14 e20/17 2:287034c4:::1.00000000.inode:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.201895351, 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056846830000000003, 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.449 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218746+0000", 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218747+0000", 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2389999999999999e-06 2024-04-25T15:52:25.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218816+0000", 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9423000000000003e-05 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218841+0000", 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218860+0000", 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9335999999999999e-05 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220944+0000", 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020839019999999999 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221003+0000", 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8740999999999999e-05 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224354+0000", 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033505169999999999 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224430+0000", 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6729000000000003e-05 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.14 epoch 20/17 rep_tid 7 entries 20'5 (0'0) error 2:2e2fa760:::200.00000001:head by mds.0.4:32 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 109.19994712499999, 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037786650000000001, 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220694+0000", 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220696+0000", 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220723+0000", 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7090999999999999e-05 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220797+0000", 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3386999999999994e-05 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220875+0000", 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8578999999999994e-05 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222046+0000", 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011710220000000001 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224473+0000", 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024263790000000002 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "pg_update_log_missing(2.1 epoch 20/17 rep_tid 11 entries 20'1 (0'0) error 2:810cebc8:::201.00000004:head by mds.1.10:31 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.17191949399999, 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039111700000000003, 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248722+0000", 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248724+0000", 2024-04-25T15:52:25.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2699999999999999e-06 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248749+0000", 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5316e-05 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248817+0000", 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8283999999999999e-05 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248875+0000", 2024-04-25T15:52:25.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7654000000000001e-05 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249239+0000", 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000364354 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252633+0000", 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003393292 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.459 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:11 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.169896603, 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032814200000000002, 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250744+0000", 2024-04-25T15:52:25.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250747+0000", 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1890000000000001e-06 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250784+0000", 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7082000000000002e-05 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250817+0000", 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3025000000000003e-05 2024-04-25T15:52:25.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250845+0000", 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8079e-05 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251154+0000", 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030885999999999999 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251214+0000", 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9874999999999998e-05 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253903+0000", 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002689922 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254026+0000", 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012238799999999999 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.463 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:12 2.12 e20/17 2:48cd7578:::60e.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.16972605399999, 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032445320000000001, 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250915+0000", 2024-04-25T15:52:25.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250916+0000", 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3629999999999999e-06 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250932+0000", 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5563e-05 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250957+0000", 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4672e-05 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250977+0000", 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0268999999999999e-05 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251870+0000", 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00089335200000000003 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251925+0000", 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254036+0000", 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021114340000000001 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254160+0000", 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012336500000000001 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:24 2.1b 2:dbae76b5:::mds1_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.167370558, 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077994370000000002, 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253270+0000", 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253274+0000", 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8700000000000002e-06 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253295+0000", 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0792999999999999e-05 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253387+0000", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2056000000000001e-05 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253415+0000", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7849000000000001e-05 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253727+0000", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031149599999999997 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254094+0000", 2024-04-25T15:52:25.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036768100000000002 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254538+0000", 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000443545 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258012+0000", 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003474441 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260845+0000", 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028323419999999998 2024-04-25T15:52:25.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260948+0000", 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103259 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.261070+0000", 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012210500000000001 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:28 2.1b 2:db457a8a:::201.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:25.472 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.167335846, 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099959000000000003, 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253305+0000", 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253306+0000", 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253321+0000", 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4266e-05 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253500+0000", 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017927199999999999 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253521+0000", 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1299000000000001e-05 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255909+0000", 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023879249999999999 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256412+0000", 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050270499999999999 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263301+0000", 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068891830000000001 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:29 2.1b 2:d80f8093:::201.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:25.476 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 103.16731503699999, 2024-04-25T15:52:25.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010276995000000001, 2024-04-25T15:52:25.477 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.477 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.477 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:25.477 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:25.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253326+0000", 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.003e-06 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253334+0000", 2024-04-25T15:52:25.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3370000000000002e-06 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253590+0000", 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025537500000000001 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253610+0000", 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0136000000000002e-05 2024-04-25T15:52:25.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257193+0000", 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035835340000000002 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257421+0000", 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022810300000000001 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.263603+0000", 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061815070000000001 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:15 2.19 2:9ec52373:::61a.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 93.090427989999995, 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060955260000000004, 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.482 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330213+0000", 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330216+0000", 2024-04-25T15:52:25.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0489999999999999e-06 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330230+0000", 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4405e-05 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330280+0000", 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9709999999999997e-05 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330298+0000", 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7672999999999998e-05 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330498+0000", 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019969000000000001 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330731+0000", 2024-04-25T15:52:25.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000233577 2024-04-25T15:52:25.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:25.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330935+0000", 2024-04-25T15:52:25.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020401100000000001 2024-04-25T15:52:25.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335053+0000", 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041174319999999999 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336097+0000", 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001044618 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336228+0000", 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130615 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336309+0000", 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0746000000000002e-05 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.488 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:16 2.19 2:9ec52373:::61a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 93.090403907999999, 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006325977, 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330237+0000", 2024-04-25T15:52:25.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330238+0000", 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1200000000000002e-07 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330244+0000", 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8300000000000001e-06 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330360+0000", 2024-04-25T15:52:25.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115938 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330373+0000", 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2986000000000001e-05 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331576+0000", 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012036390000000001 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331708+0000", 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013137399999999999 2024-04-25T15:52:25.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331959+0000", 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025145199999999999 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335065+0000", 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031059690000000001 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336412+0000", 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013468919999999999 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336497+0000", 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4844000000000003e-05 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336563+0000", 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6241000000000005e-05 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:29 2.1e 2:7d92db23:::202.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 93.090394211000003, 2024-04-25T15:52:25.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052691969999999998, 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330247+0000", 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330248+0000", 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.16e-07 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330252+0000", 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4100000000000001e-06 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330414+0000", 2024-04-25T15:52:25.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016137200000000001 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330425+0000", 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1667e-05 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330569+0000", 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014417600000000001 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330789+0000", 2024-04-25T15:52:25.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021975200000000001 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335516+0000", 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0047269039999999997 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:13 2.1 e20/17 2:81735765:::619.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 93.084499273999995, 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0033919140000000002, 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.497 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336142+0000", 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336143+0000", 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3850000000000001e-06 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336158+0000", 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4533e-05 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336177+0000", 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9215000000000002e-05 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336189+0000", 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2157999999999999e-05 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336626+0000", 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000437248 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336659+0000", 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2771000000000002e-05 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339424+0000", 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027649229999999999 2024-04-25T15:52:25.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339534+0000", 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109681 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:27 2.19 2:9e6a133d:::104.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 81.495060616999993, 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038114049999999999, 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:25.501 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925580+0000", 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:25.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925583+0000", 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.305e-06 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925601+0000", 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8476000000000001e-05 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925632+0000", 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0429999999999998e-05 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925658+0000", 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6570000000000001e-05 2024-04-25T15:52:25.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925754+0000", 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5792999999999996e-05 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925906+0000", 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015155799999999999 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926076+0000", 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017026000000000001 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928033+0000", 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001957674 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929233+0000", 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001199858 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929323+0000", 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9794000000000001e-05 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:25.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929392+0000", 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8687000000000002e-05 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:25.506 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.439+0000 7f8114b2e640 1 Processor -- start 2024-04-25T15:52:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.439+0000 7f8114b2e640 1 -- start start 2024-04-25T15:52:25.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.439+0000 7f8114b2e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f81100caf60 0x7f811015d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.439+0000 7f8114b2e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f811015d790 0x7f811006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.439+0000 7f8114b2e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81100c77a0 con 0x7f8110073bf0 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.439+0000 7f8114b2e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f81100cab60 con 0x7f81100caf60 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.439+0000 7f8114b2e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8110159620 con 0x7f811015d790 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f811015d790 0x7f811006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f810e59c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f81100caf60 0x7f811015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f811015d790 0x7f811006d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:47382/0 (socket says 172.21.15.67:47382) 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 -- 172.21.15.67:0/2259544061 learned_addr learned my addr 172.21.15.67:0/2259544061 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 -- 172.21.15.67:0/2259544061 >> v2:172.21.15.176:3300/0 conn(0x7f81100caf60 msgr2=0x7f811015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 --2- 172.21.15.67:0/2259544061 >> v2:172.21.15.176:3300/0 conn(0x7f81100caf60 0x7f811015d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 -- 172.21.15.67:0/2259544061 >> v1:172.21.15.176:6789/0 conn(0x7f8110073bf0 legacy=0x7f8110072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 -- 172.21.15.67:0/2259544061 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81100c7a60 con 0x7f811015d790 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f8107fff640 1 --2- 172.21.15.67:0/2259544061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f811015d790 0x7f811006d750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009870 tx=0x7f80f801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f18e50be26785ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f810dd9b640 1 -- 172.21.15.67:0/2259544061 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f80b2070 con 0x7f811015d790 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f810dd9b640 1 -- 172.21.15.67:0/2259544061 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f80a9070 con 0x7f811015d790 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.443+0000 7f810dd9b640 1 -- 172.21.15.67:0/2259544061 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f80ad880 con 0x7f811015d790 2024-04-25T15:52:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- 172.21.15.67:0/2259544061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f811015d790 msgr2=0x7f811006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 --2- 172.21.15.67:0/2259544061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f811015d790 0x7f811006d750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009870 tx=0x7f80f801c560 comp rx=0 tx=0).stop 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- 172.21.15.67:0/2259544061 shutdown_connections 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 --2- 172.21.15.67:0/2259544061 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f811015d790 0x7f811006d750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 --2- 172.21.15.67:0/2259544061 >> v2:172.21.15.176:3300/0 conn(0x7f81100caf60 0x7f811015d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- 172.21.15.67:0/2259544061 >> 172.21.15.67:0/2259544061 conn(0x7f8110070db0 msgr2=0x7f81100ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- 172.21.15.67:0/2259544061 shutdown_connections 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- 172.21.15.67:0/2259544061 wait complete. 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 Processor -- start 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- start start 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8110073bf0 0x7f8110149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81100caf60 0x7f8110150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811015d790 0x7f81101468e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8110167060 con 0x7f811015d790 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8110166ee0 con 0x7f8110073bf0 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8114b2e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f81101671e0 con 0x7f81100caf60 2024-04-25T15:52:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f810ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811015d790 0x7f81101468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f810ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811015d790 0x7f81101468e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40028/0 (socket says 172.21.15.67:40028) 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f810ed9d640 1 -- 172.21.15.67:0/3537559059 learned_addr learned my addr 172.21.15.67:0/3537559059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f8107fff640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81100caf60 0x7f8110150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.447+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8110073bf0 0x7f8110149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f810ed9d640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81100caf60 msgr2=0x7f8110150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f810ed9d640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81100caf60 0x7f8110150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f810ed9d640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8110073bf0 msgr2=0x7f8110149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f810ed9d640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8110073bf0 0x7f8110149530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f810ed9d640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8110148000 con 0x7f811015d790 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8110073bf0 0x7f8110149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f810ed9d640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811015d790 0x7f81101468e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f810800c790 tx=0x7f810800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108003ba0 con 0x7f811015d790 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8108003d40 con 0x7f811015d790 2024-04-25T15:52:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8114b2e640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8110159ce0 con 0x7f811015d790 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8108005310 con 0x7f811015d790 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8114b2e640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8110148270 con 0x7f811015d790 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f81080054b0 con 0x7f811015d790 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f811006a480 con 0x7f811015d790 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.451+0000 7f8106ffd640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f80f006f430 0x7f80f0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5857+0+0 (secure 0 0 0) 0x7f810808eef0 con 0x7f811015d790 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f80f006f430 0x7f80f0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v51) ==== 54+0+16 (secure 0 0 0) 0x7f810805be60 con 0x7f811015d790 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f811014ac90 0x7f81101f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f811015c9a0 con 0x7f811014ac90 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f8107fff640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f811014ac90 0x7f81101f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f80f006f430 0x7f80f0071940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8100010d30 tx=0x7f8100008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f8107fff640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f811014ac90 0x7f81101f7760 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.455+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f811015c9a0 con 0x7f811014ac90 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.471+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f81100c8a40 con 0x7f811014ac90 2024-04-25T15:52:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.471+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f81100c8a40 con 0x7f811014ac90 2024-04-25T15:52:25.516 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:25.517 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:25.507+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.471+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8110148a70 0x7f811014ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.471+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8110159220 con 0x7f8110148a70 2024-04-25T15:52:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.471+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8110148a70 0x7f811014ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.475+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8110148a70 0x7f811014ef70 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.475+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8110159220 con 0x7f8110148a70 2024-04-25T15:52:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.487+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8110147330 con 0x7f8110148a70 2024-04-25T15:52:25.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.491+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8110147330 con 0x7f8110148a70 2024-04-25T15:52:25.518 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:25.519 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:25.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.491+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8110063090 0x7f81100634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.491+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8110151490 con 0x7f8110063090 2024-04-25T15:52:25.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.491+0000 7f8107fff640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8110063090 0x7f81100634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.491+0000 7f8107fff640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8110063090 0x7f81100634a0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.491+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8110151490 con 0x7f8110063090 2024-04-25T15:52:25.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.519+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f81100639e0 con 0x7f8110063090 2024-04-25T15:52:25.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.523+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f81100639e0 con 0x7f8110063090 2024-04-25T15:52:25.527 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:25.527 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:25.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.523+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f81100c93a0 0x7f81100c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.523+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8110063fb0 con 0x7f81100c93a0 2024-04-25T15:52:25.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.523+0000 7f810ed9d640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f81100c93a0 0x7f81100c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.523+0000 7f810ed9d640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f81100c93a0 0x7f81100c97b0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.523+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8110063fb0 con 0x7f81100c93a0 2024-04-25T15:52:25.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.551+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f81100c9f40 con 0x7f81100c93a0 2024-04-25T15:52:25.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.555+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f81100c9f40 con 0x7f81100c93a0 2024-04-25T15:52:25.558 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.555+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8110204fd0 0x7f81102073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.555+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f811020a1c0 con 0x7f8110204fd0 2024-04-25T15:52:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.555+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8110204fd0 0x7f81102073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.555+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8110204fd0 0x7f81102073f0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.555+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f811020a1c0 con 0x7f8110204fd0 2024-04-25T15:52:25.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.583+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f8110207930 con 0x7f8110204fd0 2024-04-25T15:52:25.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.587+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8110207930 con 0x7f8110204fd0 2024-04-25T15:52:25.592 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:25.592 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:25.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.587+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8110207e40 0x7f811020a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.587+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f811020d120 con 0x7f8110207e40 2024-04-25T15:52:25.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.587+0000 7f8107fff640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8110207e40 0x7f811020a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.587+0000 7f8107fff640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8110207e40 0x7f811020a260 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.591+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f811020d120 con 0x7f8110207e40 2024-04-25T15:52:25.608 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:25.603+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:25.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.615+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f811020a7a0 con 0x7f8110207e40 2024-04-25T15:52:25.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.619+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f811020a7a0 con 0x7f8110207e40 2024-04-25T15:52:25.625 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:25.626 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:25.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.619+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f811020aec0 0x7f811020b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.619+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f81102104f0 con 0x7f811020aec0 2024-04-25T15:52:25.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.623+0000 7f810ed9d640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f811020aec0 0x7f811020b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.623+0000 7f810ed9d640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f811020aec0 0x7f811020b2d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.623+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81102104f0 con 0x7f811020aec0 2024-04-25T15:52:25.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.651+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f811020f370 con 0x7f811020aec0 2024-04-25T15:52:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.655+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f811020f370 con 0x7f811020aec0 2024-04-25T15:52:25.659 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.655+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f811020f9f0 0x7f811020fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.655+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f81102138c0 con 0x7f811020f9f0 2024-04-25T15:52:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.655+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f811020f9f0 0x7f811020fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.655+0000 7f810e59c640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f811020f9f0 0x7f811020fe60 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:25.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.659+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81102138c0 con 0x7f811020f9f0 2024-04-25T15:52:25.676 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:25.676 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:25.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.683+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f811020b810 con 0x7f811020f9f0 2024-04-25T15:52:25.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:25.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.687+0000 7f8106ffd640 1 -- 172.21.15.67:0/3537559059 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f811020b810 con 0x7f811020f9f0 2024-04-25T15:52:25.694 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f811014ac90 msgr2=0x7f81101f7760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f811014ac90 0x7f81101f7760 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8110148a70 msgr2=0x7f811014ef70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8110148a70 0x7f811014ef70 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8110063090 msgr2=0x7f81100634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8110063090 0x7f81100634a0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f81100c93a0 msgr2=0x7f81100c97b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f81100c93a0 0x7f81100c97b0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8110204fd0 msgr2=0x7f81102073f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8110204fd0 0x7f81102073f0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8110207e40 msgr2=0x7f811020a260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8110207e40 0x7f811020a260 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f810ed9d640 1 -- 172.21.15.67:0/3537559059 reap_dead start 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f811020aec0 msgr2=0x7f811020b2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f811020aec0 0x7f811020b2d0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f810ed9d640 1 -- 172.21.15.67:0/3537559059 reap_dead start 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f811020f9f0 msgr2=0x7f811020fe60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f811020f9f0 0x7f811020fe60 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f810ed9d640 1 -- 172.21.15.67:0/3537559059 reap_dead start 2024-04-25T15:52:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f80f006f430 msgr2=0x7f80f0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f80f006f430 0x7f80f0071940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8100010d30 tx=0x7f8100008040 comp rx=0 tx=0).stop 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811015d790 msgr2=0x7f81101468e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811015d790 0x7f81101468e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f810800c790 tx=0x7f810800cc50 comp rx=0 tx=0).stop 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 shutdown_connections 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f80f006f430 0x7f80f0071940 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 --2- 172.21.15.67:0/3537559059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f811015d790 0x7f81101468e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 >> 172.21.15.67:0/3537559059 conn(0x7f8110070db0 msgr2=0x7f81101536f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 shutdown_connections 2024-04-25T15:52:25.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.691+0000 7f8104ff9640 1 -- 172.21.15.67:0/3537559059 wait complete. 2024-04-25T15:52:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:25.715+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f512406a480 con 0x7f512406c640 2024-04-25T15:52:25.743 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:25.743+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:25.810 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:25.807+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:25.844 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:25.844 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:25.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:25.911 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:25.907+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:26.012 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:26.007+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:26.048 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:26.048 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.055+0000 7f5112ffd640 1 -- 172.21.15.67:0/1464704643 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v52) ==== 101+0+0 (secure 0 0 0) 0x7f511405b220 con 0x7f512406c640 2024-04-25T15:52:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510406f450 msgr2=0x7f5104071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510406f450 0x7f5104071960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f512415aa70 tx=0x7f5118007e40 comp rx=0 tx=0).stop 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 msgr2=0x7f5124143990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f5124143990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f511400e8a0 tx=0x7f511400eca0 comp rx=0 tx=0).stop 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 shutdown_connections 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f510406f450 0x7f5104071960 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5124161740 0x7f5124144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51240c35a0 0x7f5124144800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 --2- 172.21.15.67:0/1464704643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f512406c640 0x7f5124143990 secure :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f511400e8a0 tx=0x7f511400eca0 comp rx=0 tx=0).stop 2024-04-25T15:52:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 >> 172.21.15.67:0/1464704643 conn(0x7f51240ce1c0 msgr2=0x7f51240ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 shutdown_connections 2024-04-25T15:52:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:26.059+0000 7f5128e0c640 1 -- 172.21.15.67:0/1464704643 wait complete. 2024-04-25T15:52:26.070 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:26.115 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:26.115+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:26.213 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:26.207+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:26.213 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:26.213 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:26.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:26.315 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:26.315+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:26.413 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:26.411+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:26.416 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:26.416 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:26.434 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:26.515 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:26.511+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:26.615 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:26.611+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:26.625 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:26.626 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:26.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:26.716 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:26.711+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:26.792 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:26.792 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:26.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:26.817 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:26.811+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:26.918 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:26.915+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:26.958 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:26.958 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:26.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:27.018 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:27.015+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:27.119 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:27.119+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:27.124 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:27.124 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:27.141 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:27.227 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:27.227+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:27.320 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:27.314+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:27.334 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:27.335 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:27.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:27.421 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:27.414+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:27.522 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:27.519+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:27.531 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:27.532 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:27.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:27.622 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:27.618+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:27.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:27.722 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:27.718+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:27.732 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:27.732 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:27.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:27.823 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:27.818+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:27.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.910+0000 7eff9526e640 1 Processor -- start 2024-04-25T15:52:27.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.910+0000 7eff9526e640 1 -- start start 2024-04-25T15:52:27.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.910+0000 7eff9526e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff90160800 0x7eff9006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.910+0000 7eff9526e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7eff900c92d0 0x7eff90072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.910+0000 7eff9526e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9006d8e0 con 0x7eff900c96e0 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.910+0000 7eff9526e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7eff9006dbc0 con 0x7eff90160800 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.910+0000 7eff9526e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7eff90073c70 con 0x7eff900c92d0 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff90160800 0x7eff9006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff90160800 0x7eff9006d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40036/0 (socket says 172.21.15.67:40036) 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8effd640 1 -- 172.21.15.67:0/3784200773 learned_addr learned my addr 172.21.15.67:0/3784200773 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/3784200773 >> v2:172.21.15.176:3300/0 conn(0x7eff900c92d0 0x7eff90072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8effd640 1 -- 172.21.15.67:0/3784200773 >> v2:172.21.15.176:3300/0 conn(0x7eff900c92d0 msgr2=0x7eff90072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8effd640 1 --2- 172.21.15.67:0/3784200773 >> v2:172.21.15.176:3300/0 conn(0x7eff900c92d0 0x7eff90072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8effd640 1 -- 172.21.15.67:0/3784200773 >> v1:172.21.15.176:6789/0 conn(0x7eff900c96e0 legacy=0x7eff9015c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.914+0000 7eff8effd640 1 -- 172.21.15.67:0/3784200773 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff90073f50 con 0x7eff90160800 2024-04-25T15:52:27.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff8effd640 1 --2- 172.21.15.67:0/3784200773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff90160800 0x7eff9006d030 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7eff9006dd40 tx=0x7eff8001c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f063914e7251e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff8dffb640 1 -- 172.21.15.67:0/3784200773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff800aa650 con 0x7eff90160800 2024-04-25T15:52:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff8dffb640 1 -- 172.21.15.67:0/3784200773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff800af070 con 0x7eff90160800 2024-04-25T15:52:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff8dffb640 1 -- 172.21.15.67:0/3784200773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff800aaa10 con 0x7eff90160800 2024-04-25T15:52:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 -- 172.21.15.67:0/3784200773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff90160800 msgr2=0x7eff9006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 --2- 172.21.15.67:0/3784200773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff90160800 0x7eff9006d030 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7eff9006dd40 tx=0x7eff8001c6f0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 -- 172.21.15.67:0/3784200773 shutdown_connections 2024-04-25T15:52:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 --2- 172.21.15.67:0/3784200773 >> v2:172.21.15.176:3300/0 conn(0x7eff900c92d0 0x7eff90072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 --2- 172.21.15.67:0/3784200773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff90160800 0x7eff9006d030 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 -- 172.21.15.67:0/3784200773 >> 172.21.15.67:0/3784200773 conn(0x7eff90070db0 msgr2=0x7eff900ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 -- 172.21.15.67:0/3784200773 shutdown_connections 2024-04-25T15:52:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.918+0000 7eff9526e640 1 -- 172.21.15.67:0/3784200773 wait complete. 2024-04-25T15:52:27.924 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:27.923+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 Processor -- start 2024-04-25T15:52:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 -- start start 2024-04-25T15:52:27.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff900c92d0 0x7eff90157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff900c96e0 0x7eff9015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff90160800 0x7eff9015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7eff90160c10 con 0x7eff900c96e0 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff900c96e0 0x7eff9015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff900c96e0 0x7eff9015b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40038/0 (socket says 172.21.15.67:40038) 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 -- 172.21.15.67:0/2405030271 learned_addr learned my addr 172.21.15.67:0/2405030271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff9006d570 con 0x7eff90160800 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff9526e640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff90160d90 con 0x7eff900c92d0 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff900c92d0 0x7eff90157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff90160800 0x7eff9015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff900c92d0 msgr2=0x7eff90157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff900c92d0 0x7eff90157730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff90160800 msgr2=0x7eff9015be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff90160800 0x7eff9015be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8e7fc640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9015a240 con 0x7eff900c96e0 2024-04-25T15:52:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.922+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff90160800 0x7eff9015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.926+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff900c96e0 0x7eff9015b8f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e870 tx=0x7eff8400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.926+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84017070 con 0x7eff900c96e0 2024-04-25T15:52:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.926+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8400c940 con 0x7eff900c96e0 2024-04-25T15:52:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.926+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8400ccf0 con 0x7eff900c96e0 2024-04-25T15:52:27.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.926+0000 7eff9526e640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff90157c70 con 0x7eff900c96e0 2024-04-25T15:52:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.926+0000 7eff9526e640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff900c6b40 con 0x7eff900c96e0 2024-04-25T15:52:27.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.926+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7eff68005280 con 0x7eff900c96e0 2024-04-25T15:52:27.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7eff8401a070 con 0x7eff900c96e0 2024-04-25T15:52:27.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff777fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff7c06f430 0x7eff7c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5857+0+0 (secure 0 0 0) 0x7eff8408f5a0 con 0x7eff900c96e0 2024-04-25T15:52:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v52) ==== 54+0+16 (secure 0 0 0) 0x7eff9015a240 con 0x7eff900c96e0 2024-04-25T15:52:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff7c06f430 0x7eff7c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7eff680045f0 0x7eff68004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff68005560 con 0x7eff680045f0 2024-04-25T15:52:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7eff680045f0 0x7eff68004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff7c06f430 0x7eff7c071940 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff800ae000 tx=0x7eff800a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:27.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7eff680045f0 0x7eff68004a00 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:27.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.930+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff68005560 con 0x7eff680045f0 2024-04-25T15:52:27.940 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:27.940 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:27.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:27.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.958+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff68001280 con 0x7eff680045f0 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.962+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff68001280 con 0x7eff680045f0 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.962+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7eff68001d40 0x7eff680029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.962+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7eff68001d40 0x7eff680029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.962+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7eff68002370 con 0x7eff68001d40 2024-04-25T15:52:27.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.962+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7eff68001d40 0x7eff680029a0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:27.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.962+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff68002370 con 0x7eff68001d40 2024-04-25T15:52:27.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.994+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff680031b0 con 0x7eff68001d40 2024-04-25T15:52:28.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.998+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff680031b0 con 0x7eff68001d40 2024-04-25T15:52:28.003 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.998+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7eff68009c90 0x7eff6800c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.998+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7eff6800ee80 con 0x7eff68009c90 2024-04-25T15:52:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.998+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7eff68009c90 0x7eff6800c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.998+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7eff68009c90 0x7eff6800c0b0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:27.998+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff6800ee80 con 0x7eff68009c90 2024-04-25T15:52:28.024 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:28.018+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.026+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff6800c8c0 con 0x7eff68009c90 2024-04-25T15:52:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.030+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff6800c8c0 con 0x7eff68009c90 2024-04-25T15:52:28.035 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.030+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7eff6800d380 0x7eff6800d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.030+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7eff6800df70 con 0x7eff6800d380 2024-04-25T15:52:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.030+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7eff6800d380 0x7eff6800d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.030+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7eff6800d380 0x7eff6800d790 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.030+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff6800df70 con 0x7eff6800d380 2024-04-25T15:52:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.058+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff6800e480 con 0x7eff6800d380 2024-04-25T15:52:28.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.066+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff6800e480 con 0x7eff6800d380 2024-04-25T15:52:28.070 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:28.071 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:28.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.066+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7eff68011040 0x7eff68013460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.066+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7eff68016230 con 0x7eff68011040 2024-04-25T15:52:28.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.066+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7eff68011040 0x7eff68013460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.066+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7eff68011040 0x7eff68013460 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.070+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff68016230 con 0x7eff68011040 2024-04-25T15:52:28.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:52:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.094+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff680139a0 con 0x7eff68011040 2024-04-25T15:52:28.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.098+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff680139a0 con 0x7eff68011040 2024-04-25T15:52:28.106 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:28.106 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:28.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.102+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7eff68014020 0x7eff68016440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.107 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:28.107 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:28.107 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:28.107 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:28.107 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:28.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.102+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7eff680194b0 con 0x7eff68014020 2024-04-25T15:52:28.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.102+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7eff68014020 0x7eff68016440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.102+0000 7eff8e7fc640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7eff68014020 0x7eff68016440 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.102+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff680194b0 con 0x7eff68014020 2024-04-25T15:52:28.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.130+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff680181b0 con 0x7eff68014020 2024-04-25T15:52:28.136 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:28.130+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:28.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.134+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff680181b0 con 0x7eff68014020 2024-04-25T15:52:28.141 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.134+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7eff680188d0 0x7eff68018ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.134+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7eff6801c950 con 0x7eff680188d0 2024-04-25T15:52:28.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.134+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7eff680188d0 0x7eff68018ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.138+0000 7eff8f7fe640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7eff680188d0 0x7eff68018ce0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.138+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff6801c950 con 0x7eff680188d0 2024-04-25T15:52:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.166+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff6801c950 con 0x7eff680188d0 2024-04-25T15:52:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.170+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff6801c950 con 0x7eff680188d0 2024-04-25T15:52:28.176 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.170+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7eff6801bc10 0x7eff6801c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.170+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7eff6801fcf0 con 0x7eff6801bc10 2024-04-25T15:52:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.170+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7eff6801bc10 0x7eff6801c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.170+0000 7eff8effd640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7eff6801bc10 0x7eff6801c020 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.174+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff6801fcf0 con 0x7eff6801bc10 2024-04-25T15:52:28.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.202+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7eff68016a70 con 0x7eff6801bc10 2024-04-25T15:52:28.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff777fe640 1 -- 172.21.15.67:0/2405030271 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7eff68016a70 con 0x7eff6801bc10 2024-04-25T15:52:28.211 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7eff680045f0 msgr2=0x7eff68004a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7eff680045f0 0x7eff68004a00 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7eff68001d40 msgr2=0x7eff680029a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7eff68001d40 0x7eff680029a0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7eff68009c90 msgr2=0x7eff6800c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7eff68009c90 0x7eff6800c0b0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7eff6800d380 msgr2=0x7eff6800d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7eff6800d380 0x7eff6800d790 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7eff68011040 msgr2=0x7eff68013460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff8f7fe640 1 -- 172.21.15.67:0/2405030271 reap_dead start 2024-04-25T15:52:28.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7eff68011040 0x7eff68013460 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff8f7fe640 1 -- 172.21.15.67:0/2405030271 reap_dead start 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7eff68014020 msgr2=0x7eff68016440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7eff68014020 0x7eff68016440 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7eff680188d0 msgr2=0x7eff68018ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7eff680188d0 0x7eff68018ce0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7eff6801bc10 msgr2=0x7eff6801c020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7eff6801bc10 0x7eff6801c020 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff7c06f430 msgr2=0x7eff7c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7eff7c06f430 0x7eff7c071940 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff800ae000 tx=0x7eff800a7000 comp rx=0 tx=0).stop 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff8f7fe640 1 -- 172.21.15.67:0/2405030271 reap_dead start 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff900c96e0 msgr2=0x7eff9015b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff900c96e0 0x7eff9015b8f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e870 tx=0x7eff8400ec70 comp rx=0 tx=0).stop 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.206+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 shutdown_connections 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.210+0000 7eff757fa640 1 --2- 172.21.15.67:0/2405030271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff900c96e0 0x7eff9015b8f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.210+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 >> 172.21.15.67:0/2405030271 conn(0x7eff90070db0 msgr2=0x7eff90151610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:28.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.210+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 shutdown_connections 2024-04-25T15:52:28.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.210+0000 7eff757fa640 1 -- 172.21.15.67:0/2405030271 wait complete. 2024-04-25T15:52:28.226 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:28.222+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:28.296 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:28.296 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.29229530399999, 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'48, mlcod=15'42)", 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.29066608599999, 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012230665999999999, 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004239+0000", 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004250+0000", 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1518999999999999e-05 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004274+0000", 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004292+0000", 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7458000000000001e-05 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005373+0000", 2024-04-25T15:52:28.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010819359999999999 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005450+0000", 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6200999999999998e-05 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016441+0000", 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991445000000001 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016468+0000", 2024-04-25T15:52:28.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6801999999999999e-05 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:71 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'49, mlcod=15'42)", 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.289875647, 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011466877, 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005029+0000", 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999999999999999e-06 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005040+0000", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1199000000000001e-05 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005063+0000", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2342000000000001e-05 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005081+0000", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7711999999999999e-05 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006374+0000", 2024-04-25T15:52:28.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001293492 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006429+0000", 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5315999999999997e-05 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016469+0000", 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010039984 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5232000000000002e-05 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.308 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:72 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'50, mlcod=15'44)", 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.28859876999999, 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010219404, 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:28.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006306+0000", 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.48e-06 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006368+0000", 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1532000000000007e-05 2024-04-25T15:52:28.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006394+0000", 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6917999999999999e-05 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006423+0000", 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8087000000000001e-05 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007111+0000", 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068813299999999995 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007172+0000", 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0918000000000002e-05 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016500+0000", 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093286540000000005 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016524+0000", 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3682000000000001e-05 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:73 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'51, mlcod=15'44)", 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.28736683700001, 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090122410000000007, 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007538+0000", 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-06 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007559+0000", 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0302e-05 2024-04-25T15:52:28.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007589+0000", 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0088000000000001e-05 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007643+0000", 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4419000000000002e-05 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010532+0000", 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028890249999999999 2024-04-25T15:52:28.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010603+0000", 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1144999999999997e-05 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016525+0000", 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059219279999999999 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016549+0000", 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3444000000000001e-05 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.316 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:76 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'44)", 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.2843134, 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010878550000000001, 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010591+0000", 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7899999999999997e-07 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010599+0000", 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6000000000000007e-06 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010611+0000", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2194999999999999e-05 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010620+0000", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8529999999999998e-06 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016013+0000", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053922229999999998 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016045+0000", 2024-04-25T15:52:28.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1933000000000001e-05 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021402+0000", 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053573559999999998 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021468+0000", 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6611000000000003e-05 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.083450226, 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010088613, 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211464+0000", 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0740000000000001e-05 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211486+0000", 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1722e-05 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211654+0000", 2024-04-25T15:52:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016800800000000001 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211678+0000", 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4283000000000001e-05 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212032+0000", 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035369900000000002 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212515+0000", 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048357899999999999 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024946400000000002 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217815+0000", 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050506989999999996 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221231+0000", 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003415567 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221412+0000", 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000180681 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221542+0000", 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130171 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.083407661, 2024-04-25T15:52:28.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:28.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:27 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.08339400200001, 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010202413, 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.330 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211510+0000", 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211538+0000", 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7475e-05 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211876+0000", 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033851299999999998 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211898+0000", 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1648999999999999e-05 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212060+0000", 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016183899999999999 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212463+0000", 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040314700000000002 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212983+0000", 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052057999999999998 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220727+0000", 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077435919999999997 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221245+0000", 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051843400000000002 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221590+0000", 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034479999999999998 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221712+0000", 2024-04-25T15:52:28.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121477 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.08336027599999, 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:28.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.083348194, 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:28.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:28.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.083328022, 2024-04-25T15:52:28.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:28.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.083089684, 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.348 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:17 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.04535576799999, 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010948628, 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249551+0000", 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2329999999999998e-06 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249572+0000", 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0979e-05 2024-04-25T15:52:28.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249640+0000", 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8326999999999998e-05 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249668+0000", 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8368999999999999e-05 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250082+0000", 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041399200000000002 2024-04-25T15:52:28.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250425+0000", 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034239200000000001 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253136+0000", 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027107210000000001 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256784+0000", 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036481980000000001 2024-04-25T15:52:28.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260324+0000", 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035401909999999998 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260418+0000", 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3692999999999999e-05 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260496+0000", 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8533000000000002e-05 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:18 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.045321547, 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011222823999999999, 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249583+0000", 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1799999999999999e-06 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249591+0000", 2024-04-25T15:52:28.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0669999999999992e-06 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249819+0000", 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022795000000000001 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249865+0000", 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5561e-05 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254040+0000", 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041751720000000004 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254349+0000", 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030965399999999997 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254663+0000", 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031403600000000001 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258978+0000", 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043149959999999998 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260612+0000", 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00163383 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260715+0000", 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010311599999999999 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260805+0000", 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9262e-05 2024-04-25T15:52:28.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:7 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.964045275000004, 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086917899999999996, 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:28.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330859+0000", 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7000000000000005e-07 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330864+0000", 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1120000000000004e-06 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331053+0000", 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189261 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331064+0000", 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1260000000000001e-05 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331313+0000", 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024867500000000001 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331526+0000", 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000212721 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331732+0000", 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000206839 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335642+0000", 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039095639999999999 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339325+0000", 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036827790000000002 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339449+0000", 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123924 2024-04-25T15:52:28.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339550+0000", 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101085 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:8 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.964037184000006, 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010026810000000001, 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330867+0000", 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8899999999999999e-07 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330871+0000", 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2089999999999999e-06 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331105+0000", 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000234381 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331119+0000", 2024-04-25T15:52:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4165000000000001e-05 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332225+0000", 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011058140000000001 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332348+0000", 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122872 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332557+0000", 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000208364 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336449+0000", 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038921329999999999 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340746+0000", 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042971779999999996 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340840+0000", 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4109000000000003e-05 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340893+0000", 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2995999999999999e-05 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:32 2.1f 2:fe09a0b8:::202.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.964017420999994, 2024-04-25T15:52:28.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090313700000000004, 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330887+0000", 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1799999999999995e-07 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330890+0000", 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.427e-06 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331211+0000", 2024-04-25T15:52:28.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000320639 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331220+0000", 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4669999999999993e-06 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335755+0000", 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349500000000003 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336042+0000", 2024-04-25T15:52:28.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028675400000000001 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339917+0000", 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038755149999999999 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:1 2.1f 2:fb684bcd:::204.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.369287348, 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012576726, 2024-04-25T15:52:28.376 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000002e-07 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925629+0000", 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3002e-05 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925764+0000", 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013477399999999999 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925779+0000", 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228000000000001e-05 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925847+0000", 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7980999999999998e-05 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925951+0000", 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010333299999999999 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:28.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926085+0000", 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013462200000000001 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929258+0000", 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003172607 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938074+0000", 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088162910000000004 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:28.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938129+0000", 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5234e-05 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938193+0000", 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3367000000000005e-05 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:31 2.1f 2:ff4f7703:::204.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 84.369272611, 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012700365, 2024-04-25T15:52:28.381 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5499999999999999e-07 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925633+0000", 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.796e-06 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925809+0000", 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017651200000000001 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925824+0000", 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5071e-05 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926689+0000", 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086453799999999998 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926825+0000", 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013552799999999999 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938331+0000", 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011506565 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:28.385 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.290+0000 7f93a3c01640 1 Processor -- start 2024-04-25T15:52:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.290+0000 7f93a3c01640 1 -- start start 2024-04-25T15:52:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a3c01640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f939c0c8e20 0x7f939c164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a3c01640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f939c164880 0x7f939c0c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a3c01640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f939c070390 con 0x7f939c0c3cc0 2024-04-25T15:52:28.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a3c01640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f939c0c89b0 con 0x7f939c164880 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a3c01640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f939c06f8b0 con 0x7f939c0c8e20 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a119c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f939c164880 0x7f939c0c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a119c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f939c164880 0x7f939c0c3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40054/0 (socket says 172.21.15.67:40054) 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a119c640 1 -- 172.21.15.67:0/944207420 learned_addr learned my addr 172.21.15.67:0/944207420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a199d640 1 --2- 172.21.15.67:0/944207420 >> v2:172.21.15.176:3300/0 conn(0x7f939c0c8e20 0x7f939c164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a119c640 1 -- 172.21.15.67:0/944207420 >> v2:172.21.15.176:3300/0 conn(0x7f939c0c8e20 msgr2=0x7f939c164340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a119c640 1 --2- 172.21.15.67:0/944207420 >> v2:172.21.15.176:3300/0 conn(0x7f939c0c8e20 0x7f939c164340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a119c640 1 -- 172.21.15.67:0/944207420 >> v1:172.21.15.176:6789/0 conn(0x7f939c0c3cc0 legacy=0x7f939c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.294+0000 7f93a119c640 1 -- 172.21.15.67:0/944207420 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c06fb90 con 0x7f939c164880 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a119c640 1 --2- 172.21.15.67:0/944207420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c164880 0x7f939c0c3780 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f939c06d9c0 tx=0x7f938c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c47c7c7a7fbd8f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a099b640 1 -- 172.21.15.67:0/944207420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c0b5070 con 0x7f939c164880 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a099b640 1 -- 172.21.15.67:0/944207420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f938c0a8a10 con 0x7f939c164880 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a099b640 1 -- 172.21.15.67:0/944207420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c0a8de0 con 0x7f939c164880 2024-04-25T15:52:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a3c01640 1 -- 172.21.15.67:0/944207420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c164880 msgr2=0x7f939c0c3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a3c01640 1 --2- 172.21.15.67:0/944207420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c164880 0x7f939c0c3780 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f939c06d9c0 tx=0x7f938c01ed10 comp rx=0 tx=0).stop 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a3c01640 1 -- 172.21.15.67:0/944207420 shutdown_connections 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a3c01640 1 --2- 172.21.15.67:0/944207420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c164880 0x7f939c0c3780 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.298+0000 7f93a3c01640 1 --2- 172.21.15.67:0/944207420 >> v2:172.21.15.176:3300/0 conn(0x7f939c0c8e20 0x7f939c164340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.302+0000 7f93a3c01640 1 -- 172.21.15.67:0/944207420 >> 172.21.15.67:0/944207420 conn(0x7f939c0cdd60 msgr2=0x7f939c0ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.302+0000 7f93a3c01640 1 -- 172.21.15.67:0/944207420 shutdown_connections 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.302+0000 7f93a3c01640 1 -- 172.21.15.67:0/944207420 wait complete. 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.302+0000 7f93a3c01640 1 Processor -- start 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.302+0000 7f93a3c01640 1 -- start start 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a3c01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c0c3cc0 0x7f939c1fb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a3c01640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939c0c8e20 0x7f939c1eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a199d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c0c3cc0 0x7f939c1fb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a199d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c0c3cc0 0x7f939c1fb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40066/0 (socket says 172.21.15.67:40066) 2024-04-25T15:52:28.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a199d640 1 -- 172.21.15.67:0/541139768 learned_addr learned my addr 172.21.15.67:0/541139768 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a3c01640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f939c164880 0x7f939c15f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a3c01640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f939c074cc0 con 0x7f939c0c3cc0 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a3c01640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f939c074b40 con 0x7f939c0c8e20 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a3c01640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f939c0c2e10 con 0x7f939c164880 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a119c640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939c0c8e20 0x7f939c1eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.306+0000 7f93a219e640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f939c164880 0x7f939c15f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a119c640 1 -- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f939c164880 msgr2=0x7f939c15f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a119c640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f939c164880 0x7f939c15f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a119c640 1 -- 172.21.15.67:0/541139768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c0c3cc0 msgr2=0x7f939c1fb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a119c640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c0c3cc0 0x7f939c1fb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a119c640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c0ca170 con 0x7f939c0c8e20 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a199d640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c0c3cc0 0x7f939c1fb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a119c640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939c0c8e20 0x7f939c1eae40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f938c0af000 tx=0x7f938c01bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93927fc640 1 -- 172.21.15.67:0/541139768 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c0c0070 con 0x7f939c0c8e20 2024-04-25T15:52:28.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93927fc640 1 -- 172.21.15.67:0/541139768 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f938c0b0940 con 0x7f939c0c8e20 2024-04-25T15:52:28.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93927fc640 1 -- 172.21.15.67:0/541139768 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c0b5040 con 0x7f939c0c8e20 2024-04-25T15:52:28.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a3c01640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939c1fb5d0 con 0x7f939c0c8e20 2024-04-25T15:52:28.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93a3c01640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939c1ea440 con 0x7f939c0c8e20 2024-04-25T15:52:28.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93780052a0 con 0x7f939c0c8e20 2024-04-25T15:52:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.310+0000 7f93927fc640 1 -- 172.21.15.67:0/541139768 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f938c0bb430 con 0x7f939c0c8e20 2024-04-25T15:52:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.314+0000 7f93927fc640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f938006f400 0x7f9380071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.314+0000 7f93927fc640 1 -- 172.21.15.67:0/541139768 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5857+0+0 (secure 0 0 0) 0x7f938c12c3e0 con 0x7f939c0c8e20 2024-04-25T15:52:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.314+0000 7f93a199d640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f938006f400 0x7f9380071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.314+0000 7f93a199d640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f938006f400 0x7f9380071910 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9398007e60 tx=0x7f9398008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:28.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.326+0000 7f93927fc640 1 -- 172.21.15.67:0/541139768 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f938c0ca080 con 0x7f939c0c8e20 2024-04-25T15:52:28.395 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:28.382+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:28.487 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:28.482+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:28.588 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:28.582+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:28.646 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:28.647 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:28.647 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:28.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:28.662+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f9378005590 con 0x7f939c0c8e20 2024-04-25T15:52:28.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:52:28.668 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:52:28.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:28.706 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:28.702+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:28.794 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:28.790+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:28.891 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:28.886+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:28.953 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:28.953 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:28.953 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:28.953 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:28.953 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:28.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:28.992 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:28.991+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:29.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.082+0000 7f93927fc640 1 -- 172.21.15.67:0/541139768 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v53) ==== 87+0+0 (secure 0 0 0) 0x7f938c018870 con 0x7f939c0c8e20 2024-04-25T15:52:29.090 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:52:29.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f938006f400 msgr2=0x7f9380071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:29.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f938006f400 0x7f9380071910 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9398007e60 tx=0x7f9398008040 comp rx=0 tx=0).stop 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939c0c8e20 msgr2=0x7f939c1eae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939c0c8e20 0x7f939c1eae40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f938c0af000 tx=0x7f938c01bab0 comp rx=0 tx=0).stop 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 shutdown_connections 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f938006f400 0x7f9380071910 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f939c164880 0x7f939c15f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939c0c8e20 0x7f939c1eae40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 --2- 172.21.15.67:0/541139768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f939c0c3cc0 0x7f939c1fb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 >> 172.21.15.67:0/541139768 conn(0x7f939c0cdd60 msgr2=0x7f939c072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 shutdown_connections 2024-04-25T15:52:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:29.086+0000 7f937ffff640 1 -- 172.21.15.67:0/541139768 wait complete. 2024-04-25T15:52:29.106 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:29.102+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:29.194 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:29.190+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.011675704, 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013243108, 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.225 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0946e-05 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211617+0000", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211761+0000", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014392699999999999 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211793+0000", 2024-04-25T15:52:29.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1568e-05 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212109+0000", 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031626100000000003 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212422+0000", 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031341099999999998 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213414+0000", 2024-04-25T15:52:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099192799999999999 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223830+0000", 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010415654999999999 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224524+0000", 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069455899999999998 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224662+0000", 2024-04-25T15:52:29.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013742600000000001 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224828+0000", 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016641799999999999 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.011634399, 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013508035, 2024-04-25T15:52:29.230 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211628+0000", 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3400000000000001e-06 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211641+0000", 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2866e-05 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027786899999999998 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211973+0000", 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4401000000000003e-05 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214895+0000", 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029218109999999999 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215130+0000", 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023500499999999999 2024-04-25T15:52:29.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215556+0000", 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042636900000000001 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224481+0000", 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089248869999999994 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224861+0000", 2024-04-25T15:52:29.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038038699999999999 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225017+0000", 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155296 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225135+0000", 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117804 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:29.235 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.01161702, 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013948439999999999, 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:29.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211645+0000", 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1699999999999997e-07 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211680+0000", 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5374999999999999e-05 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212061+0000", 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038129700000000001 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212135+0000", 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4103000000000005e-05 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218486+0000", 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063501709999999999 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218861+0000", 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037518799999999999 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219238+0000", 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037732500000000002 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225340+0000", 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061016430000000003 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225414+0000", 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.436e-05 2024-04-25T15:52:29.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225489+0000", 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4882999999999994e-05 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225592+0000", 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103378 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:29.240 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.011575295, 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013711255, 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211687+0000", 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-06 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211694+0000", 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6329999999999996e-06 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212201+0000", 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050685200000000002 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212239+0000", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7537999999999999e-05 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004023726 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216562+0000", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029964199999999997 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:29.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033712999999999999 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223552+0000", 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066527330000000001 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225162+0000", 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016101869999999999 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225266+0000", 2024-04-25T15:52:29.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010402400000000001 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225397+0000", 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130684 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:6 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.00730196799999, 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198140000000006, 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.245 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215961+0000", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5469999999999999e-06 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215983+0000", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2014999999999998e-05 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216017+0000", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3114999999999997e-05 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216043+0000", 2024-04-25T15:52:29.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6755999999999999e-05 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216409+0000", 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036599799999999997 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216466+0000", 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6947000000000002e-05 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224337+0000", 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078708230000000007 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224479+0000", 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014161299999999999 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:7 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.00715176200001, 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198700000000006, 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216110+0000", 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3570000000000001e-06 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216125+0000", 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4772e-05 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216151+0000", 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5496000000000001e-05 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216172+0000", 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1095e-05 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217019+0000", 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084760800000000002 2024-04-25T15:52:29.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217071+0000", 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1894000000000002e-05 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224489+0000", 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074175710000000004 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224629+0000", 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014007700000000001 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:8 2.10 e20/17 2:09401a81:::601.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.00546686600001, 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069206529999999997, 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217796+0000", 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9069999999999999e-06 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217816+0000", 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0296e-05 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217847+0000", 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0877999999999998e-05 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217872+0000", 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4749e-05 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218099+0000", 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022737199999999999 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8900000000000002e-05 2024-04-25T15:52:29.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224635+0000", 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064771480000000003 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224715+0000", 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9402999999999999e-05 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:9 2.10 e20/17 2:09401a81:::601.00000000:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.004351988, 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058821409999999996, 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218911+0000", 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218930+0000", 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8916e-05 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218960+0000", 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0065000000000001e-05 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218984+0000", 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4528999999999998e-05 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219206+0000", 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022124300000000001 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219263+0000", 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7003000000000003e-05 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224719+0000", 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054565830000000001 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224791+0000", 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1928999999999995e-05 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.260 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:11 2.10 e20/17 2:0bd6d154:::602.00000000:head v 20'6, mlcod=0'0)", 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.00197435600001, 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050276770000000004, 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221288+0000", 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221306+0000", 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719000000000001e-05 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221336+0000", 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0026e-05 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221365+0000", 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8725e-05 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222395+0000", 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001030078 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222455+0000", 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9803999999999999e-05 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226163+0000", 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037085629999999998 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226314+0000", 2024-04-25T15:52:29.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150829 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:19 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.97562265800001, 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010474059000000001, 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247641+0000", 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.129e-06 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247667+0000", 2024-04-25T15:52:29.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6089e-05 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247827+0000", 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000159977 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247854+0000", 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6971999999999999e-05 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248122+0000", 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026745499999999999 2024-04-25T15:52:29.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248469+0000", 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034735899999999999 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248784+0000", 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031528900000000001 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251768+0000", 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002983383 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257863+0000", 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006095339 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258009+0000", 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000145921 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258112+0000", 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103146 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:20 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.975583887, 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010923117, 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:29.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247688+0000", 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3659999999999998e-06 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247937+0000", 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024977700000000002 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247958+0000", 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249612+0000", 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016540089999999999 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249860+0000", 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024840600000000001 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:29.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250352+0000", 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049215499999999998 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255543+0000", 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051905060000000001 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258273+0000", 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027298460000000002 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258467+0000", 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019409900000000001 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258600+0000", 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013348099999999999 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:23 2.15 2:aa0f8bf4:::101.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.975568834, 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082242619999999995, 2024-04-25T15:52:29.274 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247728+0000", 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5049000000000003e-05 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248028+0000", 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029943400000000001 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248050+0000", 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2722e-05 2024-04-25T15:52:29.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248203+0000", 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015264 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248563+0000", 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036047500000000002 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248940+0000", 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037679400000000001 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252184+0000", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003243821 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255610+0000", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034255370000000002 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255793+0000", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018366899999999999 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255916+0000", 2024-04-25T15:52:29.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122998 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.975526899, 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:29.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:4 2.10 e20/17 2:0f65c9b8:::60a.00000000:head v 20'10, mlcod=20'7)", 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.972169126, 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054254350000000002, 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251094+0000", 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.136e-06 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251114+0000", 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0003000000000001e-05 2024-04-25T15:52:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251147+0000", 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2947999999999997e-05 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251171+0000", 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3845999999999998e-05 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251347+0000", 2024-04-25T15:52:29.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176813 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251408+0000", 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0470000000000002e-05 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256382+0000", 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049743510000000001 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256517+0000", 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013486800000000001 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:3 2.0 e20/17 2:03625d61:::614.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.892619858000003, 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060303220000000003, 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330643+0000", 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9850000000000001e-06 2024-04-25T15:52:29.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330658+0000", 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5386000000000001e-05 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330686+0000", 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7971e-05 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330702+0000", 2024-04-25T15:52:29.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5268999999999999e-05 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330841+0000", 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013921800000000001 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330881+0000", 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0052000000000002e-05 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336596+0000", 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057151320000000004 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336671+0000", 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5308999999999993e-05 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:4 2.0 e20/17 2:03625d61:::614.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:29.292 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.889348366999997, 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068526890000000003, 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333914+0000", 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3859999999999999e-06 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333929+0000", 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5508000000000001e-05 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333945+0000", 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5968999999999999e-05 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333961+0000", 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5160999999999999e-05 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334268+0000", 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030782800000000001 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334291+0000", 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2459e-05 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340696+0000", 2024-04-25T15:52:29.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064053749999999996 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340765+0000", 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9003000000000003e-05 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:6 2.9 2:941d59bc:::61f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.336298067000001, 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054913510000000002, 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.296 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886964+0000", 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9999999999999996e-07 2024-04-25T15:52:29.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886973+0000", 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1400000000000006e-06 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887182+0000", 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020973699999999999 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887195+0000", 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2502e-05 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888149+0000", 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095407200000000004 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888285+0000", 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013629399999999999 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888481+0000", 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000195343 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890469+0000", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019881650000000001 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892371+0000", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019022620000000001 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892426+0000", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4404999999999998e-05 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892454+0000", 2024-04-25T15:52:29.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8631000000000001e-05 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.336217222000002, 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052764769999999999, 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:29.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8400000000000004e-07 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887048+0000", 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7390000000000001e-06 2024-04-25T15:52:29.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887454+0000", 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040556499999999998 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887460+0000", 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8639999999999998e-06 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888968+0000", 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015077599999999999 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889063+0000", 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4658000000000001e-05 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889206+0000", 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014365600000000001 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890596+0000", 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390023 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892161+0000", 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015643219999999999 2024-04-25T15:52:29.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892259+0000", 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7955000000000007e-05 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892320+0000", 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1351000000000003e-05 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:24 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.336210825999999, 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055188479999999998, 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887051+0000", 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1699999999999998e-07 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887063+0000", 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2575000000000001e-05 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887475+0000", 2024-04-25T15:52:29.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041185899999999999 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887481+0000", 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0970000000000002e-06 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887550+0000", 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9258000000000007e-05 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887693+0000", 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014290400000000001 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887898+0000", 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020492000000000001 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889987+0000", 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020884929999999999 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892508+0000", 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002521644 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892542+0000", 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3654e-05 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892569+0000", 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6927000000000002e-05 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 85.296043983999994, 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010008022, 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:29.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927218+0000", 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:29.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927222+0000", 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6519999999999997e-06 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927339+0000", 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116705 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927350+0000", 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0995000000000001e-05 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928080+0000", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072958300000000001 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928207+0000", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012715 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928433+0000", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022608599999999999 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:29.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070747400000000004 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937109+0000", 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079685439999999993 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9806999999999998e-05 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937225+0000", 2024-04-25T15:52:29.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6313000000000001e-05 2024-04-25T15:52:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:29.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:29.315 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:29.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:29.317 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:29.310+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:29.417 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:29.410+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:29.518 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:29.515+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:29.561 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:29.561 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:29.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:29.619 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:29.619+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:29.720 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:29.719+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:29.821 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:29.814+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:29.829 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:29.829 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:29.830 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:29.830 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:29.830 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:29.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:29.922 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:29.919+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:30.023 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:30.018+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:30.088 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:30.088 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:30.088 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 3, 4, 5, 6, 7, 1] 2024-04-25T15:52:30.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:6 2.10 2:0d82a743:::600.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.886824412, 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937687, 2024-04-25T15:52:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212210+0000", 2024-04-25T15:52:30.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212220+0000", 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0416e-05 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212245+0000", 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4796000000000001e-05 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212458+0000", 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021240899999999999 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212488+0000", 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0715e-05 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212836+0000", 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000348096 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213194+0000", 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000357413 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213591+0000", 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039711599999999998 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224723+0000", 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011132177 2024-04-25T15:52:30.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224892+0000", 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016872100000000001 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225031+0000", 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013966 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225148+0000", 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116168 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:7 2.10 2:0d82a743:::600.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.886778111, 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013374822, 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212256+0000", 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212257+0000", 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212263+0000", 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0900000000000001e-06 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212576+0000", 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031227099999999998 2024-04-25T15:52:30.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212590+0000", 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4133e-05 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214544+0000", 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001953939 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214793+0000", 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024941399999999999 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215181+0000", 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038823800000000001 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224739+0000", 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095574579999999996 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225301+0000", 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00056223199999999997 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225459+0000", 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015786199999999999 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225631+0000", 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172242 2024-04-25T15:52:30.112 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:8 2.10 2:09401a81:::601.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.886767698, 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013916951, 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.113 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212267+0000", 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8299999999999996e-07 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.114 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212272+0000", 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6630000000000001e-06 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212665+0000", 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039338399999999999 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212687+0000", 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1882000000000002e-05 2024-04-25T15:52:30.115 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215937+0000", 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032501119999999999 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032469999999999998 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.116 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216646+0000", 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038413800000000001 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224753+0000", 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0081070039999999993 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225911+0000", 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001158002 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.117 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226062+0000", 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015119600000000001 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226184+0000", 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121187 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:9 2.10 2:09401a81:::601.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.118 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.886759729, 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01435588, 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:30.119 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212275+0000", 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6000000000000003e-07 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212279+0000", 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1169999999999997e-06 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.120 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212723+0000", 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00044335799999999998 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212733+0000", 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0349e-05 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217516+0000", 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004782606 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.121 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217766+0000", 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025045699999999999 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039180399999999999 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224766+0000", 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066076449999999997 2024-04-25T15:52:30.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226308+0000", 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00154228 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226483+0000", 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174938 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.123 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226630+0000", 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014766599999999999 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:10 2.10 2:0bd6d154:::602.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.886752749, 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014867514, 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.124 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212282+0000", 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2699999999999999e-07 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212287+0000", 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6229999999999996e-06 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.126 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212756+0000", 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046907999999999997 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9560000000000003e-06 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218829+0000", 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060638690000000004 2024-04-25T15:52:30.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219182+0000", 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035291899999999999 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219507+0000", 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032545399999999998 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224779+0000", 2024-04-25T15:52:30.128 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052717279999999998 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226860+0000", 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020809790000000002 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227036+0000", 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176062 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.129 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227149+0000", 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113217 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:11 2.10 2:0bd6d154:::602.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.886745279, 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015281197, 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.130 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212289+0000", 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212290+0000", 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3e-07 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212333+0000", 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3251000000000003e-05 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212784+0000", 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00045139399999999998 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212793+0000", 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7960000000000001e-06 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220253+0000", 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074595699999999996 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220482+0000", 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000229554 2024-04-25T15:52:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220879+0000", 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00039646400000000002 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225181+0000", 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00430205 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227277+0000", 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020961119999999998 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227449+0000", 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017166800000000001 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227570+0000", 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012170799999999999 2024-04-25T15:52:30.135 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:12 2.10 2:0d89b25e:::603.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.88669579899999, 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016239407000000001, 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212339+0000", 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212340+0000", 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1200000000000001e-06 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212346+0000", 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.584e-06 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212813+0000", 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00046677600000000002 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212821+0000", 2024-04-25T15:52:30.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1529999999999997e-06 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221805+0000", 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089835999999999996 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222154+0000", 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034940299999999997 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222503+0000", 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034907900000000002 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227840+0000", 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367349999999996 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228310+0000", 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00047011999999999998 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228439+0000", 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012899999999999999 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228578+0000", 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138837 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:13 2.10 2:0d89b25e:::603.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.886684589, 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016443972000000001, 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212350+0000", 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212351+0000", 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4300000000000001e-07 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212357+0000", 2024-04-25T15:52:30.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2500000000000003e-06 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212848+0000", 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049076600000000001 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212863+0000", 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5498999999999998e-05 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223300+0000", 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010437093999999999 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223537+0000", 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000236665 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224027+0000", 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048967900000000003 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228184+0000", 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041571209999999997 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228599+0000", 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000415392 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228676+0000", 2024-04-25T15:52:30.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7095999999999995e-05 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.228794+0000", 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011746700000000001 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:22 2.e 2:717a0223:::608.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.88667361, 2024-04-25T15:52:30.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013451848000000001, 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212361+0000", 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212362+0000", 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9400000000000004e-07 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212368+0000", 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6229999999999999e-06 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212925+0000", 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00055690199999999996 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212960+0000", 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5045000000000001e-05 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213123+0000", 2024-04-25T15:52:30.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163222 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213447+0000", 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000323627 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213759+0000", 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031232800000000001 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216671+0000", 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002911145 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223654+0000", 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069831370000000004 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223829+0000", 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174857 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225813+0000", 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019840679999999999 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:23 2.e 2:717a0223:::608.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 113.88666244700001, 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014242332, 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212372+0000", 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212373+0000", 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4e-07 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212384+0000", 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1443e-05 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213026+0000", 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00064212500000000001 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213045+0000", 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8621999999999999e-05 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214708+0000", 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016628109999999999 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214970+0000", 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026197699999999999 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215380+0000", 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000410789 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218599+0000", 2024-04-25T15:52:30.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0032185270000000001 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226296+0000", 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076965269999999999 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226473+0000", 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017755200000000001 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226614+0000", 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:4 2.10 2:0f65c9b8:::60a.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.851859113, 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010394548999999999, 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247176+0000", 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2100000000000001e-06 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247209+0000", 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2945999999999999e-05 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247394+0000", 2024-04-25T15:52:30.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018468799999999999 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247417+0000", 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2733000000000001e-05 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249734+0000", 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002317684 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250032+0000", 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029727699999999998 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250453+0000", 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042143500000000001 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255742+0000", 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052886809999999999 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257232+0000", 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014906260000000001 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257393+0000", 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016088699999999999 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257570+0000", 2024-04-25T15:52:30.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017638199999999999 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:13 2.1 2:81735765:::619.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767890546999993, 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094210270000000002, 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331144+0000", 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6699999999999999e-07 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331165+0000", 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1186e-05 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331418+0000", 2024-04-25T15:52:30.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025271799999999997 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331428+0000", 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6460000000000005e-06 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333229+0000", 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018009499999999999 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334396+0000", 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011673269999999999 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334597+0000", 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020135300000000001 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339905+0000", 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053077610000000002 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340210+0000", 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030499999999999999 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340385+0000", 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017450199999999999 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340565+0000", 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018011700000000001 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:14 2.1 2:81735765:::619.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767865800999999, 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095762039999999996, 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.171 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331169+0000", 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3200000000000004e-07 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331174+0000", 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-06 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331467+0000", 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029356 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331478+0000", 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336604+0000", 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051267170000000003 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336735+0000", 2024-04-25T15:52:30.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013052699999999999 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336996+0000", 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000260511 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339986+0000", 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002990487 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340579+0000", 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00059299899999999998 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340673+0000", 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4241999999999997e-05 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340745+0000", 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1484999999999997e-05 2024-04-25T15:52:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:17 2.10 2:0e9da45e:::61b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767858555000004, 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010814343000000001, 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.177 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331176+0000", 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0399999999999996e-07 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331180+0000", 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6650000000000001e-06 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331501+0000", 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032061600000000002 2024-04-25T15:52:30.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331509+0000", 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8240000000000006e-06 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333383+0000", 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018733020000000001 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333652+0000", 2024-04-25T15:52:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026890399999999999 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333865+0000", 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021341 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338313+0000", 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044478269999999997 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341868+0000", 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035546729999999999 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341920+0000", 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2507999999999999e-05 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341990+0000", 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0110000000000005e-05 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.182 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:18 2.10 2:0e9da45e:::61b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767848882999999, 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011086621, 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331185+0000", 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331186+0000", 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.51e-07 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331191+0000", 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3970000000000002e-06 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331546+0000", 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035447799999999998 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331557+0000", 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0791000000000001e-05 2024-04-25T15:52:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334408+0000", 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028516520000000001 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334546+0000", 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013802700000000001 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334877+0000", 2024-04-25T15:52:30.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000331081 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339910+0000", 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050323449999999997 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342080+0000", 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021703899999999999 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342184+0000", 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103767 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342272+0000", 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8242e-05 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:1 2.1 2:85bbe569:::202.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767840863000004, 2024-04-25T15:52:30.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010331643, 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331193+0000", 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331194+0000", 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331198+0000", 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1400000000000002e-06 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331608+0000", 2024-04-25T15:52:30.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040962799999999999 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331620+0000", 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2721000000000001e-05 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337545+0000", 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005924726 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338432+0000", 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00088720699999999999 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.338639+0000", 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020665000000000001 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341318+0000", 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026785860000000002 2024-04-25T15:52:30.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341406+0000", 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8344000000000007e-05 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341458+0000", 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1805999999999998e-05 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.341525+0000", 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7336999999999996e-05 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:2 2.10 2:0daba267:::402.00000000:head [writefull 0~22] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767834403999998, 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011245289, 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331200+0000", 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331215+0000", 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331647+0000", 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043194799999999998 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331656+0000", 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3440000000000001e-06 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335316+0000", 2024-04-25T15:52:30.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036607710000000002 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335504+0000", 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018793299999999999 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335699+0000", 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000194544 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339945+0000", 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042457600000000003 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342333+0000", 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023885730000000002 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342383+0000", 2024-04-25T15:52:30.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9314999999999998e-05 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342445+0000", 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2691e-05 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:23 2.c 2:3631e767:::102.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.200 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767815643000006, 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087706009999999994, 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331219+0000", 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.75e-07 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331239+0000", 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9653e-05 2024-04-25T15:52:30.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331678+0000", 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043868999999999999 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331701+0000", 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3019000000000001e-05 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333753+0000", 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020522050000000001 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334131+0000", 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000378484 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334455+0000", 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00032319899999999998 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337268+0000", 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028133199999999998 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339816+0000", 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025479140000000001 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339907+0000", 2024-04-25T15:52:30.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1150000000000007e-05 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339989+0000", 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2392000000000007e-05 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.206 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:27 2.10 2:091e33df:::502.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767792602, 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011447874, 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331242+0000", 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4199999999999996e-07 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331249+0000", 2024-04-25T15:52:30.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2259999999999999e-06 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331730+0000", 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048009600000000001 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331739+0000", 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1840000000000005e-06 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336653+0000", 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049141510000000003 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336830+0000", 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017740099999999999 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 1", 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337033+0000", 2024-04-25T15:52:30.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020295999999999999 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340803+0000", 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037692480000000002 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342567+0000", 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017649250000000001 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342650+0000", 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2748000000000001e-05 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.342690+0000", 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9393e-05 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.212 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:31 2.10 2:0eee8b97:::202.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.767782745000005, 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012346048, 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331252+0000", 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5300000000000004e-07 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:30.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331255+0000", 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8380000000000002e-06 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331760+0000", 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050452600000000004 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331768+0000", 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:30.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337613+0000", 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058448700000000003 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.337836+0000", 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000223043 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:30.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.343598+0000", 2024-04-25T15:52:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057612679999999999 2024-04-25T15:52:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:30.217 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:30.217 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:30.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:30.219 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:30.214+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:30.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.282+0000 7fa350339640 1 Processor -- start 2024-04-25T15:52:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.282+0000 7fa350339640 1 -- start start 2024-04-25T15:52:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa350339640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa3480ca780 0x7fa348164a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa350339640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa348074720 con 0x7fa3480c99e0 2024-04-25T15:52:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa350339640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3480cab90 con 0x7fa3480733f0 2024-04-25T15:52:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa350339640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa34806d050 con 0x7fa3480ca780 2024-04-25T15:52:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d8d4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3480c99e0 0x7fa348164300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56084/0 (socket says 172.21.15.67:56084) 2024-04-25T15:52:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d8d4640 1 -- 172.21.15.67:0/859535084 learned_addr learned my addr 172.21.15.67:0/859535084 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34e8d6640 1 --2- 172.21.15.67:0/859535084 >> v2:172.21.15.176:3300/0 conn(0x7fa3480ca780 0x7fa348164a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210506439 0 0) 0x7fa348074720 con 0x7fa3480c99e0 2024-04-25T15:52:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa334003530 con 0x7fa3480c99e0 2024-04-25T15:52:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311869369 0 0) 0x7fa338001120 con 0x7fa3480733f0 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa348074720 con 0x7fa3480733f0 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1268148965 0 0) 0x7fa334003530 con 0x7fa3480c99e0 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 >> v2:172.21.15.176:3300/0 conn(0x7fa3480ca780 msgr2=0x7fa348164a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 --2- 172.21.15.67:0/859535084 >> v2:172.21.15.176:3300/0 conn(0x7fa3480ca780 0x7fa348164a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 >> v1:172.21.15.176:6789/0 conn(0x7fa3480733f0 legacy=0x7fa3480c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa348074990 con 0x7fa3480c99e0 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fa344001680 con 0x7fa3480c99e0 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 >> v1:172.21.15.67:6789/0 conn(0x7fa3480c99e0 legacy=0x7fa348164300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa334003b50 0x7fa334003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa334004810 0x7fa334004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334005280 0x7fa33400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa34806d050 con 0x7fa334004810 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa33400c1f0 con 0x7fa334003b50 2024-04-25T15:52:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa33400c3f0 con 0x7fa334005280 2024-04-25T15:52:30.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34e0d5640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa334003b50 0x7fa334003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d8d4640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa334004810 0x7fa334004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34e0d5640 1 -- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334005280 msgr2=0x7fa33400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:52:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34e0d5640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334005280 0x7fa33400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34e0d5640 1 -- 172.21.15.67:0/859535084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa334004810 msgr2=0x7fa334004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34e0d5640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa334004810 0x7fa334004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34e0d5640 1 -- 172.21.15.67:0/859535084 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa334003530 con 0x7fa334003b50 2024-04-25T15:52:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.286+0000 7fa34d8d4640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa334004810 0x7fa334004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.290+0000 7fa34e8d6640 1 --2- 172.21.15.67:0/859535084 >> v2:172.21.15.176:3300/0 conn(0x7fa3480ca780 0x7fa348164a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.290+0000 7fa34e8d6640 1 -- 172.21.15.67:0/859535084 reap_dead start 2024-04-25T15:52:30.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.290+0000 7fa34e0d5640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa334003b50 0x7fa334003fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa338001ae0 tx=0x7fa33801ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=618b637a86341d36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.290+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa338001700 con 0x7fa334003b50 2024-04-25T15:52:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.290+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3380052d0 con 0x7fa334003b50 2024-04-25T15:52:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.290+0000 7fa34d0d3640 1 -- 172.21.15.67:0/859535084 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3380054c0 con 0x7fa334003b50 2024-04-25T15:52:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.294+0000 7fa350339640 1 -- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa334003b50 msgr2=0x7fa334003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.294+0000 7fa350339640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa334003b50 0x7fa334003fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa338001ae0 tx=0x7fa33801ecd0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.294+0000 7fa350339640 1 -- 172.21.15.67:0/859535084 shutdown_connections 2024-04-25T15:52:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.294+0000 7fa350339640 1 --2- 172.21.15.67:0/859535084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa334003b50 0x7fa334003fa0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.294+0000 7fa350339640 1 -- 172.21.15.67:0/859535084 >> 172.21.15.67:0/859535084 conn(0x7fa348070db0 msgr2=0x7fa3480ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.294+0000 7fa350339640 1 -- 172.21.15.67:0/859535084 shutdown_connections 2024-04-25T15:52:30.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.294+0000 7fa350339640 1 -- 172.21.15.67:0/859535084 wait complete. 2024-04-25T15:52:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 Processor -- start 2024-04-25T15:52:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 -- start start 2024-04-25T15:52:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334003b50 0x7fa3481604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3481eb5b0 0x7fa3481e8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3481f9c00 0x7fa3481fa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa34806d860 con 0x7fa3481eb5b0 2024-04-25T15:52:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa34806d580 con 0x7fa3481f9c00 2024-04-25T15:52:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa350339640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa3480c92d0 con 0x7fa334003b50 2024-04-25T15:52:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa34d8d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3481eb5b0 0x7fa3481e8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa34d8d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3481eb5b0 0x7fa3481e8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40072/0 (socket says 172.21.15.67:40072) 2024-04-25T15:52:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa34d8d4640 1 -- 172.21.15.67:0/3497713090 learned_addr learned my addr 172.21.15.67:0/3497713090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa34e0d5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334003b50 0x7fa3481604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.298+0000 7fa34e8d6640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3481f9c00 0x7fa3481fa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa34d8d4640 1 -- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334003b50 msgr2=0x7fa3481604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa34d8d4640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334003b50 0x7fa3481604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa34d8d4640 1 -- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3481f9c00 msgr2=0x7fa3481fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa34d8d4640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3481f9c00 0x7fa3481fa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa34d8d4640 1 -- 172.21.15.67:0/3497713090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa348160a30 con 0x7fa3481eb5b0 2024-04-25T15:52:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa34d8d4640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3481eb5b0 0x7fa3481e8bd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa3440017f0 tx=0x7fa344007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa33effd640 1 -- 172.21.15.67:0/3497713090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34400fa80 con 0x7fa3481eb5b0 2024-04-25T15:52:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa33effd640 1 -- 172.21.15.67:0/3497713090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa344018070 con 0x7fa3481eb5b0 2024-04-25T15:52:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa33effd640 1 -- 172.21.15.67:0/3497713090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34400fe40 con 0x7fa3481eb5b0 2024-04-25T15:52:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3481e7060 con 0x7fa3481eb5b0 2024-04-25T15:52:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3481e8550 con 0x7fa3481eb5b0 2024-04-25T15:52:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa34806a480 con 0x7fa3481eb5b0 2024-04-25T15:52:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.302+0000 7fa33effd640 1 -- 172.21.15.67:0/3497713090 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa34401d460 con 0x7fa3481eb5b0 2024-04-25T15:52:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.306+0000 7fa33effd640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa32c06f430 0x7fa32c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.306+0000 7fa34e0d5640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa32c06f430 0x7fa32c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.306+0000 7fa34e0d5640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa32c06f430 0x7fa32c071940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa338001ab0 tx=0x7fa338009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.314+0000 7fa33effd640 1 -- 172.21.15.67:0/3497713090 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5857+0+0 (secure 0 0 0) 0x7fa34408d000 con 0x7fa3481eb5b0 2024-04-25T15:52:30.319 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:30.315+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:30.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.314+0000 7fa33effd640 1 -- 172.21.15.67:0/3497713090 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa34405b3b0 con 0x7fa3481eb5b0 2024-04-25T15:52:30.420 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:30.414+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:30.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.446+0000 7f1baa00e640 1 Processor -- start 2024-04-25T15:52:30.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.446+0000 7f1baa00e640 1 -- start start 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1baa00e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1ba4164440 0x7f1ba40cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1baa00e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ba40cc600 0x7f1ba40cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1baa00e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ba4164a10 con 0x7f1ba415b050 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1baa00e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f1ba4161230 con 0x7f1ba40cc600 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1baa00e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1ba4164cf0 con 0x7f1ba4164440 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ba40cc600 0x7f1ba40cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1ba40cc600 0x7f1ba40cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40084/0 (socket says 172.21.15.67:40084) 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba3fff640 1 -- 172.21.15.67:0/2270758777 learned_addr learned my addr 172.21.15.67:0/2270758777 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/2270758777 >> v2:172.21.15.176:3300/0 conn(0x7f1ba4164440 0x7f1ba40cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba3fff640 1 -- 172.21.15.67:0/2270758777 >> v2:172.21.15.176:3300/0 conn(0x7f1ba4164440 msgr2=0x7f1ba40cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/2270758777 >> v2:172.21.15.176:3300/0 conn(0x7f1ba4164440 0x7f1ba40cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba3fff640 1 -- 172.21.15.67:0/2270758777 >> v1:172.21.15.176:6789/0 conn(0x7f1ba415b050 legacy=0x7f1ba41616c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:52:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.450+0000 7f1ba3fff640 1 -- 172.21.15.67:0/2270758777 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba4164fd0 con 0x7f1ba40cc600 2024-04-25T15:52:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/2270758777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba40cca10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4074ca0 tx=0x7f1b9c01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d036d87041f144 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1ba27fc640 1 -- 172.21.15.67:0/2270758777 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c0b4070 con 0x7f1ba40cc600 2024-04-25T15:52:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1ba27fc640 1 -- 172.21.15.67:0/2270758777 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9c0af400 con 0x7f1ba40cc600 2024-04-25T15:52:30.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1ba27fc640 1 -- 172.21.15.67:0/2270758777 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c0af7d0 con 0x7f1ba40cc600 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 -- 172.21.15.67:0/2270758777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 msgr2=0x7f1ba40cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 --2- 172.21.15.67:0/2270758777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba40cca10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4074ca0 tx=0x7f1b9c01f1a0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 -- 172.21.15.67:0/2270758777 shutdown_connections 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 --2- 172.21.15.67:0/2270758777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba40cca10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 --2- 172.21.15.67:0/2270758777 >> v2:172.21.15.176:3300/0 conn(0x7f1ba4164440 0x7f1ba40cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 -- 172.21.15.67:0/2270758777 >> 172.21.15.67:0/2270758777 conn(0x7f1ba4068c70 msgr2=0x7f1ba4071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 -- 172.21.15.67:0/2270758777 shutdown_connections 2024-04-25T15:52:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.454+0000 7f1baa00e640 1 -- 172.21.15.67:0/2270758777 wait complete. 2024-04-25T15:52:30.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.458+0000 7f1baa00e640 1 Processor -- start 2024-04-25T15:52:30.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.458+0000 7f1baa00e640 1 -- start start 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1baa00e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba41fb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1baa00e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba415b050 0x7f1ba41ec980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1baa00e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4164440 0x7f1ba4157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba41fb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba41fb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40094/0 (socket says 172.21.15.67:40094) 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 -- 172.21.15.67:0/70968659 learned_addr learned my addr 172.21.15.67:0/70968659 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1baa00e640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba406d750 con 0x7f1ba40cc600 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1baa00e640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba4161dd0 con 0x7f1ba4164440 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1baa00e640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba4160e50 con 0x7f1ba415b050 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba2ffd640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba415b050 0x7f1ba41ec980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4164440 0x7f1ba4157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba415b050 msgr2=0x7f1ba41ec980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1ba415b050 0x7f1ba41ec980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4164440 msgr2=0x7f1ba4157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4164440 0x7f1ba4157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba41fb930 con 0x7f1ba40cc600 2024-04-25T15:52:30.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4164440 0x7f1ba4157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:30.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.462+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba41fb3f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800e8a0 tx=0x7f1b9800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98017070 con 0x7f1ba40cc600 2024-04-25T15:52:30.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1baa00e640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba41f74c0 con 0x7f1ba40cc600 2024-04-25T15:52:30.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1baa00e640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba41f7780 con 0x7f1ba40cc600 2024-04-25T15:52:30.472 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:30.472 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:30.472 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:30.472 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9800c940 con 0x7f1ba40cc600 2024-04-25T15:52:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1baa00e640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1ba406a480 con 0x7f1ba40cc600 2024-04-25T15:52:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9800ccf0 con 0x7f1ba40cc600 2024-04-25T15:52:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1b9801a070 con 0x7f1ba40cc600 2024-04-25T15:52:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1b8bfff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9006f450 0x7f1b90071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.466+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5857+0+0 (secure 0 0 0) 0x7f1b9808f1b0 con 0x7f1ba40cc600 2024-04-25T15:52:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.470+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v53) ==== 54+0+16 (secure 0 0 0) 0x7f1b98091750 con 0x7f1ba40cc600 2024-04-25T15:52:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.470+0000 7f1ba2ffd640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9006f450 0x7f1b90071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.470+0000 7f1ba2ffd640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9006f450 0x7f1b90071960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1ba415e9a0 tx=0x7f1b94008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.470+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1ba41f8930 0x7f1ba40c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.470+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ba4157b70 con 0x7f1ba41f8930 2024-04-25T15:52:30.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.470+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1ba41f8930 0x7f1ba40c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.474+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1ba41f8930 0x7f1ba40c7aa0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.474+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba4157b70 con 0x7f1ba41f8930 2024-04-25T15:52:30.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:30.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.502+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba40cba60 con 0x7f1ba41f8930 2024-04-25T15:52:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.506+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba40cba60 con 0x7f1ba41f8930 2024-04-25T15:52:30.510 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.506+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1ba40c84f0 0x7f1ba4063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.506+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1ba40c8900 con 0x7f1ba40c84f0 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.506+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1ba40c84f0 0x7f1ba4063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.506+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1ba40c84f0 0x7f1ba4063090 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.506+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba40c8900 con 0x7f1ba40c84f0 2024-04-25T15:52:30.521 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:30.519+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:30.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.534+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba40cb260 con 0x7f1ba40c84f0 2024-04-25T15:52:30.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.538+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba40cb260 con 0x7f1ba40c84f0 2024-04-25T15:52:30.542 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.538+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1ba4063ae0 0x7f1ba41eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.538+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1ba406da70 con 0x7f1ba4063ae0 2024-04-25T15:52:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.538+0000 7f1ba2ffd640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1ba4063ae0 0x7f1ba41eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.538+0000 7f1ba2ffd640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1ba4063ae0 0x7f1ba41eb630 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.538+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba406da70 con 0x7f1ba4063ae0 2024-04-25T15:52:30.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.566+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba406da70 con 0x7f1ba4063ae0 2024-04-25T15:52:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.570+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba406da70 con 0x7f1ba4063ae0 2024-04-25T15:52:30.577 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:30.578 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:30.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.570+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1ba41ec080 0x7f1ba4209da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.574+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1ba40cccb0 con 0x7f1ba41ec080 2024-04-25T15:52:30.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.574+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1ba41ec080 0x7f1ba4209da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.574+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1ba41ec080 0x7f1ba4209da0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.574+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba40cccb0 con 0x7f1ba41ec080 2024-04-25T15:52:30.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.602+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba40cccb0 con 0x7f1ba41ec080 2024-04-25T15:52:30.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.606+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba40cccb0 con 0x7f1ba41ec080 2024-04-25T15:52:30.610 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:30.610 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:30.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.606+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1ba420a7f0 0x7f1ba420cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.606+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1ba41f8260 con 0x7f1ba420a7f0 2024-04-25T15:52:30.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.606+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1ba420a7f0 0x7f1ba420cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.606+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1ba420a7f0 0x7f1ba420cc10 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.606+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba41f8260 con 0x7f1ba420a7f0 2024-04-25T15:52:30.622 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:30.619+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:30.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.634+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba41f8260 con 0x7f1ba420a7f0 2024-04-25T15:52:30.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.638+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba41f8260 con 0x7f1ba420a7f0 2024-04-25T15:52:30.644 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.638+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1ba420d660 0x7f1ba420fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.638+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1ba41f8e50 con 0x7f1ba420d660 2024-04-25T15:52:30.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.638+0000 7f1ba2ffd640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1ba420d660 0x7f1ba420fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.642+0000 7f1ba2ffd640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1ba420d660 0x7f1ba420fa80 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.642+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba41f8e50 con 0x7f1ba420d660 2024-04-25T15:52:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.650+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd out", "ids": ["4"]} v 0) -- 0x7fa348074120 con 0x7fa3481eb5b0 2024-04-25T15:52:30.656 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:52:30.650+0000 7fa4fdaf0640 -1 mon.a@0(leader).osd e53 definitely_dead 0 2024-04-25T15:52:30.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.670+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba41f8dd0 con 0x7f1ba420d660 2024-04-25T15:52:30.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.674+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba41f8dd0 con 0x7f1ba420d660 2024-04-25T15:52:30.680 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.674+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1ba42104d0 0x7f1ba42128f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.674+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1ba41f7bf0 con 0x7f1ba42104d0 2024-04-25T15:52:30.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.674+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1ba42104d0 0x7f1ba42128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.678+0000 7f1ba3fff640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1ba42104d0 0x7f1ba42128f0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.678+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b9c0c03b0 con 0x7f1ba42104d0 2024-04-25T15:52:30.694 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:30.694 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:30.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.706+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba4213100 con 0x7f1ba42104d0 2024-04-25T15:52:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.710+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba4213100 con 0x7f1ba42104d0 2024-04-25T15:52:30.716 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.710+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1ba4213820 0x7f1ba4213c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.710+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1ba4215370 con 0x7f1ba4213820 2024-04-25T15:52:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.710+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1ba4213820 0x7f1ba4213c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.714+0000 7f1ba37fe640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1ba4213820 0x7f1ba4213c30 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.714+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1ba4215370 con 0x7f1ba4213820 2024-04-25T15:52:30.722 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:30.718+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:30.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.742+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1ba4214170 con 0x7f1ba4213820 2024-04-25T15:52:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b8bfff640 1 -- 172.21.15.67:0/70968659 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1ba4214170 con 0x7f1ba4213820 2024-04-25T15:52:30.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1ba41f8930 msgr2=0x7f1ba40c7aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1ba41f8930 0x7f1ba40c7aa0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1ba40c84f0 msgr2=0x7f1ba4063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1ba40c84f0 0x7f1ba4063090 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1ba4063ae0 msgr2=0x7f1ba41eb630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1ba4063ae0 0x7f1ba41eb630 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1ba3fff640 1 -- 172.21.15.67:0/70968659 reap_dead start 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1ba41ec080 msgr2=0x7f1ba4209da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1ba41ec080 0x7f1ba4209da0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1ba420a7f0 msgr2=0x7f1ba420cc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1ba420a7f0 0x7f1ba420cc10 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1ba420d660 msgr2=0x7f1ba420fa80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1ba420d660 0x7f1ba420fa80 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1ba42104d0 msgr2=0x7f1ba42128f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1ba42104d0 0x7f1ba42128f0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1ba4213820 msgr2=0x7f1ba4213c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1ba4213820 0x7f1ba4213c30 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1ba3fff640 1 -- 172.21.15.67:0/70968659 reap_dead start 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9006f450 msgr2=0x7f1b90071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9006f450 0x7f1b90071960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1ba415e9a0 tx=0x7f1b94008040 comp rx=0 tx=0).stop 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 msgr2=0x7f1ba41fb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba41fb3f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800e8a0 tx=0x7f1b9800eca0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 shutdown_connections 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9006f450 0x7f1b90071960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 --2- 172.21.15.67:0/70968659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba40cc600 0x7f1ba41fb3f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 >> 172.21.15.67:0/70968659 conn(0x7f1ba4068c70 msgr2=0x7f1ba406e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:30.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 shutdown_connections 2024-04-25T15:52:30.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:30.746+0000 7f1b89ffb640 1 -- 172.21.15.67:0/70968659 wait complete. 2024-04-25T15:52:30.823 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:30.823+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:30.847 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:30.847 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:30.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:30.925 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:30.923+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:31.024 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:31.024 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:31.025 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:31.018+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:31.041 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.094+0000 7fa33effd640 1 -- 172.21.15.67:0/3497713090 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["4"]}]=0 marked out osd.4. v54) ==== 87+0+0 (secure 0 0 0) 0x7fa34405f6b0 con 0x7fa3481eb5b0 2024-04-25T15:52:31.101 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa32c06f430 msgr2=0x7fa32c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa32c06f430 0x7fa32c071940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa338001ab0 tx=0x7fa338009040 comp rx=0 tx=0).stop 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3481eb5b0 msgr2=0x7fa3481e8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3481eb5b0 0x7fa3481e8bd0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa3440017f0 tx=0x7fa344007720 comp rx=0 tx=0).stop 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 shutdown_connections 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa32c06f430 0x7fa32c071940 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3481f9c00 0x7fa3481fa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3481eb5b0 0x7fa3481e8bd0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 --2- 172.21.15.67:0/3497713090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa334003b50 0x7fa3481604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 >> 172.21.15.67:0/3497713090 conn(0x7fa348070db0 msgr2=0x7fa3481ebfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 shutdown_connections 2024-04-25T15:52:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.098+0000 7fa350339640 1 -- 172.21.15.67:0/3497713090 wait complete. 2024-04-25T15:52:31.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:52:31.138 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:31.134+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:31.227 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:31.222+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:31.242 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:31.242 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:31.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.314+0000 7f60363b4640 1 Processor -- start 2024-04-25T15:52:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.314+0000 7f60363b4640 1 -- start start 2024-04-25T15:52:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f60363b4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6030074370 0x7f6030074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f60363b4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60300c7fc0 0x7f60300c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f60363b4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60300c92d0 con 0x7f603006c640 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f60363b4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f603006e210 con 0x7f60300c7fc0 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f60363b4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f603006e4d0 con 0x7f6030074370 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f6034951640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60300c7fc0 0x7f60300c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f6034951640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60300c7fc0 0x7f60300c83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40108/0 (socket says 172.21.15.67:40108) 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f6034951640 1 -- 172.21.15.67:0/693654171 learned_addr learned my addr 172.21.15.67:0/693654171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f602f7fe640 1 --2- 172.21.15.67:0/693654171 >> v2:172.21.15.176:3300/0 conn(0x7f6030074370 0x7f6030074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f602effd640 1 -- 172.21.15.67:0/693654171 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2710896418 0 0) 0x7f60300c92d0 con 0x7f603006c640 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f602effd640 1 -- 172.21.15.67:0/693654171 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6018003530 con 0x7f603006c640 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f602f7fe640 1 -- 172.21.15.67:0/693654171 >> v2:172.21.15.67:3300/0 conn(0x7f60300c7fc0 msgr2=0x7f60300c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f602f7fe640 1 --2- 172.21.15.67:0/693654171 >> v2:172.21.15.67:3300/0 conn(0x7f60300c7fc0 0x7f60300c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f602f7fe640 1 -- 172.21.15.67:0/693654171 >> v1:172.21.15.67:6789/0 conn(0x7f603006c640 legacy=0x7f603015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.318+0000 7f602f7fe640 1 -- 172.21.15.67:0/693654171 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603006e7b0 con 0x7f6030074370 2024-04-25T15:52:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f602f7fe640 1 --2- 172.21.15.67:0/693654171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6030074370 0x7f6030074780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f60300c8f70 tx=0x7f602401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbfad058cf20e4cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f602effd640 1 -- 172.21.15.67:0/693654171 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60240b5070 con 0x7f6030074370 2024-04-25T15:52:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f602effd640 1 -- 172.21.15.67:0/693654171 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60240a8970 con 0x7f6030074370 2024-04-25T15:52:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f602effd640 1 -- 172.21.15.67:0/693654171 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60240a8d20 con 0x7f6030074370 2024-04-25T15:52:31.328 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:31.327+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:31.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 -- 172.21.15.67:0/693654171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6030074370 msgr2=0x7f6030074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:31.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 --2- 172.21.15.67:0/693654171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6030074370 0x7f6030074780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f60300c8f70 tx=0x7f602401ec90 comp rx=0 tx=0).stop 2024-04-25T15:52:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 -- 172.21.15.67:0/693654171 shutdown_connections 2024-04-25T15:52:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 --2- 172.21.15.67:0/693654171 >> v2:172.21.15.67:3300/0 conn(0x7f60300c7fc0 0x7f60300c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 --2- 172.21.15.67:0/693654171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6030074370 0x7f6030074780 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 -- 172.21.15.67:0/693654171 >> 172.21.15.67:0/693654171 conn(0x7f6030070db0 msgr2=0x7f60300ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 -- 172.21.15.67:0/693654171 shutdown_connections 2024-04-25T15:52:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.322+0000 7f60363b4640 1 -- 172.21.15.67:0/693654171 wait complete. 2024-04-25T15:52:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 Processor -- start 2024-04-25T15:52:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 -- start start 2024-04-25T15:52:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f603006c640 0x7f603014dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6030074370 0x7f60301537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60300c7fc0 0x7f603014a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6030073e50 con 0x7f6030074370 2024-04-25T15:52:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6030073b70 con 0x7f603006c640 2024-04-25T15:52:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.326+0000 7f60363b4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6030073420 con 0x7f60300c7fc0 2024-04-25T15:52:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f603006c640 0x7f603014dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f603006c640 0x7f603014dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:44180/0 (socket says 172.21.15.67:44180) 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 -- 172.21.15.67:0/1845060032 learned_addr learned my addr 172.21.15.67:0/1845060032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f6034951640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60300c7fc0 0x7f603014a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602f7fe640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6030074370 0x7f60301537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 -- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60300c7fc0 msgr2=0x7f603014a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60300c7fc0 0x7f603014a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 -- 172.21.15.67:0/1845060032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6030074370 msgr2=0x7f60301537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6030074370 0x7f60301537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 -- 172.21.15.67:0/1845060032 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603015be20 con 0x7f603006c640 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602ffff640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f603006c640 0x7f603014dc00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f602000c420 tx=0x7f602000c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602cff9640 1 -- 172.21.15.67:0/1845060032 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6020017070 con 0x7f603006c640 2024-04-25T15:52:31.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602cff9640 1 -- 172.21.15.67:0/1845060032 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60200052d0 con 0x7f603006c640 2024-04-25T15:52:31.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603015c100 con 0x7f603006c640 2024-04-25T15:52:31.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602cff9640 1 -- 172.21.15.67:0/1845060032 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6020003840 con 0x7f603006c640 2024-04-25T15:52:31.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f603014d6e0 con 0x7f603006c640 2024-04-25T15:52:31.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f603006a480 con 0x7f603006c640 2024-04-25T15:52:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.330+0000 7f602cff9640 1 -- 172.21.15.67:0/1845060032 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f602001a070 con 0x7f603006c640 2024-04-25T15:52:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.334+0000 7f602cff9640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f601c06f400 0x7f601c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.334+0000 7f602cff9640 1 -- 172.21.15.67:0/1845060032 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5857+0+0 (secure 0 0 0) 0x7f602008e6d0 con 0x7f603006c640 2024-04-25T15:52:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.334+0000 7f602f7fe640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f601c06f400 0x7f601c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.334+0000 7f602f7fe640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f601c06f400 0x7f601c071910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f60240b2000 tx=0x7f6024008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:31.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.338+0000 7f602cff9640 1 -- 172.21.15.67:0/1845060032 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6020005480 con 0x7f603006c640 2024-04-25T15:52:31.409 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:31.409 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:31.426 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:31.428 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:31.422+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:31.529 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:31.527+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:31.617 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:31.617 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:31.630 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:31.627+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:31.634 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:31.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:31.670+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f60300cb070 con 0x7f603006c640 2024-04-25T15:52:31.731 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:31.727+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:31.824 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:31.824 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:31.832 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:31.831+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:31.841 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:31.932 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:31.931+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:32.032 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:32.026+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:32.036 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:32.036 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:32.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:32.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.114+0000 7f602cff9640 1 -- 172.21.15.67:0/1845060032 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v55) ==== 97+0+0 (secure 0 0 0) 0x7f602005af60 con 0x7f603006c640 2024-04-25T15:52:32.123 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:52:32.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f601c06f400 msgr2=0x7f601c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:32.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f601c06f400 0x7f601c071910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f60240b2000 tx=0x7f6024008040 comp rx=0 tx=0).stop 2024-04-25T15:52:32.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f603006c640 msgr2=0x7f603014dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f603006c640 0x7f603014dc00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f602000c420 tx=0x7f602000c8e0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 shutdown_connections 2024-04-25T15:52:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f601c06f400 0x7f601c071910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60300c7fc0 0x7f603014a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6030074370 0x7f60301537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 --2- 172.21.15.67:0/1845060032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f603006c640 0x7f603014dc00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 >> 172.21.15.67:0/1845060032 conn(0x7f6030070db0 msgr2=0x7f60300ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:32.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 shutdown_connections 2024-04-25T15:52:32.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.126+0000 7f60363b4640 1 -- 172.21.15.67:0/1845060032 wait complete. 2024-04-25T15:52:32.134 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:32.130+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:32.235 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:32.235+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:32.260 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:32.260 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:32.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:32.367 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:32.367+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:32.436 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:32.430+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:32.471 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:32.489 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:32.537 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:32.530+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:32.638 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:32.635+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:32.682 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:32.682 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:32.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:32.739 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:32.734+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:32.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:32.839 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:32.839+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:32.893 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:32.893 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:32.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:32.959 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:32.959+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda89adc640 1 Processor -- start 2024-04-25T15:52:32.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda89adc640 1 -- start start 2024-04-25T15:52:32.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda89adc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda84156f10 0x7fda840cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:32.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda89adc640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fda8415d620 0x7fda841673f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:32.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda89adc640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda84160fa0 con 0x7fda8415d210 2024-04-25T15:52:32.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda89adc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fda84157320 con 0x7fda84156f10 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda89adc640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fda840c6d10 con 0x7fda8415d620 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda84156f10 0x7fda840cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda84156f10 0x7fda840cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42994/0 (socket says 172.21.15.67:42994) 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda837fe640 1 -- 172.21.15.67:0/174350811 learned_addr learned my addr 172.21.15.67:0/174350811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda83fff640 1 --2- 172.21.15.67:0/174350811 >> v2:172.21.15.176:3300/0 conn(0x7fda8415d620 0x7fda841673f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda837fe640 1 -- 172.21.15.67:0/174350811 >> v2:172.21.15.176:3300/0 conn(0x7fda8415d620 msgr2=0x7fda841673f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda837fe640 1 --2- 172.21.15.67:0/174350811 >> v2:172.21.15.176:3300/0 conn(0x7fda8415d620 0x7fda841673f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda837fe640 1 -- 172.21.15.67:0/174350811 >> v1:172.21.15.176:6789/0 conn(0x7fda8415d210 legacy=0x7fda840cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:32.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.970+0000 7fda837fe640 1 -- 172.21.15.67:0/174350811 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda84160420 con 0x7fda84156f10 2024-04-25T15:52:32.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.974+0000 7fda837fe640 1 --2- 172.21.15.67:0/174350811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda840cbfc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fda84071f00 tx=0x7fda7401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e04d3e7a5d55982 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:32.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.974+0000 7fda827fc640 1 -- 172.21.15.67:0/174350811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda740ab610 con 0x7fda84156f10 2024-04-25T15:52:32.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.974+0000 7fda827fc640 1 -- 172.21.15.67:0/174350811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda740b0070 con 0x7fda84156f10 2024-04-25T15:52:32.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.974+0000 7fda827fc640 1 -- 172.21.15.67:0/174350811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda740ab9d0 con 0x7fda84156f10 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 -- 172.21.15.67:0/174350811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 msgr2=0x7fda840cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 --2- 172.21.15.67:0/174350811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda840cbfc0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fda84071f00 tx=0x7fda7401c520 comp rx=0 tx=0).stop 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 -- 172.21.15.67:0/174350811 shutdown_connections 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 --2- 172.21.15.67:0/174350811 >> v2:172.21.15.176:3300/0 conn(0x7fda8415d620 0x7fda841673f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 --2- 172.21.15.67:0/174350811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda840cbfc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 -- 172.21.15.67:0/174350811 >> 172.21.15.67:0/174350811 conn(0x7fda84068c70 msgr2=0x7fda84071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 -- 172.21.15.67:0/174350811 shutdown_connections 2024-04-25T15:52:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 -- 172.21.15.67:0/174350811 wait complete. 2024-04-25T15:52:32.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 Processor -- start 2024-04-25T15:52:32.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.978+0000 7fda89adc640 1 -- start start 2024-04-25T15:52:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda89adc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda8415dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda89adc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415d210 0x7fda840c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda8415dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda8415dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42998/0 (socket says 172.21.15.67:42998) 2024-04-25T15:52:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 -- 172.21.15.67:0/4171688276 learned_addr learned my addr 172.21.15.67:0/4171688276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda8415d620 0x7fda8414b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:32.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda8415da30 con 0x7fda84156f10 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda8415ccf0 con 0x7fda8415d620 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda841600c0 con 0x7fda8415d210 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415d210 0x7fda840c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda8415d620 0x7fda8414b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415d210 msgr2=0x7fda840c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415d210 0x7fda840c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda8415d620 msgr2=0x7fda8414b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda8415d620 0x7fda8414b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda840c8370 con 0x7fda84156f10 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda8415d620 0x7fda8414b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda8415dcb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fda740ad040 tx=0x7fda7401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda740b4800 con 0x7fda84156f10 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda740b49c0 con 0x7fda84156f10 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8415e630 con 0x7fda84156f10 2024-04-25T15:52:32.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.982+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda740b4db0 con 0x7fda84156f10 2024-04-25T15:52:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8415ebf0 con 0x7fda84156f10 2024-04-25T15:52:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fda8406a480 con 0x7fda84156f10 2024-04-25T15:52:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fda740c7390 con 0x7fda84156f10 2024-04-25T15:52:32.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda6bfff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f400 0x7fda70071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:32.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5857+0+0 (secure 0 0 0) 0x7fda740fa440 con 0x7fda84156f10 2024-04-25T15:52:32.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f400 0x7fda70071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v55) ==== 54+0+16 (secure 0 0 0) 0x7fda740c4060 con 0x7fda84156f10 2024-04-25T15:52:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fda8415fb60 0x7fda8415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda840ca010 con 0x7fda8415fb60 2024-04-25T15:52:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fda8415fb60 0x7fda8415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.986+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f400 0x7fda70071910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fda840c9060 tx=0x7fda78009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.990+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fda8415fb60 0x7fda8415edc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:32.990+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda840ca010 con 0x7fda8415fb60 2024-04-25T15:52:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.018+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda840c9be0 con 0x7fda8415fb60 2024-04-25T15:52:33.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.018+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda840c9be0 con 0x7fda8415fb60 2024-04-25T15:52:33.026 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:33.026 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.022+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fda840cb840 0x7fda840cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.022+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fda840cb260 con 0x7fda840cb840 2024-04-25T15:52:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.022+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fda840cb840 0x7fda840cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.022+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fda840cb840 0x7fda840cd970 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.022+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda840cb260 con 0x7fda840cb840 2024-04-25T15:52:33.040 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:33.039+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:33.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.050+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda84149500 con 0x7fda840cb840 2024-04-25T15:52:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.050+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda84149500 con 0x7fda840cb840 2024-04-25T15:52:33.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.050+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fda840ce3c0 0x7fda841f9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.050+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fda840ce7d0 con 0x7fda840ce3c0 2024-04-25T15:52:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.054+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fda840ce3c0 0x7fda841f9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.054+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fda840ce3c0 0x7fda841f9f10 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.054+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda840ce7d0 con 0x7fda840ce3c0 2024-04-25T15:52:33.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.078+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda841fa450 con 0x7fda840ce3c0 2024-04-25T15:52:33.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.082+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda841fa450 con 0x7fda840ce3c0 2024-04-25T15:52:33.088 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:33.089 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:33.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.082+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fda84063090 0x7fda840634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:33.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.082+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fda840639e0 con 0x7fda84063090 2024-04-25T15:52:33.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.082+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fda84063090 0x7fda840634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:33.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.082+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fda84063090 0x7fda840634a0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:33.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.086+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda840639e0 con 0x7fda84063090 2024-04-25T15:52:33.109 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:33.110 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.110+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda84063ef0 con 0x7fda84063090 2024-04-25T15:52:33.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.114+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda84063ef0 con 0x7fda84063090 2024-04-25T15:52:33.119 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.114+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fda841552e0 0x7fda841556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.114+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fda84155c30 con 0x7fda841552e0 2024-04-25T15:52:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.114+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fda841552e0 0x7fda841556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.114+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fda841552e0 0x7fda841556f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.118+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda84155c30 con 0x7fda841552e0 2024-04-25T15:52:33.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:33.141 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:33.134+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.142+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda84155c30 con 0x7fda841552e0 2024-04-25T15:52:33.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.150+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda84155c30 con 0x7fda841552e0 2024-04-25T15:52:33.154 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:33.154 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:33.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.150+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fda8402cef0 0x7fda8402d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:33.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.150+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fda8402d840 con 0x7fda8402cef0 2024-04-25T15:52:33.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.150+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fda8402cef0 0x7fda8402d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:33.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.150+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fda8402cef0 0x7fda8402d300 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:33.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.150+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda8402d840 con 0x7fda8402cef0 2024-04-25T15:52:33.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.178+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda8402d840 con 0x7fda8402cef0 2024-04-25T15:52:33.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.178+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda8402d840 con 0x7fda8402cef0 2024-04-25T15:52:33.185 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.178+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fda8420d1c0 0x7fda8420f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.178+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fda842123b0 con 0x7fda8420d1c0 2024-04-25T15:52:33.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.178+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fda8420d1c0 0x7fda8420f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:33.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.182+0000 7fda83fff640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fda8420d1c0 0x7fda8420f5e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:33.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.182+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda842123b0 con 0x7fda8420d1c0 2024-04-25T15:52:33.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.210+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda8420fb20 con 0x7fda8420d1c0 2024-04-25T15:52:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.210+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda8420fb20 con 0x7fda8420d1c0 2024-04-25T15:52:33.217 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.210+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fda842101a0 0x7fda842125c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.210+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fda84215570 con 0x7fda842101a0 2024-04-25T15:52:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.214+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fda842101a0 0x7fda842125c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.214+0000 7fda837fe640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fda842101a0 0x7fda842125c0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.214+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda84215570 con 0x7fda842101a0 2024-04-25T15:52:33.242 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:33.238+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:33.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.238+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fda84000960 con 0x7fda842101a0 2024-04-25T15:52:33.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda6bfff640 1 -- 172.21.15.67:0/4171688276 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fda84000960 con 0x7fda842101a0 2024-04-25T15:52:33.248 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fda8415fb60 msgr2=0x7fda8415edc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fda8415fb60 0x7fda8415edc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fda840cb840 msgr2=0x7fda840cd970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fda840cb840 0x7fda840cd970 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fda840ce3c0 msgr2=0x7fda841f9f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fda840ce3c0 0x7fda841f9f10 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda83fff640 1 -- 172.21.15.67:0/4171688276 reap_dead start 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fda84063090 msgr2=0x7fda840634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fda84063090 0x7fda840634a0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fda841552e0 msgr2=0x7fda841556f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fda841552e0 0x7fda841556f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fda8402cef0 msgr2=0x7fda8402d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fda8402cef0 0x7fda8402d300 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fda8420d1c0 msgr2=0x7fda8420f5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fda8420d1c0 0x7fda8420f5e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fda842101a0 msgr2=0x7fda842125c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fda842101a0 0x7fda842125c0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda83fff640 1 -- 172.21.15.67:0/4171688276 reap_dead start 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f400 msgr2=0x7fda70071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f400 0x7fda70071910 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fda840c9060 tx=0x7fda78009040 comp rx=0 tx=0).stop 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 msgr2=0x7fda8415dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda8415dcb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fda740ad040 tx=0x7fda7401ad60 comp rx=0 tx=0).stop 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 shutdown_connections 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f400 0x7fda70071910 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 --2- 172.21.15.67:0/4171688276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84156f10 0x7fda8415dcb0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 >> 172.21.15.67:0/4171688276 conn(0x7fda84068c70 msgr2=0x7fda8414ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 shutdown_connections 2024-04-25T15:52:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:33.242+0000 7fda89adc640 1 -- 172.21.15.67:0/4171688276 wait complete. 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 134.22166491199999, 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.324 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:52:33.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.75806409699999, 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01409815, 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 25 2024-04-25T15:52:33.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562066+0000", 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562072+0000", 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.703e-06 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562156+0000", 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3486000000000007e-05 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562165+0000", 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.568966+0000", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006801372 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569110+0000", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014385799999999999 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569344+0000", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023376500000000001 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.573357+0000", 2024-04-25T15:52:33.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040127870000000003 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.575955+0000", 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025982969999999998 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576077+0000", 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121966 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576163+0000", 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6058999999999997e-05 2024-04-25T15:52:33.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.631258661, 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013368089999999999, 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.335 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688872+0000", 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0470000000000001e-06 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688892+0000", 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9143e-05 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688940+0000", 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8343000000000001e-05 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688961+0000", 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0925999999999999e-05 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694131+0000", 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051698639999999997 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694283+0000", 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015229800000000001 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694506+0000", 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222995 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.698950+0000", 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044438300000000002 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702065+0000", 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031151719999999998 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702159+0000", 2024-04-25T15:52:33.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4089e-05 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702238+0000", 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9382999999999995e-05 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.63113210899999, 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013712703999999999, 2024-04-25T15:52:33.340 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 44 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688998+0000", 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689301+0000", 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000302684 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689729+0000", 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000428486 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689746+0000", 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6690000000000001e-05 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.695997+0000", 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062505779999999997 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696164+0000", 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167045 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696383+0000", 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.701919+0000", 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055357399999999999 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702410+0000", 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00049136500000000001 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702581+0000", 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017146899999999999 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702710+0000", 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012822199999999999 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.345 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.63081976800001, 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013745535999999999, 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.346 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 45 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689310+0000", 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.127e-06 2024-04-25T15:52:33.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689317+0000", 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.973e-06 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689793+0000", 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000475468 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689804+0000", 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.082e-05 2024-04-25T15:52:33.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697359+0000", 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075556 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697536+0000", 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176658 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697753+0000", 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021705499999999999 2024-04-25T15:52:33.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702734+0000", 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049807779999999999 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702850+0000", 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116109 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702934+0000", 2024-04-25T15:52:33.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4429000000000001e-05 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703055+0000", 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120519 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.63080772199999, 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013995720999999999, 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.351 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 46 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:33.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689322+0000", 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8999999999999996e-07 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689329+0000", 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1899999999999998e-06 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689831+0000", 2024-04-25T15:52:33.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050143900000000001 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689839+0000", 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4069999999999993e-06 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699042+0000", 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092026810000000007 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699214+0000", 2024-04-25T15:52:33.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017275100000000001 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699442+0000", 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022718200000000001 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702745+0000", 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003303266 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703161+0000", 2024-04-25T15:52:33.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000416253 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703250+0000", 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8806000000000003e-05 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703317+0000", 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7056000000000003e-05 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.356 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.630796455, 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014300548, 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:33.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689491+0000", 2024-04-25T15:52:33.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015768600000000001 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689857+0000", 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000366313 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689865+0000", 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1470000000000001e-06 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700215+0000", 2024-04-25T15:52:33.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010350177 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700471+0000", 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521499999999999 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700879+0000", 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040780599999999998 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703339+0000", 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024602140000000001 2024-04-25T15:52:33.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703443+0000", 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104393 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703524+0000", 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0745e-05 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703633+0000", 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109226 2024-04-25T15:52:33.361 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.630632177, 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019412740000000001, 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.362 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 48 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689498+0000", 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5999999999999991e-07 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689505+0000", 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1339999999999998e-06 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689885+0000", 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000380553 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689894+0000", 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0660000000000004e-06 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703735+0000", 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013841509 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703880+0000", 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014450899999999999 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704121+0000", 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024184799999999999 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708286+0000", 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041643369999999997 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708530+0000", 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024477400000000001 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708710+0000", 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179715 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708910+0000", 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199335 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.367 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.63062016699999, 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019597107999999999, 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 49 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4700000000000001e-07 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689515+0000", 2024-04-25T15:52:33.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3770000000000004e-06 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689911+0000", 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039616199999999998 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689920+0000", 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2939999999999994e-06 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704840+0000", 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014919412999999999 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704990+0000", 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150388 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.705238+0000", 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000248392 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708485+0000", 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032464740000000001 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708936+0000", 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045072599999999998 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709037+0000", 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101482 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709106+0000", 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8752999999999997e-05 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.63061060999999, 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.373 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:33.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:52:33.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:52:33.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:52:33.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.378 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.63046198800001, 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.63045090899999, 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:52:33.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:52:33.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:52:33.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:52:33.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.32361242900001, 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019231747, 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.389 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 69 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996518+0000", 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6840000000000001e-06 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996645+0000", 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012677899999999999 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996696+0000", 2024-04-25T15:52:33.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0912999999999999e-05 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996718+0000", 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1931999999999999e-05 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000632+0000", 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003914509 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000756+0000", 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012338099999999999 2024-04-25T15:52:33.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000879+0000", 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012301599999999999 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009110+0000", 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082309819999999995 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015495+0000", 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063848969999999996 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015700+0000", 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020508399999999999 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015748+0000", 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8569999999999997e-05 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:33.394 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.32336038700001, 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019263037, 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 70 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996770+0000", 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7060000000000001e-06 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996786+0000", 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5497000000000001e-05 2024-04-25T15:52:33.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996990+0000", 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000203938 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997014+0000", 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4617000000000002e-05 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002337+0000", 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053230450000000002 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002417+0000", 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9548999999999993e-05 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002579+0000", 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162385 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009114+0000", 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00653442 2024-04-25T15:52:33.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015848+0000", 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067339909999999999 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015951+0000", 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102796 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016032+0000", 2024-04-25T15:52:33.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1093000000000004e-05 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.32333448700001, 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019380720000000001, 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.400 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 71 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996796+0000", 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.063e-06 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996810+0000", 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4100000000000001e-05 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997061+0000", 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000251797 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997076+0000", 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4970000000000001e-05 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003008+0000", 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005931487 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003148+0000", 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140097 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003371+0000", 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022273400000000001 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009117+0000", 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057466929999999998 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016106+0000", 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069880919999999996 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016144+0000", 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8686000000000002e-05 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016175+0000", 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1000999999999997e-05 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:33.405 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.323311821, 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019495743999999999, 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:33.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996818+0000", 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.046e-06 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996944+0000", 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012572199999999999 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997139+0000", 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019529899999999999 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997151+0000", 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2116000000000001e-05 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004407+0000", 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072560519999999998 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004474+0000", 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6810000000000006e-05 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004576+0000", 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101431 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009121+0000", 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045451099999999998 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016239+0000", 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071178359999999998 2024-04-25T15:52:33.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016280+0000", 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0914000000000002e-05 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016313+0000", 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3408000000000003e-05 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.410 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.32293255499999, 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019262988000000002, 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 73 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:33.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997198+0000", 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.347e-06 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997208+0000", 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0427e-05 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997419+0000", 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021052399999999999 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997433+0000", 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4687e-05 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005473+0000", 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080393610000000001 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005619+0000", 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146157 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005848+0000", 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229001 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009124+0000", 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003276288 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016366+0000", 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072413759999999999 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016408+0000", 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2119000000000001e-05 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016459+0000", 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1700999999999998e-05 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.415 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.32291508099999, 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019348008999999999, 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 74 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997215+0000", 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997220+0000", 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7200000000000003e-06 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997480+0000", 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026014000000000001 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997493+0000", 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2306999999999999e-05 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006512+0000", 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090195669999999992 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006591+0000", 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9058000000000001e-05 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006724+0000", 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013295000000000001 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009130+0000", 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024053659999999999 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073654790000000003 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016533+0000", 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7432999999999999e-05 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016562+0000", 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9379000000000001e-05 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.32290527500001, 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020191183000000001, 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:52:33.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997376+0000", 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015132899999999999 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997528+0000", 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152275 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997538+0000", 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0137e-05 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007159+0000", 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096209090000000004 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007248+0000", 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9078000000000003e-05 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007394+0000", 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014618699999999999 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009133+0000", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001739008 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017311+0000", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081775159999999993 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017360+0000", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9202000000000001e-05 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017415+0000", 2024-04-25T15:52:33.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4932000000000002e-05 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 133.32274787, 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:33.426 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:52:33.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:52:33.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:33.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:33.431 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:33.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:33.433 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:33.426+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:33.533 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:33.531+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:33.633 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:33.631+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:33.637 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:33.637 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:33.637 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:33.637 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:33.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:33.734 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:33.730+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:33.836 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:33.835+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:33.864 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:33.864 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:33.864 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:33.864 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:33.864 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:33.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:33.974 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:33.970+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:34.038 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:34.034+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:34.136 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 134.132205185, 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:34.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'48, mlcod=15'42)", 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 134.130575967, 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012230665999999999, 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:34.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004239+0000", 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004250+0000", 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1518999999999999e-05 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004274+0000", 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004292+0000", 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7458000000000001e-05 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005373+0000", 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010819359999999999 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005450+0000", 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6200999999999998e-05 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016441+0000", 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991445000000001 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016468+0000", 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6801999999999999e-05 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:71 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'49, mlcod=15'42)", 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 134.12978552800001, 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011466877, 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:34.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005029+0000", 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999999999999999e-06 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005040+0000", 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1199000000000001e-05 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005063+0000", 2024-04-25T15:52:34.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2342000000000001e-05 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005081+0000", 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7711999999999999e-05 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006374+0000", 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001293492 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006429+0000", 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5315999999999997e-05 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016469+0000", 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010039984 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5232000000000002e-05 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:72 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'50, mlcod=15'44)", 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:34.148 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 134.128508651, 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010219404, 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006306+0000", 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.48e-06 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006368+0000", 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1532000000000007e-05 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006394+0000", 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6917999999999999e-05 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006423+0000", 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8087000000000001e-05 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007111+0000", 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068813299999999995 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007172+0000", 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0918000000000002e-05 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016500+0000", 2024-04-25T15:52:34.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093286540000000005 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016524+0000", 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3682000000000001e-05 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:73 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'51, mlcod=15'44)", 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 134.12727671799999, 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090122410000000007, 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.152 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007538+0000", 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-06 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007559+0000", 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0302e-05 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007589+0000", 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0088000000000001e-05 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007643+0000", 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4419000000000002e-05 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010532+0000", 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028890249999999999 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010603+0000", 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1144999999999997e-05 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016525+0000", 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059219279999999999 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016549+0000", 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3444000000000001e-05 2024-04-25T15:52:34.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:76 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'44)", 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 134.12422328100001, 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010878550000000001, 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:34.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010591+0000", 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7899999999999997e-07 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010599+0000", 2024-04-25T15:52:34.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6000000000000007e-06 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010611+0000", 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2194999999999999e-05 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010620+0000", 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8529999999999998e-06 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016013+0000", 2024-04-25T15:52:34.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053922229999999998 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016045+0000", 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1933000000000001e-05 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021402+0000", 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053573559999999998 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021468+0000", 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6611000000000003e-05 2024-04-25T15:52:34.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.92336010699999, 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010088613, 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211464+0000", 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0740000000000001e-05 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211486+0000", 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1722e-05 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211654+0000", 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016800800000000001 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211678+0000", 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4283000000000001e-05 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212032+0000", 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035369900000000002 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212515+0000", 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048357899999999999 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024946400000000002 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217815+0000", 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050506989999999996 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221231+0000", 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003415567 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221412+0000", 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000180681 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221542+0000", 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130171 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.164 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.92331754200001, 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:34.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:34.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:27 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.923303883, 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010202413, 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211510+0000", 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211538+0000", 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7475e-05 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211876+0000", 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033851299999999998 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211898+0000", 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1648999999999999e-05 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212060+0000", 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016183899999999999 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212463+0000", 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040314700000000002 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:34.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212983+0000", 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052057999999999998 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220727+0000", 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077435919999999997 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221245+0000", 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051843400000000002 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221590+0000", 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034479999999999998 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221712+0000", 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121477 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.923270157, 2024-04-25T15:52:34.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:34.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:34.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.92325807500001, 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:34.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.923237903, 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:34.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.922999565, 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:34.187 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:34.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:34.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:34.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:17 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 111.885265649, 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010948628, 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.191 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249551+0000", 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2329999999999998e-06 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249572+0000", 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0979e-05 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249640+0000", 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8326999999999998e-05 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249668+0000", 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8368999999999999e-05 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250082+0000", 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041399200000000002 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250425+0000", 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034239200000000001 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253136+0000", 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027107210000000001 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256784+0000", 2024-04-25T15:52:34.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036481980000000001 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260324+0000", 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035401909999999998 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260418+0000", 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3692999999999999e-05 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260496+0000", 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8533000000000002e-05 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.195 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:18 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 111.885231428, 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011222823999999999, 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249583+0000", 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1799999999999999e-06 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249591+0000", 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0669999999999992e-06 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249819+0000", 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022795000000000001 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249865+0000", 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5561e-05 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254040+0000", 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041751720000000004 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254349+0000", 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030965399999999997 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254663+0000", 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031403600000000001 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258978+0000", 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043149959999999998 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260612+0000", 2024-04-25T15:52:34.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00163383 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260715+0000", 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010311599999999999 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260805+0000", 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9262e-05 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.200 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:7 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.803955156, 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086917899999999996, 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330859+0000", 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7000000000000005e-07 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330864+0000", 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1120000000000004e-06 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331053+0000", 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189261 2024-04-25T15:52:34.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331064+0000", 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1260000000000001e-05 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331313+0000", 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024867500000000001 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331526+0000", 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000212721 2024-04-25T15:52:34.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331732+0000", 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000206839 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335642+0000", 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039095639999999999 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339325+0000", 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036827790000000002 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339449+0000", 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123924 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339550+0000", 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101085 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:8 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.803947065, 2024-04-25T15:52:34.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010026810000000001, 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330867+0000", 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8899999999999999e-07 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330871+0000", 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2089999999999999e-06 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331105+0000", 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000234381 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331119+0000", 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4165000000000001e-05 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332225+0000", 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011058140000000001 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332348+0000", 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122872 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332557+0000", 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000208364 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336449+0000", 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038921329999999999 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340746+0000", 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042971779999999996 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340840+0000", 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4109000000000003e-05 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340893+0000", 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2995999999999999e-05 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:32 2.1f 2:fe09a0b8:::202.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:34.210 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.80392730200001, 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090313700000000004, 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330887+0000", 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1799999999999995e-07 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330890+0000", 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.427e-06 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331211+0000", 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000320639 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331220+0000", 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4669999999999993e-06 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335755+0000", 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349500000000003 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336042+0000", 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028675400000000001 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339917+0000", 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038755149999999999 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:1 2.1f 2:fb684bcd:::204.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.209197228999997, 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012576726, 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000002e-07 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925629+0000", 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3002e-05 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925764+0000", 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013477399999999999 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925779+0000", 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228000000000001e-05 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925847+0000", 2024-04-25T15:52:34.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7980999999999998e-05 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925951+0000", 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010333299999999999 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926085+0000", 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013462200000000001 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929258+0000", 2024-04-25T15:52:34.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003172607 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938074+0000", 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088162910000000004 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938129+0000", 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5234e-05 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938193+0000", 2024-04-25T15:52:34.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3367000000000005e-05 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:31 2.1f 2:ff4f7703:::204.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 90.209182491999997, 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012700365, 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5499999999999999e-07 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925633+0000", 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.796e-06 2024-04-25T15:52:34.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925809+0000", 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017651200000000001 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925824+0000", 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5071e-05 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926689+0000", 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086453799999999998 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926825+0000", 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013552799999999999 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938331+0000", 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011506565 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.222 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:34.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:52:34.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:34.224 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:34.223+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:34.324 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:34.323+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:34.425 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:34.423+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:34.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.462+0000 7fe6f23cb640 1 Processor -- start 2024-04-25T15:52:34.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.462+0000 7fe6f23cb640 1 -- start start 2024-04-25T15:52:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6f23cb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec074370 0x7fe6ec074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6f23cb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ec0c92d0 con 0x7fe6ec06c640 2024-04-25T15:52:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6f23cb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ec06e210 con 0x7fe6ec0c7fc0 2024-04-25T15:52:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6f23cb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ec06e4d0 con 0x7fe6ec074370 2024-04-25T15:52:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6ebfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe6ec06c640 0x7fe6ec15b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49788/0 (socket says 172.21.15.67:49788) 2024-04-25T15:52:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6ebfff640 1 -- 172.21.15.67:0/2186751071 learned_addr learned my addr 172.21.15.67:0/2186751071 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:34.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6eb7fe640 1 --2- 172.21.15.67:0/2186751071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec074370 0x7fe6ec074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6eaffd640 1 -- 172.21.15.67:0/2186751071 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2102633770 0 0) 0x7fe6ec0c92d0 con 0x7fe6ec06c640 2024-04-25T15:52:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6eaffd640 1 -- 172.21.15.67:0/2186751071 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6d4003530 con 0x7fe6ec06c640 2024-04-25T15:52:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6eb7fe640 1 -- 172.21.15.67:0/2186751071 >> v1:172.21.15.176:6789/0 conn(0x7fe6ec0c7fc0 legacy=0x7fe6ec164f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6eb7fe640 1 -- 172.21.15.67:0/2186751071 >> v1:172.21.15.67:6789/0 conn(0x7fe6ec06c640 legacy=0x7fe6ec15b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.466+0000 7fe6eb7fe640 1 -- 172.21.15.67:0/2186751071 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec06e7b0 con 0x7fe6ec074370 2024-04-25T15:52:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6eb7fe640 1 --2- 172.21.15.67:0/2186751071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec074370 0x7fe6ec074780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097f0 tx=0x7fe6e001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2c16b1a85ba394c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6eaffd640 1 -- 172.21.15.67:0/2186751071 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b5070 con 0x7fe6ec074370 2024-04-25T15:52:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6eaffd640 1 -- 172.21.15.67:0/2186751071 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e00a89d0 con 0x7fe6ec074370 2024-04-25T15:52:34.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6eaffd640 1 -- 172.21.15.67:0/2186751071 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00a8d80 con 0x7fe6ec074370 2024-04-25T15:52:34.476 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:34.476 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:34.476 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6f23cb640 1 -- 172.21.15.67:0/2186751071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec074370 msgr2=0x7fe6ec074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6f23cb640 1 --2- 172.21.15.67:0/2186751071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec074370 0x7fe6ec074780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097f0 tx=0x7fe6e001ecd0 comp rx=0 tx=0).stop 2024-04-25T15:52:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6f23cb640 1 -- 172.21.15.67:0/2186751071 shutdown_connections 2024-04-25T15:52:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6f23cb640 1 --2- 172.21.15.67:0/2186751071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec074370 0x7fe6ec074780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6f23cb640 1 -- 172.21.15.67:0/2186751071 >> 172.21.15.67:0/2186751071 conn(0x7fe6ec070db0 msgr2=0x7fe6ec0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6f23cb640 1 -- 172.21.15.67:0/2186751071 shutdown_connections 2024-04-25T15:52:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.470+0000 7fe6f23cb640 1 -- 172.21.15.67:0/2186751071 wait complete. 2024-04-25T15:52:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.474+0000 7fe6f23cb640 1 Processor -- start 2024-04-25T15:52:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.474+0000 7fe6f23cb640 1 -- start start 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec06c640 0x7fe6ec1484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6ec074370 0x7fe6ec141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6ec0c7fc0 0x7fe6ec14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6ec074370 0x7fe6ec141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6ec074370 0x7fe6ec141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43010/0 (socket says 172.21.15.67:43010) 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 -- 172.21.15.67:0/111837388 learned_addr learned my addr 172.21.15.67:0/111837388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec073e50 con 0x7fe6ec074370 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec073b70 con 0x7fe6ec0c7fc0 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6ebfff640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec06c640 0x7fe6ec1484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ec073420 con 0x7fe6ec06c640 2024-04-25T15:52:34.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f0968640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6ec0c7fc0 0x7fe6ec14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 -- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec06c640 msgr2=0x7fe6ec1484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec06c640 0x7fe6ec1484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 -- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6ec0c7fc0 msgr2=0x7fe6ec14f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6ec0c7fc0 0x7fe6ec14f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec148c30 con 0x7fe6ec074370 2024-04-25T15:52:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6eb7fe640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6ec074370 0x7fe6ec141ea0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097f0 tx=0x7fe6e001ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:34.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6e8ff9640 1 -- 172.21.15.67:0/111837388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00bf070 con 0x7fe6ec074370 2024-04-25T15:52:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6e8ff9640 1 -- 172.21.15.67:0/111837388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e00b0700 con 0x7fe6ec074370 2024-04-25T15:52:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6e8ff9640 1 -- 172.21.15.67:0/111837388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00b5040 con 0x7fe6ec074370 2024-04-25T15:52:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ec148a20 con 0x7fe6ec074370 2024-04-25T15:52:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.478+0000 7fe6f23cb640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ec147a00 con 0x7fe6ec074370 2024-04-25T15:52:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.482+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6c40052a0 con 0x7fe6ec074370 2024-04-25T15:52:34.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.482+0000 7fe6e8ff9640 1 -- 172.21.15.67:0/111837388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe6e00b0ad0 con 0x7fe6ec074370 2024-04-25T15:52:34.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.482+0000 7fe6e8ff9640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6d806f400 0x7fe6d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:34.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.482+0000 7fe6e8ff9640 1 -- 172.21.15.67:0/111837388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5857+0+0 (secure 0 0 0) 0x7fe6e012baa0 con 0x7fe6ec074370 2024-04-25T15:52:34.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.482+0000 7fe6ebfff640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6d806f400 0x7fe6d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:34.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.486+0000 7fe6ebfff640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6d806f400 0x7fe6d8071910 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0011f0 tx=0x7fe6dc007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.490+0000 7fe6e8ff9640 1 -- 172.21.15.67:0/111837388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe6e0018830 con 0x7fe6ec074370 2024-04-25T15:52:34.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:34.525 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:34.523+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:34.626 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:34.623+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:34.707 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:34.707 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:34.707 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:34.707 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:34.707 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:34.727 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:34.727+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:34.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:34.828 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:34.822+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:34.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:34.826+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fe6c4005590 con 0x7fe6ec074370 2024-04-25T15:52:34.929 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:34.927+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:34.940 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:34.940 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:34.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:34.940 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:34.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.727203785, 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013243108, 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0946e-05 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211617+0000", 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211761+0000", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014392699999999999 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211793+0000", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1568e-05 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212109+0000", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031626100000000003 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212422+0000", 2024-04-25T15:52:34.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031341099999999998 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213414+0000", 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099192799999999999 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223830+0000", 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010415654999999999 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224524+0000", 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069455899999999998 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224662+0000", 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013742600000000001 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224828+0000", 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016641799999999999 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.72716248, 2024-04-25T15:52:34.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013508035, 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211628+0000", 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3400000000000001e-06 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211641+0000", 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2866e-05 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027786899999999998 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211973+0000", 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4401000000000003e-05 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214895+0000", 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029218109999999999 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215130+0000", 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023500499999999999 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215556+0000", 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042636900000000001 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224481+0000", 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089248869999999994 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224861+0000", 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038038699999999999 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225017+0000", 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155296 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225135+0000", 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117804 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.727145101, 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013948439999999999, 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211645+0000", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1699999999999997e-07 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211680+0000", 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5374999999999999e-05 2024-04-25T15:52:34.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212061+0000", 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038129700000000001 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212135+0000", 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4103000000000005e-05 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218486+0000", 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063501709999999999 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218861+0000", 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037518799999999999 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219238+0000", 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037732500000000002 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225340+0000", 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061016430000000003 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225414+0000", 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.436e-05 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225489+0000", 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4882999999999994e-05 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225592+0000", 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103378 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.727103376, 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013711255, 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211687+0000", 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-06 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211694+0000", 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6329999999999996e-06 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212201+0000", 2024-04-25T15:52:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050685200000000002 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212239+0000", 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7537999999999999e-05 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004023726 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216562+0000", 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029964199999999997 2024-04-25T15:52:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033712999999999999 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223552+0000", 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066527330000000001 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225162+0000", 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016101869999999999 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225266+0000", 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010402400000000001 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225397+0000", 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130684 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:6 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:34.959 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.722830049, 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198140000000006, 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215961+0000", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5469999999999999e-06 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215983+0000", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2014999999999998e-05 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216017+0000", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3114999999999997e-05 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216043+0000", 2024-04-25T15:52:34.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6755999999999999e-05 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216409+0000", 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036599799999999997 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216466+0000", 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6947000000000002e-05 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224337+0000", 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078708230000000007 2024-04-25T15:52:34.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224479+0000", 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014161299999999999 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:7 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.72267984299999, 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198700000000006, 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.963 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216110+0000", 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3570000000000001e-06 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216125+0000", 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4772e-05 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216151+0000", 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5496000000000001e-05 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216172+0000", 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1095e-05 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217019+0000", 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084760800000000002 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217071+0000", 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1894000000000002e-05 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224489+0000", 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074175710000000004 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224629+0000", 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014007700000000001 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:8 2.10 e20/17 2:09401a81:::601.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.72099494699999, 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069206529999999997, 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217796+0000", 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9069999999999999e-06 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217816+0000", 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0296e-05 2024-04-25T15:52:34.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217847+0000", 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0877999999999998e-05 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217872+0000", 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4749e-05 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218099+0000", 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022737199999999999 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8900000000000002e-05 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224635+0000", 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064771480000000003 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224715+0000", 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9402999999999999e-05 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:9 2.10 e20/17 2:09401a81:::601.00000000:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.719880069, 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058821409999999996, 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218911+0000", 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218930+0000", 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8916e-05 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218960+0000", 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0065000000000001e-05 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218984+0000", 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4528999999999998e-05 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219206+0000", 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022124300000000001 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219263+0000", 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7003000000000003e-05 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224719+0000", 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054565830000000001 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224791+0000", 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1928999999999995e-05 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:11 2.10 e20/17 2:0bd6d154:::602.00000000:head v 20'6, mlcod=0'0)", 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.71750243699999, 2024-04-25T15:52:34.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050276770000000004, 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221288+0000", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221306+0000", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719000000000001e-05 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221336+0000", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0026e-05 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221365+0000", 2024-04-25T15:52:34.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8725e-05 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222395+0000", 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001030078 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222455+0000", 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9803999999999999e-05 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226163+0000", 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037085629999999998 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226314+0000", 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150829 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:19 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.69115073899999, 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010474059000000001, 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247641+0000", 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.129e-06 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247667+0000", 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6089e-05 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247827+0000", 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000159977 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247854+0000", 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6971999999999999e-05 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248122+0000", 2024-04-25T15:52:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026745499999999999 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248469+0000", 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034735899999999999 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248784+0000", 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031528900000000001 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251768+0000", 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002983383 2024-04-25T15:52:34.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257863+0000", 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006095339 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258009+0000", 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000145921 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258112+0000", 2024-04-25T15:52:34.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103146 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:20 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.691111968, 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010923117, 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247688+0000", 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3659999999999998e-06 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247937+0000", 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024977700000000002 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247958+0000", 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249612+0000", 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016540089999999999 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249860+0000", 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024840600000000001 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250352+0000", 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049215499999999998 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255543+0000", 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051905060000000001 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258273+0000", 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027298460000000002 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258467+0000", 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019409900000000001 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258600+0000", 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013348099999999999 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:23 2.15 2:aa0f8bf4:::101.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.691096915, 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082242619999999995, 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.988 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247728+0000", 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5049000000000003e-05 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248028+0000", 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029943400000000001 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248050+0000", 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2722e-05 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248203+0000", 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015264 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248563+0000", 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036047500000000002 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248940+0000", 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037679400000000001 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252184+0000", 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003243821 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255610+0000", 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034255370000000002 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255793+0000", 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018366899999999999 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255916+0000", 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122998 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.69105498, 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:52:34.993 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:52:34.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:34.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:4 2.10 e20/17 2:0f65c9b8:::60a.00000000:head v 20'10, mlcod=20'7)", 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 112.687697207, 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054254350000000002, 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251094+0000", 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.136e-06 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:34.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251114+0000", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0003000000000001e-05 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251147+0000", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2947999999999997e-05 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251171+0000", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3845999999999998e-05 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251347+0000", 2024-04-25T15:52:35.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176813 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251408+0000", 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0470000000000002e-05 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256382+0000", 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049743510000000001 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256517+0000", 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013486800000000001 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.001 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:3 2.0 e20/17 2:03625d61:::614.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.60814793900001, 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060303220000000003, 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:35.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330643+0000", 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9850000000000001e-06 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330658+0000", 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5386000000000001e-05 2024-04-25T15:52:35.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330686+0000", 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7971e-05 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330702+0000", 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5268999999999999e-05 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330841+0000", 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013921800000000001 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330881+0000", 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0052000000000002e-05 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336596+0000", 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057151320000000004 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336671+0000", 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5308999999999993e-05 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:4 2.0 e20/17 2:03625d61:::614.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.604876448, 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068526890000000003, 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:35.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333914+0000", 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3859999999999999e-06 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333929+0000", 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5508000000000001e-05 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333945+0000", 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5968999999999999e-05 2024-04-25T15:52:35.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333961+0000", 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5160999999999999e-05 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334268+0000", 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030782800000000001 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334291+0000", 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2459e-05 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340696+0000", 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064053749999999996 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340765+0000", 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9003000000000003e-05 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:6 2.9 2:941d59bc:::61f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.051826148000004, 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054913510000000002, 2024-04-25T15:52:35.009 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886964+0000", 2024-04-25T15:52:35.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9999999999999996e-07 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886973+0000", 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1400000000000006e-06 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887182+0000", 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020973699999999999 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887195+0000", 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2502e-05 2024-04-25T15:52:35.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888149+0000", 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095407200000000004 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888285+0000", 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013629399999999999 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888481+0000", 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000195343 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890469+0000", 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019881650000000001 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892371+0000", 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019022620000000001 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892426+0000", 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4404999999999998e-05 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:35.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892454+0000", 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8631000000000001e-05 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.051745303000004, 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052764769999999999, 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:35.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8400000000000004e-07 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887048+0000", 2024-04-25T15:52:35.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7390000000000001e-06 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887454+0000", 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040556499999999998 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887460+0000", 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8639999999999998e-06 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888968+0000", 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015077599999999999 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889063+0000", 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4658000000000001e-05 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889206+0000", 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014365600000000001 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890596+0000", 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390023 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:35.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892161+0000", 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015643219999999999 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892259+0000", 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7955000000000007e-05 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892320+0000", 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1351000000000003e-05 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.018 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:24 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 97.051738907000001, 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055188479999999998, 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887051+0000", 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1699999999999998e-07 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887063+0000", 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2575000000000001e-05 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887475+0000", 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041185899999999999 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887481+0000", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0970000000000002e-06 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887550+0000", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9258000000000007e-05 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887693+0000", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014290400000000001 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887898+0000", 2024-04-25T15:52:35.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020492000000000001 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889987+0000", 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020884929999999999 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892508+0000", 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002521644 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892542+0000", 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3654e-05 2024-04-25T15:52:35.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892569+0000", 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6927000000000002e-05 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.011572064999996, 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010008022, 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:35.023 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927218+0000", 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927222+0000", 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6519999999999997e-06 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927339+0000", 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116705 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927350+0000", 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0995000000000001e-05 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:35.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928080+0000", 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072958300000000001 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928207+0000", 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012715 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928433+0000", 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022608599999999999 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:35.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070747400000000004 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937109+0000", 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079685439999999993 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9806999999999998e-05 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937225+0000", 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6313000000000001e-05 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.027 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:35.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:35.028 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:35.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:35.029 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:35.022+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:35.130 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:35.126+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6e8ff9640 1 -- 172.21.15.67:0/111837388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v56) ==== 91+0+0 (secure 0 0 0) 0x7fe6e00f8320 con 0x7fe6ec074370 2024-04-25T15:52:35.154 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:52:35.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6d806f400 msgr2=0x7fe6d8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6d806f400 0x7fe6d8071910 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0011f0 tx=0x7fe6dc007d40 comp rx=0 tx=0).stop 2024-04-25T15:52:35.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6ec074370 msgr2=0x7fe6ec141ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6ec074370 0x7fe6ec141ea0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00097f0 tx=0x7fe6e001ba70 comp rx=0 tx=0).stop 2024-04-25T15:52:35.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 shutdown_connections 2024-04-25T15:52:35.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6d806f400 0x7fe6d8071910 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6ec0c7fc0 0x7fe6ec14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6ec074370 0x7fe6ec141ea0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 --2- 172.21.15.67:0/111837388 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6ec06c640 0x7fe6ec1484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 >> 172.21.15.67:0/111837388 conn(0x7fe6ec070db0 msgr2=0x7fe6ec152970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:35.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.150+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 shutdown_connections 2024-04-25T15:52:35.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.154+0000 7fe6d27fc640 1 -- 172.21.15.67:0/111837388 wait complete. 2024-04-25T15:52:35.230 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:35.226+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:35.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:35.275 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:35.275 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:35.275 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:35.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:35.342 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:35.338+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:35.432 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:35.431+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:35.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.462+0000 7faede7cf640 1 Processor -- start 2024-04-25T15:52:35.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.462+0000 7faede7cf640 1 -- start start 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faede7cf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faed8156330 0x7faed806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faede7cf640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7faed806cb80 0x7faed80c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faede7cf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faed806fa10 con 0x7faed80c4f30 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faede7cf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faed80ca6c0 con 0x7faed8156330 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faede7cf640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7faed806cf90 con 0x7faed806cb80 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faed7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faed8156330 0x7faed806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faed7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faed8156330 0x7faed806c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43016/0 (socket says 172.21.15.67:43016) 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faed7fff640 1 -- 172.21.15.67:0/418386975 learned_addr learned my addr 172.21.15.67:0/418386975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faecffff640 1 --2- 172.21.15.67:0/418386975 >> v2:172.21.15.176:3300/0 conn(0x7faed806cb80 0x7faed80c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faed7fff640 1 -- 172.21.15.67:0/418386975 >> v2:172.21.15.176:3300/0 conn(0x7faed806cb80 msgr2=0x7faed80c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:35.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faed7fff640 1 --2- 172.21.15.67:0/418386975 >> v2:172.21.15.176:3300/0 conn(0x7faed806cb80 0x7faed80c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faed7fff640 1 -- 172.21.15.67:0/418386975 >> v1:172.21.15.176:6789/0 conn(0x7faed80c4f30 legacy=0x7faed8158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.466+0000 7faed7fff640 1 -- 172.21.15.67:0/418386975 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed80ca8f0 con 0x7faed8156330 2024-04-25T15:52:35.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faed7fff640 1 --2- 172.21.15.67:0/418386975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed806c640 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7faed80b1130 tx=0x7faec8021180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7febf17cba3c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faed77fe640 1 -- 172.21.15.67:0/418386975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80b7070 con 0x7faed8156330 2024-04-25T15:52:35.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faed77fe640 1 -- 172.21.15.67:0/418386975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec80b2440 con 0x7faed8156330 2024-04-25T15:52:35.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faed77fe640 1 -- 172.21.15.67:0/418386975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80b2810 con 0x7faed8156330 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 -- 172.21.15.67:0/418386975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 msgr2=0x7faed806c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 --2- 172.21.15.67:0/418386975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed806c640 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7faed80b1130 tx=0x7faec8021180 comp rx=0 tx=0).stop 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 -- 172.21.15.67:0/418386975 shutdown_connections 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 --2- 172.21.15.67:0/418386975 >> v2:172.21.15.176:3300/0 conn(0x7faed806cb80 0x7faed80c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 --2- 172.21.15.67:0/418386975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed806c640 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 -- 172.21.15.67:0/418386975 >> 172.21.15.67:0/418386975 conn(0x7faed80ce1c0 msgr2=0x7faed80ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 -- 172.21.15.67:0/418386975 shutdown_connections 2024-04-25T15:52:35.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.470+0000 7faede7cf640 1 -- 172.21.15.67:0/418386975 wait complete. 2024-04-25T15:52:35.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.474+0000 7faede7cf640 1 Processor -- start 2024-04-25T15:52:35.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.474+0000 7faede7cf640 1 -- start start 2024-04-25T15:52:35.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.474+0000 7faede7cf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faed806cb80 0x7faed80b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.474+0000 7faede7cf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faed80c4f30 0x7faed81f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.474+0000 7faede7cf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faede7cf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faed80c3d80 con 0x7faed8156330 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed80b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43022/0 (socket says 172.21.15.67:43022) 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 -- 172.21.15.67:0/4110305668 learned_addr learned my addr 172.21.15.67:0/4110305668 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faed806cb80 0x7faed80b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faecffff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faed80c4f30 0x7faed81f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faede7cf640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faed80c43d0 con 0x7faed80c4f30 2024-04-25T15:52:35.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faede7cf640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faed80c4060 con 0x7faed806cb80 2024-04-25T15:52:35.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faed806cb80 msgr2=0x7faed80b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faed806cb80 0x7faed80b25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faed80c4f30 msgr2=0x7faed81f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faed80c4f30 0x7faed81f3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed8158760 con 0x7faed8156330 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faedcd6c640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed80b1bc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faed000c760 tx=0x7faed000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0016020 con 0x7faed8156330 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed0006d80 con 0x7faed8156330 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0003bc0 con 0x7faed8156330 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faede7cf640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed81f3860 con 0x7faed8156330 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faede7cf640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed80b32e0 con 0x7faed8156330 2024-04-25T15:52:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.478+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7faeb0005280 con 0x7faed8156330 2024-04-25T15:52:35.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7faed000a850 con 0x7faed8156330 2024-04-25T15:52:35.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faed57fa640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faeb806f430 0x7faeb8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5857+0+0 (secure 0 0 0) 0x7faed008fbf0 con 0x7faed8156330 2024-04-25T15:52:35.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v56) ==== 54+0+16 (secure 0 0 0) 0x7faed0020020 con 0x7faed8156330 2024-04-25T15:52:35.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faeb806f430 0x7faeb8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faeb00045f0 0x7faeb0004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faeb0005560 con 0x7faeb00045f0 2024-04-25T15:52:35.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faecffff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faeb00045f0 0x7faeb0004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faecffff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faeb00045f0 0x7faeb0004a00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.482+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faeb806f430 0x7faeb8071940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faec80b4000 tx=0x7faec8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.486+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb0005560 con 0x7faeb00045f0 2024-04-25T15:52:35.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.514+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb0001280 con 0x7faeb00045f0 2024-04-25T15:52:35.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.514+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb0001280 con 0x7faeb00045f0 2024-04-25T15:52:35.521 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:35.522 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:35.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.518+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faeb0001d40 0x7faeb00029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.518+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7faeb000b3a0 con 0x7faeb0001d40 2024-04-25T15:52:35.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.518+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faeb0001d40 0x7faeb00029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.518+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faeb0001d40 0x7faeb00029a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.518+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb000b3a0 con 0x7faeb0001d40 2024-04-25T15:52:35.533 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:35.531+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:35.547 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:35.547 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:35.547 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:35.547 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:35.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.546+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb00031b0 con 0x7faeb0001d40 2024-04-25T15:52:35.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.550+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb00031b0 con 0x7faeb0001d40 2024-04-25T15:52:35.554 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.550+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faeb0003c30 0x7faeb0009c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.550+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7faeb000a3f0 con 0x7faeb0003c30 2024-04-25T15:52:35.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.550+0000 7faecffff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faeb0003c30 0x7faeb0009c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.550+0000 7faecffff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faeb0003c30 0x7faeb0009c90 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.550+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb000a3f0 con 0x7faeb0003c30 2024-04-25T15:52:35.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.578+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb000a9c0 con 0x7faeb0003c30 2024-04-25T15:52:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.582+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb000a9c0 con 0x7faeb0003c30 2024-04-25T15:52:35.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.582+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faeb000d560 0x7faeb000f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.582+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7faeb0012900 con 0x7faeb000d560 2024-04-25T15:52:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.582+0000 7faedcd6c640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faeb000d560 0x7faeb000f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.582+0000 7faedcd6c640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faeb000d560 0x7faeb000f980 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.582+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb0012900 con 0x7faeb000d560 2024-04-25T15:52:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.610+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb00101e0 con 0x7faeb000d560 2024-04-25T15:52:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.614+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb00101e0 con 0x7faeb000d560 2024-04-25T15:52:35.618 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.614+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faeb0010930 0x7faeb0010d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.614+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7faeb0012590 con 0x7faeb0010930 2024-04-25T15:52:35.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.614+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faeb0010930 0x7faeb0010d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.614+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faeb0010930 0x7faeb0010d40 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.618+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb0012590 con 0x7faeb0010930 2024-04-25T15:52:35.634 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:35.630+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.642+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb0012560 con 0x7faeb0010930 2024-04-25T15:52:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.646+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb0012560 con 0x7faeb0010930 2024-04-25T15:52:35.653 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.646+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faeb00117c0 0x7faeb0011bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.646+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7faeb0019420 con 0x7faeb00117c0 2024-04-25T15:52:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.650+0000 7faecffff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faeb00117c0 0x7faeb0011bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.650+0000 7faecffff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faeb00117c0 0x7faeb0011bf0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.650+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb0019420 con 0x7faeb00117c0 2024-04-25T15:52:35.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.678+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb0014b50 con 0x7faeb00117c0 2024-04-25T15:52:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.682+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb0014b50 con 0x7faeb00117c0 2024-04-25T15:52:35.688 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:35.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.682+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faeb0015270 0x7faeb0015680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.682+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7faeb0015f30 con 0x7faeb0015270 2024-04-25T15:52:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.682+0000 7faedcd6c640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faeb0015270 0x7faeb0015680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.686+0000 7faedcd6c640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faeb0015270 0x7faeb0015680 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.686+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb0015f30 con 0x7faeb0015270 2024-04-25T15:52:35.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.714+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb0015f30 con 0x7faeb0015270 2024-04-25T15:52:35.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.718+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb0015f30 con 0x7faeb0015270 2024-04-25T15:52:35.722 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:35.723 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:35.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.718+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faeb001a7c0 0x7faeb001cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.718+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7faeb001fa30 con 0x7faeb001a7c0 2024-04-25T15:52:35.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.718+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faeb001a7c0 0x7faeb001cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.718+0000 7faed7fff640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faeb001a7c0 0x7faeb001cbe0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.722+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faeb001fa30 con 0x7faeb001a7c0 2024-04-25T15:52:35.735 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:35.730+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:35.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.746+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7faeb001d120 con 0x7faeb001a7c0 2024-04-25T15:52:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faed57fa640 1 -- 172.21.15.67:0/4110305668 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7faeb001d120 con 0x7faeb001a7c0 2024-04-25T15:52:35.757 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faeb00045f0 msgr2=0x7faeb0004a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faeb00045f0 0x7faeb0004a00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faeb0001d40 msgr2=0x7faeb00029a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7faeb0001d40 0x7faeb00029a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faeb0003c30 msgr2=0x7faeb0009c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7faeb0003c30 0x7faeb0009c90 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.750+0000 7faedcd6c640 1 -- 172.21.15.67:0/4110305668 reap_dead start 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faeb000d560 msgr2=0x7faeb000f980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7faeb000d560 0x7faeb000f980 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faeb0010930 msgr2=0x7faeb0010d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7faeb0010930 0x7faeb0010d40 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faeb00117c0 msgr2=0x7faeb0011bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7faeb00117c0 0x7faeb0011bf0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faeb0015270 msgr2=0x7faeb0015680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7faeb0015270 0x7faeb0015680 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faeb001a7c0 msgr2=0x7faeb001cbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7faeb001a7c0 0x7faeb001cbe0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faedcd6c640 1 -- 172.21.15.67:0/4110305668 reap_dead start 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faeb806f430 msgr2=0x7faeb8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faeb806f430 0x7faeb8071940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faec80b4000 tx=0x7faec8008040 comp rx=0 tx=0).stop 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 msgr2=0x7faed80b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed80b1bc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faed000c760 tx=0x7faed000cc20 comp rx=0 tx=0).stop 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 shutdown_connections 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faeb806f430 0x7faeb8071940 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 --2- 172.21.15.67:0/4110305668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faed8156330 0x7faed80b1bc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 >> 172.21.15.67:0/4110305668 conn(0x7faed80ce1c0 msgr2=0x7faed8154e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 shutdown_connections 2024-04-25T15:52:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:35.754+0000 7faece7fc640 1 -- 172.21.15.67:0/4110305668 wait complete. 2024-04-25T15:52:35.776 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:35.776 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:35.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:35.836 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:35.835+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:35.937 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:35.935+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:36.037 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:36.034+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:36.047 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:36.048 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:36.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:36.139 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:36.139 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:36.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:52:36.158 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:36.154+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:36.239 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:36.234+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:36.320 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:36.320 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:36.340 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:36.339+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:36.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:36.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.354+0000 7f86ca5a3640 1 Processor -- start 2024-04-25T15:52:36.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.354+0000 7f86ca5a3640 1 -- start start 2024-04-25T15:52:36.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.354+0000 7f86ca5a3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86c4071bc0 0x7f86c40cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.354+0000 7f86ca5a3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f86c40cc100 0x7f86c415a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86ca5a3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86c415de10 con 0x7f86c415ac60 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86ca5a3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f86c415ebb0 con 0x7f86c4071bc0 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86ca5a3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f86c415e720 con 0x7f86c40cc100 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86c4071bc0 0x7f86c40cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86c4071bc0 0x7f86c40cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43024/0 (socket says 172.21.15.67:43024) 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 -- 172.21.15.67:0/129889131 learned_addr learned my addr 172.21.15.67:0/129889131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c8b40640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f86c415ac60 0x7f86c40ca540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49798/0 (socket says 172.21.15.67:49798) 2024-04-25T15:52:36.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86bbfff640 1 --2- 172.21.15.67:0/129889131 >> v2:172.21.15.176:3300/0 conn(0x7f86c40cc100 0x7f86c415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 -- 172.21.15.67:0/129889131 >> v2:172.21.15.176:3300/0 conn(0x7f86c40cc100 msgr2=0x7f86c415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 --2- 172.21.15.67:0/129889131 >> v2:172.21.15.176:3300/0 conn(0x7f86c40cc100 0x7f86c415a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 -- 172.21.15.67:0/129889131 >> v1:172.21.15.67:6789/0 conn(0x7f86c415ac60 legacy=0x7f86c40ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:36.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 -- 172.21.15.67:0/129889131 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c4072390 con 0x7f86c4071bc0 2024-04-25T15:52:36.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.358+0000 7f86c3fff640 1 --2- 172.21.15.67:0/129889131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c40cbbc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f86c4071000 tx=0x7f86b401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c324fcf856276058 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:36.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86c37fe640 1 -- 172.21.15.67:0/129889131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b40ab650 con 0x7f86c4071bc0 2024-04-25T15:52:36.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86c37fe640 1 -- 172.21.15.67:0/129889131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86b40b0070 con 0x7f86c4071bc0 2024-04-25T15:52:36.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86c37fe640 1 -- 172.21.15.67:0/129889131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b40aba50 con 0x7f86c4071bc0 2024-04-25T15:52:36.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 -- 172.21.15.67:0/129889131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 msgr2=0x7f86c40cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:36.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/129889131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c40cbbc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f86c4071000 tx=0x7f86b401c540 comp rx=0 tx=0).stop 2024-04-25T15:52:36.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 -- 172.21.15.67:0/129889131 shutdown_connections 2024-04-25T15:52:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/129889131 >> v2:172.21.15.176:3300/0 conn(0x7f86c40cc100 0x7f86c415a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/129889131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c40cbbc0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 -- 172.21.15.67:0/129889131 >> 172.21.15.67:0/129889131 conn(0x7f86c40700b0 msgr2=0x7f86c40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 -- 172.21.15.67:0/129889131 shutdown_connections 2024-04-25T15:52:36.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.362+0000 7f86ca5a3640 1 -- 172.21.15.67:0/129889131 wait complete. 2024-04-25T15:52:36.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.366+0000 7f86ca5a3640 1 Processor -- start 2024-04-25T15:52:36.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.366+0000 7f86ca5a3640 1 -- start start 2024-04-25T15:52:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c41f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c40cc100 0x7f86c41475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c41f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c41f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43040/0 (socket says 172.21.15.67:43040) 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 -- 172.21.15.67:0/2411269074 learned_addr learned my addr 172.21.15.67:0/2411269074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c415ac60 0x7f86c4148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86bbfff640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c40cc100 0x7f86c41475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86c4072150 con 0x7f86c4071bc0 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86c4071fd0 con 0x7f86c415ac60 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86c40cac50 con 0x7f86c40cc100 2024-04-25T15:52:36.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c8b40640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c415ac60 0x7f86c4148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 -- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c40cc100 msgr2=0x7f86c41475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c40cc100 0x7f86c41475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 -- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c415ac60 msgr2=0x7f86c4148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c415ac60 0x7f86c4148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c4147f00 con 0x7f86c4071bc0 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c3fff640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c41f3c40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f86b401a170 tx=0x7f86b401ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c17fa640 1 -- 172.21.15.67:0/2411269074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b40be5f0 con 0x7f86c4071bc0 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c17fa640 1 -- 172.21.15.67:0/2411269074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86b40be790 con 0x7f86c4071bc0 2024-04-25T15:52:36.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86c4147ae0 con 0x7f86c4071bc0 2024-04-25T15:52:36.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86c17fa640 1 -- 172.21.15.67:0/2411269074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b40beb60 con 0x7f86c4071bc0 2024-04-25T15:52:36.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.370+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86c4143a00 con 0x7f86c4071bc0 2024-04-25T15:52:36.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.374+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86c406a480 con 0x7f86c4071bc0 2024-04-25T15:52:36.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.374+0000 7f86c17fa640 1 -- 172.21.15.67:0/2411269074 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f86b40b43f0 con 0x7f86c4071bc0 2024-04-25T15:52:36.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.374+0000 7f86c17fa640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86a406f430 0x7f86a4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:36.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.374+0000 7f86c17fa640 1 -- 172.21.15.67:0/2411269074 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5857+0+0 (secure 0 0 0) 0x7f86b412d170 con 0x7f86c4071bc0 2024-04-25T15:52:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.382+0000 7f86bbfff640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86a406f430 0x7f86a4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.382+0000 7f86bbfff640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86a406f430 0x7f86a4071940 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f86c41f4db0 tx=0x7f86ac009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:36.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.382+0000 7f86c17fa640 1 -- 172.21.15.67:0/2411269074 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f86b4016130 con 0x7f86c4071bc0 2024-04-25T15:52:36.441 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:36.439+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:36.542 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:36.538+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:36.564 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:36.564 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:36.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:36.642 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:36.638+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.698+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f86c4154260 con 0x7f86a406f430 2024-04-25T15:52:36.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.710+0000 7f86c17fa640 1 -- 172.21.15.67:0/2411269074 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251983 (secure 0 0 0) 0x7f86c4154260 con 0x7f86a406f430 2024-04-25T15:52:36.719 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:36.720 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:52:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86a406f430 msgr2=0x7f86a4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86a406f430 0x7f86a4071940 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f86c41f4db0 tx=0x7f86ac009190 comp rx=0 tx=0).stop 2024-04-25T15:52:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 msgr2=0x7f86c41f3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c41f3c40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f86b401a170 tx=0x7f86b401ad80 comp rx=0 tx=0).stop 2024-04-25T15:52:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 shutdown_connections 2024-04-25T15:52:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86a406f430 0x7f86a4071940 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c415ac60 0x7f86c4148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c40cc100 0x7f86c41475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 --2- 172.21.15.67:0/2411269074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86c4071bc0 0x7f86c41f3c40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 >> 172.21.15.67:0/2411269074 conn(0x7f86c40700b0 msgr2=0x7f86c40c9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 shutdown_connections 2024-04-25T15:52:36.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:36.714+0000 7f86ca5a3640 1 -- 172.21.15.67:0/2411269074 wait complete. 2024-04-25T15:52:36.743 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:36.743+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:36.746 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-04-25T15:52:35.842327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604666,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":319,"num_write_kb":1704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4003,"num_keys_recovered":22,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262,"ondisk_log_size":262,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":199040,"kb_used_data":5712,"kb_used_omap":10,"kb_used_meta":193205,"kb_avail":660403840,"statfs":{"total":676457349120,"available":676253532160,"internally_reserved":0,"allocated":5849088,"data_stored":3174536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":197842027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002076"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":81,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.115575+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:52:32.115575+0000","last_peered":"2024-04-25T15:52:32.115575+0000","last_clean":"2024-04-25T15:52:32.115575+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:52:32.115575+0000","last_undegraded":"2024-04-25T15:52:32.115575+0000","last_fullsized":"2024-04-25T15:52:32.115575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112621+0000","last_change":"2024-04-25T15:51:47.264956+0000","last_active":"2024-04-25T15:52:32.112621+0000","last_peered":"2024-04-25T15:52:32.112621+0000","last_clean":"2024-04-25T15:52:32.112621+0000","last_became_active":"2024-04-25T15:51:47.264571+0000","last_became_peered":"2024-04-25T15:51:47.264571+0000","last_unstale":"2024-04-25T15:52:32.112621+0000","last_undegraded":"2024-04-25T15:52:32.112621+0000","last_fullsized":"2024-04-25T15:52:32.112621+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:39.343444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":38,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.111705+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:52:32.111705+0000","last_peered":"2024-04-25T15:52:32.111705+0000","last_clean":"2024-04-25T15:52:32.111705+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:52:32.111705+0000","last_undegraded":"2024-04-25T15:52:32.111705+0000","last_fullsized":"2024-04-25T15:52:32.111705+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":83,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121375+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:52:32.121375+0000","last_peered":"2024-04-25T15:52:32.121375+0000","last_clean":"2024-04-25T15:52:32.121375+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:52:32.121375+0000","last_undegraded":"2024-04-25T15:52:32.121375+0000","last_fullsized":"2024-04-25T15:52:32.121375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":81,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.116266+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:52:32.116266+0000","last_peered":"2024-04-25T15:52:32.116266+0000","last_clean":"2024-04-25T15:52:32.116266+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:52:32.116266+0000","last_undegraded":"2024-04-25T15:52:32.116266+0000","last_fullsized":"2024-04-25T15:52:32.116266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121194+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:52:32.121194+0000","last_peered":"2024-04-25T15:52:32.121194+0000","last_clean":"2024-04-25T15:52:32.121194+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:52:32.121194+0000","last_undegraded":"2024-04-25T15:52:32.121194+0000","last_fullsized":"2024-04-25T15:52:32.121194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":81,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113596+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:52:32.113596+0000","last_peered":"2024-04-25T15:52:32.113596+0000","last_clean":"2024-04-25T15:52:32.113596+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:52:32.113596+0000","last_undegraded":"2024-04-25T15:52:32.113596+0000","last_fullsized":"2024-04-25T15:52:32.113596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112518+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:52:32.112518+0000","last_peered":"2024-04-25T15:52:32.112518+0000","last_clean":"2024-04-25T15:52:32.112518+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:52:32.112518+0000","last_undegraded":"2024-04-25T15:52:32.112518+0000","last_fullsized":"2024-04-25T15:52:32.112518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":75,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112373+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:52:32.112373+0000","last_peered":"2024-04-25T15:52:32.112373+0000","last_clean":"2024-04-25T15:52:32.112373+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:52:32.112373+0000","last_undegraded":"2024-04-25T15:52:32.112373+0000","last_fullsized":"2024-04-25T15:52:32.112373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113469+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:52:32.113469+0000","last_peered":"2024-04-25T15:52:32.113469+0000","last_clean":"2024-04-25T15:52:32.113469+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:52:32.113469+0000","last_undegraded":"2024-04-25T15:52:32.113469+0000","last_fullsized":"2024-04-25T15:52:32.113469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112357+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:52:32.112357+0000","last_peered":"2024-04-25T15:52:32.112357+0000","last_clean":"2024-04-25T15:52:32.112357+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:52:32.112357+0000","last_undegraded":"2024-04-25T15:52:32.112357+0000","last_fullsized":"2024-04-25T15:52:32.112357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.127494+0000","last_change":"2024-04-25T15:52:32.125058+0000","last_active":"2024-04-25T15:52:32.127494+0000","last_peered":"2024-04-25T15:52:32.127494+0000","last_clean":"2024-04-25T15:52:32.127494+0000","last_became_active":"2024-04-25T15:52:32.124575+0000","last_became_peered":"2024-04-25T15:52:32.124575+0000","last_unstale":"2024-04-25T15:52:32.127494+0000","last_undegraded":"2024-04-25T15:52:32.127494+0000","last_fullsized":"2024-04-25T15:52:32.127494+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:24:57.698573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.117753+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:52:32.117753+0000","last_peered":"2024-04-25T15:52:32.117753+0000","last_clean":"2024-04-25T15:52:32.117753+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:52:32.117753+0000","last_undegraded":"2024-04-25T15:52:32.117753+0000","last_fullsized":"2024-04-25T15:52:32.117753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":85,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114050+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:52:32.114050+0000","last_peered":"2024-04-25T15:52:32.114050+0000","last_clean":"2024-04-25T15:52:32.114050+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:52:32.114050+0000","last_undegraded":"2024-04-25T15:52:32.114050+0000","last_fullsized":"2024-04-25T15:52:32.114050+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":53,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.129421+0000","last_change":"2024-04-25T15:52:32.129421+0000","last_active":"2024-04-25T15:52:32.129421+0000","last_peered":"2024-04-25T15:52:32.129421+0000","last_clean":"2024-04-25T15:52:32.129421+0000","last_became_active":"2024-04-25T15:52:32.128747+0000","last_became_peered":"2024-04-25T15:52:32.128747+0000","last_unstale":"2024-04-25T15:52:32.129421+0000","last_undegraded":"2024-04-25T15:52:32.129421+0000","last_fullsized":"2024-04-25T15:52:32.129421+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:44.731855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121189+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:52:32.121189+0000","last_peered":"2024-04-25T15:52:32.121189+0000","last_clean":"2024-04-25T15:52:32.121189+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:52:32.121189+0000","last_undegraded":"2024-04-25T15:52:32.121189+0000","last_fullsized":"2024-04-25T15:52:32.121189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.122089+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:52:32.122089+0000","last_peered":"2024-04-25T15:52:32.122089+0000","last_clean":"2024-04-25T15:52:32.122089+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:52:32.122089+0000","last_undegraded":"2024-04-25T15:52:32.122089+0000","last_fullsized":"2024-04-25T15:52:32.122089+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":81,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112172+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:52:32.112172+0000","last_peered":"2024-04-25T15:52:32.112172+0000","last_clean":"2024-04-25T15:52:32.112172+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:52:32.112172+0000","last_undegraded":"2024-04-25T15:52:32.112172+0000","last_fullsized":"2024-04-25T15:52:32.112172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":81,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114668+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:52:32.114668+0000","last_peered":"2024-04-25T15:52:32.114668+0000","last_clean":"2024-04-25T15:52:32.114668+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:52:32.114668+0000","last_undegraded":"2024-04-25T15:52:32.114668+0000","last_fullsized":"2024-04-25T15:52:32.114668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.120016+0000","last_change":"2024-04-25T15:52:32.120016+0000","last_active":"2024-04-25T15:52:32.120016+0000","last_peered":"2024-04-25T15:52:32.120016+0000","last_clean":"2024-04-25T15:52:32.120016+0000","last_became_active":"2024-04-25T15:52:32.119425+0000","last_became_peered":"2024-04-25T15:52:32.119425+0000","last_unstale":"2024-04-25T15:52:32.120016+0000","last_undegraded":"2024-04-25T15:52:32.120016+0000","last_fullsized":"2024-04-25T15:52:32.120016+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:42:15.118421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":38,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.116196+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:52:32.116196+0000","last_peered":"2024-04-25T15:52:32.116196+0000","last_clean":"2024-04-25T15:52:32.116196+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:52:32.116196+0000","last_undegraded":"2024-04-25T15:52:32.116196+0000","last_fullsized":"2024-04-25T15:52:32.116196+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":75,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.111860+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:52:32.111860+0000","last_peered":"2024-04-25T15:52:32.111860+0000","last_clean":"2024-04-25T15:52:32.111860+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:52:32.111860+0000","last_undegraded":"2024-04-25T15:52:32.111860+0000","last_fullsized":"2024-04-25T15:52:32.111860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.123939+0000","last_change":"2024-04-25T15:52:32.123939+0000","last_active":"2024-04-25T15:52:32.123939+0000","last_peered":"2024-04-25T15:52:32.123939+0000","last_clean":"2024-04-25T15:52:32.123939+0000","last_became_active":"2024-04-25T15:52:32.122698+0000","last_became_peered":"2024-04-25T15:52:32.122698+0000","last_unstale":"2024-04-25T15:52:32.123939+0000","last_undegraded":"2024-04-25T15:52:32.123939+0000","last_fullsized":"2024-04-25T15:52:32.123939+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:09.597867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121344+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:52:32.121344+0000","last_peered":"2024-04-25T15:52:32.121344+0000","last_clean":"2024-04-25T15:52:32.121344+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:52:32.121344+0000","last_undegraded":"2024-04-25T15:52:32.121344+0000","last_fullsized":"2024-04-25T15:52:32.121344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.120389+0000","last_change":"2024-04-25T15:52:32.120389+0000","last_active":"2024-04-25T15:52:32.120389+0000","last_peered":"2024-04-25T15:52:32.120389+0000","last_clean":"2024-04-25T15:52:32.120389+0000","last_became_active":"2024-04-25T15:52:32.120052+0000","last_became_peered":"2024-04-25T15:52:32.120052+0000","last_unstale":"2024-04-25T15:52:32.120389+0000","last_undegraded":"2024-04-25T15:52:32.120389+0000","last_fullsized":"2024-04-25T15:52:32.120389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:45.982199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113601+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:52:32.113601+0000","last_peered":"2024-04-25T15:52:32.113601+0000","last_clean":"2024-04-25T15:52:32.113601+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:52:32.113601+0000","last_undegraded":"2024-04-25T15:52:32.113601+0000","last_fullsized":"2024-04-25T15:52:32.113601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112452+0000","last_change":"2024-04-25T15:51:47.271333+0000","last_active":"2024-04-25T15:52:32.112452+0000","last_peered":"2024-04-25T15:52:32.112452+0000","last_clean":"2024-04-25T15:52:32.112452+0000","last_became_active":"2024-04-25T15:51:47.270971+0000","last_became_peered":"2024-04-25T15:51:47.270971+0000","last_unstale":"2024-04-25T15:52:32.112452+0000","last_undegraded":"2024-04-25T15:52:32.112452+0000","last_fullsized":"2024-04-25T15:52:32.112452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:23.860435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.125522+0000","last_change":"2024-04-25T15:52:32.125522+0000","last_active":"2024-04-25T15:52:32.125522+0000","last_peered":"2024-04-25T15:52:32.125522+0000","last_clean":"2024-04-25T15:52:32.125522+0000","last_became_active":"2024-04-25T15:52:32.125187+0000","last_became_peered":"2024-04-25T15:52:32.125187+0000","last_unstale":"2024-04-25T15:52:32.125522+0000","last_undegraded":"2024-04-25T15:52:32.125522+0000","last_fullsized":"2024-04-25T15:52:32.125522+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.964571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.117673+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:52:32.117673+0000","last_peered":"2024-04-25T15:52:32.117673+0000","last_clean":"2024-04-25T15:52:32.117673+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:52:32.117673+0000","last_undegraded":"2024-04-25T15:52:32.117673+0000","last_fullsized":"2024-04-25T15:52:32.117673+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":28,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113465+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:52:32.113465+0000","last_peered":"2024-04-25T15:52:32.113465+0000","last_clean":"2024-04-25T15:52:32.113465+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:52:32.113465+0000","last_undegraded":"2024-04-25T15:52:32.113465+0000","last_fullsized":"2024-04-25T15:52:32.113465+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":91,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112121+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:52:32.112121+0000","last_peered":"2024-04-25T15:52:32.112121+0000","last_clean":"2024-04-25T15:52:32.112121+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:52:32.112121+0000","last_undegraded":"2024-04-25T15:52:32.112121+0000","last_fullsized":"2024-04-25T15:52:32.112121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.110704+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:52:32.110704+0000","last_peered":"2024-04-25T15:52:32.110704+0000","last_clean":"2024-04-25T15:52:32.110704+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:52:32.110704+0000","last_undegraded":"2024-04-25T15:52:32.110704+0000","last_fullsized":"2024-04-25T15:52:32.110704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":57,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.129719+0000","last_change":"2024-04-25T15:52:32.123541+0000","last_active":"2024-04-25T15:52:32.129719+0000","last_peered":"2024-04-25T15:52:32.129719+0000","last_clean":"2024-04-25T15:52:32.129719+0000","last_became_active":"2024-04-25T15:52:32.123158+0000","last_became_peered":"2024-04-25T15:52:32.123158+0000","last_unstale":"2024-04-25T15:52:32.129719+0000","last_undegraded":"2024-04-25T15:52:32.129719+0000","last_fullsized":"2024-04-25T15:52:32.129719+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.858171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":90,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.117788+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:52:32.117788+0000","last_peered":"2024-04-25T15:52:32.117788+0000","last_clean":"2024-04-25T15:52:32.117788+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:52:32.117788+0000","last_undegraded":"2024-04-25T15:52:32.117788+0000","last_fullsized":"2024-04-25T15:52:32.117788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.136630+0000","last_change":"2024-04-25T15:52:32.131021+0000","last_active":"2024-04-25T15:52:32.136630+0000","last_peered":"2024-04-25T15:52:32.136630+0000","last_clean":"2024-04-25T15:52:32.136630+0000","last_became_active":"2024-04-25T15:52:32.129920+0000","last_became_peered":"2024-04-25T15:52:32.129920+0000","last_unstale":"2024-04-25T15:52:32.136630+0000","last_undegraded":"2024-04-25T15:52:32.136630+0000","last_fullsized":"2024-04-25T15:52:32.136630+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:00.291863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":104,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.152478+0000","last_change":"2024-04-25T15:52:32.152478+0000","last_active":"2024-04-25T15:52:32.152478+0000","last_peered":"2024-04-25T15:52:32.152478+0000","last_clean":"2024-04-25T15:52:32.152478+0000","last_became_active":"2024-04-25T15:52:32.137069+0000","last_became_peered":"2024-04-25T15:52:32.137069+0000","last_unstale":"2024-04-25T15:52:32.152478+0000","last_undegraded":"2024-04-25T15:52:32.152478+0000","last_fullsized":"2024-04-25T15:52:32.152478+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:26.946219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114922+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:52:32.114922+0000","last_peered":"2024-04-25T15:52:32.114922+0000","last_clean":"2024-04-25T15:52:32.114922+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:52:32.114922+0000","last_undegraded":"2024-04-25T15:52:32.114922+0000","last_fullsized":"2024-04-25T15:52:32.114922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112625+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:52:32.112625+0000","last_peered":"2024-04-25T15:52:32.112625+0000","last_clean":"2024-04-25T15:52:32.112625+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:52:32.112625+0000","last_undegraded":"2024-04-25T15:52:32.112625+0000","last_fullsized":"2024-04-25T15:52:32.112625+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":96,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.145033+0000","last_change":"2024-04-25T15:52:32.145033+0000","last_active":"2024-04-25T15:52:32.145033+0000","last_peered":"2024-04-25T15:52:32.145033+0000","last_clean":"2024-04-25T15:52:32.145033+0000","last_became_active":"2024-04-25T15:52:32.136902+0000","last_became_peered":"2024-04-25T15:52:32.136902+0000","last_unstale":"2024-04-25T15:52:32.145033+0000","last_undegraded":"2024-04-25T15:52:32.145033+0000","last_fullsized":"2024-04-25T15:52:32.145033+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:59.330286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113283+0000","last_change":"2024-04-25T15:51:47.271089+0000","last_active":"2024-04-25T15:52:32.113283+0000","last_peered":"2024-04-25T15:52:32.113283+0000","last_clean":"2024-04-25T15:52:32.113283+0000","last_became_active":"2024-04-25T15:51:47.270705+0000","last_became_peered":"2024-04-25T15:51:47.270705+0000","last_unstale":"2024-04-25T15:52:32.113283+0000","last_undegraded":"2024-04-25T15:52:32.113283+0000","last_fullsized":"2024-04-25T15:52:32.113283+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:08.380566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":89,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113925+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:52:32.113925+0000","last_peered":"2024-04-25T15:52:32.113925+0000","last_clean":"2024-04-25T15:52:32.113925+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:52:32.113925+0000","last_undegraded":"2024-04-25T15:52:32.113925+0000","last_fullsized":"2024-04-25T15:52:32.113925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121788+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:52:32.121788+0000","last_peered":"2024-04-25T15:52:32.121788+0000","last_clean":"2024-04-25T15:52:32.121788+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:52:32.121788+0000","last_undegraded":"2024-04-25T15:52:32.121788+0000","last_fullsized":"2024-04-25T15:52:32.121788+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121920+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:52:32.121920+0000","last_peered":"2024-04-25T15:52:32.121920+0000","last_clean":"2024-04-25T15:52:32.121920+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:52:32.121920+0000","last_undegraded":"2024-04-25T15:52:32.121920+0000","last_fullsized":"2024-04-25T15:52:32.121920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.122367+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:52:32.122367+0000","last_peered":"2024-04-25T15:52:32.122367+0000","last_clean":"2024-04-25T15:52:32.122367+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:52:32.122367+0000","last_undegraded":"2024-04-25T15:52:32.122367+0000","last_fullsized":"2024-04-25T15:52:32.122367+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":58,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113807+0000","last_change":"2024-04-25T15:51:47.302272+0000","last_active":"2024-04-25T15:52:32.113807+0000","last_peered":"2024-04-25T15:52:32.113807+0000","last_clean":"2024-04-25T15:52:32.113807+0000","last_became_active":"2024-04-25T15:51:47.266158+0000","last_became_peered":"2024-04-25T15:51:47.266158+0000","last_unstale":"2024-04-25T15:52:32.113807+0000","last_undegraded":"2024-04-25T15:52:32.113807+0000","last_fullsized":"2024-04-25T15:52:32.113807+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:28.386086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":672,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.136289+0000","last_change":"2024-04-25T15:52:32.129340+0000","last_active":"2024-04-25T15:52:32.136289+0000","last_peered":"2024-04-25T15:52:32.136289+0000","last_clean":"2024-04-25T15:52:32.136289+0000","last_became_active":"2024-04-25T15:52:32.128859+0000","last_became_peered":"2024-04-25T15:52:32.128859+0000","last_unstale":"2024-04-25T15:52:32.136289+0000","last_undegraded":"2024-04-25T15:52:32.136289+0000","last_fullsized":"2024-04-25T15:52:32.136289+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:59:42.834204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'14","reported_seq":98,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112086+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:52:32.112086+0000","last_peered":"2024-04-25T15:52:32.112086+0000","last_clean":"2024-04-25T15:52:32.112086+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:52:32.112086+0000","last_undegraded":"2024-04-25T15:52:32.112086+0000","last_fullsized":"2024-04-25T15:52:32.112086+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":81,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113925+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:52:32.113925+0000","last_peered":"2024-04-25T15:52:32.113925+0000","last_clean":"2024-04-25T15:52:32.113925+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:52:32.113925+0000","last_undegraded":"2024-04-25T15:52:32.113925+0000","last_fullsized":"2024-04-25T15:52:32.113925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":24,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.151538+0000","last_change":"2024-04-25T15:52:32.150681+0000","last_active":"2024-04-25T15:52:32.151538+0000","last_peered":"2024-04-25T15:52:32.151538+0000","last_clean":"2024-04-25T15:52:32.151538+0000","last_became_active":"2024-04-25T15:52:32.125799+0000","last_became_peered":"2024-04-25T15:52:32.125799+0000","last_unstale":"2024-04-25T15:52:32.151538+0000","last_undegraded":"2024-04-25T15:52:32.151538+0000","last_fullsized":"2024-04-25T15:52:32.151538+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.858701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":38,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114096+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:52:32.114096+0000","last_peered":"2024-04-25T15:52:32.114096+0000","last_clean":"2024-04-25T15:52:32.114096+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:52:32.114096+0000","last_undegraded":"2024-04-25T15:52:32.114096+0000","last_fullsized":"2024-04-25T15:52:32.114096+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":89,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121285+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:52:32.121285+0000","last_peered":"2024-04-25T15:52:32.121285+0000","last_clean":"2024-04-25T15:52:32.121285+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:52:32.121285+0000","last_undegraded":"2024-04-25T15:52:32.121285+0000","last_fullsized":"2024-04-25T15:52:32.121285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":86,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.135585+0000","last_change":"2024-04-25T15:52:32.135585+0000","last_active":"2024-04-25T15:52:32.135585+0000","last_peered":"2024-04-25T15:52:32.135585+0000","last_clean":"2024-04-25T15:52:32.135585+0000","last_became_active":"2024-04-25T15:52:32.134857+0000","last_became_peered":"2024-04-25T15:52:32.134857+0000","last_unstale":"2024-04-25T15:52:32.135585+0000","last_undegraded":"2024-04-25T15:52:32.135585+0000","last_fullsized":"2024-04-25T15:52:32.135585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:58.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.144332+0000","last_change":"2024-04-25T15:52:32.144332+0000","last_active":"2024-04-25T15:52:32.144332+0000","last_peered":"2024-04-25T15:52:32.144332+0000","last_clean":"2024-04-25T15:52:32.144332+0000","last_became_active":"2024-04-25T15:52:32.124713+0000","last_became_peered":"2024-04-25T15:52:32.124713+0000","last_unstale":"2024-04-25T15:52:32.144332+0000","last_undegraded":"2024-04-25T15:52:32.144332+0000","last_fullsized":"2024-04-25T15:52:32.144332+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:15.176691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1029,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.110634+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:52:32.110634+0000","last_peered":"2024-04-25T15:52:32.110634+0000","last_clean":"2024-04-25T15:52:32.110634+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:52:32.110634+0000","last_undegraded":"2024-04-25T15:52:32.110634+0000","last_fullsized":"2024-04-25T15:52:32.110634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":146,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.111730+0000","last_change":"2024-04-25T15:51:47.287562+0000","last_active":"2024-04-25T15:52:32.111730+0000","last_peered":"2024-04-25T15:52:32.111730+0000","last_clean":"2024-04-25T15:52:32.111730+0000","last_became_active":"2024-04-25T15:51:47.264513+0000","last_became_peered":"2024-04-25T15:51:47.264513+0000","last_unstale":"2024-04-25T15:52:32.111730+0000","last_undegraded":"2024-04-25T15:52:32.111730+0000","last_fullsized":"2024-04-25T15:52:32.111730+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:51:11.555766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":86,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.130800+0000","last_change":"2024-04-25T15:52:32.130800+0000","last_active":"2024-04-25T15:52:32.130800+0000","last_peered":"2024-04-25T15:52:32.130800+0000","last_clean":"2024-04-25T15:52:32.130800+0000","last_became_active":"2024-04-25T15:52:32.129969+0000","last_became_peered":"2024-04-25T15:52:32.129969+0000","last_unstale":"2024-04-25T15:52:32.130800+0000","last_undegraded":"2024-04-25T15:52:32.130800+0000","last_fullsized":"2024-04-25T15:52:32.130800+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:45.521306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.157124+0000","last_change":"2024-04-25T15:52:32.155801+0000","last_active":"2024-04-25T15:52:32.157124+0000","last_peered":"2024-04-25T15:52:32.157124+0000","last_clean":"2024-04-25T15:52:32.157124+0000","last_became_active":"2024-04-25T15:52:32.129444+0000","last_became_peered":"2024-04-25T15:52:32.129444+0000","last_unstale":"2024-04-25T15:52:32.157124+0000","last_undegraded":"2024-04-25T15:52:32.157124+0000","last_fullsized":"2024-04-25T15:52:32.157124+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:46.358507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":86,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.129065+0000","last_change":"2024-04-25T15:52:32.129065+0000","last_active":"2024-04-25T15:52:32.129065+0000","last_peered":"2024-04-25T15:52:32.129065+0000","last_clean":"2024-04-25T15:52:32.129065+0000","last_became_active":"2024-04-25T15:52:32.128801+0000","last_became_peered":"2024-04-25T15:52:32.128801+0000","last_unstale":"2024-04-25T15:52:32.129065+0000","last_undegraded":"2024-04-25T15:52:32.129065+0000","last_fullsized":"2024-04-25T15:52:32.129065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:09.896373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112281+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:52:32.112281+0000","last_peered":"2024-04-25T15:52:32.112281+0000","last_clean":"2024-04-25T15:52:32.112281+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:52:32.112281+0000","last_undegraded":"2024-04-25T15:52:32.112281+0000","last_fullsized":"2024-04-25T15:52:32.112281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":86,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.129750+0000","last_change":"2024-04-25T15:52:32.129750+0000","last_active":"2024-04-25T15:52:32.129750+0000","last_peered":"2024-04-25T15:52:32.129750+0000","last_clean":"2024-04-25T15:52:32.129750+0000","last_became_active":"2024-04-25T15:52:32.129402+0000","last_became_peered":"2024-04-25T15:52:32.129402+0000","last_unstale":"2024-04-25T15:52:32.129750+0000","last_undegraded":"2024-04-25T15:52:32.129750+0000","last_fullsized":"2024-04-25T15:52:32.129750+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:48.203567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":61,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.135358+0000","last_change":"2024-04-25T15:52:32.135358+0000","last_active":"2024-04-25T15:52:32.135358+0000","last_peered":"2024-04-25T15:52:32.135358+0000","last_clean":"2024-04-25T15:52:32.135358+0000","last_became_active":"2024-04-25T15:52:32.129558+0000","last_became_peered":"2024-04-25T15:52:32.129558+0000","last_unstale":"2024-04-25T15:52:32.135358+0000","last_undegraded":"2024-04-25T15:52:32.135358+0000","last_fullsized":"2024-04-25T15:52:32.135358+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:49.940252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121516+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:52:32.121516+0000","last_peered":"2024-04-25T15:52:32.121516+0000","last_clean":"2024-04-25T15:52:32.121516+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:52:32.121516+0000","last_undegraded":"2024-04-25T15:52:32.121516+0000","last_fullsized":"2024-04-25T15:52:32.121516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":89,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112069+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:52:32.112069+0000","last_peered":"2024-04-25T15:52:32.112069+0000","last_clean":"2024-04-25T15:52:32.112069+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:52:32.112069+0000","last_undegraded":"2024-04-25T15:52:32.112069+0000","last_fullsized":"2024-04-25T15:52:32.112069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":81,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114200+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:52:32.114200+0000","last_peered":"2024-04-25T15:52:32.114200+0000","last_clean":"2024-04-25T15:52:32.114200+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:52:32.114200+0000","last_undegraded":"2024-04-25T15:52:32.114200+0000","last_fullsized":"2024-04-25T15:52:32.114200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":180,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113644+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:52:32.113644+0000","last_peered":"2024-04-25T15:52:32.113644+0000","last_clean":"2024-04-25T15:52:32.113644+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:52:32.113644+0000","last_undegraded":"2024-04-25T15:52:32.113644+0000","last_fullsized":"2024-04-25T15:52:32.113644+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":90,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.125266+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:52:32.125266+0000","last_peered":"2024-04-25T15:52:32.125266+0000","last_clean":"2024-04-25T15:52:32.125266+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:52:32.125266+0000","last_undegraded":"2024-04-25T15:52:32.125266+0000","last_fullsized":"2024-04-25T15:52:32.125266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113028+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:52:32.113028+0000","last_peered":"2024-04-25T15:52:32.113028+0000","last_clean":"2024-04-25T15:52:32.113028+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:52:32.113028+0000","last_undegraded":"2024-04-25T15:52:32.113028+0000","last_fullsized":"2024-04-25T15:52:32.113028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":106,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113599+0000","last_change":"2024-04-25T15:51:47.292263+0000","last_active":"2024-04-25T15:52:32.113599+0000","last_peered":"2024-04-25T15:52:32.113599+0000","last_clean":"2024-04-25T15:52:32.113599+0000","last_became_active":"2024-04-25T15:51:47.264780+0000","last_became_peered":"2024-04-25T15:51:47.264780+0000","last_unstale":"2024-04-25T15:52:32.113599+0000","last_undegraded":"2024-04-25T15:52:32.113599+0000","last_fullsized":"2024-04-25T15:52:32.113599+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:40.269146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.132000+0000","last_change":"2024-04-25T15:52:32.125579+0000","last_active":"2024-04-25T15:52:32.132000+0000","last_peered":"2024-04-25T15:52:32.132000+0000","last_clean":"2024-04-25T15:52:32.132000+0000","last_became_active":"2024-04-25T15:52:32.125247+0000","last_became_peered":"2024-04-25T15:52:32.125247+0000","last_unstale":"2024-04-25T15:52:32.132000+0000","last_undegraded":"2024-04-25T15:52:32.132000+0000","last_fullsized":"2024-04-25T15:52:32.132000+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:10.258830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":89,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112241+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:52:32.112241+0000","last_peered":"2024-04-25T15:52:32.112241+0000","last_clean":"2024-04-25T15:52:32.112241+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:52:32.112241+0000","last_undegraded":"2024-04-25T15:52:32.112241+0000","last_fullsized":"2024-04-25T15:52:32.112241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.111673+0000","last_change":"2024-04-25T15:51:47.255396+0000","last_active":"2024-04-25T15:52:32.111673+0000","last_peered":"2024-04-25T15:52:32.111673+0000","last_clean":"2024-04-25T15:52:32.111673+0000","last_became_active":"2024-04-25T15:51:47.254596+0000","last_became_peered":"2024-04-25T15:51:47.254596+0000","last_unstale":"2024-04-25T15:52:32.111673+0000","last_undegraded":"2024-04-25T15:52:32.111673+0000","last_fullsized":"2024-04-25T15:52:32.111673+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:06:53.573154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":111,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.121409+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:52:32.121409+0000","last_peered":"2024-04-25T15:52:32.121409+0000","last_clean":"2024-04-25T15:52:32.121409+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:52:32.121409+0000","last_undegraded":"2024-04-25T15:52:32.121409+0000","last_fullsized":"2024-04-25T15:52:32.121409+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.123848+0000","last_change":"2024-04-25T15:52:32.118757+0000","last_active":"2024-04-25T15:52:32.123848+0000","last_peered":"2024-04-25T15:52:32.123848+0000","last_clean":"2024-04-25T15:52:32.123848+0000","last_became_active":"2024-04-25T15:52:32.118380+0000","last_became_peered":"2024-04-25T15:52:32.118380+0000","last_unstale":"2024-04-25T15:52:32.123848+0000","last_undegraded":"2024-04-25T15:52:32.123848+0000","last_fullsized":"2024-04-25T15:52:32.123848+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:18.978899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112573+0000","last_change":"2024-04-25T15:51:47.263905+0000","last_active":"2024-04-25T15:52:32.112573+0000","last_peered":"2024-04-25T15:52:32.112573+0000","last_clean":"2024-04-25T15:52:32.112573+0000","last_became_active":"2024-04-25T15:51:47.263546+0000","last_became_peered":"2024-04-25T15:51:47.263546+0000","last_unstale":"2024-04-25T15:52:32.112573+0000","last_undegraded":"2024-04-25T15:52:32.112573+0000","last_fullsized":"2024-04-25T15:52:32.112573+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:14.832346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.122035+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:52:32.122035+0000","last_peered":"2024-04-25T15:52:32.122035+0000","last_clean":"2024-04-25T15:52:32.122035+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:52:32.122035+0000","last_undegraded":"2024-04-25T15:52:32.122035+0000","last_fullsized":"2024-04-25T15:52:32.122035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114770+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:52:32.114770+0000","last_peered":"2024-04-25T15:52:32.114770+0000","last_clean":"2024-04-25T15:52:32.114770+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:52:32.114770+0000","last_undegraded":"2024-04-25T15:52:32.114770+0000","last_fullsized":"2024-04-25T15:52:32.114770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":48,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112681+0000","last_change":"2024-04-25T15:51:47.263756+0000","last_active":"2024-04-25T15:52:32.112681+0000","last_peered":"2024-04-25T15:52:32.112681+0000","last_clean":"2024-04-25T15:52:32.112681+0000","last_became_active":"2024-04-25T15:51:47.263130+0000","last_became_peered":"2024-04-25T15:51:47.263130+0000","last_unstale":"2024-04-25T15:52:32.112681+0000","last_undegraded":"2024-04-25T15:52:32.112681+0000","last_fullsized":"2024-04-25T15:52:32.112681+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:38.024802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'9","reported_seq":58,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112724+0000","last_change":"2024-04-25T15:51:47.314396+0000","last_active":"2024-04-25T15:52:32.112724+0000","last_peered":"2024-04-25T15:52:32.112724+0000","last_clean":"2024-04-25T15:52:32.112724+0000","last_became_active":"2024-04-25T15:51:47.274984+0000","last_became_peered":"2024-04-25T15:51:47.274984+0000","last_unstale":"2024-04-25T15:52:32.112724+0000","last_undegraded":"2024-04-25T15:52:32.112724+0000","last_fullsized":"2024-04-25T15:52:32.112724+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:53.936857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":82,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113369+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:52:32.113369+0000","last_peered":"2024-04-25T15:52:32.113369+0000","last_clean":"2024-04-25T15:52:32.113369+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:52:32.113369+0000","last_undegraded":"2024-04-25T15:52:32.113369+0000","last_fullsized":"2024-04-25T15:52:32.113369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":91,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114072+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:52:32.114072+0000","last_peered":"2024-04-25T15:52:32.114072+0000","last_clean":"2024-04-25T15:52:32.114072+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:52:32.114072+0000","last_undegraded":"2024-04-25T15:52:32.114072+0000","last_fullsized":"2024-04-25T15:52:32.114072+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":85,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112511+0000","last_change":"2024-04-25T15:51:59.936821+0000","last_active":"2024-04-25T15:52:32.112511+0000","last_peered":"2024-04-25T15:52:32.112511+0000","last_clean":"2024-04-25T15:52:32.112511+0000","last_became_active":"2024-04-25T15:51:59.936474+0000","last_became_peered":"2024-04-25T15:51:59.936474+0000","last_unstale":"2024-04-25T15:52:32.112511+0000","last_undegraded":"2024-04-25T15:52:32.112511+0000","last_fullsized":"2024-04-25T15:52:32.112511+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:05.147722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":85,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113413+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:52:32.113413+0000","last_peered":"2024-04-25T15:52:32.113413+0000","last_clean":"2024-04-25T15:52:32.113413+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:52:32.113413+0000","last_undegraded":"2024-04-25T15:52:32.113413+0000","last_fullsized":"2024-04-25T15:52:32.113413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.132442+0000","last_change":"2024-04-25T15:52:32.131460+0000","last_active":"2024-04-25T15:52:32.132442+0000","last_peered":"2024-04-25T15:52:32.132442+0000","last_clean":"2024-04-25T15:52:32.132442+0000","last_became_active":"2024-04-25T15:52:32.131087+0000","last_became_peered":"2024-04-25T15:52:32.131087+0000","last_unstale":"2024-04-25T15:52:32.132442+0000","last_undegraded":"2024-04-25T15:52:32.132442+0000","last_fullsized":"2024-04-25T15:52:32.132442+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:14.840887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":58,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112803+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:52:32.112803+0000","last_peered":"2024-04-25T15:52:32.112803+0000","last_clean":"2024-04-25T15:52:32.112803+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:52:32.112803+0000","last_undegraded":"2024-04-25T15:52:32.112803+0000","last_fullsized":"2024-04-25T15:52:32.112803+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":92,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.112721+0000","last_change":"2024-04-25T15:51:47.270024+0000","last_active":"2024-04-25T15:52:32.112721+0000","last_peered":"2024-04-25T15:52:32.112721+0000","last_clean":"2024-04-25T15:52:32.112721+0000","last_became_active":"2024-04-25T15:51:47.269733+0000","last_became_peered":"2024-04-25T15:51:47.269733+0000","last_unstale":"2024-04-25T15:52:32.112721+0000","last_undegraded":"2024-04-25T15:52:32.112721+0000","last_fullsized":"2024-04-25T15:52:32.112721+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:23.505183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":54,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114154+0000","last_change":"2024-04-25T15:51:47.286937+0000","last_active":"2024-04-25T15:52:32.114154+0000","last_peered":"2024-04-25T15:52:32.114154+0000","last_clean":"2024-04-25T15:52:32.114154+0000","last_became_active":"2024-04-25T15:51:47.266040+0000","last_became_peered":"2024-04-25T15:51:47.266040+0000","last_unstale":"2024-04-25T15:52:32.114154+0000","last_undegraded":"2024-04-25T15:52:32.114154+0000","last_fullsized":"2024-04-25T15:52:32.114154+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:48.792187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":75,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.111870+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:52:32.111870+0000","last_peered":"2024-04-25T15:52:32.111870+0000","last_clean":"2024-04-25T15:52:32.111870+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:52:32.111870+0000","last_undegraded":"2024-04-25T15:52:32.111870+0000","last_fullsized":"2024-04-25T15:52:32.111870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":98,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.159964+0000","last_change":"2024-04-25T15:52:32.159964+0000","last_active":"2024-04-25T15:52:32.159964+0000","last_peered":"2024-04-25T15:52:32.159964+0000","last_clean":"2024-04-25T15:52:32.159964+0000","last_became_active":"2024-04-25T15:52:32.134918+0000","last_became_peered":"2024-04-25T15:52:32.134918+0000","last_unstale":"2024-04-25T15:52:32.159964+0000","last_undegraded":"2024-04-25T15:52:32.159964+0000","last_fullsized":"2024-04-25T15:52:32.159964+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:26:55.807079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.126151+0000","last_change":"2024-04-25T15:52:32.126151+0000","last_active":"2024-04-25T15:52:32.126151+0000","last_peered":"2024-04-25T15:52:32.126151+0000","last_clean":"2024-04-25T15:52:32.126151+0000","last_became_active":"2024-04-25T15:52:32.125517+0000","last_became_peered":"2024-04-25T15:52:32.125517+0000","last_unstale":"2024-04-25T15:52:32.126151+0000","last_undegraded":"2024-04-25T15:52:32.126151+0000","last_fullsized":"2024-04-25T15:52:32.126151+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:13.443149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.129077+0000","last_change":"2024-04-25T15:52:32.129077+0000","last_active":"2024-04-25T15:52:32.129077+0000","last_peered":"2024-04-25T15:52:32.129077+0000","last_clean":"2024-04-25T15:52:32.129077+0000","last_became_active":"2024-04-25T15:52:32.128629+0000","last_became_peered":"2024-04-25T15:52:32.128629+0000","last_unstale":"2024-04-25T15:52:32.129077+0000","last_undegraded":"2024-04-25T15:52:32.129077+0000","last_fullsized":"2024-04-25T15:52:32.129077+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:05:51.807066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":88,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114559+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:52:32.114559+0000","last_peered":"2024-04-25T15:52:32.114559+0000","last_clean":"2024-04-25T15:52:32.114559+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:52:32.114559+0000","last_undegraded":"2024-04-25T15:52:32.114559+0000","last_fullsized":"2024-04-25T15:52:32.114559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.132711+0000","last_change":"2024-04-25T15:52:32.132711+0000","last_active":"2024-04-25T15:52:32.132711+0000","last_peered":"2024-04-25T15:52:32.132711+0000","last_clean":"2024-04-25T15:52:32.132711+0000","last_became_active":"2024-04-25T15:52:32.132406+0000","last_became_peered":"2024-04-25T15:52:32.132406+0000","last_unstale":"2024-04-25T15:52:32.132711+0000","last_undegraded":"2024-04-25T15:52:32.132711+0000","last_fullsized":"2024-04-25T15:52:32.132711+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:30.616697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":95,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.114467+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:52:32.114467+0000","last_peered":"2024-04-25T15:52:32.114467+0000","last_clean":"2024-04-25T15:52:32.114467+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:52:32.114467+0000","last_undegraded":"2024-04-25T15:52:32.114467+0000","last_fullsized":"2024-04-25T15:52:32.114467+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.122256+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:52:32.122256+0000","last_peered":"2024-04-25T15:52:32.122256+0000","last_clean":"2024-04-25T15:52:32.122256+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:52:32.122256+0000","last_undegraded":"2024-04-25T15:52:32.122256+0000","last_fullsized":"2024-04-25T15:52:32.122256+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":87,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113163+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:52:32.113163+0000","last_peered":"2024-04-25T15:52:32.113163+0000","last_clean":"2024-04-25T15:52:32.113163+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:52:32.113163+0000","last_undegraded":"2024-04-25T15:52:32.113163+0000","last_fullsized":"2024-04-25T15:52:32.113163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":83,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.113049+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:52:32.113049+0000","last_peered":"2024-04-25T15:52:32.113049+0000","last_clean":"2024-04-25T15:52:32.113049+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:52:32.113049+0000","last_undegraded":"2024-04-25T15:52:32.113049+0000","last_fullsized":"2024-04-25T15:52:32.113049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":62,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T15:52:32.116078+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:52:32.116078+0000","last_peered":"2024-04-25T15:52:32.116078+0000","last_clean":"2024-04-25T15:52:32.116078+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:52:32.116078+0000","last_undegraded":"2024-04-25T15:52:32.116078+0000","last_fullsized":"2024-04-25T15:52:32.116078+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14298,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":218,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4003,"num_keys_recovered":22,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":30367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28272,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343568,"statfs":{"total":96636764160,"available":96607813632,"internally_reserved":0,"allocated":1273856,"data_stored":877892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":655360,"data_stored":286058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":288752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574879,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":292335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":655360,"data_stored":280177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":279806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1245184,"data_stored":869516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":7956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":7784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:52:36.750 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:36.750 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:36.767 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:36.844 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:36.838+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:36.944 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:36.938+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:36.962 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:36.962 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:36.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:37.045 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:37.042+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:37.127 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:37.127 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:37.145 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:37.146 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:37.142+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:37.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:52:37.247 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:37.247+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:37.290 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:37.290 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:37.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:37.349 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:37.342+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.378+0000 7f3d8ed45640 1 Processor -- start 2024-04-25T15:52:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.378+0000 7f3d8ed45640 1 -- start start 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8ed45640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3d8815a720 0x7f3d8815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8ed45640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d880ca540 0x7f3d880ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8ed45640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8815de10 con 0x7f3d88071bc0 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8ed45640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d8815ebb0 con 0x7f3d880ca540 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8d2e2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d880ca540 0x7f3d880ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8ed45640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3d8815e720 con 0x7f3d8815a720 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8d2e2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d880ca540 0x7f3d880ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43048/0 (socket says 172.21.15.67:43048) 2024-04-25T15:52:37.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/3746004616 learned_addr learned my addr 172.21.15.67:0/3746004616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:37.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d87fff640 1 --2- 172.21.15.67:0/3746004616 >> v2:172.21.15.176:3300/0 conn(0x7f3d8815a720 0x7f3d8815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/3746004616 >> v2:172.21.15.176:3300/0 conn(0x7f3d8815a720 msgr2=0x7f3d8815ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:37.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8d2e2640 1 --2- 172.21.15.67:0/3746004616 >> v2:172.21.15.176:3300/0 conn(0x7f3d8815a720 0x7f3d8815ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/3746004616 >> v1:172.21.15.176:6789/0 conn(0x7f3d88071bc0 legacy=0x7f3d880cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/3746004616 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d88072390 con 0x7f3d880ca540 2024-04-25T15:52:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.382+0000 7f3d877fe640 1 -- 172.21.15.67:0/3746004616 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485880773 0 0) 0x7f3d8815de10 con 0x7f3d88071bc0 2024-04-25T15:52:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8d2e2640 1 --2- 172.21.15.67:0/3746004616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d880ca540 0x7f3d880ca950 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3d88071000 tx=0x7f3d8001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1676f77161da37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d877fe640 1 -- 172.21.15.67:0/3746004616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d800a57d0 con 0x7f3d880ca540 2024-04-25T15:52:37.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d877fe640 1 -- 172.21.15.67:0/3746004616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d800a5970 con 0x7f3d880ca540 2024-04-25T15:52:37.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d877fe640 1 -- 172.21.15.67:0/3746004616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d800a5d40 con 0x7f3d880ca540 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 -- 172.21.15.67:0/3746004616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d880ca540 msgr2=0x7f3d880ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/3746004616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d880ca540 0x7f3d880ca950 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3d88071000 tx=0x7f3d8001c9f0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 -- 172.21.15.67:0/3746004616 shutdown_connections 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/3746004616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d880ca540 0x7f3d880ca950 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/3746004616 >> v2:172.21.15.176:3300/0 conn(0x7f3d8815a720 0x7f3d8815ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 -- 172.21.15.67:0/3746004616 >> 172.21.15.67:0/3746004616 conn(0x7f3d880700b0 msgr2=0x7f3d880704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 -- 172.21.15.67:0/3746004616 shutdown_connections 2024-04-25T15:52:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.386+0000 7f3d8ed45640 1 -- 172.21.15.67:0/3746004616 wait complete. 2024-04-25T15:52:37.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.390+0000 7f3d8ed45640 1 Processor -- start 2024-04-25T15:52:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.390+0000 7f3d8ed45640 1 -- start start 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d88071bc0 0x7f3d8814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d880ca540 0x7f3d8814b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8815a720 0x7f3d8814a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d88071fd0 con 0x7f3d8815a720 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d880cc2d0 con 0x7f3d88071bc0 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d88072150 con 0x7f3d880ca540 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8815a720 0x7f3d8814a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8815a720 0x7f3d8814a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43054/0 (socket says 172.21.15.67:43054) 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8cae1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d88071bc0 0x7f3d8814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/1006063558 learned_addr learned my addr 172.21.15.67:0/1006063558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d87fff640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d880ca540 0x7f3d8814b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d880ca540 msgr2=0x7f3d8814b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d880ca540 0x7f3d8814b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d88071bc0 msgr2=0x7f3d8814fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d88071bc0 0x7f3d8814fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 -- 172.21.15.67:0/1006063558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d880d0480 con 0x7f3d8815a720 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8cae1640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d88071bc0 0x7f3d8814fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8d2e2640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8815a720 0x7f3d8814a5e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3d800af040 tx=0x7f3d80019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d857fa640 1 -- 172.21.15.67:0/1006063558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d800b8470 con 0x7f3d8815a720 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d857fa640 1 -- 172.21.15.67:0/1006063558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d800b8630 con 0x7f3d8815a720 2024-04-25T15:52:37.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d857fa640 1 -- 172.21.15.67:0/1006063558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d800b8a20 con 0x7f3d8815a720 2024-04-25T15:52:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d880d0760 con 0x7f3d8815a720 2024-04-25T15:52:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d8814d1c0 con 0x7f3d8815a720 2024-04-25T15:52:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.394+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d8806a480 con 0x7f3d8815a720 2024-04-25T15:52:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.398+0000 7f3d857fa640 1 -- 172.21.15.67:0/1006063558 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3d800c0030 con 0x7f3d8815a720 2024-04-25T15:52:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.398+0000 7f3d857fa640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6c06f430 0x7f3d6c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.398+0000 7f3d857fa640 1 -- 172.21.15.67:0/1006063558 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5857+0+0 (secure 0 0 0) 0x7f3d80129d10 con 0x7f3d8815a720 2024-04-25T15:52:37.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.398+0000 7f3d8cae1640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6c06f430 0x7f3d6c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.398+0000 7f3d8cae1640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6c06f430 0x7f3d6c071940 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c007fd0 tx=0x7f3d7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.406+0000 7f3d857fa640 1 -- 172.21.15.67:0/1006063558 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3d800c7020 con 0x7f3d8815a720 2024-04-25T15:52:37.449 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:37.442+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:37.503 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:37.503 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:37.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:37.549 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:37.546+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:37.650 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:37.647+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:37.669 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:37.669 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:37.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.662+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f3d880caad0 con 0x7f3d8815a720 2024-04-25T15:52:37.686 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:37.751 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:37.746+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:37.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:37.851 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:37.851+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:37.880 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:37.880 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:37.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.926+0000 7f75336e1640 1 Processor -- start 2024-04-25T15:52:37.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f75336e1640 1 -- start start 2024-04-25T15:52:37.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f75336e1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f752c0b5d70 0x7f752c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f75336e1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f752c06d780 con 0x7f752c06ca60 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f75336e1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f752c160920 con 0x7f752c15fe00 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f75336e1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f752c15d510 con 0x7f752c0b5d70 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f753147d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f752c06ca60 0x7f752c0b5660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49806/0 (socket says 172.21.15.67:49806) 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f753147d640 1 -- 172.21.15.67:0/3663022607 learned_addr learned my addr 172.21.15.67:0/3663022607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3663022607 >> v2:172.21.15.176:3300/0 conn(0x7f752c0b5d70 0x7f752c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176594479 0 0) 0x7f752c06d780 con 0x7f752c06ca60 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7518003530 con 0x7f752c06ca60 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1023461675 0 0) 0x7f7518003530 con 0x7f752c06ca60 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 >> v2:172.21.15.176:3300/0 conn(0x7f752c0b5d70 msgr2=0x7f752c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 --2- 172.21.15.67:0/3663022607 >> v2:172.21.15.176:3300/0 conn(0x7f752c0b5d70 0x7f752c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 >> v1:172.21.15.176:6789/0 conn(0x7f752c15fe00 legacy=0x7f752c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c160be0 con 0x7f752c06ca60 2024-04-25T15:52:37.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f7528001a20 con 0x7f752c06ca60 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 >> v1:172.21.15.67:6789/0 conn(0x7f752c06ca60 legacy=0x7f752c0b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75180039f0 0x7f7518003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518004810 0x7f7518004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7518005280 0x7f751800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f752c15d510 con 0x7f7518005280 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f751800c1a0 con 0x7f7518004810 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f751800c3a0 con 0x7f75180039f0 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7518005280 0x7f751800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518004810 0x7f7518004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f753147d640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75180039f0 0x7f7518003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7531c7e640 1 -- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75180039f0 msgr2=0x7f7518003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75180039f0 0x7f7518003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7531c7e640 1 -- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518004810 msgr2=0x7f7518004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518004810 0x7f7518004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f7531c7e640 1 -- 172.21.15.67:0/3663022607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7518003530 con 0x7f7518005280 2024-04-25T15:52:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.930+0000 7f753147d640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75180039f0 0x7f7518003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f7531c7e640 1 -- 172.21.15.67:0/3663022607 reap_dead start 2024-04-25T15:52:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7518005280 0x7f751800b9d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7524001750 tx=0x7f752401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2731217179192f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75240013a0 con 0x7f7518005280 2024-04-25T15:52:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75240052d0 con 0x7f7518005280 2024-04-25T15:52:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f7523fff640 1 -- 172.21.15.67:0/3663022607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524003820 con 0x7f7518005280 2024-04-25T15:52:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f75336e1640 1 -- 172.21.15.67:0/3663022607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7518005280 msgr2=0x7f751800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f75336e1640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7518005280 0x7f751800b9d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7524001750 tx=0x7f752401c3e0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f75336e1640 1 -- 172.21.15.67:0/3663022607 shutdown_connections 2024-04-25T15:52:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f75336e1640 1 --2- 172.21.15.67:0/3663022607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7518005280 0x7f751800b9d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f75336e1640 1 -- 172.21.15.67:0/3663022607 >> 172.21.15.67:0/3663022607 conn(0x7f752c06e420 msgr2=0x7f752c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f75336e1640 1 -- 172.21.15.67:0/3663022607 shutdown_connections 2024-04-25T15:52:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.934+0000 7f75336e1640 1 -- 172.21.15.67:0/3663022607 wait complete. 2024-04-25T15:52:37.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 Processor -- start 2024-04-25T15:52:37.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 -- start start 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7518005280 0x7f752c158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f752c159950 0x7f752c152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f752c0ca7a0 0x7f752c0cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f752c15d1a0 con 0x7f752c0ca7a0 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f752c15cec0 con 0x7f752c159950 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f752c15c890 con 0x7f7518005280 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f752c0ca7a0 0x7f752c0cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f752c0ca7a0 0x7f752c0cabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43064/0 (socket says 172.21.15.67:43064) 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 -- 172.21.15.67:0/3990598067 learned_addr learned my addr 172.21.15.67:0/3990598067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:37.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f752c159950 0x7f752c152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7518005280 0x7f752c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7518005280 msgr2=0x7f752c158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7518005280 0x7f752c158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f752c159950 msgr2=0x7f752c152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f752c159950 0x7f752c152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c157db0 con 0x7f752c0ca7a0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f752c0ca7a0 0x7f752c0cabb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f752401cc80 tx=0x7f7518005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75240013a0 con 0x7f752c0ca7a0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7524005480 con 0x7f752c0ca7a0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524007a40 con 0x7f752c0ca7a0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752c1f6240 con 0x7f752c0ca7a0 2024-04-25T15:52:37.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.938+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752c0ca530 con 0x7f752c0ca7a0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f752c06a480 con 0x7f752c0ca7a0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f75240bb060 con 0x7f752c0ca7a0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f7521ffb640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f751006f430 0x7f7510071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5857+0+0 (secure 0 0 0) 0x7f75241281c0 con 0x7f752c0ca7a0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v56) ==== 54+0+16 (secure 0 0 0) 0x7f75240160f0 con 0x7f752c0ca7a0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f751006f430 0x7f7510071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f752c0cb980 0x7f752c14e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f752c1572a0 con 0x7f752c0cb980 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f752c0cb980 0x7f752c14e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f751006f430 0x7f7510071940 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75280013c0 tx=0x7f7528009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f752c0cb980 0x7f752c14e170 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.942+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f751c0096f0 con 0x7f752c0cb980 2024-04-25T15:52:37.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.970+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c0b5160 con 0x7f752c0cb980 2024-04-25T15:52:37.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.970+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c0b5160 con 0x7f752c0cb980 2024-04-25T15:52:37.977 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.974+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f752c14ebc0 0x7f752c063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.974+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f752c14efd0 con 0x7f752c14ebc0 2024-04-25T15:52:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.974+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f752c14ebc0 0x7f752c063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.974+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f752c14ebc0 0x7f752c063090 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.974+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752c14efd0 con 0x7f752c14ebc0 2024-04-25T15:52:37.987 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:37.987+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:37.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.990+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c0cd380 con 0x7f752c14ebc0 2024-04-25T15:52:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.990+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c0cd380 con 0x7f752c14ebc0 2024-04-25T15:52:37.995 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.990+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f752c063ae0 0x7f752c15dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.990+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f752c152f50 con 0x7f752c063ae0 2024-04-25T15:52:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.990+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f752c063ae0 0x7f752c15dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.990+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f752c063ae0 0x7f752c15dd70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:37.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:37.990+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752c152f50 con 0x7f752c063ae0 2024-04-25T15:52:38.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.006+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c06c740 con 0x7f752c063ae0 2024-04-25T15:52:38.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.006+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c06c740 con 0x7f752c063ae0 2024-04-25T15:52:38.013 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.010+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c15e7c0 0x7f752c02c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.010+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f752c15ebd0 con 0x7f752c15e7c0 2024-04-25T15:52:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.010+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c15e7c0 0x7f752c02c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:38.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.010+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c15e7c0 0x7f752c02c9e0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:38.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.010+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752c15ebd0 con 0x7f752c15e7c0 2024-04-25T15:52:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.022+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c1583d0 con 0x7f752c15e7c0 2024-04-25T15:52:38.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.026+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c1583d0 con 0x7f752c15e7c0 2024-04-25T15:52:38.031 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:38.031 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:38.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.026+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f752c02d430 0x7f752c02d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:38.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.026+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f752c158540 con 0x7f752c02d430 2024-04-25T15:52:38.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.026+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f752c02d430 0x7f752c02d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:38.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.026+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f752c02d430 0x7f752c02d840 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:38.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.026+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752c158540 con 0x7f752c02d430 2024-04-25T15:52:38.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.042+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c158540 con 0x7f752c02d430 2024-04-25T15:52:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.046+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c158540 con 0x7f752c02d430 2024-04-25T15:52:38.050 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.046+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f752c207af0 0x7f752c209f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.046+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f752c02df10 con 0x7f752c207af0 2024-04-25T15:52:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.046+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f752c207af0 0x7f752c209f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:38.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.046+0000 7f7530c7c640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f752c207af0 0x7f752c209f10 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:38.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.046+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752c02df10 con 0x7f752c207af0 2024-04-25T15:52:38.053 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:38.051+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:38.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.062+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c158050 con 0x7f752c207af0 2024-04-25T15:52:38.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.066+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c158050 con 0x7f752c207af0 2024-04-25T15:52:38.071 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.066+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f752c20a960 0x7f752c20cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.066+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f752c20fcb0 con 0x7f752c20a960 2024-04-25T15:52:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.066+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f752c20a960 0x7f752c20cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.066+0000 7f7531c7e640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f752c20a960 0x7f752c20cd80 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.070+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752c20fcb0 con 0x7f752c20a960 2024-04-25T15:52:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.082+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c20d2c0 con 0x7f752c20a960 2024-04-25T15:52:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.086+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c20d2c0 con 0x7f752c20a960 2024-04-25T15:52:38.091 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.086+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f752c20d9e0 0x7f752c20ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.086+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f752c213100 con 0x7f752c20d9e0 2024-04-25T15:52:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.086+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f752c20d9e0 0x7f752c20ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:38.092 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:38.093 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.086+0000 7f753147d640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f752c20d9e0 0x7f752c20ddf0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.090+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f752c213100 con 0x7f752c20d9e0 2024-04-25T15:52:38.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.102+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f752c211f80 con 0x7f752c20d9e0 2024-04-25T15:52:38.109 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:38.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.106+0000 7f7521ffb640 1 -- 172.21.15.67:0/3990598067 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f752c211f80 con 0x7f752c20d9e0 2024-04-25T15:52:38.113 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f752c0cb980 msgr2=0x7f752c14e170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f752c0cb980 0x7f752c14e170 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f752c14ebc0 msgr2=0x7f752c063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f752c14ebc0 0x7f752c063090 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f752c063ae0 msgr2=0x7f752c15dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f752c063ae0 0x7f752c15dd70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c15e7c0 msgr2=0x7f752c02c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c15e7c0 0x7f752c02c9e0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f752c02d430 msgr2=0x7f752c02d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f752c02d430 0x7f752c02d840 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f7531c7e640 1 -- 172.21.15.67:0/3990598067 reap_dead start 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f752c207af0 msgr2=0x7f752c209f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f752c207af0 0x7f752c209f10 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f7531c7e640 1 -- 172.21.15.67:0/3990598067 reap_dead start 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f752c20a960 msgr2=0x7f752c20cd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f752c20a960 0x7f752c20cd80 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f752c20d9e0 msgr2=0x7f752c20ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f752c20d9e0 0x7f752c20ddf0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f7531c7e640 1 -- 172.21.15.67:0/3990598067 reap_dead start 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f751006f430 msgr2=0x7f7510071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f751006f430 0x7f7510071940 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75280013c0 tx=0x7f7528009040 comp rx=0 tx=0).stop 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f752c0ca7a0 msgr2=0x7f752c0cabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f752c0ca7a0 0x7f752c0cabb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f752401cc80 tx=0x7f7518005840 comp rx=0 tx=0).stop 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 shutdown_connections 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f751006f430 0x7f7510071940 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 --2- 172.21.15.67:0/3990598067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f752c0ca7a0 0x7f752c0cabb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 >> 172.21.15.67:0/3990598067 conn(0x7f752c06e420 msgr2=0x7f752c0ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 shutdown_connections 2024-04-25T15:52:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.110+0000 7f75336e1640 1 -- 172.21.15.67:0/3990598067 wait complete. 2024-04-25T15:52:38.153 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:38.150+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.178+0000 7f3d857fa640 1 -- 172.21.15.67:0/1006063558 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v57) ==== 101+0+0 (secure 0 0 0) 0x7f3d800165e0 con 0x7f3d8815a720 2024-04-25T15:52:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6c06f430 msgr2=0x7f3d6c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6c06f430 0x7f3d6c071940 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c007fd0 tx=0x7f3d7c009040 comp rx=0 tx=0).stop 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8815a720 msgr2=0x7f3d8814a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8815a720 0x7f3d8814a5e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3d800af040 tx=0x7f3d80019790 comp rx=0 tx=0).stop 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 shutdown_connections 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6c06f430 0x7f3d6c071940 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8815a720 0x7f3d8814a5e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d880ca540 0x7f3d8814b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 --2- 172.21.15.67:0/1006063558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d88071bc0 0x7f3d8814fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 >> 172.21.15.67:0/1006063558 conn(0x7f3d880700b0 msgr2=0x7f3d881597b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 shutdown_connections 2024-04-25T15:52:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:38.182+0000 7f3d8ed45640 1 -- 172.21.15.67:0/1006063558 wait complete. 2024-04-25T15:52:38.255 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:38.250+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:38.310 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 122.09694401199999, 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:52:38.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:52:38.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:52:38.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 122.096914748, 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:52:38.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:52:38.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:24 2.7 2:e0b41d06:::609.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 122.09690546, 2024-04-25T15:52:38.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010024903999999999, 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210673+0000", 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9500000000000002e-07 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210678+0000", 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.532e-06 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081834099999999999 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211529+0000", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2539999999999997e-05 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211765+0000", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000236371 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211996+0000", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023101900000000001 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212199+0000", 2024-04-25T15:52:38.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020299399999999999 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215531+0000", 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003332284 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220519+0000", 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049877339999999997 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220617+0000", 2024-04-25T15:52:38.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7572999999999996e-05 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220698+0000", 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0921000000000007e-05 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 122.096897884, 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:52:38.325 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:38.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 122.09688925099999, 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:52:38.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 122.096839418, 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:52:38.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:52:38.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:38.339 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 122.09683154, 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:52:38.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:52:38.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:5 2.7 2:e455cf45:::60b.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.060403533, 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077529449999999998, 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:38.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247175+0000", 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247179+0000", 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9689999999999996e-06 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247196+0000", 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7802e-05 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247314+0000", 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011734 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247345+0000", 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1469e-05 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247478+0000", 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013243799999999999 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247586+0000", 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108614 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247729+0000", 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143098 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253327+0000", 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055976630000000001 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254748+0000", 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014212700000000001 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254859+0000", 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011050199999999999 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.254928+0000", 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8780000000000002e-05 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.347 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:6 2.7 2:e455cf45:::60b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.06037286900001, 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079096879999999998, 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247205+0000", 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247206+0000", 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.093e-06 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247214+0000", 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2370000000000003e-06 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247724+0000", 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050982999999999996 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247752+0000", 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8895e-05 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248104+0000", 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035201999999999998 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248196+0000", 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2032000000000002e-05 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:38.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248347+0000", 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015051499999999999 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253336+0000", 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049891839999999998 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255009+0000", 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016728190000000001 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255066+0000", 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7085e-05 2024-04-25T15:52:38.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.255115+0000", 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8977999999999998e-05 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:9 2.17 2:e8b9846e:::60d.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.060360339, 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011968626, 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.352 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247218+0000", 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247219+0000", 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1699999999999997e-07 2024-04-25T15:52:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247225+0000", 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3249999999999996e-06 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247817+0000", 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059212000000000002 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247836+0000", 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:52:38.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248267+0000", 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043196100000000002 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248846+0000", 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057846000000000004 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249084+0000", 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238349 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256794+0000", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077093309999999998 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259004+0000", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022102329999999998 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259134+0000", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129899 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259187+0000", 2024-04-25T15:52:38.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2774e-05 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:10 2.17 2:e8b9846e:::60d.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.060349707, 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012065226, 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:38.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247229+0000", 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4300000000000002e-07 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247235+0000", 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7620000000000001e-06 2024-04-25T15:52:38.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247872+0000", 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063666700000000005 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247886+0000", 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.464e-05 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249809+0000", 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019227230000000001 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249935+0000", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125606 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251282+0000", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001347701 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256885+0000", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056025190000000002 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259198+0000", 2024-04-25T15:52:38.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023128070000000001 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259241+0000", 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3646999999999997e-05 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259294+0000", 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2311000000000001e-05 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:16 2.14 2:2e2b5214:::610.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.060330162, 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011931158000000001, 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247248+0000", 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247249+0000", 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.85e-07 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247254+0000", 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4330000000000003e-06 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248005+0000", 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000750915 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248021+0000", 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5509999999999999e-05 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249654+0000", 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00163306 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.249857+0000", 2024-04-25T15:52:38.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020345699999999999 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250205+0000", 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034819400000000002 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.253702+0000", 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034963730000000001 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259034+0000", 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005332346 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259111+0000", 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7191e-05 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259179+0000", 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7893999999999997e-05 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.06032127, 2024-04-25T15:52:38.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:52:38.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:13 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.05675865400001, 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008280272, 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250820+0000", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250822+0000", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250839+0000", 2024-04-25T15:52:38.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7753000000000001e-05 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250868+0000", 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8877000000000002e-05 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250889+0000", 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0528999999999999e-05 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251025+0000", 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013676300000000001 2024-04-25T15:52:38.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251063+0000", 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7088999999999997e-05 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259044+0000", 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079817080000000006 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259100+0000", 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5572999999999998e-05 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.1.10:14 2.16 e20/17 2:6f2c300e:::60f.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.056638042, 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082080079999999993, 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250940+0000", 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250941+0000", 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0219999999999999e-06 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250954+0000", 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2544999999999999e-05 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250979+0000", 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4783e-05 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.250994+0000", 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5760000000000002e-05 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251452+0000", 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045743800000000001 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251537+0000", 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4926000000000006e-05 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259106+0000", 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075695930000000003 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259148+0000", 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1940999999999997e-05 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.3.14:1 2.17 2:eb104683:::203.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 100.42164002200001, 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086696540000000006, 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885938+0000", 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885939+0000", 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9800000000000004e-07 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.885942+0000", 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0170000000000001e-06 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886274+0000", 2024-04-25T15:52:38.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033199899999999998 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886286+0000", 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.182e-05 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886354+0000", 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886538+0000", 2024-04-25T15:52:38.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018443600000000001 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.886724+0000", 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018579800000000001 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.889081+0000", 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023575520000000002 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894496+0000", 2024-04-25T15:52:38.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054143239999999999 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894540+0000", 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4339999999999999e-05 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:57.894608+0000", 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7850999999999998e-05 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:19 2.7 2:e196673b:::630.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.383 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 94.381070624000003, 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010597863000000001, 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:38.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926508+0000", 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-07 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926512+0000", 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5360000000000001e-06 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926626+0000", 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011425300000000001 2024-04-25T15:52:38.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926636+0000", 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7440000000000002e-06 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927479+0000", 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00084336299999999997 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927599+0000", 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119931 2024-04-25T15:52:38.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927720+0000", 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012076199999999999 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930802+0000", 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003082698 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936962+0000", 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061595929999999997 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937030+0000", 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7792000000000004e-05 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937105+0000", 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5751e-05 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.4.16:20 2.7 2:e196673b:::630.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:52:38.388 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 94.381065116000002, 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011020623, 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926513+0000", 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926514+0000", 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3799999999999998e-07 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926516+0000", 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8279999999999999e-06 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926704+0000", 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018758800000000001 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.926737+0000", 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3297999999999999e-05 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.927960+0000", 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012225280000000001 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928046+0000", 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6543000000000001e-05 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:38.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928175+0000", 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012911500000000001 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936189+0000", 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080134869999999997 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937351+0000", 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001161624 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937462+0000", 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111501 2024-04-25T15:52:38.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937534+0000", 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1673000000000003e-05 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:7 2.16 e20/17 2:6b7c230a:::62a.00000000:head v 20'3, mlcod=20'1)", 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 94.378650601999993, 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076874389999999999, 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.393 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928928+0000", 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0599999999999999e-07 2024-04-25T15:52:38.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928936+0000", 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6550000000000004e-06 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928953+0000", 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7314000000000001e-05 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.928965+0000", 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1161999999999999e-05 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929134+0000", 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016923900000000001 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929230+0000", 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6112000000000004e-05 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936573+0000", 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073427800000000001 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.936615+0000", 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2370999999999998e-05 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(mds.4.16:16 2.1d e20/17 2:b9436051:::62e.00000000:head v 20'9, mlcod=20'6)", 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 94.377688715999994, 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079200339999999994, 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:38.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929890+0000", 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929891+0000", 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.392e-06 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929902+0000", 2024-04-25T15:52:38.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0703999999999999e-05 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929920+0000", 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8567999999999999e-05 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.929935+0000", 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4594e-05 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930584+0000", 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00064965200000000002 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.930611+0000", 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6468000000000001e-05 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937633+0000", 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070224839999999998 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:51:03.937810+0000", 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017617199999999999 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:38.401 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:38.401 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:38.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:38.402 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:38.399+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:38.503 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:38.499+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:38.596 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:38.596 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:38.603 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:38.603+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:38.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:38.703 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:38.703+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:38.801 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:38.801 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:38.801 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:38.801 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:38.802 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:38.805 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:38.798+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:38.821 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:38.907 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:38.907+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:39.007 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:39.002+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.91187360399999, 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.014 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:52:39.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:52:39.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:52:39.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:52:39.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.021 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.44827278899999, 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01409815, 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 25 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562066+0000", 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562072+0000", 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.703e-06 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562156+0000", 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3486000000000007e-05 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562165+0000", 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.568966+0000", 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006801372 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569110+0000", 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014385799999999999 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569344+0000", 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023376500000000001 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.573357+0000", 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040127870000000003 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.575955+0000", 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025982969999999998 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576077+0000", 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121966 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576163+0000", 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6058999999999997e-05 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:43 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.321467353, 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013368089999999999, 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.027 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688870+0000", 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688872+0000", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0470000000000001e-06 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688892+0000", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9143e-05 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688940+0000", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8343000000000001e-05 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688961+0000", 2024-04-25T15:52:39.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0925999999999999e-05 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694131+0000", 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051698639999999997 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694283+0000", 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015229800000000001 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.694506+0000", 2024-04-25T15:52:39.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222995 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.698950+0000", 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044438300000000002 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702065+0000", 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031151719999999998 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702159+0000", 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4089e-05 2024-04-25T15:52:39.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702238+0000", 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9382999999999995e-05 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:44 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32134080099999, 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013712703999999999, 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.032 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 44 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688997+0000", 2024-04-25T15:52:39.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.688998+0000", 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689301+0000", 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000302684 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689729+0000", 2024-04-25T15:52:39.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000428486 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689746+0000", 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6690000000000001e-05 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.695997+0000", 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062505779999999997 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696164+0000", 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167045 2024-04-25T15:52:39.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.696383+0000", 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.701919+0000", 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055357399999999999 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702410+0000", 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00049136500000000001 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702581+0000", 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017146899999999999 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702710+0000", 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012822199999999999 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:45 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:52:39.037 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32102846000001, 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013745535999999999, 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 45 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689309+0000", 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689310+0000", 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.127e-06 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689317+0000", 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.973e-06 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689793+0000", 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000475468 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689804+0000", 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.082e-05 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697359+0000", 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075556 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697536+0000", 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176658 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.697753+0000", 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021705499999999999 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702734+0000", 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049807779999999999 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702850+0000", 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116109 2024-04-25T15:52:39.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702934+0000", 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4429000000000001e-05 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703055+0000", 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120519 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.042 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32101641400001, 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013995720999999999, 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 46 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:39.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689322+0000", 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8999999999999996e-07 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689329+0000", 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1899999999999998e-06 2024-04-25T15:52:39.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689831+0000", 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050143900000000001 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689839+0000", 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4069999999999993e-06 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699042+0000", 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092026810000000007 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699214+0000", 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017275100000000001 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699442+0000", 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022718200000000001 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702745+0000", 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003303266 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703161+0000", 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000416253 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703250+0000", 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8806000000000003e-05 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703317+0000", 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7056000000000003e-05 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.047 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32100514699999, 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014300548, 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.048 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689491+0000", 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015768600000000001 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689857+0000", 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000366313 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689865+0000", 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1470000000000001e-06 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700215+0000", 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010350177 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700471+0000", 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521499999999999 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700879+0000", 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040780599999999998 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703339+0000", 2024-04-25T15:52:39.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024602140000000001 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703443+0000", 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104393 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703524+0000", 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0745e-05 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703633+0000", 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109226 2024-04-25T15:52:39.052 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32084086899999, 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019412740000000001, 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 48 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689498+0000", 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5999999999999991e-07 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689505+0000", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1339999999999998e-06 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689885+0000", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000380553 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689894+0000", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0660000000000004e-06 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703735+0000", 2024-04-25T15:52:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013841509 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703880+0000", 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014450899999999999 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704121+0000", 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024184799999999999 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708286+0000", 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041643369999999997 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708530+0000", 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024477400000000001 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708710+0000", 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179715 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708910+0000", 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199335 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.057 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32082885899999, 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019597107999999999, 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 49 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.058 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4700000000000001e-07 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689515+0000", 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3770000000000004e-06 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689911+0000", 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039616199999999998 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689920+0000", 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2939999999999994e-06 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704840+0000", 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014919412999999999 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704990+0000", 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150388 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.705238+0000", 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000248392 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708485+0000", 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032464740000000001 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708936+0000", 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045072599999999998 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709037+0000", 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101482 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709106+0000", 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8752999999999997e-05 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32081930199999, 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.063 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:52:39.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:52:39.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32067068000001, 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.068 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:39.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:52:39.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.32065960099999, 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.073 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:39.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:52:39.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.01382112100001, 2024-04-25T15:52:39.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019231747, 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 69 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996518+0000", 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6840000000000001e-06 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996645+0000", 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012677899999999999 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996696+0000", 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0912999999999999e-05 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996718+0000", 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1931999999999999e-05 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000632+0000", 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003914509 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000756+0000", 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012338099999999999 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000879+0000", 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012301599999999999 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009110+0000", 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082309819999999995 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015495+0000", 2024-04-25T15:52:39.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063848969999999996 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015700+0000", 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020508399999999999 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015748+0000", 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8569999999999997e-05 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.083 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.01356907900001, 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019263037, 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 70 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:39.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996770+0000", 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7060000000000001e-06 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996786+0000", 2024-04-25T15:52:39.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5497000000000001e-05 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996990+0000", 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000203938 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997014+0000", 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4617000000000002e-05 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002337+0000", 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053230450000000002 2024-04-25T15:52:39.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002417+0000", 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9548999999999993e-05 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002579+0000", 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162385 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009114+0000", 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00653442 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015848+0000", 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067339909999999999 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015951+0000", 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102796 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016032+0000", 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1093000000000004e-05 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.01354317900001, 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019380720000000001, 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 71 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996796+0000", 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.063e-06 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996810+0000", 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4100000000000001e-05 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997061+0000", 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000251797 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997076+0000", 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4970000000000001e-05 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003008+0000", 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005931487 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003148+0000", 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140097 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003371+0000", 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022273400000000001 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009117+0000", 2024-04-25T15:52:39.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057466929999999998 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016106+0000", 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069880919999999996 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016144+0000", 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8686000000000002e-05 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016175+0000", 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1000999999999997e-05 2024-04-25T15:52:39.093 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.013520513, 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019495743999999999, 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.094 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996818+0000", 2024-04-25T15:52:39.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.046e-06 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996944+0000", 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012572199999999999 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997139+0000", 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019529899999999999 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997151+0000", 2024-04-25T15:52:39.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2116000000000001e-05 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004407+0000", 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072560519999999998 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004474+0000", 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6810000000000006e-05 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004576+0000", 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101431 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009121+0000", 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045451099999999998 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016239+0000", 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071178359999999998 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016280+0000", 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0914000000000002e-05 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016313+0000", 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3408000000000003e-05 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.01314124699999, 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019262988000000002, 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.099 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 73 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997198+0000", 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.347e-06 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997208+0000", 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0427e-05 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997419+0000", 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021052399999999999 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997433+0000", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4687e-05 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005473+0000", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080393610000000001 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005619+0000", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146157 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005848+0000", 2024-04-25T15:52:39.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229001 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009124+0000", 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003276288 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016366+0000", 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072413759999999999 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016408+0000", 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2119000000000001e-05 2024-04-25T15:52:39.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016459+0000", 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1700999999999998e-05 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.01312377299999, 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019348008999999999, 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.104 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 74 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:39.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997215+0000", 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997220+0000", 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7200000000000003e-06 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997480+0000", 2024-04-25T15:52:39.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026014000000000001 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997493+0000", 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2306999999999999e-05 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006512+0000", 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090195669999999992 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006591+0000", 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9058000000000001e-05 2024-04-25T15:52:39.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006724+0000", 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013295000000000001 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009130+0000", 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024053659999999999 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073654790000000003 2024-04-25T15:52:39.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016533+0000", 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7432999999999999e-05 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016562+0000", 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9379000000000001e-05 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.109 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.01311396700001, 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020191183000000001, 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997376+0000", 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015132899999999999 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997528+0000", 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152275 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997538+0000", 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0137e-05 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007159+0000", 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096209090000000004 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007248+0000", 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9078000000000003e-05 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007394+0000", 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014618699999999999 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009133+0000", 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001739008 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017311+0000", 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081775159999999993 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017360+0000", 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9202000000000001e-05 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017415+0000", 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4932000000000002e-05 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.114 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 139.012956562, 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.115 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:52:39.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:52:39.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:52:39.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:39.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:39.120 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:39.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:39.122 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:39.118+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:39.223 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:39.223+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:39.276 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:39.276 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:39.276 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:39.276 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:39.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:39.337 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:39.334+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:39.425 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:39.423+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:39.450 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:39.450 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:39.450 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:39.450 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:39.451 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:39.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:39.526 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:39.523+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:39.627 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:39.622+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:39.674 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:39.674 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:39.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.66922453800001, 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:39.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'48, mlcod=15'42)", 2024-04-25T15:52:39.678 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.66759532, 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012230665999999999, 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004239+0000", 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004250+0000", 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1518999999999999e-05 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004274+0000", 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004292+0000", 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7458000000000001e-05 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005373+0000", 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010819359999999999 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005450+0000", 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6200999999999998e-05 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016441+0000", 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991445000000001 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016468+0000", 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6801999999999999e-05 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:71 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'49, mlcod=15'42)", 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.66680488099999, 2024-04-25T15:52:39.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011466877, 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005029+0000", 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999999999999999e-06 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005040+0000", 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1199000000000001e-05 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005063+0000", 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2342000000000001e-05 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005081+0000", 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7711999999999999e-05 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006374+0000", 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001293492 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006429+0000", 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5315999999999997e-05 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016469+0000", 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010039984 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5232000000000002e-05 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:72 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'50, mlcod=15'44)", 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.66552800400001, 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010219404, 2024-04-25T15:52:39.686 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006305+0000", 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006306+0000", 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.48e-06 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006368+0000", 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1532000000000007e-05 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006394+0000", 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6917999999999999e-05 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006423+0000", 2024-04-25T15:52:39.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8087000000000001e-05 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007111+0000", 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00068813299999999995 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007172+0000", 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0918000000000002e-05 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016500+0000", 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093286540000000005 2024-04-25T15:52:39.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016524+0000", 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3682000000000001e-05 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:73 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'51, mlcod=15'44)", 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.664296071, 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090122410000000007, 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007536+0000", 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007538+0000", 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8899999999999999e-06 2024-04-25T15:52:39.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007559+0000", 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0302e-05 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007589+0000", 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0088000000000001e-05 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.007643+0000", 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4419000000000002e-05 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010532+0000", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028890249999999999 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010603+0000", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1144999999999997e-05 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016525+0000", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059219279999999999 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016549+0000", 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3444000000000001e-05 2024-04-25T15:52:39.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:76 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'54, mlcod=15'44)", 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 139.66124263399999, 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010878550000000001, 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:39.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010590+0000", 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010591+0000", 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7899999999999997e-07 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010599+0000", 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6000000000000007e-06 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010611+0000", 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2194999999999999e-05 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.010620+0000", 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8529999999999998e-06 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016013+0000", 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053922229999999998 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016045+0000", 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1933000000000001e-05 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021402+0000", 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053573559999999998 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.021468+0000", 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6611000000000003e-05 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.697 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:4 2.1f 2:ff5b34d6:::1.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.46037946, 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010088613, 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211453+0000", 2024-04-25T15:52:39.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211464+0000", 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0740000000000001e-05 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211486+0000", 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1722e-05 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211654+0000", 2024-04-25T15:52:39.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016800800000000001 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211678+0000", 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4283000000000001e-05 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212032+0000", 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00035369900000000002 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212515+0000", 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00048357899999999999 2024-04-25T15:52:39.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212765+0000", 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024946400000000002 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217815+0000", 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050506989999999996 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221231+0000", 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003415567 2024-04-25T15:52:39.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221412+0000", 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000180681 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221542+0000", 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130171 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:39.702 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.460336895, 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:27 2.13 2:cd5a64a3:::100.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.46032323599999, 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010202413, 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.707 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211509+0000", 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211510+0000", 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211538+0000", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7475e-05 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211876+0000", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033851299999999998 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211898+0000", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1648999999999999e-05 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212060+0000", 2024-04-25T15:52:39.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016183899999999999 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212463+0000", 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040314700000000002 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212983+0000", 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00052057999999999998 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.220727+0000", 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077435919999999997 2024-04-25T15:52:39.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221245+0000", 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00051843400000000002 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221590+0000", 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034479999999999998 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221712+0000", 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121477 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.46028951, 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:39.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:39.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:39.716 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.460277428, 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:39.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.46025725600001, 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 123.460018918, 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:17 2.13 2:ca4dd7f5:::611.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.422285002, 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010948628, 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249548+0000", 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249551+0000", 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2329999999999998e-06 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249572+0000", 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0979e-05 2024-04-25T15:52:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249640+0000", 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8326999999999998e-05 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249668+0000", 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8368999999999999e-05 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250082+0000", 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041399200000000002 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250425+0000", 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034239200000000001 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.253136+0000", 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027107210000000001 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256784+0000", 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036481980000000001 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260324+0000", 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035401909999999998 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260418+0000", 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3692999999999999e-05 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260496+0000", 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8533000000000002e-05 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:18 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 117.422250781, 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011222823999999999, 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249583+0000", 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1799999999999999e-06 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249591+0000", 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0669999999999992e-06 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249819+0000", 2024-04-25T15:52:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022795000000000001 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249865+0000", 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5561e-05 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254040+0000", 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041751720000000004 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254349+0000", 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030965399999999997 2024-04-25T15:52:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254663+0000", 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031403600000000001 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258978+0000", 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043149959999999998 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260612+0000", 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00163383 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260715+0000", 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010311599999999999 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260805+0000", 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9262e-05 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:7 2.13 2:cdf3ac81:::616.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.34097450900001, 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086917899999999996, 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330858+0000", 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330859+0000", 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7000000000000005e-07 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330864+0000", 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1120000000000004e-06 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331053+0000", 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000189261 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331064+0000", 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1260000000000001e-05 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331313+0000", 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024867500000000001 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331526+0000", 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000212721 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331732+0000", 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000206839 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335642+0000", 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039095639999999999 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339325+0000", 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0036827790000000002 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339449+0000", 2024-04-25T15:52:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123924 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339550+0000", 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101085 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:8 2.13 2:cdf3ac81:::616.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.34096641799999, 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010026810000000001, 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330866+0000", 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330867+0000", 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8899999999999999e-07 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330871+0000", 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2089999999999999e-06 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331105+0000", 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000234381 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331119+0000", 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4165000000000001e-05 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332225+0000", 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0011058140000000001 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332348+0000", 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122872 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.332557+0000", 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000208364 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336449+0000", 2024-04-25T15:52:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038921329999999999 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340746+0000", 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0042971779999999996 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340840+0000", 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4109000000000003e-05 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340893+0000", 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2995999999999999e-05 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.2.12:32 2.1f 2:fe09a0b8:::202.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.340946655, 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090313700000000004, 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330886+0000", 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330887+0000", 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1799999999999995e-07 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330890+0000", 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.427e-06 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331211+0000", 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000320639 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.331220+0000", 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4669999999999993e-06 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.335755+0000", 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045349500000000003 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336042+0000", 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028675400000000001 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.339917+0000", 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038755149999999999 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:1 2.1f 2:fb684bcd:::204.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.746216582000002, 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012576726, 2024-04-25T15:52:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000002e-07 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925629+0000", 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3002e-05 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925764+0000", 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013477399999999999 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925779+0000", 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228000000000001e-05 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925847+0000", 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7980999999999998e-05 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925951+0000", 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010333299999999999 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926085+0000", 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013462200000000001 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929258+0000", 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003172607 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938074+0000", 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088162910000000004 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938129+0000", 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5234e-05 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938193+0000", 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3367000000000005e-05 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:31 2.1f 2:ff4f7703:::204.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 95.746201845000002, 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012700365, 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5499999999999999e-07 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925633+0000", 2024-04-25T15:52:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.796e-06 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925809+0000", 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017651200000000001 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925824+0000", 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5071e-05 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926689+0000", 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086453799999999998 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926825+0000", 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013552799999999999 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938331+0000", 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011506565 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:39.759 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:39.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:39.761 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:39.759+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:39.861 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:39.859+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:39.952 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:39.953 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:39.953 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:39.962 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:39.959+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:39.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:40.063 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:40.058+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:40.108 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:40.109 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:40.109 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:40.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:40.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:40.155 INFO:tasks.workunit.client.0.smithi067.stderr:Updating files: 95% (11686/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-25T15:52:40.164 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:40.158+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:40.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:52:40.265 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:40.263+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:40.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f2830156640 1 Processor -- start 2024-04-25T15:52:40.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f2830156640 1 -- start start 2024-04-25T15:52:40.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f2830156640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f28280c92d0 0x7f2828072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f2830156640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28280c96e0 0x7f2828073300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f2830156640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282806d8e0 con 0x7f2828160800 2024-04-25T15:52:40.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f2830156640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f282806dbc0 con 0x7f28280c96e0 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f2830156640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2828073c70 con 0x7f28280c92d0 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f282e6f3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28280c96e0 0x7f2828073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f282def2640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2828160800 0x7f282806d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49812/0 (socket says 172.21.15.67:49812) 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f282def2640 1 -- 172.21.15.67:0/3656350339 learned_addr learned my addr 172.21.15.67:0/3656350339 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f282e6f3640 1 -- 172.21.15.67:0/3656350339 >> v2:172.21.15.176:3300/0 conn(0x7f28280c92d0 msgr2=0x7f2828072dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f282e6f3640 1 --2- 172.21.15.67:0/3656350339 >> v2:172.21.15.176:3300/0 conn(0x7f28280c92d0 0x7f2828072dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f282e6f3640 1 -- 172.21.15.67:0/3656350339 >> v1:172.21.15.67:6789/0 conn(0x7f2828160800 legacy=0x7f282806d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.282+0000 7f282e6f3640 1 -- 172.21.15.67:0/3656350339 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2828073f50 con 0x7f28280c96e0 2024-04-25T15:52:40.289 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:40.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.077163423, 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013243108, 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0946e-05 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211617+0000", 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211761+0000", 2024-04-25T15:52:40.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014392699999999999 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211793+0000", 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1568e-05 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212109+0000", 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031626100000000003 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212422+0000", 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031341099999999998 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213414+0000", 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099192799999999999 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223830+0000", 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010415654999999999 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224524+0000", 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069455899999999998 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224662+0000", 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013742600000000001 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224828+0000", 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016641799999999999 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.07712211800001, 2024-04-25T15:52:40.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013508035, 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211628+0000", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3400000000000001e-06 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211641+0000", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2866e-05 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027786899999999998 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211973+0000", 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4401000000000003e-05 2024-04-25T15:52:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214895+0000", 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029218109999999999 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215130+0000", 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023500499999999999 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215556+0000", 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042636900000000001 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224481+0000", 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089248869999999994 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224861+0000", 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038038699999999999 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225017+0000", 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155296 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225135+0000", 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117804 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.07710473900001, 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013948439999999999, 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211645+0000", 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1699999999999997e-07 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211680+0000", 2024-04-25T15:52:40.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5374999999999999e-05 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212061+0000", 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038129700000000001 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212135+0000", 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4103000000000005e-05 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218486+0000", 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063501709999999999 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218861+0000", 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037518799999999999 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219238+0000", 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037732500000000002 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225340+0000", 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061016430000000003 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225414+0000", 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.436e-05 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225489+0000", 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4882999999999994e-05 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225592+0000", 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103378 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.303 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.077063014, 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013711255, 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211687+0000", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-06 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211694+0000", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6329999999999996e-06 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212201+0000", 2024-04-25T15:52:40.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050685200000000002 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212239+0000", 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7537999999999999e-05 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004023726 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216562+0000", 2024-04-25T15:52:40.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029964199999999997 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033712999999999999 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223552+0000", 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066527330000000001 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225162+0000", 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016101869999999999 2024-04-25T15:52:40.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225266+0000", 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010402400000000001 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225397+0000", 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130684 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:6 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:40.308 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.072789687, 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198140000000006, 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215961+0000", 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5469999999999999e-06 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215983+0000", 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2014999999999998e-05 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216017+0000", 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3114999999999997e-05 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216043+0000", 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6755999999999999e-05 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216409+0000", 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036599799999999997 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216466+0000", 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6947000000000002e-05 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224337+0000", 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078708230000000007 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224479+0000", 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014161299999999999 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:7 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:40.312 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.072639481, 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198700000000006, 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:40.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216110+0000", 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3570000000000001e-06 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216125+0000", 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4772e-05 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216151+0000", 2024-04-25T15:52:40.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5496000000000001e-05 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216172+0000", 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1095e-05 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217019+0000", 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084760800000000002 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217071+0000", 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1894000000000002e-05 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224489+0000", 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074175710000000004 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224629+0000", 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014007700000000001 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:8 2.10 e20/17 2:09401a81:::601.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.070954585, 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069206529999999997, 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217796+0000", 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9069999999999999e-06 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217816+0000", 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0296e-05 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217847+0000", 2024-04-25T15:52:40.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0877999999999998e-05 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217872+0000", 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4749e-05 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218099+0000", 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022737199999999999 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8900000000000002e-05 2024-04-25T15:52:40.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224635+0000", 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064771480000000003 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224715+0000", 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9402999999999999e-05 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:9 2.10 e20/17 2:09401a81:::601.00000000:head v 20'4, mlcod=0'0)", 2024-04-25T15:52:40.320 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.069839707, 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058821409999999996, 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218909+0000", 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218911+0000", 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218930+0000", 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8916e-05 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218960+0000", 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0065000000000001e-05 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218984+0000", 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4528999999999998e-05 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219206+0000", 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022124300000000001 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219263+0000", 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7003000000000003e-05 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224719+0000", 2024-04-25T15:52:40.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054565830000000001 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224791+0000", 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1928999999999995e-05 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:11 2.10 e20/17 2:0bd6d154:::602.00000000:head v 20'6, mlcod=0'0)", 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 124.06746207499999, 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050276770000000004, 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.324 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221287+0000", 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221288+0000", 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221306+0000", 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7719000000000001e-05 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221336+0000", 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0026e-05 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.221365+0000", 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8725e-05 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222395+0000", 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001030078 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222455+0000", 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9803999999999999e-05 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226163+0000", 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0037085629999999998 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226314+0000", 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150829 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:19 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.041110377, 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010474059000000001, 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247641+0000", 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.129e-06 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247667+0000", 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6089e-05 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247827+0000", 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000159977 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247854+0000", 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6971999999999999e-05 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248122+0000", 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026745499999999999 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248469+0000", 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034735899999999999 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248784+0000", 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031528900000000001 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251768+0000", 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002983383 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257863+0000", 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006095339 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258009+0000", 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000145921 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258112+0000", 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103146 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:20 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.041071606, 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010923117, 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:40.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247688+0000", 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3659999999999998e-06 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247937+0000", 2024-04-25T15:52:40.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024977700000000002 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247958+0000", 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249612+0000", 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016540089999999999 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249860+0000", 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024840600000000001 2024-04-25T15:52:40.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250352+0000", 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049215499999999998 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255543+0000", 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051905060000000001 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258273+0000", 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027298460000000002 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258467+0000", 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019409900000000001 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258600+0000", 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013348099999999999 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:23 2.15 2:aa0f8bf4:::101.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.041056553, 2024-04-25T15:52:40.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082242619999999995, 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247728+0000", 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5049000000000003e-05 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248028+0000", 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029943400000000001 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248050+0000", 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2722e-05 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248203+0000", 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015264 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248563+0000", 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036047500000000002 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248940+0000", 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037679400000000001 2024-04-25T15:52:40.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252184+0000", 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003243821 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255610+0000", 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034255370000000002 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255793+0000", 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018366899999999999 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255916+0000", 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122998 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.04101461800001, 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:52:40.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.346 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.1.10:4 2.10 e20/17 2:0f65c9b8:::60a.00000000:head v 20'10, mlcod=20'7)", 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 118.037656845, 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054254350000000002, 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251092+0000", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251094+0000", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.136e-06 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251114+0000", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0003000000000001e-05 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251147+0000", 2024-04-25T15:52:40.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2947999999999997e-05 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251171+0000", 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3845999999999998e-05 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251347+0000", 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000176813 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251408+0000", 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0470000000000002e-05 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256382+0000", 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049743510000000001 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.256517+0000", 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013486800000000001 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.350 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:3 2.0 e20/17 2:03625d61:::614.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.95810757700001, 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060303220000000003, 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330641+0000", 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330643+0000", 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9850000000000001e-06 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330658+0000", 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5386000000000001e-05 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330686+0000", 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7971e-05 2024-04-25T15:52:40.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330702+0000", 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5268999999999999e-05 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330841+0000", 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013921800000000001 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.330881+0000", 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0052000000000002e-05 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336596+0000", 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057151320000000004 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.336671+0000", 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5308999999999993e-05 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.2.12:4 2.0 e20/17 2:03625d61:::614.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 107.954836086, 2024-04-25T15:52:40.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068526890000000003, 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333913+0000", 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333914+0000", 2024-04-25T15:52:40.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3859999999999999e-06 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333929+0000", 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5508000000000001e-05 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333945+0000", 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5968999999999999e-05 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.333961+0000", 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5160999999999999e-05 2024-04-25T15:52:40.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334268+0000", 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030782800000000001 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.334291+0000", 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2459e-05 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340696+0000", 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064053749999999996 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:52.340765+0000", 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9003000000000003e-05 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:6 2.9 2:941d59bc:::61f.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.401785786, 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054913510000000002, 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886963+0000", 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886964+0000", 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9999999999999996e-07 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.886973+0000", 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1400000000000006e-06 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887182+0000", 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020973699999999999 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887195+0000", 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2502e-05 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888149+0000", 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00095407200000000004 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888285+0000", 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013629399999999999 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888481+0000", 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000195343 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890469+0000", 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019881650000000001 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892371+0000", 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019022620000000001 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892426+0000", 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4404999999999998e-05 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892454+0000", 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8631000000000001e-05 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:22 2.6 2:62eebe7c:::627.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.40170494100001, 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052764769999999999, 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887044+0000", 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8400000000000004e-07 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887048+0000", 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7390000000000001e-06 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887454+0000", 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040556499999999998 2024-04-25T15:52:40.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887460+0000", 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8639999999999998e-06 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.888968+0000", 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015077599999999999 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889063+0000", 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4658000000000001e-05 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889206+0000", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014365600000000001 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.890596+0000", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001390023 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892161+0000", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015643219999999999 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892259+0000", 2024-04-25T15:52:40.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7955000000000007e-05 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892320+0000", 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1351000000000003e-05 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.3.14:24 2.15 2:afbd2f49:::103.00000000:head [omap-set-header,omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 102.401698545, 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0055188479999999998, 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887050+0000", 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887051+0000", 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1699999999999998e-07 2024-04-25T15:52:40.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887063+0000", 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2575000000000001e-05 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887475+0000", 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041185899999999999 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887481+0000", 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0970000000000002e-06 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887550+0000", 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9258000000000007e-05 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887693+0000", 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014290400000000001 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.887898+0000", 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020492000000000001 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.889987+0000", 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020884929999999999 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892508+0000", 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002521644 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892542+0000", 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3654e-05 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:57.892569+0000", 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6927000000000002e-05 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 96.361531702999997, 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010008022, 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:40.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927218+0000", 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927222+0000", 2024-04-25T15:52:40.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6519999999999997e-06 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927339+0000", 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116705 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927350+0000", 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0995000000000001e-05 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928080+0000", 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072958300000000001 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928207+0000", 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012715 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928433+0000", 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022608599999999999 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070747400000000004 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937109+0000", 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079685439999999993 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9806999999999998e-05 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937225+0000", 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6313000000000001e-05 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:40.376 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:40.377 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f282e6f3640 1 --2- 172.21.15.67:0/3656350339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828073300 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f282806dd40 tx=0x7f282001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb2aa4042c80e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f282cef0640 1 -- 172.21.15.67:0/3656350339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28200b2070 con 0x7f28280c96e0 2024-04-25T15:52:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f282cef0640 1 -- 172.21.15.67:0/3656350339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28200a5870 con 0x7f28280c96e0 2024-04-25T15:52:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f282cef0640 1 -- 172.21.15.67:0/3656350339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28200a5c40 con 0x7f28280c96e0 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 -- 172.21.15.67:0/3656350339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 msgr2=0x7f2828073300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 --2- 172.21.15.67:0/3656350339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828073300 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f282806dd40 tx=0x7f282001c8c0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 -- 172.21.15.67:0/3656350339 shutdown_connections 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 --2- 172.21.15.67:0/3656350339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828073300 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 --2- 172.21.15.67:0/3656350339 >> v2:172.21.15.176:3300/0 conn(0x7f28280c92d0 0x7f2828072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 -- 172.21.15.67:0/3656350339 >> 172.21.15.67:0/3656350339 conn(0x7f2828070db0 msgr2=0x7f28280ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 -- 172.21.15.67:0/3656350339 shutdown_connections 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.286+0000 7f2830156640 1 -- 172.21.15.67:0/3656350339 wait complete. 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 Processor -- start 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 -- start start 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c92d0 0x7f28281578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2828160800 0x7f282814cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2828160d90 con 0x7f28280c96e0 2024-04-25T15:52:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2828160c10 con 0x7f28280c92d0 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f28281699a0 con 0x7f2828160800 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43082/0 (socket says 172.21.15.67:43082) 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 -- 172.21.15.67:0/959502837 learned_addr learned my addr 172.21.15.67:0/959502837 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2828160800 0x7f282814cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c92d0 0x7f28281578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2828160800 msgr2=0x7f282814cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2828160800 0x7f282814cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c92d0 msgr2=0x7f28281578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c92d0 0x7f28281578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28281f6bf0 con 0x7f28280c96e0 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c92d0 0x7f28281578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2828160800 0x7f282814cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f282d6f1640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828157e30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f281800b410 tx=0x7f281800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2818015020 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2818011830 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f2830156640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28281f5e30 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.290+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2818011c20 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f2830156640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28281f63f0 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2804005280 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2818018020 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f281e7fc640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f280c06f400 0x7f280c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5857+0+0 (secure 0 0 0) 0x7f281808f350 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v57) ==== 54+0+16 (secure 0 0 0) 0x7f281801f020 con 0x7f28280c96e0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f280c06f400 0x7f280c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f28040045f0 0x7f2804004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2804004f40 con 0x7f28040045f0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f28040045f0 0x7f2804004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f280c06f400 0x7f280c071910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2824000f70 tx=0x7f2824009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.294+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f28040045f0 0x7f2804004a00 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.298+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2804004f40 con 0x7f28040045f0 2024-04-25T15:52:40.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:40.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.314+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f28040013e0 con 0x7f28040045f0 2024-04-25T15:52:40.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.314+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f28040013e0 con 0x7f28040045f0 2024-04-25T15:52:40.385 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:40.386 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:40.375+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.314+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2804001e60 0x7f28040029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.314+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f28040023a0 con 0x7f2804001e60 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.314+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2804001e60 0x7f28040029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.318+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2804001e60 0x7f28040029a0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.318+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f28040023a0 con 0x7f2804001e60 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.342+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2804003270 con 0x7f2804001e60 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.342+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2804003270 con 0x7f2804001e60 2024-04-25T15:52:40.387 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.342+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2804009c90 0x7f280400a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.342+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f280400a5e0 con 0x7f2804009c90 2024-04-25T15:52:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.342+0000 7f282d6f1640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2804009c90 0x7f280400a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.346+0000 7f282d6f1640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2804009c90 0x7f280400a0a0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.346+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f280400a5e0 con 0x7f2804009c90 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 Processor -- start 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 -- start start 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff634071bc0 0x7ff6340cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff6340cc100 0x7ff63415a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff63415ac60 0x7ff6340ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff63415ebb0 con 0x7ff63415ac60 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff63415e720 con 0x7ff6340cc100 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6393f0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff63415de10 con 0x7ff634071bc0 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff63415ac60 0x7ff6340ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff63415ac60 0x7ff6340ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43094/0 (socket says 172.21.15.67:43094) 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 -- 172.21.15.67:0/2593008555 learned_addr learned my addr 172.21.15.67:0/2593008555 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff632ffd640 1 --2- 172.21.15.67:0/2593008555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff634071bc0 0x7ff6340cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6327fc640 1 --2- 172.21.15.67:0/2593008555 >> v2:172.21.15.176:3300/0 conn(0x7ff6340cc100 0x7ff63415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 -- 172.21.15.67:0/2593008555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff634071bc0 msgr2=0x7ff6340cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 --2- 172.21.15.67:0/2593008555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff634071bc0 0x7ff6340cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 -- 172.21.15.67:0/2593008555 >> v2:172.21.15.176:3300/0 conn(0x7ff6340cc100 msgr2=0x7ff63415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 --2- 172.21.15.67:0/2593008555 >> v2:172.21.15.176:3300/0 conn(0x7ff6340cc100 0x7ff63415a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.350+0000 7ff6337fe640 1 -- 172.21.15.67:0/2593008555 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff634072390 con 0x7ff63415ac60 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.354+0000 7ff6337fe640 1 --2- 172.21.15.67:0/2593008555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63415ac60 0x7ff6340ca540 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff6340d0ba0 tx=0x7ff62c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a238023b8fe990 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.354+0000 7ff631ffb640 1 -- 172.21.15.67:0/2593008555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0a5780 con 0x7ff63415ac60 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.354+0000 7ff631ffb640 1 -- 172.21.15.67:0/2593008555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62c0a5920 con 0x7ff63415ac60 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.354+0000 7ff631ffb640 1 -- 172.21.15.67:0/2593008555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0a5cf0 con 0x7ff63415ac60 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- 172.21.15.67:0/2593008555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63415ac60 msgr2=0x7ff6340ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 --2- 172.21.15.67:0/2593008555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63415ac60 0x7ff6340ca540 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff6340d0ba0 tx=0x7ff62c01c9f0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- 172.21.15.67:0/2593008555 shutdown_connections 2024-04-25T15:52:40.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 --2- 172.21.15.67:0/2593008555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63415ac60 0x7ff6340ca540 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 --2- 172.21.15.67:0/2593008555 >> v2:172.21.15.176:3300/0 conn(0x7ff6340cc100 0x7ff63415a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 --2- 172.21.15.67:0/2593008555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff634071bc0 0x7ff6340cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- 172.21.15.67:0/2593008555 >> 172.21.15.67:0/2593008555 conn(0x7ff6340700b0 msgr2=0x7ff6340704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- 172.21.15.67:0/2593008555 shutdown_connections 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- 172.21.15.67:0/2593008555 wait complete. 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 Processor -- start 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- start start 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff634071bc0 0x7ff634150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6340cc100 0x7ff63414dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff63415ac60 0x7ff634155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff634072150 con 0x7ff6340cc100 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff634071fd0 con 0x7ff634071bc0 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6393f0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff6340d02e0 con 0x7ff63415ac60 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6327fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6340cc100 0x7ff63414dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6327fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6340cc100 0x7ff63414dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43110/0 (socket says 172.21.15.67:43110) 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.358+0000 7ff6327fc640 1 -- 172.21.15.67:0/3362030353 learned_addr learned my addr 172.21.15.67:0/3362030353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6337fe640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff63415ac60 0x7ff634155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff632ffd640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff634071bc0 0x7ff634150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6327fc640 1 -- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff63415ac60 msgr2=0x7ff634155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6327fc640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff63415ac60 0x7ff634155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6327fc640 1 -- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff634071bc0 msgr2=0x7ff634150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6327fc640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff634071bc0 0x7ff634150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6327fc640 1 -- 172.21.15.67:0/3362030353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff634150c40 con 0x7ff6340cc100 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6327fc640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6340cc100 0x7ff63414dea0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d5d0 tx=0x7ff62800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff61b7fe640 1 -- 172.21.15.67:0/3362030353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6280125f0 con 0x7ff6340cc100 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff61b7fe640 1 -- 172.21.15.67:0/3362030353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff628019070 con 0x7ff6340cc100 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63414f6a0 con 0x7ff6340cc100 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff61b7fe640 1 -- 172.21.15.67:0/3362030353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6280129b0 con 0x7ff6340cc100 2024-04-25T15:52:40.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63414fc60 con 0x7ff6340cc100 2024-04-25T15:52:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63415c380 con 0x7ff6340cc100 2024-04-25T15:52:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff61b7fe640 1 -- 172.21.15.67:0/3362030353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff628012b50 con 0x7ff6340cc100 2024-04-25T15:52:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f280400aaf0 con 0x7f2804009c90 2024-04-25T15:52:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff61b7fe640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff62006f400 0x7ff620071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff61b7fe640 1 -- 172.21.15.67:0/3362030353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5857+0+0 (secure 0 0 0) 0x7ff62808f640 con 0x7ff6340cc100 2024-04-25T15:52:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.362+0000 7ff632ffd640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff62006f400 0x7ff620071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.366+0000 7ff632ffd640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff62006f400 0x7ff620071910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff62400b990 tx=0x7ff624008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.366+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f280400aaf0 con 0x7f2804009c90 2024-04-25T15:52:40.397 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.366+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f280400aec0 0x7f280400f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.366+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f28040128d0 con 0x7f280400aec0 2024-04-25T15:52:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.366+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f280400aec0 0x7f280400f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.366+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f280400aec0 0x7f280400f900 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.370+0000 7ff61b7fe640 1 -- 172.21.15.67:0/3362030353 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff63415c380 con 0x7ff6340cc100 2024-04-25T15:52:40.479 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:40.479+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:40.535 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:40.535 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.538+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7ff634150340 con 0x7ff6340cc100 2024-04-25T15:52:40.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:40.568 INFO:tasks.workunit.client.0.smithi067.stderr:Note: switching to '994e2f20102a20fa9a27966c14b625cc01884df8'. 2024-04-25T15:52:40.568 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-25T15:52:40.568 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T15:52:40.568 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T15:52:40.568 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: git switch -c 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr:Or undo this operation with: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: git switch - 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-25T15:52:40.569 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at 994e2f20102 Merge PR #56810 into wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:52:40.576 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T15:52:40.622 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:40.618+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:40.625 INFO:tasks.workunit.client.0.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T15:52:40.627 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T15:52:40.627 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T15:52:40.681 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:40.679+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:40.691 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:40.691 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:40.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:40.794 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:40.790+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:40.883 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:40.883+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.886+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f28040128d0 con 0x7f280400aec0 2024-04-25T15:52:40.901 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:40.902 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.902+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f28040101b0 con 0x7f280400aec0 2024-04-25T15:52:40.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.902+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f28040101b0 con 0x7f280400aec0 2024-04-25T15:52:40.909 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:40.910 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:40.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.906+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f28040108d0 0x7f2804010ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.906+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2804012470 con 0x7f28040108d0 2024-04-25T15:52:40.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.906+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f28040108d0 0x7f2804010ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.906+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f28040108d0 0x7f2804010ce0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.906+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2804012470 con 0x7f28040108d0 2024-04-25T15:52:40.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:40.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.926+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f28040123f0 con 0x7f28040108d0 2024-04-25T15:52:40.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.930+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f28040123f0 con 0x7f28040108d0 2024-04-25T15:52:40.934 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:40.935 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:40.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.930+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2804011730 0x7f2804011b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.930+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f28040193c0 con 0x7f2804011730 2024-04-25T15:52:40.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.930+0000 7f282d6f1640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2804011730 0x7f2804011b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.930+0000 7f282d6f1640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2804011730 0x7f2804011b40 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.934+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f28040193c0 con 0x7f2804011730 2024-04-25T15:52:40.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.946+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2804016b30 con 0x7f2804011730 2024-04-25T15:52:40.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.950+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2804016b30 con 0x7f2804011730 2024-04-25T15:52:40.955 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.950+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f28040171b0 0x7f28040195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.950+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f280401c530 con 0x7f28040171b0 2024-04-25T15:52:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.950+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f28040171b0 0x7f28040195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.950+0000 7f282e6f3640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f28040171b0 0x7f28040195d0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.954+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f280401c530 con 0x7f28040171b0 2024-04-25T15:52:40.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.966+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f280401b250 con 0x7f28040171b0 2024-04-25T15:52:40.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.970+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f280401b250 con 0x7f28040171b0 2024-04-25T15:52:40.977 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.970+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f280401b970 0x7f280401bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.970+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f280401f900 con 0x7f280401b970 2024-04-25T15:52:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.974+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f280401b970 0x7f280401bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:40.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.974+0000 7f282def2640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f280401b970 0x7f280401bd80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:40.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.974+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f280401f900 con 0x7f280401b970 2024-04-25T15:52:40.984 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:40.978+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.990+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f280401e780 con 0x7f280401b970 2024-04-25T15:52:40.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f281e7fc640 1 -- 172.21.15.67:0/959502837 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f280401e780 con 0x7f280401b970 2024-04-25T15:52:40.998 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f28040045f0 msgr2=0x7f2804004a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f28040045f0 0x7f2804004a00 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2804001e60 msgr2=0x7f28040029a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2804001e60 0x7f28040029a0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2804009c90 msgr2=0x7f280400a0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2804009c90 0x7f280400a0a0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f280400aec0 msgr2=0x7f280400f900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f280400aec0 0x7f280400f900 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f28040108d0 msgr2=0x7f2804010ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f28040108d0 0x7f2804010ce0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2804011730 msgr2=0x7f2804011b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2804011730 0x7f2804011b40 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f282e6f3640 1 -- 172.21.15.67:0/959502837 reap_dead start 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f282e6f3640 1 -- 172.21.15.67:0/959502837 reap_dead start 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f282e6f3640 1 -- 172.21.15.67:0/959502837 reap_dead start 2024-04-25T15:52:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f28040171b0 msgr2=0x7f28040195d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f28040171b0 0x7f28040195d0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f280401b970 msgr2=0x7f280401bd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f280401b970 0x7f280401bd80 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f280c06f400 msgr2=0x7f280c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f280c06f400 0x7f280c071910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2824000f70 tx=0x7f2824009040 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 msgr2=0x7f2828157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828157e30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f281800b410 tx=0x7f281800b8d0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 shutdown_connections 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f28040171b0 0x7f28040195d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f280401b970 0x7f280401bd80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f280c06f400 0x7f280c071910 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 --2- 172.21.15.67:0/959502837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28280c96e0 0x7f2828157e30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 >> 172.21.15.67:0/959502837 conn(0x7f2828070db0 msgr2=0x7f2828154600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 shutdown_connections 2024-04-25T15:52:41.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:40.994+0000 7f280bfff640 1 -- 172.21.15.67:0/959502837 wait complete. 2024-04-25T15:52:41.068 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:41.068 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:41.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:41.085 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:41.078+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:41.185 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:41.183+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:41.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff61b7fe640 1 -- 172.21.15.67:0/3362030353 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v58) ==== 87+0+0 (secure 0 0 0) 0x7ff62805c520 con 0x7ff6340cc100 2024-04-25T15:52:41.198 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:52:41.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff62006f400 msgr2=0x7ff620071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff62006f400 0x7ff620071910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff62400b990 tx=0x7ff624008040 comp rx=0 tx=0).stop 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6340cc100 msgr2=0x7ff63414dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6340cc100 0x7ff63414dea0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d5d0 tx=0x7ff62800da90 comp rx=0 tx=0).stop 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 shutdown_connections 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff62006f400 0x7ff620071910 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff63415ac60 0x7ff634155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6340cc100 0x7ff63414dea0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 --2- 172.21.15.67:0/3362030353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff634071bc0 0x7ff634150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 >> 172.21.15.67:0/3362030353 conn(0x7ff6340700b0 msgr2=0x7ff63414e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 shutdown_connections 2024-04-25T15:52:41.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.194+0000 7ff6393f0640 1 -- 172.21.15.67:0/3362030353 wait complete. 2024-04-25T15:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:41.270 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:41.286 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:41.283+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:41.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:41.330 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T15:52:41.387 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:41.382+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:41.402 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T15:52:41.423 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:41.424 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:41.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:41.446 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T15:52:41.447 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T15:52:41.447 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T15:52:41.486 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T15:52:41.491 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:52:41.491 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T15:52:41.491 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:41.491+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:41.538 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-04-25T15:52:41.539 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-04-25T15:52:41.539 DEBUG:teuthology.orchestra.run.smithi067:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-04-25T15:52:41.589 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:41.587+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:41.599 INFO:tasks.workunit.client.0.smithi067.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-25T15:52:41.601 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:41.601 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:41.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Iozone: Performance Test of File I/O 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Version $Revision: 3.489 $ 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Compiled for 64 bit mode. 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Build: linux-AMD64 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-25T15:52:41.703 INFO:tasks.workunit.client.0.smithi067.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Run began: Thu Apr 25 15:52:41 2024 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Include close in write timing 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Include fsync in write timing 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: File size set to 1048576 kB 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Record Size 16 kB 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Output is in kBytes/sec 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Time Resolution = 0.000001 seconds. 2024-04-25T15:52:41.704 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache size set to 1024 kBytes. 2024-04-25T15:52:41.705 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache line size set to 32 bytes. 2024-04-25T15:52:41.705 INFO:tasks.workunit.client.0.smithi067.stdout: File stride size set to 17 * record size. 2024-04-25T15:52:41.705 INFO:tasks.workunit.client.0.smithi067.stdout: Throughput test with 1 process 2024-04-25T15:52:41.705 INFO:tasks.workunit.client.0.smithi067.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-04-25T15:52:41.707 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:41.707+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:41.749 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:41.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:41.750 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-25T15:52:41.750 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-04-25T15:52:41.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-25T15:52:41.790 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:41.787+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:41.811 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:41.811 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:41.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:41.891 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:41.886+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc219a25640 1 Processor -- start 2024-04-25T15:52:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc219a25640 1 -- start start 2024-04-25T15:52:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc219a25640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc214071bc0 0x7fc2140cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc219a25640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc2140ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc219a25640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc21415de10 con 0x7fc2140cc100 2024-04-25T15:52:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc219a25640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc21415ebb0 con 0x7fc214071bc0 2024-04-25T15:52:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc219a25640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc21415e720 con 0x7fc2140ca540 2024-04-25T15:52:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc212ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc214071bc0 0x7fc2140cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc2127fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc2140cc100 0x7fc21415a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49816/0 (socket says 172.21.15.67:49816) 2024-04-25T15:52:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc2127fc640 1 -- 172.21.15.67:0/3076162049 learned_addr learned my addr 172.21.15.67:0/3076162049 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc2137fe640 1 --2- 172.21.15.67:0/3076162049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc2140ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc212ffd640 1 -- 172.21.15.67:0/3076162049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 msgr2=0x7fc2140ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:41.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc212ffd640 1 --2- 172.21.15.67:0/3076162049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc2140ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc212ffd640 1 -- 172.21.15.67:0/3076162049 >> v1:172.21.15.67:6789/0 conn(0x7fc2140cc100 legacy=0x7fc21415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc212ffd640 1 -- 172.21.15.67:0/3076162049 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc214072390 con 0x7fc214071bc0 2024-04-25T15:52:41.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.910+0000 7fc211ffb640 1 -- 172.21.15.67:0/3076162049 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452342072 0 0) 0x7fc21415de10 con 0x7fc2140cc100 2024-04-25T15:52:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc212ffd640 1 --2- 172.21.15.67:0/3076162049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc214071bc0 0x7fc2140cbbc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc2040097c0 tx=0x7fc20401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbeda923f04424f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc211ffb640 1 -- 172.21.15.67:0/3076162049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2040b1070 con 0x7fc214071bc0 2024-04-25T15:52:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc211ffb640 1 -- 172.21.15.67:0/3076162049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2040ac410 con 0x7fc214071bc0 2024-04-25T15:52:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc211ffb640 1 -- 172.21.15.67:0/3076162049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2040ac7e0 con 0x7fc214071bc0 2024-04-25T15:52:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 -- 172.21.15.67:0/3076162049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc214071bc0 msgr2=0x7fc2140cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 --2- 172.21.15.67:0/3076162049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc214071bc0 0x7fc2140cbbc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc2040097c0 tx=0x7fc20401c4a0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 -- 172.21.15.67:0/3076162049 shutdown_connections 2024-04-25T15:52:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 --2- 172.21.15.67:0/3076162049 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc2140ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 --2- 172.21.15.67:0/3076162049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc214071bc0 0x7fc2140cbbc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 -- 172.21.15.67:0/3076162049 >> 172.21.15.67:0/3076162049 conn(0x7fc2140700b0 msgr2=0x7fc2140704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 -- 172.21.15.67:0/3076162049 shutdown_connections 2024-04-25T15:52:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.914+0000 7fc219a25640 1 -- 172.21.15.67:0/3076162049 wait complete. 2024-04-25T15:52:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 Processor -- start 2024-04-25T15:52:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 -- start start 2024-04-25T15:52:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc214071bc0 0x7fc2141f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc214148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140cc100 0x7fc214142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc214072150 con 0x7fc2140cc100 2024-04-25T15:52:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc214071fd0 con 0x7fc214071bc0 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc219a25640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc21415ae30 con 0x7fc2140ca540 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc2137fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140cc100 0x7fc214142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc2137fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140cc100 0x7fc214142180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43140/0 (socket says 172.21.15.67:43140) 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc2137fe640 1 -- 172.21.15.67:0/1246193534 learned_addr learned my addr 172.21.15.67:0/1246193534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc2127fc640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc214148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.918+0000 7fc212ffd640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc214071bc0 0x7fc2141f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc2137fe640 1 -- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 msgr2=0x7fc214148650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc2137fe640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc214148650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc2137fe640 1 -- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc214071bc0 msgr2=0x7fc2141f42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc2137fe640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc214071bc0 0x7fc2141f42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc2137fe640 1 -- 172.21.15.67:0/1246193534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2141f4810 con 0x7fc2140cc100 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc2137fe640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140cc100 0x7fc214142180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00efc0 tx=0x7fc20c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc1fbfff640 1 -- 172.21.15.67:0/1246193534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c018020 con 0x7fc2140cc100 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2141f7830 con 0x7fc2140cc100 2024-04-25T15:52:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc1fbfff640 1 -- 172.21.15.67:0/1246193534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc20c014480 con 0x7fc2140cc100 2024-04-25T15:52:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc1fbfff640 1 -- 172.21.15.67:0/1246193534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20c009e60 con 0x7fc2140cc100 2024-04-25T15:52:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2141f7df0 con 0x7fc2140cc100 2024-04-25T15:52:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2141f7ff0 con 0x7fc2140cc100 2024-04-25T15:52:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc1fbfff640 1 -- 172.21.15.67:0/1246193534 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc20c01b020 con 0x7fc2140cc100 2024-04-25T15:52:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc1fbfff640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc20006f430 0x7fc200071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.922+0000 7fc1fbfff640 1 -- 172.21.15.67:0/1246193534 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5857+0+0 (secure 0 0 0) 0x7fc20c08fec0 con 0x7fc2140cc100 2024-04-25T15:52:41.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.926+0000 7fc212ffd640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc20006f430 0x7fc200071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.926+0000 7fc212ffd640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc20006f430 0x7fc200071940 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc2040ae000 tx=0x7fc204008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:41.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:41.930+0000 7fc1fbfff640 1 -- 172.21.15.67:0/1246193534 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc2141f7ff0 con 0x7fc2140cc100 2024-04-25T15:52:41.992 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:41.986+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:42.020 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:42.020 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:42.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:42.093 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:42.086+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:42.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.142+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7fc21406a480 con 0x7fc2140cc100 2024-04-25T15:52:42.191 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:42.191 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:42.193 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:42.191+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc1fbfff640 1 -- 172.21.15.67:0/1246193534 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v59) ==== 133+0+0 (secure 0 0 0) 0x7fc20c022030 con 0x7fc2140cc100 2024-04-25T15:52:42.201 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc20006f430 msgr2=0x7fc200071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc20006f430 0x7fc200071940 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc2040ae000 tx=0x7fc204008040 comp rx=0 tx=0).stop 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140cc100 msgr2=0x7fc214142180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140cc100 0x7fc214142180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00efc0 tx=0x7fc20c00c490 comp rx=0 tx=0).stop 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 shutdown_connections 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc20006f430 0x7fc200071940 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2140cc100 0x7fc214142180 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc2140ca540 0x7fc214148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 --2- 172.21.15.67:0/1246193534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc214071bc0 0x7fc2141f42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:42.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 >> 172.21.15.67:0/1246193534 conn(0x7fc2140700b0 msgr2=0x7fc2140704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:42.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 shutdown_connections 2024-04-25T15:52:42.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:42.198+0000 7fc219a25640 1 -- 172.21.15.67:0/1246193534 wait complete. 2024-04-25T15:52:42.211 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:42.299 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:42.299+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:42.395 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:42.395+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:42.411 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:42.411 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:42.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:42.515 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:42.515+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:42.597 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:42.590+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:42.623 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:42.624 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:42.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:42.698 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:42.694+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:42.798 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:42.795+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:42.834 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:42.834 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:42.855 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:42.902 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:42.898+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:43.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:42.998+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:43.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:43.043 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:43.043 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:43.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:43.103 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:43.098+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:43.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.182+0000 7fdedcf53640 1 Processor -- start 2024-04-25T15:52:43.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.182+0000 7fdedcf53640 1 -- start start 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fdedcf53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded806c640 0x7fded8154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fdedcf53640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fded815e4a0 0x7fded80c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fdedcf53640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fded80749d0 con 0x7fded815e8b0 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fdedcf53640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fded806e510 con 0x7fded815e4a0 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fdedcf53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fded815dca0 con 0x7fded806c640 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fded815e4a0 0x7fded80c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fded815e4a0 0x7fded80c7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49056/0 (socket says 172.21.15.67:49056) 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 -- 172.21.15.67:0/2376709708 learned_addr learned my addr 172.21.15.67:0/2376709708 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded659c640 1 --2- 172.21.15.67:0/2376709708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded806c640 0x7fded8154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 -- 172.21.15.67:0/2376709708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded806c640 msgr2=0x7fded8154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2376709708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded806c640 0x7fded8154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 -- 172.21.15.67:0/2376709708 >> v1:172.21.15.67:6789/0 conn(0x7fded815e8b0 legacy=0x7fded815a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 -- 172.21.15.67:0/2376709708 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded815df80 con 0x7fded815e4a0 2024-04-25T15:52:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2376709708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded815e4a0 0x7fded80c7fc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009870 tx=0x7fdecc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b44aef3876397b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded559a640 1 -- 172.21.15.67:0/2376709708 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc0b1070 con 0x7fded815e4a0 2024-04-25T15:52:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded559a640 1 -- 172.21.15.67:0/2376709708 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdecc0ac600 con 0x7fded815e4a0 2024-04-25T15:52:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.186+0000 7fded559a640 1 -- 172.21.15.67:0/2376709708 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc0ac9d0 con 0x7fded815e4a0 2024-04-25T15:52:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- 172.21.15.67:0/2376709708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded815e4a0 msgr2=0x7fded80c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2376709708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded815e4a0 0x7fded80c7fc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009870 tx=0x7fdecc01c4d0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- 172.21.15.67:0/2376709708 shutdown_connections 2024-04-25T15:52:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2376709708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded815e4a0 0x7fded80c7fc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2376709708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded806c640 0x7fded8154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- 172.21.15.67:0/2376709708 >> 172.21.15.67:0/2376709708 conn(0x7fded8070db0 msgr2=0x7fded80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- 172.21.15.67:0/2376709708 shutdown_connections 2024-04-25T15:52:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- 172.21.15.67:0/2376709708 wait complete. 2024-04-25T15:52:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 Processor -- start 2024-04-25T15:52:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- start start 2024-04-25T15:52:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded806c640 0x7fded81f9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded815e4a0 0x7fded80cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fded815e8b0 0x7fded814d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fded80cad60 con 0x7fded806c640 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fded8072bc0 con 0x7fded815e8b0 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.190+0000 7fdedcf53640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fded81fa270 con 0x7fded815e4a0 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded806c640 0x7fded81f9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded806c640 0x7fded81f9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49068/0 (socket says 172.21.15.67:49068) 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 -- 172.21.15.67:0/2981063982 learned_addr learned my addr 172.21.15.67:0/2981063982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded815e4a0 0x7fded80cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded6d9d640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fded815e8b0 0x7fded814d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded815e4a0 msgr2=0x7fded80cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fded815e4a0 0x7fded80cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fded815e8b0 msgr2=0x7fded814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fded815e8b0 0x7fded814d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded81f9da0 con 0x7fded806c640 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded806c640 0x7fded81f9860 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdec800d5f0 tx=0x7fdec800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec80125f0 con 0x7fded806c640 2024-04-25T15:52:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec8019070 con 0x7fded806c640 2024-04-25T15:52:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec80129b0 con 0x7fded806c640 2024-04-25T15:52:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded80ce290 con 0x7fded806c640 2024-04-25T15:52:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded80c7bb0 con 0x7fded806c640 2024-04-25T15:52:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fded806a480 con 0x7fded806c640 2024-04-25T15:52:43.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdec8012b50 con 0x7fded806c640 2024-04-25T15:52:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdec6ffd640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdeb806f400 0x7fdeb8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5857+0+0 (secure 0 0 0) 0x7fdec8090160 con 0x7fded806c640 2024-04-25T15:52:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v59) ==== 54+0+16 (secure 0 0 0) 0x7fdec80905f0 con 0x7fded806c640 2024-04-25T15:52:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.194+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdeb806f400 0x7fdeb8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.198+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fded814b3c0 0x7fded80cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.198+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fded81586b0 con 0x7fded814b3c0 2024-04-25T15:52:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.198+0000 7fded6d9d640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fded814b3c0 0x7fded80cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.198+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdeb806f400 0x7fdeb8071910 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdecc0ae000 tx=0x7fdecc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.198+0000 7fded6d9d640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fded814b3c0 0x7fded80cb3b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.198+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded81586b0 con 0x7fded814b3c0 2024-04-25T15:52:43.203 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:43.198+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:43.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.214+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded80c9650 con 0x7fded814b3c0 2024-04-25T15:52:43.219 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:43.219 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:43.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:52:43.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.214+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded80c9650 con 0x7fded814b3c0 2024-04-25T15:52:43.222 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:43.222 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:43.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.214+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fded813e560 0x7fded80c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.218+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fded813e970 con 0x7fded813e560 2024-04-25T15:52:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.218+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fded813e560 0x7fded80c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.218+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fded813e560 0x7fded80c5db0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.218+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded813e970 con 0x7fded813e560 2024-04-25T15:52:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.234+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded814dd10 con 0x7fded813e560 2024-04-25T15:52:43.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.234+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded814dd10 con 0x7fded813e560 2024-04-25T15:52:43.240 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:43.240 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.234+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fded80c6800 0x7fded8063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.234+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fded80c6c10 con 0x7fded80c6800 2024-04-25T15:52:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.234+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fded80c6800 0x7fded8063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.234+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fded80c6800 0x7fded8063090 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.238+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded80c6c10 con 0x7fded80c6800 2024-04-25T15:52:43.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.254+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded80635d0 con 0x7fded80c6800 2024-04-25T15:52:43.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.254+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded80635d0 con 0x7fded80c6800 2024-04-25T15:52:43.260 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:43.261 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:43.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.254+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fded8152300 0x7fded8152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.254+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fded80635d0 con 0x7fded8152300 2024-04-25T15:52:43.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.254+0000 7fded6d9d640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fded8152300 0x7fded8152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.254+0000 7fded6d9d640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fded8152300 0x7fded8152710 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.258+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded80635d0 con 0x7fded8152300 2024-04-25T15:52:43.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.274+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded80635d0 con 0x7fded8152300 2024-04-25T15:52:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.278+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded80635d0 con 0x7fded8152300 2024-04-25T15:52:43.283 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.278+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fded802c9e0 0x7fded802cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.278+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fded802d330 con 0x7fded802c9e0 2024-04-25T15:52:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.278+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fded802c9e0 0x7fded802cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.278+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fded802c9e0 0x7fded802cdf0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.278+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded802d330 con 0x7fded802c9e0 2024-04-25T15:52:43.305 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:43.298+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:43.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.302+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded802d330 con 0x7fded802c9e0 2024-04-25T15:52:43.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.306+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded802d330 con 0x7fded802c9e0 2024-04-25T15:52:43.310 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.306+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fded802d9b0 0x7fded820ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.306+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fded802ddc0 con 0x7fded802d9b0 2024-04-25T15:52:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.306+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fded802d9b0 0x7fded820ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.306+0000 7fded5d9b640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fded802d9b0 0x7fded820ca40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.306+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded802ddc0 con 0x7fded802d9b0 2024-04-25T15:52:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.326+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded802ddc0 con 0x7fded802d9b0 2024-04-25T15:52:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.330+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded802ddc0 con 0x7fded802d9b0 2024-04-25T15:52:43.335 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.330+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fded820d490 0x7fded820f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.330+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fded8212700 con 0x7fded820d490 2024-04-25T15:52:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.330+0000 7fded6d9d640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fded820d490 0x7fded820f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.330+0000 7fded6d9d640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fded820d490 0x7fded820f8b0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.330+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded8212700 con 0x7fded820d490 2024-04-25T15:52:43.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.350+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded82100c0 con 0x7fded820d490 2024-04-25T15:52:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.378+0000 7fbe4468c640 1 Processor -- start 2024-04-25T15:52:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.378+0000 7fbe4468c640 1 -- start start 2024-04-25T15:52:43.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe4468c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe3c073bf0 0x7fbe3c06d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe4468c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe4468c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe3c0c77a0 con 0x7fbe3c0caf60 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe4468c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbe3c0cab60 con 0x7fbe3c073bf0 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe4468c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3c159620 con 0x7fbe3c06dc90 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41c27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe3c073bf0 0x7fbe3c06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe42428640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbe3c0caf60 0x7fbe3c15d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44668/0 (socket says 172.21.15.67:44668) 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe42428640 1 -- 172.21.15.67:0/3822170350 learned_addr learned my addr 172.21.15.67:0/3822170350 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe42c29640 1 --2- 172.21.15.67:0/3822170350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41c27640 1 -- 172.21.15.67:0/3822170350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 msgr2=0x7fbe3c163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41c27640 1 --2- 172.21.15.67:0/3822170350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41c27640 1 -- 172.21.15.67:0/3822170350 >> v1:172.21.15.67:6789/0 conn(0x7fbe3c0caf60 legacy=0x7fbe3c15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41c27640 1 -- 172.21.15.67:0/3822170350 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c0c7a60 con 0x7fbe3c073bf0 2024-04-25T15:52:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41426640 1 -- 172.21.15.67:0/3822170350 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256030940 0 0) 0x7fbe3c0c77a0 con 0x7fbe3c0caf60 2024-04-25T15:52:43.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41c27640 1 --2- 172.21.15.67:0/3822170350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c073bf0 0x7fbe3c06d750 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c0097c0 tx=0x7fbe2c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5edc6e1ab8110cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41426640 1 -- 172.21.15.67:0/3822170350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c0b0070 con 0x7fbe3c073bf0 2024-04-25T15:52:43.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41426640 1 -- 172.21.15.67:0/3822170350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe2c0ab600 con 0x7fbe3c073bf0 2024-04-25T15:52:43.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.382+0000 7fbe41426640 1 -- 172.21.15.67:0/3822170350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c0ab9d0 con 0x7fbe3c073bf0 2024-04-25T15:52:43.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 -- 172.21.15.67:0/3822170350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c073bf0 msgr2=0x7fbe3c06d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 --2- 172.21.15.67:0/3822170350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c073bf0 0x7fbe3c06d750 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c0097c0 tx=0x7fbe2c01c4a0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 -- 172.21.15.67:0/3822170350 shutdown_connections 2024-04-25T15:52:43.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 --2- 172.21.15.67:0/3822170350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 --2- 172.21.15.67:0/3822170350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c073bf0 0x7fbe3c06d750 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 -- 172.21.15.67:0/3822170350 >> 172.21.15.67:0/3822170350 conn(0x7fbe3c070db0 msgr2=0x7fbe3c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:43.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 -- 172.21.15.67:0/3822170350 shutdown_connections 2024-04-25T15:52:43.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 -- 172.21.15.67:0/3822170350 wait complete. 2024-04-25T15:52:43.391 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:43.391 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 Processor -- start 2024-04-25T15:52:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.386+0000 7fbe4468c640 1 -- start start 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe3c073bf0 0x7fbe3c0ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c0caf60 0x7fbe3c14d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c15a570 con 0x7fbe3c0caf60 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c15a3f0 con 0x7fbe3c073bf0 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3c166c20 con 0x7fbe3c06dc90 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c0caf60 0x7fbe3c14d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c0caf60 0x7fbe3c14d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49094/0 (socket says 172.21.15.67:49094) 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 -- 172.21.15.67:0/2662879081 learned_addr learned my addr 172.21.15.67:0/2662879081 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42428640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe41c27640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe3c073bf0 0x7fbe3c0ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 -- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 msgr2=0x7fbe3c155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 -- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe3c073bf0 msgr2=0x7fbe3c0ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe3c073bf0 0x7fbe3c0ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 -- 172.21.15.67:0/2662879081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c155790 con 0x7fbe3c0caf60 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe42c29640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c0caf60 0x7fbe3c14d670 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbe34010f20 tx=0x7fbe3400e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe32ffd640 1 -- 172.21.15.67:0/2662879081 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34012840 con 0x7fbe3c0caf60 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe32ffd640 1 -- 172.21.15.67:0/2662879081 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe340129e0 con 0x7fbe3c0caf60 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3c14ff20 con 0x7fbe3c0caf60 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe32ffd640 1 -- 172.21.15.67:0/2662879081 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34012dd0 con 0x7fbe3c0caf60 2024-04-25T15:52:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3c1510b0 con 0x7fbe3c0caf60 2024-04-25T15:52:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.390+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe3c06a480 con 0x7fbe3c0caf60 2024-04-25T15:52:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.394+0000 7fbe32ffd640 1 -- 172.21.15.67:0/2662879081 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbe34021020 con 0x7fbe3c0caf60 2024-04-25T15:52:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.394+0000 7fbe32ffd640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbe2006f400 0x7fbe20071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.394+0000 7fbe32ffd640 1 -- 172.21.15.67:0/2662879081 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5857+0+0 (secure 0 0 0) 0x7fbe34092890 con 0x7fbe3c0caf60 2024-04-25T15:52:43.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.394+0000 7fbe42428640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbe2006f400 0x7fbe20071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.394+0000 7fbe42428640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbe2006f400 0x7fbe20071910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000f70 tx=0x7fbe38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.398+0000 7fbe32ffd640 1 -- 172.21.15.67:0/2662879081 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbe34028070 con 0x7fbe3c0caf60 2024-04-25T15:52:43.405 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:43.403+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:43.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:43.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.438+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded82100c0 con 0x7fded820d490 2024-04-25T15:52:43.444 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:43.444 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:43.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.438+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fded82107e0 0x7fded8210bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:43.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.438+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fded8215b50 con 0x7fded82107e0 2024-04-25T15:52:43.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.438+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fded82107e0 0x7fded8210bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:43.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.438+0000 7fded659c640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fded82107e0 0x7fded8210bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:43.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.442+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fded8215b50 con 0x7fded82107e0 2024-04-25T15:52:43.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.454+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fded82149d0 con 0x7fded82107e0 2024-04-25T15:52:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdec6ffd640 1 -- 172.21.15.67:0/2981063982 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fded82149d0 con 0x7fded82107e0 2024-04-25T15:52:43.462 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fded814b3c0 msgr2=0x7fded80cb3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fded814b3c0 0x7fded80cb3b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fded813e560 msgr2=0x7fded80c5db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fded813e560 0x7fded80c5db0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fded80c6800 msgr2=0x7fded8063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fded80c6800 0x7fded8063090 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fded8152300 msgr2=0x7fded8152710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fded8152300 0x7fded8152710 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fded802c9e0 msgr2=0x7fded802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fded802c9e0 0x7fded802cdf0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fded6d9d640 1 -- 172.21.15.67:0/2981063982 reap_dead start 2024-04-25T15:52:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fded802d9b0 msgr2=0x7fded820ca40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fded802d9b0 0x7fded820ca40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fded6d9d640 1 -- 172.21.15.67:0/2981063982 reap_dead start 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fded6d9d640 1 -- 172.21.15.67:0/2981063982 reap_dead start 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fded820d490 msgr2=0x7fded820f8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fded820d490 0x7fded820f8b0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fded82107e0 msgr2=0x7fded8210bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fded82107e0 0x7fded8210bf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdeb806f400 msgr2=0x7fdeb8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdeb806f400 0x7fdeb8071910 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdecc0ae000 tx=0x7fdecc008040 comp rx=0 tx=0).stop 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded806c640 msgr2=0x7fded81f9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded806c640 0x7fded81f9860 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdec800d5f0 tx=0x7fdec800dab0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 shutdown_connections 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fded820d490 0x7fded820f8b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fded82107e0 0x7fded8210bf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdeb806f400 0x7fdeb8071910 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 --2- 172.21.15.67:0/2981063982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded806c640 0x7fded81f9860 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 >> 172.21.15.67:0/2981063982 conn(0x7fded8070db0 msgr2=0x7fded806d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 shutdown_connections 2024-04-25T15:52:43.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.458+0000 7fdedcf53640 1 -- 172.21.15.67:0/2981063982 wait complete. 2024-04-25T15:52:43.529 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:43.526+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:43.607 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:43.602+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:43.610 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:43.611 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:43.611 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:43.611 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:43.611 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:43.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:43.658+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fbe3c0c8a40 con 0x7fbe3c0caf60 2024-04-25T15:52:43.708 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:43.707+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.682596886, 2024-04-25T15:52:43.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:52:43.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:52:43.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:52:43.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:25 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.21899607099999, 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01409815, 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.791 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 25 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562065+0000", 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562066+0000", 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0699999999999996e-07 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562072+0000", 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.703e-06 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562156+0000", 2024-04-25T15:52:43.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3486000000000007e-05 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.562165+0000", 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9500000000000007e-06 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.568966+0000", 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006801372 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569110+0000", 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014385799999999999 2024-04-25T15:52:43.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.569344+0000", 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023376500000000001 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.573357+0000", 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040127870000000003 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.575955+0000", 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025982969999999998 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576077+0000", 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121966 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.576163+0000", 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6058999999999997e-05 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:46 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:43.796 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.09173969599999, 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013995720999999999, 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 46 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689321+0000", 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689322+0000", 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8999999999999996e-07 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689329+0000", 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1899999999999998e-06 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689831+0000", 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050143900000000001 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689839+0000", 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4069999999999993e-06 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699042+0000", 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092026810000000007 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699214+0000", 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017275100000000001 2024-04-25T15:52:43.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.699442+0000", 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022718200000000001 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.702745+0000", 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003303266 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703161+0000", 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000416253 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703250+0000", 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8806000000000003e-05 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703317+0000", 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7056000000000003e-05 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.801 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.091728429, 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014300548, 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689333+0000", 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689491+0000", 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015768600000000001 2024-04-25T15:52:43.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689857+0000", 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000366313 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689865+0000", 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1470000000000001e-06 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700215+0000", 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010350177 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700471+0000", 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521499999999999 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.700879+0000", 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040780599999999998 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703339+0000", 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024602140000000001 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703443+0000", 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104393 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703524+0000", 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0745e-05 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703633+0000", 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109226 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.806 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.091564151, 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019412740000000001, 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 48 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689498+0000", 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5999999999999991e-07 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689505+0000", 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1339999999999998e-06 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689885+0000", 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000380553 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689894+0000", 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0660000000000004e-06 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703735+0000", 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013841509 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703880+0000", 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014450899999999999 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704121+0000", 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024184799999999999 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708286+0000", 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041643369999999997 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708530+0000", 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024477400000000001 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708710+0000", 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179715 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708910+0000", 2024-04-25T15:52:43.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199335 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.09155214099999, 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019597107999999999, 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.812 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 49 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4700000000000001e-07 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689515+0000", 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3770000000000004e-06 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689911+0000", 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039616199999999998 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689920+0000", 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2939999999999994e-06 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704840+0000", 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014919412999999999 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704990+0000", 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150388 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.705238+0000", 2024-04-25T15:52:43.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000248392 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708485+0000", 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032464740000000001 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708936+0000", 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045072599999999998 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709037+0000", 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101482 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709106+0000", 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8752999999999997e-05 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.091542584, 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:52:43.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:52:43.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.09139396200001, 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:52:43.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:52:43.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:52:43.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 144.09138288299999, 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.828 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:52:43.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.78454440300001, 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019231747, 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.833 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 69 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996517+0000", 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996518+0000", 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6840000000000001e-06 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996645+0000", 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012677899999999999 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996696+0000", 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0912999999999999e-05 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996718+0000", 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1931999999999999e-05 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000632+0000", 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003914509 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000756+0000", 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012338099999999999 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.000879+0000", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012301599999999999 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009110+0000", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082309819999999995 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015495+0000", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063848969999999996 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015700+0000", 2024-04-25T15:52:43.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020508399999999999 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015748+0000", 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8569999999999997e-05 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.78429236100001, 2024-04-25T15:52:43.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019263037, 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 70 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996769+0000", 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996770+0000", 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7060000000000001e-06 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996786+0000", 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5497000000000001e-05 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996990+0000", 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000203938 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997014+0000", 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4617000000000002e-05 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002337+0000", 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053230450000000002 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002417+0000", 2024-04-25T15:52:43.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9548999999999993e-05 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.002579+0000", 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162385 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009114+0000", 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00653442 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015848+0000", 2024-04-25T15:52:43.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067339909999999999 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.015951+0000", 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102796 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016032+0000", 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1093000000000004e-05 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.843 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.78426646099999, 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019380720000000001, 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 71 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996796+0000", 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.063e-06 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996810+0000", 2024-04-25T15:52:43.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4100000000000001e-05 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997061+0000", 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000251797 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997076+0000", 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4970000000000001e-05 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003008+0000", 2024-04-25T15:52:43.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005931487 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003148+0000", 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140097 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003371+0000", 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022273400000000001 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009117+0000", 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057466929999999998 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016106+0000", 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069880919999999996 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016144+0000", 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8686000000000002e-05 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016175+0000", 2024-04-25T15:52:43.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1000999999999997e-05 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.78424379500001, 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019495743999999999, 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.849 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996818+0000", 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.046e-06 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996944+0000", 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012572199999999999 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997139+0000", 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019529899999999999 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997151+0000", 2024-04-25T15:52:43.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2116000000000001e-05 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004407+0000", 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072560519999999998 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004474+0000", 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6810000000000006e-05 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004576+0000", 2024-04-25T15:52:43.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101431 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009121+0000", 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045451099999999998 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016239+0000", 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071178359999999998 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016280+0000", 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0914000000000002e-05 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016313+0000", 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3408000000000003e-05 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.783864529, 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019262988000000002, 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.854 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 73 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997196+0000", 2024-04-25T15:52:43.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997198+0000", 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.347e-06 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997208+0000", 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0427e-05 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997419+0000", 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021052399999999999 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997433+0000", 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4687e-05 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005473+0000", 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080393610000000001 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005619+0000", 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146157 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.005848+0000", 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229001 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009124+0000", 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003276288 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016366+0000", 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072413759999999999 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016408+0000", 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2119000000000001e-05 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016459+0000", 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1700999999999998e-05 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.783847055, 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019348008999999999, 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.859 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 74 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997214+0000", 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997215+0000", 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997220+0000", 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7200000000000003e-06 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997480+0000", 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026014000000000001 2024-04-25T15:52:43.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997493+0000", 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2306999999999999e-05 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006512+0000", 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090195669999999992 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006591+0000", 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9058000000000001e-05 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.006724+0000", 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013295000000000001 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009130+0000", 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024053659999999999 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073654790000000003 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016533+0000", 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7432999999999999e-05 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016562+0000", 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9379000000000001e-05 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.78383724899999, 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020191183000000001, 2024-04-25T15:52:43.864 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997376+0000", 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015132899999999999 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997528+0000", 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152275 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997538+0000", 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0137e-05 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007159+0000", 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096209090000000004 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007248+0000", 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9078000000000003e-05 2024-04-25T15:52:43.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007394+0000", 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014618699999999999 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009133+0000", 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001739008 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017311+0000", 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081775159999999993 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017360+0000", 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9202000000000001e-05 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017415+0000", 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4932000000000002e-05 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 143.78367984400001, 2024-04-25T15:52:43.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:52:43.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:35 3.2f e59/51 3:f540378d:::10000000002.0000001f:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.521891+0000", 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.25917025199999999, 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014669592, 2024-04-25T15:52:43.874 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.521891+0000", 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.521891+0000", 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.521894+0000", 2024-04-25T15:52:43.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.123e-06 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.525258+0000", 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003363932 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.525288+0000", 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0012999999999999e-05 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.525310+0000", 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2663e-05 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.525466+0000", 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015551200000000001 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.525514+0000", 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8335999999999999e-05 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.536452+0000", 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010937854 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.536560+0000", 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010815900000000001 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:36 3.1b 3:d88a10eb:::10000000002.00000020:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.559831+0000", 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.221230276, 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019257494, 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.878 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 36 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.559831+0000", 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.559831+0000", 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.559834+0000", 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.27e-06 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.564246+0000", 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044122600000000003 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.564309+0000", 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2835000000000004e-05 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.564335+0000", 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5899999999999999e-05 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.564478+0000", 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014315899999999999 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.564727+0000", 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000249015 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.564957+0000", 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229731 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.573382+0000", 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008424895 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.578904+0000", 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055220670000000003 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.579002+0000", 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8009000000000003e-05 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.579088+0000", 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6353000000000005e-05 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:37 3.3b 3:ddac2f4e:::10000000002.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.606740+0000", 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.17432131300000001, 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020101346999999999, 2024-04-25T15:52:43.883 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 37 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.606740+0000", 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.606740+0000", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.606742+0000", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4590000000000001e-06 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611280+0000", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00453792 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611315+0000", 2024-04-25T15:52:43.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5033000000000002e-05 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611329+0000", 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3481e-05 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611482+0000", 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000153857 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611751+0000", 2024-04-25T15:52:43.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026831799999999997 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611967+0000", 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021583799999999999 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.622715+0000", 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010748186 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.626645+0000", 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039301199999999996 2024-04-25T15:52:43.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.626751+0000", 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010613200000000001 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.626841+0000", 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0002999999999999e-05 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:43.888 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:43.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:43.890 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:43.887+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:43.990 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:43.987+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:44.066 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:44.066 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:44.066 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:44.066 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:44.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:44.129 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:44.126+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:44.192 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:44.186+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:44.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.210+0000 7fbe32ffd640 1 -- 172.21.15.67:0/2662879081 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v60) ==== 97+0+0 (secure 0 0 0) 0x7fbe3405f7c0 con 0x7fbe3c0caf60 2024-04-25T15:52:44.215 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbe2006f400 msgr2=0x7fbe20071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbe2006f400 0x7fbe20071910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe38000f70 tx=0x7fbe38009040 comp rx=0 tx=0).stop 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c0caf60 msgr2=0x7fbe3c14d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c0caf60 0x7fbe3c14d670 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbe34010f20 tx=0x7fbe3400e760 comp rx=0 tx=0).stop 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 shutdown_connections 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbe2006f400 0x7fbe20071910 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe3c0caf60 0x7fbe3c14d670 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe3c073bf0 0x7fbe3c0ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 --2- 172.21.15.67:0/2662879081 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe3c06dc90 0x7fbe3c155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 >> 172.21.15.67:0/2662879081 conn(0x7fbe3c070db0 msgr2=0x7fbe3c157f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:44.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 shutdown_connections 2024-04-25T15:52:44.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:44.214+0000 7fbe4468c640 1 -- 172.21.15.67:0/2662879081 wait complete. 2024-04-25T15:52:44.251 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:44.252 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:44.252 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:44.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:44.313 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:44.310+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:44.394 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:44.391+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:44.467 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 144.46438786600001, 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:44.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:44.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:70 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'48, mlcod=15'42)", 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 144.462758648, 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012230665999999999, 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.472 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004237+0000", 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004239+0000", 2024-04-25T15:52:44.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004250+0000", 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1518999999999999e-05 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004274+0000", 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3672e-05 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004292+0000", 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7458000000000001e-05 2024-04-25T15:52:44.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005373+0000", 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0010819359999999999 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005450+0000", 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6200999999999998e-05 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016441+0000", 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991445000000001 2024-04-25T15:52:44.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016468+0000", 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6801999999999999e-05 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:71 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'49, mlcod=15'42)", 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 144.46196820899999, 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011466877, 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.476 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005028+0000", 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005029+0000", 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999999999999999e-06 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005040+0000", 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1199000000000001e-05 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005063+0000", 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2342000000000001e-05 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.005081+0000", 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7711999999999999e-05 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006374+0000", 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001293492 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.006429+0000", 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5315999999999997e-05 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016469+0000", 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010039984 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016495+0000", 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5232000000000002e-05 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.25550022300001, 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:44.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:44.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.255452838, 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.25544075600001, 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:44.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.494 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.25542058400001, 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:44.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:44.498 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 128.255182246, 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:18 2.13 2:ca4dd7f5:::611.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 122.217414109, 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011222823999999999, 2024-04-25T15:52:44.502 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249582+0000", 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249583+0000", 2024-04-25T15:52:44.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1799999999999999e-06 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249591+0000", 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0669999999999992e-06 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249819+0000", 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022795000000000001 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249865+0000", 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5561e-05 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254040+0000", 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041751720000000004 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254349+0000", 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030965399999999997 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.254663+0000", 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031403600000000001 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:44.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258978+0000", 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043149959999999998 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260612+0000", 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00163383 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260715+0000", 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010311599999999999 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.260805+0000", 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9262e-05 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:1 2.1f 2:fb684bcd:::204.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.54137991, 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012576726, 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.507 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925616+0000", 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000002e-07 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925629+0000", 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3002e-05 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925764+0000", 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013477399999999999 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925779+0000", 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5228000000000001e-05 2024-04-25T15:52:44.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925847+0000", 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7980999999999998e-05 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925951+0000", 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010333299999999999 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926085+0000", 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013462200000000001 2024-04-25T15:52:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929258+0000", 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003172607 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938074+0000", 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088162910000000004 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938129+0000", 2024-04-25T15:52:44.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5234e-05 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938193+0000", 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3367000000000005e-05 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:31 2.1f 2:ff4f7703:::204.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 100.541365173, 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012700365, 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.512 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925631+0000", 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5499999999999999e-07 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925633+0000", 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.796e-06 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925809+0000", 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017651200000000001 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.925824+0000", 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5071e-05 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926689+0000", 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00086453799999999998 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.926825+0000", 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013552799999999999 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.938331+0000", 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011506565 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:32 3.6 e59/40 3:628433a0:::10000000002.0000001c:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.366429+0000", 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.1005668799999999, 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012114961, 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.366429+0000", 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.366429+0000", 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.366429+0000", 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.53e-07 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.368256+0000", 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001826584 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.368280+0000", 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3944e-05 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.368297+0000", 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7586999999999999e-05 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.368382+0000", 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4186999999999999e-05 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.368404+0000", 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2260999999999999e-05 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.378495+0000", 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010090721 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.378544+0000", 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9323999999999997e-05 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:36 3.1b e59/54 3:d88a10eb:::10000000002.00000020:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.90145029200000004, 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013408461999999999, 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565548+0000", 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5620000000000002e-06 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572005+0000", 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064569370000000003 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572034+0000", 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8586999999999999e-05 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572056+0000", 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2481e-05 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572135+0000", 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9264999999999994e-05 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572159+0000", 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3275e-05 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.578912+0000", 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067533030000000004 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.578954+0000", 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2052000000000003e-05 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.523 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:39 3.24 e59/54 3:2742a876:::10000000002.00000023:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.709273+0000", 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.75772337199999995, 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012640779, 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.709273+0000", 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.709273+0000", 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.709274+0000", 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5269999999999999e-06 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.711024+0000", 2024-04-25T15:52:44.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017498419999999999 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.711045+0000", 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1580999999999999e-05 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.711058+0000", 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2473999999999999e-05 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.711138+0000", 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0331e-05 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.711165+0000", 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6312000000000002e-05 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.721847+0000", 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010682763 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.721913+0000", 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5949000000000002e-05 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.65123052199999998, 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:52:44.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.36043237900000002, 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:52:44.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:49 3.1 e59/54 3:827302d4:::10000000002.0000002d:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.195403+0000", 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.27159269899999999, 2024-04-25T15:52:44.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012556329, 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.195403+0000", 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.195403+0000", 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.195405+0000", 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.058e-06 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.196814+0000", 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001408315 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.196846+0000", 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2069000000000001e-05 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.196867+0000", 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1749000000000001e-05 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.196944+0000", 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6265000000000002e-05 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.196975+0000", 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1136999999999997e-05 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.207883+0000", 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010908334 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.207960+0000", 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6402000000000005e-05 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:51 3.17 e60/40 3:eaf1bb0b:::10000000002.0000002f:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.295889+0000", 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.171107377, 2024-04-25T15:52:44.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01203037, 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.295889+0000", 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.295889+0000", 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.295889+0000", 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4700000000000002e-07 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.299706+0000", 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0038168719999999998 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.299723+0000", 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7521999999999999e-05 2024-04-25T15:52:44.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.299740+0000", 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6376999999999999e-05 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.299814+0000", 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4690000000000005e-05 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.299835+0000", 2024-04-25T15:52:44.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0115999999999998e-05 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.307851+0000", 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080167060000000002 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.307919+0000", 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7639999999999996e-05 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.544 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:52 3.12 e60/40 3:499536aa:::10000000002.00000030:head v 60'5, mlcod=59'3)", 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.130421333, 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01319507, 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4200000000000002e-07 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.342934+0000", 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006359211 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.342953+0000", 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9256999999999999e-05 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.342968+0000", 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.343057+0000", 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9151000000000001e-05 2024-04-25T15:52:44.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.343087+0000", 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9241999999999999e-05 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.349681+0000", 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065946440000000002 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.349770+0000", 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8323000000000001e-05 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.548 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:53 3.20 e60/18 3:068d0381:::10000000002.00000031:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.082974895000000007, 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014486163, 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389720+0000", 2024-04-25T15:52:44.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056985300000000003 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389742+0000", 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1838999999999999e-05 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389758+0000", 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6058000000000001e-05 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389844+0000", 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6293999999999997e-05 2024-04-25T15:52:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389866+0000", 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2033000000000001e-05 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398429+0000", 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085631660000000005 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398507+0000", 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7869999999999998e-05 2024-04-25T15:52:44.552 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.553 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:44.553 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:44.553 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:44.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:44.555 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:44.550+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:44.655 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:44.655+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:44.745 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:44.745 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:44.746 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:44.756 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:44.750+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:44.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:44.857 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:44.855+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:44.958 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:44.954+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:44.986 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:44.986 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:45.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:45.058 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:45.055+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:45.159 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:45.154+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:45.227 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:14 2.15 2:a93a17c2:::604.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 129.01547460099999, 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013243108, 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211585+0000", 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0946e-05 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211617+0000", 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211761+0000", 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014392699999999999 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211793+0000", 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1568e-05 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212109+0000", 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031626100000000003 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212422+0000", 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031341099999999998 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213414+0000", 2024-04-25T15:52:45.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099192799999999999 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223830+0000", 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010415654999999999 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224524+0000", 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00069455899999999998 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224662+0000", 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013742600000000001 2024-04-25T15:52:45.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224828+0000", 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016641799999999999 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:15 2.15 2:a93a17c2:::604.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 129.015433296, 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013508035, 2024-04-25T15:52:45.233 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211626+0000", 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211628+0000", 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3400000000000001e-06 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211641+0000", 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2866e-05 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027786899999999998 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211973+0000", 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4401000000000003e-05 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214895+0000", 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029218109999999999 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215130+0000", 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00023500499999999999 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215556+0000", 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00042636900000000001 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224481+0000", 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089248869999999994 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224861+0000", 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038038699999999999 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225017+0000", 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155296 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225135+0000", 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117804 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:30 2.15 2:aa448500:::500.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 129.01541591700001, 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013948439999999999, 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211644+0000", 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211645+0000", 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1699999999999997e-07 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211680+0000", 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5374999999999999e-05 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212061+0000", 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038129700000000001 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212135+0000", 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4103000000000005e-05 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218486+0000", 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063501709999999999 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218861+0000", 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037518799999999999 2024-04-25T15:52:45.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219238+0000", 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037732500000000002 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225340+0000", 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061016430000000003 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225414+0000", 2024-04-25T15:52:45.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.436e-05 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225489+0000", 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4882999999999994e-05 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225592+0000", 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103378 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:31 2.d 2:b50e409b:::mds_snaptable:head [writefull 0~70] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 129.015374192, 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013711255, 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211686+0000", 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211687+0000", 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1060000000000001e-06 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211694+0000", 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6329999999999996e-06 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212201+0000", 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050685200000000002 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212239+0000", 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7537999999999999e-05 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216262+0000", 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004023726 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216562+0000", 2024-04-25T15:52:45.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029964199999999997 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216899+0000", 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033712999999999999 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.223552+0000", 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066527330000000001 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225162+0000", 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016101869999999999 2024-04-25T15:52:45.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225266+0000", 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010402400000000001 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225397+0000", 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130684 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:6 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:45.248 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 129.011100865, 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198140000000006, 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215959+0000", 2024-04-25T15:52:45.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215961+0000", 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5469999999999999e-06 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215983+0000", 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2014999999999998e-05 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216017+0000", 2024-04-25T15:52:45.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3114999999999997e-05 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216043+0000", 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6755999999999999e-05 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216409+0000", 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036599799999999997 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216466+0000", 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6947000000000002e-05 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224337+0000", 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078708230000000007 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224479+0000", 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014161299999999999 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:7 2.10 e20/17 2:0d82a743:::600.00000000:head v 20'2, mlcod=0'0)", 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 129.010950659, 2024-04-25T15:52:45.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085198700000000006, 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216109+0000", 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216110+0000", 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3570000000000001e-06 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216125+0000", 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4772e-05 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216151+0000", 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5496000000000001e-05 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216172+0000", 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1095e-05 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217019+0000", 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00084760800000000002 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217071+0000", 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1894000000000002e-05 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224489+0000", 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074175710000000004 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224629+0000", 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014007700000000001 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:8 2.10 e20/17 2:09401a81:::601.00000000:head v 20'3, mlcod=0'0)", 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 129.009265763, 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069206529999999997, 2024-04-25T15:52:45.256 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217794+0000", 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217796+0000", 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9069999999999999e-06 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217816+0000", 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0296e-05 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217847+0000", 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0877999999999998e-05 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.217872+0000", 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4749e-05 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218099+0000", 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022737199999999999 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.218158+0000", 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8900000000000002e-05 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224635+0000", 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064771480000000003 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224715+0000", 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9402999999999999e-05 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:19 2.9 2:92e99a1d:::612.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 122.979421555, 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010474059000000001, 2024-04-25T15:52:45.260 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247638+0000", 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247641+0000", 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.129e-06 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247667+0000", 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6089e-05 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247827+0000", 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000159977 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247854+0000", 2024-04-25T15:52:45.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6971999999999999e-05 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248122+0000", 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00026745499999999999 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248469+0000", 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034735899999999999 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248784+0000", 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00031528900000000001 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.251768+0000", 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002983383 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.257863+0000", 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006095339 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258009+0000", 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000145921 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258112+0000", 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103146 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:20 2.9 2:92e99a1d:::612.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 122.97938278399999, 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010923117, 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.265 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247677+0000", 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247678+0000", 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247688+0000", 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3659999999999998e-06 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247937+0000", 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024977700000000002 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247958+0000", 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0242e-05 2024-04-25T15:52:45.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249612+0000", 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016540089999999999 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.249860+0000", 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024840600000000001 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.250352+0000", 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049215499999999998 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255543+0000", 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051905060000000001 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258273+0000", 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027298460000000002 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258467+0000", 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019409900000000001 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.258600+0000", 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013348099999999999 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:23 2.15 2:aa0f8bf4:::101.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 122.979367731, 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0082242619999999995, 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247692+0000", 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247693+0000", 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1230000000000001e-06 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247728+0000", 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5049000000000003e-05 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248028+0000", 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029943400000000001 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248050+0000", 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2722e-05 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248203+0000", 2024-04-25T15:52:45.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015264 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248563+0000", 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036047500000000002 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.248940+0000", 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037679400000000001 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.252184+0000", 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003243821 2024-04-25T15:52:45.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255610+0000", 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0034255370000000002 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255793+0000", 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018366899999999999 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.255916+0000", 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122998 2024-04-25T15:52:45.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 122.979325796, 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:45.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.4.16:10 2.9 2:954a9a2b:::62b.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 101.299842881, 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010008022, 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927217+0000", 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927218+0000", 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927222+0000", 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6519999999999997e-06 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927339+0000", 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116705 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.927350+0000", 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0995000000000001e-05 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928080+0000", 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00072958300000000001 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928207+0000", 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012715 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.928433+0000", 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022608599999999999 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.929140+0000", 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00070747400000000004 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937109+0000", 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079685439999999993 2024-04-25T15:52:45.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937169+0000", 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9806999999999998e-05 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:51:03.937225+0000", 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6313000000000001e-05 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:32 3.6 3:628433a0:::10000000002.0000001c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:45.284 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.364111+0000", 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.8629489850000001, 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014786298, 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 32 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.364111+0000", 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.364111+0000", 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.364114+0000", 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4460000000000001e-06 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.365868+0000", 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017539789999999999 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.365916+0000", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7970999999999999e-05 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.365935+0000", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8814e-05 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.366029+0000", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3740999999999996e-05 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.366151+0000", 2024-04-25T15:52:45.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121787 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.366258+0000", 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010782599999999999 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.376722+0000", 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010463301 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.378794+0000", 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020721429999999998 2024-04-25T15:52:45.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.378836+0000", 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2063e-05 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.378897+0000", 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1227000000000002e-05 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.289 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:33 3.31 3:8d4d39bf:::10000000002.0000001d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.413110+0000", 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.8139497900000001, 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011692048, 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 33 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.413110+0000", 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.413110+0000", 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.413113+0000", 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3979999999999999e-06 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.414478+0000", 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001365053 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.414520+0000", 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1928999999999998e-05 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.414535+0000", 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5140000000000001e-05 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.414626+0000", 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0675999999999997e-05 2024-04-25T15:52:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.414810+0000", 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018426400000000001 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.414985+0000", 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000174573 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.423846+0000", 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088610029999999992 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.424670+0000", 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00082378300000000002 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.424743+0000", 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3609999999999995e-05 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.424802+0000", 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8619000000000002e-05 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.294 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:44 3.1a 3:5917aad8:::10000000002.00000028:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.960475+0000", 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.2665848719999999, 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015912984000000002, 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 44 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.295 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.960475+0000", 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.960475+0000", 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.960478+0000", 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3419999999999999e-06 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.963095+0000", 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026164999999999999 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.963130+0000", 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5314000000000001e-05 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.963144+0000", 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3499000000000001e-05 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.963232+0000", 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8800000000000004e-05 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.963355+0000", 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012295399999999999 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.963461+0000", 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010566899999999999 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.972225+0000", 2024-04-25T15:52:45.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087644660000000003 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.976249+0000", 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040230819999999999 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.976323+0000", 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4581999999999998e-05 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.976388+0000", 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4776000000000005e-05 2024-04-25T15:52:45.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:45 3.19 3:9961cbc1:::10000000002.00000029:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.002499+0000", 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.224560404, 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012937195, 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 45 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.300 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.002499+0000", 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.002499+0000", 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.002502+0000", 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5940000000000001e-06 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.003764+0000", 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012620540000000001 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.003810+0000", 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5636000000000001e-05 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.003829+0000", 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8878000000000001e-05 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.003945+0000", 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011634899999999999 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.004145+0000", 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020036699999999999 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.004323+0000", 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000177555 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.014956+0000", 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010632879 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.015344+0000", 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000388069 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.015397+0000", 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3050999999999997e-05 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.015437+0000", 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9762999999999999e-05 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:50 3.1a 3:596502f4:::10000000002.0000002e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.237969+0000", 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.98909085500000005, 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016204501, 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:45.305 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 50 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.237969+0000", 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.237969+0000", 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.237973+0000", 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8229999999999997e-06 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.239456+0000", 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001483413 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.239491+0000", 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5092000000000002e-05 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.239510+0000", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8295e-05 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.239602+0000", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2569000000000001e-05 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.239735+0000", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013240800000000001 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.239845+0000", 2024-04-25T15:52:45.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110297 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.251931+0000", 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012085767000000001 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.254042+0000", 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021119229999999999 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.254119+0000", 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6959000000000006e-05 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.254173+0000", 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3955000000000001e-05 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:56 3.25 3:a56fc364:::10000000002.00000034:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.520052+0000", 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.70700777999999997, 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013503591000000001, 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.310 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 56 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.520052+0000", 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.520052+0000", 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.520056+0000", 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6600000000000001e-06 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.521387+0000", 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001331151 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.521433+0000", 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6366999999999998e-05 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.521456+0000", 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3238999999999999e-05 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.521564+0000", 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010736200000000001 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.521689+0000", 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012571500000000001 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.521805+0000", 2024-04-25T15:52:45.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115615 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.532230+0000", 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010424671999999999 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.533468+0000", 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001238488 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.533512+0000", 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4143000000000001e-05 2024-04-25T15:52:45.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.533556+0000", 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3179000000000001e-05 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:60 3.1c e60/54 3:394f8a26:::10000000002.00000038:head v 60'2, mlcod=0'0)", 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.682523+0000", 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.54453693299999995, 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015453217999999999, 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.315 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.682523+0000", 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.682523+0000", 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.682525+0000", 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1270000000000001e-06 2024-04-25T15:52:45.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.688829+0000", 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063041449999999997 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.688858+0000", 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8819e-05 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.688891+0000", 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2852999999999999e-05 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.688997+0000", 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105874 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.689037+0000", 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0157000000000002e-05 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.697890+0000", 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088530529999999996 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.697976+0000", 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.619e-05 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:61 3.32 3:4fde4c4d:::10000000002.00000039:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.713897+0000", 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.51316240199999996, 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016507504999999999, 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:45.319 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 61 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.713897+0000", 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.713897+0000", 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.713901+0000", 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7570000000000002e-06 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.715272+0000", 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001371279 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.715316+0000", 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3268e-05 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.715337+0000", 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0917999999999999e-05 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.715431+0000", 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4019999999999998e-05 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.715552+0000", 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012121900000000001 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.715658+0000", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105703 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.728057+0000", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012398998 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.730274+0000", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022177680000000002 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.730340+0000", 2024-04-25T15:52:45.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5703000000000004e-05 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.730405+0000", 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4871999999999998e-05 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:45.324 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:45.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:45.326 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:45.322+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:45.426 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:45.423+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:45.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:45.528 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:45.527+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:45.532 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:45.532 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:45.532 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:45.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:45.629 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:45.627+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89ba86b640 1 Processor -- start 2024-04-25T15:52:45.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89ba86b640 1 -- start start 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89ba86b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b40caf60 0x7f89b415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89ba86b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415d790 0x7f89b406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89ba86b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b40c77a0 con 0x7f89b4073bf0 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89ba86b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f89b40cab60 con 0x7f89b40caf60 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89ba86b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89b4159620 con 0x7f89b415d790 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89b3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b40caf60 0x7f89b415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89b8e08640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f89b4073bf0 0x7f89b4072bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44670/0 (socket says 172.21.15.67:44670) 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.646+0000 7f89b8e08640 1 -- 172.21.15.67:0/3048977979 learned_addr learned my addr 172.21.15.67:0/3048977979 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89abfff640 1 --2- 172.21.15.67:0/3048977979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415d790 0x7f89b406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b37fe640 1 -- 172.21.15.67:0/3048977979 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 665475512 0 0) 0x7f89b40c77a0 con 0x7f89b4073bf0 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b3fff640 1 -- 172.21.15.67:0/3048977979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415d790 msgr2=0x7f89b406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:45.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b3fff640 1 --2- 172.21.15.67:0/3048977979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415d790 0x7f89b406d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b3fff640 1 -- 172.21.15.67:0/3048977979 >> v1:172.21.15.67:6789/0 conn(0x7f89b4073bf0 legacy=0x7f89b4072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b3fff640 1 -- 172.21.15.67:0/3048977979 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b40c7a60 con 0x7f89b40caf60 2024-04-25T15:52:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b3fff640 1 --2- 172.21.15.67:0/3048977979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40caf60 0x7f89b415d250 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f89a40097c0 tx=0x7f89a401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413f3c49b8225345 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b37fe640 1 -- 172.21.15.67:0/3048977979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a40b0070 con 0x7f89b40caf60 2024-04-25T15:52:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b37fe640 1 -- 172.21.15.67:0/3048977979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a40ab600 con 0x7f89b40caf60 2024-04-25T15:52:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89b37fe640 1 -- 172.21.15.67:0/3048977979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a40ab9d0 con 0x7f89b40caf60 2024-04-25T15:52:45.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89ba86b640 1 -- 172.21.15.67:0/3048977979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40caf60 msgr2=0x7f89b415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.650+0000 7f89ba86b640 1 --2- 172.21.15.67:0/3048977979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40caf60 0x7f89b415d250 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f89a40097c0 tx=0x7f89a401c4a0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- 172.21.15.67:0/3048977979 shutdown_connections 2024-04-25T15:52:45.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 --2- 172.21.15.67:0/3048977979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415d790 0x7f89b406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 --2- 172.21.15.67:0/3048977979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40caf60 0x7f89b415d250 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- 172.21.15.67:0/3048977979 >> 172.21.15.67:0/3048977979 conn(0x7f89b4070db0 msgr2=0x7f89b40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:45.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- 172.21.15.67:0/3048977979 shutdown_connections 2024-04-25T15:52:45.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- 172.21.15.67:0/3048977979 wait complete. 2024-04-25T15:52:45.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 Processor -- start 2024-04-25T15:52:45.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- start start 2024-04-25T15:52:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b4073bf0 0x7f89b40cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b40caf60 0x7f89b415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415d790 0x7f89b40cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89b4166ee0 con 0x7f89b415d790 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89b415a600 con 0x7f89b4073bf0 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89ba86b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89b4167060 con 0x7f89b40caf60 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415d790 0x7f89b40cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415d790 0x7f89b40cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49108/0 (socket says 172.21.15.67:49108) 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 learned_addr learned my addr 172.21.15.67:0/1404936034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89abfff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b40caf60 0x7f89b415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b4073bf0 0x7f89b40cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b40caf60 msgr2=0x7f89b415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b40caf60 0x7f89b415baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b4073bf0 msgr2=0x7f89b40cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b4073bf0 0x7f89b40cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.654+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b415bfe0 con 0x7f89b415d790 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b8e08640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415d790 0x7f89b40cd130 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00caf0 tx=0x7f89ac002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac00abd0 con 0x7f89b415d790 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89ac00adb0 con 0x7f89b415d790 2024-04-25T15:52:45.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b414c650 con 0x7f89b415d790 2024-04-25T15:52:45.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac012640 con 0x7f89b415d790 2024-04-25T15:52:45.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b414cc10 con 0x7f89b415d790 2024-04-25T15:52:45.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f89b406a480 con 0x7f89b415d790 2024-04-25T15:52:45.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f89ac0127e0 con 0x7f89b415d790 2024-04-25T15:52:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b17fa640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f400 0x7f8994071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5857+0+0 (secure 0 0 0) 0x7f89ac090290 con 0x7f89b415d790 2024-04-25T15:52:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v60) ==== 54+0+16 (secure 0 0 0) 0x7f89ac090720 con 0x7f89b415d790 2024-04-25T15:52:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f400 0x7f8994071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b4164970 0x7f89b4145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.658+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89b415b0b0 con 0x7f89b4164970 2024-04-25T15:52:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.662+0000 7f89abfff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b4164970 0x7f89b4145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.662+0000 7f89abfff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b4164970 0x7f89b4145a40 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.662+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f400 0x7f8994071910 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f89a40ad000 tx=0x7f89a40a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.662+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b415b0b0 con 0x7f89b4164970 2024-04-25T15:52:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.678+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b40c8a40 con 0x7f89b4164970 2024-04-25T15:52:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.678+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b40c8a40 con 0x7f89b4164970 2024-04-25T15:52:45.684 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.678+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b40cd670 0x7f89b4146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.678+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f89b4146ac0 con 0x7f89b40cd670 2024-04-25T15:52:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.678+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b40cd670 0x7f89b4146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.678+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b40cd670 0x7f89b4146580 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.682+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b4146ac0 con 0x7f89b40cd670 2024-04-25T15:52:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.702+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b4144550 con 0x7f89b40cd670 2024-04-25T15:52:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.702+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b4144550 con 0x7f89b40cd670 2024-04-25T15:52:45.707 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.702+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.702+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f89b4145030 con 0x7f89b4063090 2024-04-25T15:52:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.702+0000 7f89abfff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.702+0000 7f89abfff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.702+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b4145030 con 0x7f89b4063090 2024-04-25T15:52:45.713 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:45.713 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:45.713 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.718+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b4063cb0 con 0x7f89b4063090 2024-04-25T15:52:45.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.722+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b4063cb0 con 0x7f89b4063090 2024-04-25T15:52:45.726 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.722+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b40c94f0 0x7f89b40c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.722+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f89b40c9e40 con 0x7f89b40c94f0 2024-04-25T15:52:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.722+0000 7f89b8e08640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b40c94f0 0x7f89b40c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.722+0000 7f89b8e08640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b40c94f0 0x7f89b40c9900 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.722+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b40c9e40 con 0x7f89b40c94f0 2024-04-25T15:52:45.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:45.730 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:45.727+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.742+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b402ccb0 con 0x7f89b40c94f0 2024-04-25T15:52:45.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.742+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b402ccb0 con 0x7f89b40c94f0 2024-04-25T15:52:45.749 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.746+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402d3d0 0x7f89b402d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.746+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f89b40c9e40 con 0x7f89b402d3d0 2024-04-25T15:52:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.746+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402d3d0 0x7f89b402d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.746+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402d3d0 0x7f89b402d7e0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.746+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b40c9e40 con 0x7f89b402d3d0 2024-04-25T15:52:45.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.766+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b40c9e40 con 0x7f89b402d3d0 2024-04-25T15:52:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.770+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b40c9e40 con 0x7f89b402d3d0 2024-04-25T15:52:45.777 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.770+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b420a560 0x7f89b420c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.770+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f89b402dd20 con 0x7f89b420a560 2024-04-25T15:52:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.774+0000 7f89abfff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b420a560 0x7f89b420c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.774+0000 7f89abfff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b420a560 0x7f89b420c980 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.774+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b402dd20 con 0x7f89b420a560 2024-04-25T15:52:45.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.794+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b402dd20 con 0x7f89b420a560 2024-04-25T15:52:45.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.802+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b402dd20 con 0x7f89b420a560 2024-04-25T15:52:45.806 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:45.806 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:45.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.802+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420d3d0 0x7f89b420f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.802+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f89b42126c0 con 0x7f89b420d3d0 2024-04-25T15:52:45.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.802+0000 7f89b8e08640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420d3d0 0x7f89b420f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.802+0000 7f89b8e08640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420d3d0 0x7f89b420f7f0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.802+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b42126c0 con 0x7f89b420d3d0 2024-04-25T15:52:45.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.822+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b420fd30 con 0x7f89b420d3d0 2024-04-25T15:52:45.830 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:45.826+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.830+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b420fd30 con 0x7f89b420d3d0 2024-04-25T15:52:45.834 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.830+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b4210450 0x7f89b4210860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.830+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f89b4215a90 con 0x7f89b4210450 2024-04-25T15:52:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.830+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b4210450 0x7f89b4210860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.830+0000 7f89b3fff640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b4210450 0x7f89b4210860 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.834+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b4215a90 con 0x7f89b4210450 2024-04-25T15:52:45.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.846+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f89b4214910 con 0x7f89b4210450 2024-04-25T15:52:45.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.850+0000 7f89b17fa640 1 -- 172.21.15.67:0/1404936034 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b4214910 con 0x7f89b4210450 2024-04-25T15:52:45.857 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b4164970 msgr2=0x7f89b4145a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b4164970 0x7f89b4145a40 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b40cd670 msgr2=0x7f89b4146580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b40cd670 0x7f89b4146580 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 msgr2=0x7f89b40634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b40c94f0 msgr2=0x7f89b40c9900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b40c94f0 0x7f89b40c9900 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402d3d0 msgr2=0x7f89b402d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402d3d0 0x7f89b402d7e0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b420a560 msgr2=0x7f89b420c980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b420a560 0x7f89b420c980 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420d3d0 msgr2=0x7f89b420f7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420d3d0 0x7f89b420f7f0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b4210450 msgr2=0x7f89b4210860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b4210450 0x7f89b4210860 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 reap_dead start 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 reap_dead start 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 reap_dead start 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 reap_dead start 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89b8e08640 1 -- 172.21.15.67:0/1404936034 reap_dead start 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f400 msgr2=0x7f8994071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f400 0x7f8994071910 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f89a40ad000 tx=0x7f89a40a6000 comp rx=0 tx=0).stop 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415d790 msgr2=0x7f89b40cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415d790 0x7f89b40cd130 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00caf0 tx=0x7f89ac002170 comp rx=0 tx=0).stop 2024-04-25T15:52:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 shutdown_connections 2024-04-25T15:52:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f400 0x7f8994071910 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 --2- 172.21.15.67:0/1404936034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415d790 0x7f89b40cd130 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 >> 172.21.15.67:0/1404936034 conn(0x7f89b4070db0 msgr2=0x7f89b4158ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 shutdown_connections 2024-04-25T15:52:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:45.854+0000 7f89ba86b640 1 -- 172.21.15.67:0/1404936034 wait complete. 2024-04-25T15:52:45.893 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:45.893 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:45.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:45.931 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:45.927+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:46.031 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:46.026+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:46.103 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:46.104 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:46.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:46.132 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:46.131+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:46.233 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:46.231+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:46.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:52:46.326 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:46.326 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:46.334 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:46.331+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:46.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:46.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaef091640 1 Processor -- start 2024-04-25T15:52:46.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaef091640 1 -- start start 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaef091640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdae8160800 0x7fdae806d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaef091640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae80c96e0 0x7fdae8167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaef091640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdae806d8e0 con 0x7fdae80c92d0 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaef091640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdae806dbc0 con 0x7fdae8160800 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaef091640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdae8073c70 con 0x7fdae80c96e0 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaece2d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdae8160800 0x7fdae806d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdae7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdae80c92d0 0x7fdae8072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44672/0 (socket says 172.21.15.67:44672) 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdae7fff640 1 -- 172.21.15.67:0/1160495649 learned_addr learned my addr 172.21.15.67:0/1160495649 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaed62e640 1 --2- 172.21.15.67:0/1160495649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae80c96e0 0x7fdae8167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaece2d640 1 -- 172.21.15.67:0/1160495649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae80c96e0 msgr2=0x7fdae8167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaece2d640 1 --2- 172.21.15.67:0/1160495649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae80c96e0 0x7fdae8167180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:46.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaece2d640 1 -- 172.21.15.67:0/1160495649 >> v1:172.21.15.67:6789/0 conn(0x7fdae80c92d0 legacy=0x7fdae8072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:46.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdae77fe640 1 -- 172.21.15.67:0/1160495649 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1581495508 0 0) 0x7fdae806d8e0 con 0x7fdae80c92d0 2024-04-25T15:52:46.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.406+0000 7fdaece2d640 1 -- 172.21.15.67:0/1160495649 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae8073f50 con 0x7fdae8160800 2024-04-25T15:52:46.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaece2d640 1 --2- 172.21.15.67:0/1160495649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae8160800 0x7fdae806d030 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0097c0 tx=0x7fdadc01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49100a75d3fe7801 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:46.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdae77fe640 1 -- 172.21.15.67:0/1160495649 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0b0070 con 0x7fdae8160800 2024-04-25T15:52:46.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdae77fe640 1 -- 172.21.15.67:0/1160495649 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdadc0ab600 con 0x7fdae8160800 2024-04-25T15:52:46.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdae77fe640 1 -- 172.21.15.67:0/1160495649 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0ab9d0 con 0x7fdae8160800 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 -- 172.21.15.67:0/1160495649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae8160800 msgr2=0x7fdae806d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 --2- 172.21.15.67:0/1160495649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae8160800 0x7fdae806d030 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0097c0 tx=0x7fdadc01c410 comp rx=0 tx=0).stop 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 -- 172.21.15.67:0/1160495649 shutdown_connections 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 --2- 172.21.15.67:0/1160495649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae80c96e0 0x7fdae8167180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 --2- 172.21.15.67:0/1160495649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae8160800 0x7fdae806d030 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 -- 172.21.15.67:0/1160495649 >> 172.21.15.67:0/1160495649 conn(0x7fdae8070db0 msgr2=0x7fdae80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 -- 172.21.15.67:0/1160495649 shutdown_connections 2024-04-25T15:52:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.410+0000 7fdaef091640 1 -- 172.21.15.67:0/1160495649 wait complete. 2024-04-25T15:52:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 Processor -- start 2024-04-25T15:52:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 -- start start 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdae80c92d0 0x7fdae80c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae80c96e0 0x7fdae814f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae8160800 0x7fdae814f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdae8160c10 con 0x7fdae80c96e0 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdae806d570 con 0x7fdae80c92d0 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaef091640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdae8160d90 con 0x7fdae8160800 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdae7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae80c96e0 0x7fdae814f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdae7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae80c96e0 0x7fdae814f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49122/0 (socket says 172.21.15.67:49122) 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdae7fff640 1 -- 172.21.15.67:0/1029378971 learned_addr learned my addr 172.21.15.67:0/1029378971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaed62e640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae8160800 0x7fdae814f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.414+0000 7fdaece2d640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdae80c92d0 0x7fdae80c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae7fff640 1 -- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae8160800 msgr2=0x7fdae814f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae7fff640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae8160800 0x7fdae814f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae7fff640 1 -- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdae80c92d0 msgr2=0x7fdae80c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae7fff640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdae80c92d0 0x7fdae80c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae7fff640 1 -- 172.21.15.67:0/1029378971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae8157f30 con 0x7fdae80c96e0 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae7fff640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae80c96e0 0x7fdae814f490 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c440 tx=0x7fdad400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae57fa640 1 -- 172.21.15.67:0/1029378971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad40074e0 con 0x7fdae80c96e0 2024-04-25T15:52:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae57fa640 1 -- 172.21.15.67:0/1029378971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad40047f0 con 0x7fdae80c96e0 2024-04-25T15:52:46.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae81581f0 con 0x7fdae80c96e0 2024-04-25T15:52:46.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae57fa640 1 -- 172.21.15.67:0/1029378971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4007680 con 0x7fdae80c96e0 2024-04-25T15:52:46.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae815a6d0 con 0x7fdae80c96e0 2024-04-25T15:52:46.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdae57fa640 1 -- 172.21.15.67:0/1029378971 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdad4011070 con 0x7fdae80c96e0 2024-04-25T15:52:46.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.418+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae806a480 con 0x7fdae80c96e0 2024-04-25T15:52:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.422+0000 7fdae57fa640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdacc06f430 0x7fdacc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.422+0000 7fdae57fa640 1 -- 172.21.15.67:0/1029378971 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5857+0+0 (secure 0 0 0) 0x7fdad4093060 con 0x7fdae80c96e0 2024-04-25T15:52:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.422+0000 7fdaece2d640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdacc06f430 0x7fdacc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.422+0000 7fdaece2d640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdacc06f430 0x7fdacc071940 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0ad000 tx=0x7fdadc0a4f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:46.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.426+0000 7fdae57fa640 1 -- 172.21.15.67:0/1029378971 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdad405ad40 con 0x7fdae80c96e0 2024-04-25T15:52:46.435 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:46.430+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:46.526 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:46.526 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:46.535 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:46.535+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:46.542 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:46.636 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:46.635+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:46.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:46.674+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fdae80c8c50 con 0x7fdae80c96e0 2024-04-25T15:52:46.713 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:46.713 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:46.733 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:46.737 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:46.730+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:46.838 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:46.834+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:46.928 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:46.929 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:46.939 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:46.939+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:46.947 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:47.039 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:47.039+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:47.135 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:47.135 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:47.141 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:47.134+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:47.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:47.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.222+0000 7fdae57fa640 1 -- 172.21.15.67:0/1029378971 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v61) ==== 91+0+0 (secure 0 0 0) 0x7fdad405f040 con 0x7fdae80c96e0 2024-04-25T15:52:47.227 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdacc06f430 msgr2=0x7fdacc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdacc06f430 0x7fdacc071940 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0ad000 tx=0x7fdadc0a4f90 comp rx=0 tx=0).stop 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae80c96e0 msgr2=0x7fdae814f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae80c96e0 0x7fdae814f490 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c440 tx=0x7fdad400c900 comp rx=0 tx=0).stop 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 shutdown_connections 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdacc06f430 0x7fdacc071940 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdae8160800 0x7fdae814f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdae80c96e0 0x7fdae814f490 secure :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdad400c440 tx=0x7fdad400c900 comp rx=0 tx=0).stop 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 --2- 172.21.15.67:0/1029378971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdae80c92d0 0x7fdae80c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 >> 172.21.15.67:0/1029378971 conn(0x7fdae8070db0 msgr2=0x7fdae80c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:47.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 shutdown_connections 2024-04-25T15:52:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.226+0000 7fdaef091640 1 -- 172.21.15.67:0/1029378971 wait complete. 2024-04-25T15:52:47.235 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:47.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:47.235 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T15:52:47.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:52:47.243 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:47.243+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:47.343 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:47.343+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:47.356 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:47.357 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:47.376 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.386+0000 7fc0c8720640 1 Processor -- start 2024-04-25T15:52:47.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.386+0000 7fc0c8720640 1 -- start start 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.386+0000 7fc0c8720640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc0c0074370 0x7fc0c0074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.386+0000 7fc0c8720640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c00c7fc0 0x7fc0c00c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.386+0000 7fc0c8720640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c00c92d0 con 0x7fc0c006c640 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.386+0000 7fc0c8720640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc0c006e210 con 0x7fc0c0074370 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.386+0000 7fc0c8720640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c006e4d0 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc0c0074370 0x7fc0c0074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc0c0074370 0x7fc0c0074780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:54716/0 (socket says 172.21.15.67:54716) 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 -- 172.21.15.67:0/540633931 learned_addr learned my addr 172.21.15.67:0/540633931 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c64bc640 1 --1- 172.21.15.67:0/540633931 >> v1:172.21.15.176:6789/0 conn(0x7fc0c006c640 0x7fc0c015b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:33472/0 (socket says 172.21.15.67:33472) 2024-04-25T15:52:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c6cbd640 1 --2- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c00c7fc0 0x7fc0c00c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 -- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c00c7fc0 msgr2=0x7fc0c00c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:47.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 --2- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c00c7fc0 0x7fc0c00c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 -- 172.21.15.67:0/540633931 >> v1:172.21.15.176:6789/0 conn(0x7fc0c006c640 legacy=0x7fc0c015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 -- 172.21.15.67:0/540633931 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c006e7b0 con 0x7fc0c0074370 2024-04-25T15:52:47.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c5cbb640 1 --2- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 0x7fc0c0074780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009870 tx=0x7fc0b001ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=848b6c2128a4780a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:47.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c54ba640 1 -- 172.21.15.67:0/540633931 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00b5070 con 0x7fc0c0074370 2024-04-25T15:52:47.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c54ba640 1 -- 172.21.15.67:0/540633931 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b00a8a50 con 0x7fc0c0074370 2024-04-25T15:52:47.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.390+0000 7fc0c54ba640 1 -- 172.21.15.67:0/540633931 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00a8e20 con 0x7fc0c0074370 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 -- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 msgr2=0x7fc0c0074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 --2- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 0x7fc0c0074780 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009870 tx=0x7fc0b001ed50 comp rx=0 tx=0).stop 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 -- 172.21.15.67:0/540633931 shutdown_connections 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 --2- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c00c7fc0 0x7fc0c00c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 --2- 172.21.15.67:0/540633931 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 0x7fc0c0074780 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 -- 172.21.15.67:0/540633931 >> 172.21.15.67:0/540633931 conn(0x7fc0c0070db0 msgr2=0x7fc0c00ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 -- 172.21.15.67:0/540633931 shutdown_connections 2024-04-25T15:52:47.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 -- 172.21.15.67:0/540633931 wait complete. 2024-04-25T15:52:47.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 Processor -- start 2024-04-25T15:52:47.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.394+0000 7fc0c8720640 1 -- start start 2024-04-25T15:52:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c006c640 0x7fc0c01f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 0x7fc0c0159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00c7fc0 0x7fc0c0151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c0073e50 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c0073b70 con 0x7fc0c0074370 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c0073420 con 0x7fc0c006c640 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00c7fc0 0x7fc0c0151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00c7fc0 0x7fc0c0151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49126/0 (socket says 172.21.15.67:49126) 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 -- 172.21.15.67:0/239684348 learned_addr learned my addr 172.21.15.67:0/239684348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c5cbb640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 0x7fc0c0159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c64bc640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c006c640 0x7fc0c01f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 -- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c006c640 msgr2=0x7fc0c01f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c006c640 0x7fc0c01f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 -- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 msgr2=0x7fc0c0159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 0x7fc0c0159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 -- 172.21.15.67:0/239684348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c015a2c0 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c6cbd640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00c7fc0 0x7fc0c0151080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800efc0 tx=0x7fc0b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/239684348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8018020 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/239684348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b8014480 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c01f3410 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/239684348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8009e60 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c015a790 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c006a480 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.398+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/239684348 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc0b801b020 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.402+0000 7fc0b6ffd640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc0a406f400 0x7fc0a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.402+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/239684348 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5857+0+0 (secure 0 0 0) 0x7fc0b808ff30 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.402+0000 7fc0c64bc640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc0a406f400 0x7fc0a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.402+0000 7fc0c64bc640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc0a406f400 0x7fc0a4071910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc000f70 tx=0x7fc0bc007cf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:47.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.406+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/239684348 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc0b8022030 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.444 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:47.438+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:47.545 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:47.538+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:47.575 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:47.575 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:47.593 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:47.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.638+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fc0c01528f0 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.638+0000 7fc0b6ffd640 1 -- 172.21.15.67:0/239684348 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v61) ==== 81+0+35020 (secure 0 0 0) 0x7fc0b8010040 con 0x7fc0c00c7fc0 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 61, 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:52:47.226034+0000", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:52:31.099760+0000", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5804032, 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:52:47.645 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:52:47.646 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:47.647 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.648 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:47.649 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:47.650 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:52:47.651 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:47.652 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:47.653 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:47.654 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:47.655 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:47.656 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.75, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.75, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.75, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.75, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:52:47.657 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:52:47.658 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:47.659 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:47.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:47.661 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.5299999713897705, 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.5299999713897705, 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T15:52:47.662 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.663 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.664 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.665 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.667 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.668 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:52:47.671 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:52:47.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.677 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:47.678 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.680 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:52:47.682 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 0, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.686 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:52:47.687 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.689 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.690 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:52:47.692 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.693 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.696 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:52:47.697 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:47.698 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:47.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:47.702 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:47.703 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.705 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 65536, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.706 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:47.707 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [], 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:47.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:52:47.710 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:52:47.711 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:47.712 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:47.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc0a406f400 msgr2=0x7fc0a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc0a406f400 0x7fc0a4071910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc000f70 tx=0x7fc0bc007cf0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00c7fc0 msgr2=0x7fc0c0151080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00c7fc0 0x7fc0c0151080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800efc0 tx=0x7fc0b800c490 comp rx=0 tx=0).stop 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 shutdown_connections 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc0a406f400 0x7fc0a4071910 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c00c7fc0 0x7fc0c0151080 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0c0074370 0x7fc0c0159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 --2- 172.21.15.67:0/239684348 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0c006c640 0x7fc0c01f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 >> 172.21.15.67:0/239684348 conn(0x7fc0c0070db0 msgr2=0x7fc0c014a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 shutdown_connections 2024-04-25T15:52:47.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.642+0000 7fc0c8720640 1 -- 172.21.15.67:0/239684348 wait complete. 2024-04-25T15:52:47.715 INFO:tasks.thrashosds.thrasher:j is {'epoch': 61, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:52:47.226034+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:52:31.099760+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:52:47.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:52:47.716 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:47.706+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:47.744 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:47.744 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:47.761 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:47.813 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:47.810+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:47.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:47.915 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:47.911+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:47.915 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:47.915 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:47.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.910+0000 7f235180c640 1 Processor -- start 2024-04-25T15:52:47.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.910+0000 7f235180c640 1 -- start start 2024-04-25T15:52:47.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f235180c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f234c156d30 0x7f234c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f235180c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c0c5720 0x7f234c0c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f235180c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f234c1603e0 0x7f234c1607f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f235180c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f234c0cb5b0 con 0x7f234c156d30 2024-04-25T15:52:47.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f235180c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f234c15ff50 con 0x7f234c1603e0 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f235180c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f234c160d30 con 0x7f234c0c5720 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f234c156d30 0x7f234c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f234c156d30 0x7f234c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49142/0 (socket says 172.21.15.67:49142) 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 -- 172.21.15.67:0/181115954 learned_addr learned my addr 172.21.15.67:0/181115954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234b7fe640 1 --2- 172.21.15.67:0/181115954 >> v2:172.21.15.176:3300/0 conn(0x7f234c1603e0 0x7f234c1607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f2343fff640 1 --2- 172.21.15.67:0/181115954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c0c5720 0x7f234c0c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 -- 172.21.15.67:0/181115954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c0c5720 msgr2=0x7f234c0c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 --2- 172.21.15.67:0/181115954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c0c5720 0x7f234c0c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 -- 172.21.15.67:0/181115954 >> v2:172.21.15.176:3300/0 conn(0x7f234c1603e0 msgr2=0x7f234c1607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 --2- 172.21.15.67:0/181115954 >> v2:172.21.15.176:3300/0 conn(0x7f234c1603e0 0x7f234c1607f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f234affd640 1 -- 172.21.15.67:0/181115954 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234c0cb800 con 0x7f234c156d30 2024-04-25T15:52:47.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.914+0000 7f2343fff640 1 --2- 172.21.15.67:0/181115954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c0c5720 0x7f234c0c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f234affd640 1 --2- 172.21.15.67:0/181115954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c156d30 0x7f234c0ca540 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f234c0b6050 tx=0x7f233c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272d008e5a4b7e87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f234a7fc640 1 -- 172.21.15.67:0/181115954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c0b5070 con 0x7f234c156d30 2024-04-25T15:52:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f234a7fc640 1 -- 172.21.15.67:0/181115954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f233c0a89d0 con 0x7f234c156d30 2024-04-25T15:52:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f234a7fc640 1 -- 172.21.15.67:0/181115954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c0a8da0 con 0x7f234c156d30 2024-04-25T15:52:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 -- 172.21.15.67:0/181115954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c156d30 msgr2=0x7f234c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 --2- 172.21.15.67:0/181115954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c156d30 0x7f234c0ca540 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f234c0b6050 tx=0x7f233c01ed10 comp rx=0 tx=0).stop 2024-04-25T15:52:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 -- 172.21.15.67:0/181115954 shutdown_connections 2024-04-25T15:52:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 --2- 172.21.15.67:0/181115954 >> v2:172.21.15.176:3300/0 conn(0x7f234c1603e0 0x7f234c1607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 --2- 172.21.15.67:0/181115954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c0c5720 0x7f234c0c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 --2- 172.21.15.67:0/181115954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c156d30 0x7f234c0ca540 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 -- 172.21.15.67:0/181115954 >> 172.21.15.67:0/181115954 conn(0x7f234c0700b0 msgr2=0x7f234c0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:47.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 -- 172.21.15.67:0/181115954 shutdown_connections 2024-04-25T15:52:47.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.918+0000 7f235180c640 1 -- 172.21.15.67:0/181115954 wait complete. 2024-04-25T15:52:47.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 Processor -- start 2024-04-25T15:52:47.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 -- start start 2024-04-25T15:52:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c0c5720 0x7f234c150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c156d30 0x7f234c15bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f234c1603e0 0x7f234c15e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f234c0cc0a0 con 0x7f234c0c5720 2024-04-25T15:52:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f234c0cbdc0 con 0x7f234c1603e0 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f235180c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f234c0cc380 con 0x7f234c156d30 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f234affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c0c5720 0x7f234c150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f234affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c0c5720 0x7f234c150910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49144/0 (socket says 172.21.15.67:49144) 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f234affd640 1 -- 172.21.15.67:0/2193065064 learned_addr learned my addr 172.21.15.67:0/2193065064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.922+0000 7f234b7fe640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f234c1603e0 0x7f234c15e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f234affd640 1 -- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c156d30 msgr2=0x7f234c15bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f234affd640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c156d30 0x7f234c15bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f234affd640 1 -- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f234c1603e0 msgr2=0x7f234c15e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f234affd640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f234c1603e0 0x7f234c15e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f234affd640 1 -- 172.21.15.67:0/2193065064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234c15c290 con 0x7f234c0c5720 2024-04-25T15:52:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f234affd640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c0c5720 0x7f234c150910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f233c01d3c0 tx=0x7f233c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f2348ff9640 1 -- 172.21.15.67:0/2193065064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c0b5070 con 0x7f234c0c5720 2024-04-25T15:52:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f2348ff9640 1 -- 172.21.15.67:0/2193065064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f233c0b0a50 con 0x7f234c0c5720 2024-04-25T15:52:47.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f2348ff9640 1 -- 172.21.15.67:0/2193065064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c0b0e40 con 0x7f234c0c5720 2024-04-25T15:52:47.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f234c1f6120 con 0x7f234c0c5720 2024-04-25T15:52:47.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f234c1f66e0 con 0x7f234c0c5720 2024-04-25T15:52:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.926+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f234c06a480 con 0x7f234c0c5720 2024-04-25T15:52:47.934 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.930+0000 7f2348ff9640 1 -- 172.21.15.67:0/2193065064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f233c0c2060 con 0x7f234c0c5720 2024-04-25T15:52:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.930+0000 7f2348ff9640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f232c06f400 0x7f232c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.930+0000 7f2348ff9640 1 -- 172.21.15.67:0/2193065064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5857+0+0 (secure 0 0 0) 0x7f233c12c720 con 0x7f234c0c5720 2024-04-25T15:52:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.930+0000 7f2343fff640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f232c06f400 0x7f232c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.930+0000 7f2343fff640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f232c06f400 0x7f232c071910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2334006b20 tx=0x7f2334008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:47.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:47.934+0000 7f2348ff9640 1 -- 172.21.15.67:0/2193065064 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f233c018870 con 0x7f234c0c5720 2024-04-25T15:52:48.027 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:48.027+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 Processor -- start 2024-04-25T15:52:48.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 -- start start 2024-04-25T15:52:48.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3ec06ca60 0x7fe3ec06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec0b5660 0x7fe3ec0b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe3ec15fe00 0x7fe3ec160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe3ec15d510 con 0x7fe3ec06ca60 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe3ec06d780 con 0x7fe3ec15fe00 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f29bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec160be0 con 0x7fe3ec0b5660 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3ec06ca60 0x7fe3ec06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3ec06ca60 0x7fe3ec06ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49158/0 (socket says 172.21.15.67:49158) 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 -- 172.21.15.67:0/2117307468 learned_addr learned my addr 172.21.15.67:0/2117307468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2117307468 >> v2:172.21.15.176:3300/0 conn(0x7fe3ec15fe00 0x7fe3ec160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2117307468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec0b5660 0x7fe3ec0b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 -- 172.21.15.67:0/2117307468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec0b5660 msgr2=0x7fe3ec0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2117307468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec0b5660 0x7fe3ec0b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 -- 172.21.15.67:0/2117307468 >> v2:172.21.15.176:3300/0 conn(0x7fe3ec15fe00 msgr2=0x7fe3ec160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2117307468 >> v2:172.21.15.176:3300/0 conn(0x7fe3ec15fe00 0x7fe3ec160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.034+0000 7fe3ebfff640 1 -- 172.21.15.67:0/2117307468 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec15cbe0 con 0x7fe3ec06ca60 2024-04-25T15:52:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2117307468 >> v2:172.21.15.176:3300/0 conn(0x7fe3ec15fe00 0x7fe3ec160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2117307468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec06ca60 0x7fe3ec06ce70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0cf690 tx=0x7fe3dc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0bc177bd6660052 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3eaffd640 1 -- 172.21.15.67:0/2117307468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc0b1070 con 0x7fe3ec06ca60 2024-04-25T15:52:48.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3eaffd640 1 -- 172.21.15.67:0/2117307468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3dc0ac630 con 0x7fe3ec06ca60 2024-04-25T15:52:48.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3eaffd640 1 -- 172.21.15.67:0/2117307468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc0a8050 con 0x7fe3ec06ca60 2024-04-25T15:52:48.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2117307468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec06ca60 msgr2=0x7fe3ec06ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2117307468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec06ca60 0x7fe3ec06ce70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0cf690 tx=0x7fe3dc01c520 comp rx=0 tx=0).stop 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2117307468 shutdown_connections 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2117307468 >> v2:172.21.15.176:3300/0 conn(0x7fe3ec15fe00 0x7fe3ec160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2117307468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec0b5660 0x7fe3ec0b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2117307468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec06ca60 0x7fe3ec06ce70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2117307468 >> 172.21.15.67:0/2117307468 conn(0x7fe3ec06e420 msgr2=0x7fe3ec06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2117307468 shutdown_connections 2024-04-25T15:52:48.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.038+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2117307468 wait complete. 2024-04-25T15:52:48.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.042+0000 7fe3f29bd640 1 Processor -- start 2024-04-25T15:52:48.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.042+0000 7fe3f29bd640 1 -- start start 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec06ca60 0x7fe3ec14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3ec0b5660 0x7fe3ec156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec15fe00 0x7fe3ec15eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec0b6e90 con 0x7fe3ec15fe00 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec15c890 con 0x7fe3ec0b5660 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec0b7010 con 0x7fe3ec06ca60 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec15fe00 0x7fe3ec15eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec15fe00 0x7fe3ec15eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49160/0 (socket says 172.21.15.67:49160) 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 -- 172.21.15.67:0/2073643489 learned_addr learned my addr 172.21.15.67:0/2073643489 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3ec0b5660 0x7fe3ec156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec06ca60 0x7fe3ec14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec06ca60 msgr2=0x7fe3ec14ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3ec06ca60 0x7fe3ec14ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3ec0b5660 msgr2=0x7fe3ec156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3ec0b5660 0x7fe3ec156810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec14ef70 con 0x7fe3ec15fe00 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec15fe00 0x7fe3ec15eed0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4007ef0 tx=0x7fe3e400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e401a070 con 0x7fe3ec15fe00 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e4015600 con 0x7fe3ec15fe00 2024-04-25T15:52:48.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec156d50 con 0x7fe3ec15fe00 2024-04-25T15:52:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e40159f0 con 0x7fe3ec15fe00 2024-04-25T15:52:48.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec15a560 con 0x7fe3ec15fe00 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe3e401d070 con 0x7fe3ec15fe00 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe3ec06a480 con 0x7fe3ec15fe00 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3e8ff9640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe3d806f400 0x7fe3d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.046+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5857+0+0 (secure 0 0 0) 0x7fe3e408f310 con 0x7fe3ec15fe00 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v61) ==== 54+0+16 (secure 0 0 0) 0x7fe3e4010070 con 0x7fe3ec15fe00 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe3d806f400 0x7fe3d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe3ec155680 0x7fe3ec1487a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec06e9d0 con 0x7fe3ec155680 2024-04-25T15:52:48.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe3ec155680 0x7fe3ec1487a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe3ec155680 0x7fe3ec1487a0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe3d806f400 0x7fe3d8071910 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0ae000 tx=0x7fe3dc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.050+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec06e9d0 con 0x7fe3ec155680 2024-04-25T15:52:48.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.078+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec076f50 con 0x7fe3ec155680 2024-04-25T15:52:48.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.082+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec076f50 con 0x7fe3ec155680 2024-04-25T15:52:48.086 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:48.087 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:48.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.082+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe3ec1f9280 0x7fe3ec1f9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.082+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec1491f0 con 0x7fe3ec1f9280 2024-04-25T15:52:48.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.082+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe3ec1f9280 0x7fe3ec1f9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.082+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe3ec1f9280 0x7fe3ec1f9690 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.082+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec1491f0 con 0x7fe3ec1f9280 2024-04-25T15:52:48.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.110+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec15a840 con 0x7fe3ec1f9280 2024-04-25T15:52:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.110+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec15a840 con 0x7fe3ec1f9280 2024-04-25T15:52:48.117 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.110+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe3ec063090 0x7fe3ec0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.110+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec0b4830 con 0x7fe3ec063090 2024-04-25T15:52:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.110+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe3ec063090 0x7fe3ec0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.114+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe3ec063090 0x7fe3ec0634a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.118 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:48.115+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.114+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec0b4830 con 0x7fe3ec063090 2024-04-25T15:52:48.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.130+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec1fa0e0 con 0x7fe3ec063090 2024-04-25T15:52:48.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.130+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec1fa0e0 con 0x7fe3ec063090 2024-04-25T15:52:48.137 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.130+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe3ec151710 0x7fe3ec151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.130+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec063ef0 con 0x7fe3ec151710 2024-04-25T15:52:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.130+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe3ec151710 0x7fe3ec151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.134+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe3ec151710 0x7fe3ec151b20 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.138 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:48.138 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:48.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.134+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec063ef0 con 0x7fe3ec151710 2024-04-25T15:52:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.146+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f234c0c52f0 con 0x7f232c06f400 2024-04-25T15:52:48.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.154+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec063ef0 con 0x7fe3ec151710 2024-04-25T15:52:48.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.154+0000 7f2348ff9640 1 -- 172.21.15.67:0/2193065064 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252350 (secure 0 0 0) 0x7f234c0c52f0 con 0x7f232c06f400 2024-04-25T15:52:48.163 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec063ef0 con 0x7fe3ec151710 2024-04-25T15:52:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:52:48.165 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe3ec02c9e0 0x7fe3ec02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec152570 con 0x7fe3ec02c9e0 2024-04-25T15:52:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe3ec02c9e0 0x7fe3ec02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe3ec02c9e0 0x7fe3ec02cdf0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec152570 con 0x7fe3ec02c9e0 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f232c06f400 msgr2=0x7f232c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f232c06f400 0x7f232c071910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2334006b20 tx=0x7f2334008040 comp rx=0 tx=0).stop 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c0c5720 msgr2=0x7f234c150910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c0c5720 0x7f234c150910 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f233c01d3c0 tx=0x7f233c01bab0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 shutdown_connections 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f232c06f400 0x7f232c071910 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f234c1603e0 0x7f234c15e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f234c156d30 0x7f234c15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 --2- 172.21.15.67:0/2193065064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234c0c5720 0x7f234c150910 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 >> 172.21.15.67:0/2193065064 conn(0x7f234c0700b0 msgr2=0x7f234c0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 shutdown_connections 2024-04-25T15:52:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.158+0000 7f235180c640 1 -- 172.21.15.67:0/2193065064 wait complete. 2024-04-25T15:52:48.184 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-04-25T15:52:47.846411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84512645,"num_objects":120,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":350,"num_write_kb":83649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4003,"num_keys_recovered":22,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":289512,"kb_used_data":96204,"kb_used_omap":10,"kb_used_meta":193205,"kb_avail":660313368,"statfs":{"total":676457349120,"available":676160888832,"internally_reserved":0,"allocated":98512896,"data_stored":95655308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":197842027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":83907979,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":81945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002096"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258699+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:52:42.258699+0000","last_peered":"2024-04-25T15:52:42.258699+0000","last_clean":"2024-04-25T15:52:42.258699+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:52:42.258699+0000","last_undegraded":"2024-04-25T15:52:42.258699+0000","last_fullsized":"2024-04-25T15:52:42.258699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":56,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252997+0000","last_change":"2024-04-25T15:51:47.264956+0000","last_active":"2024-04-25T15:52:42.252997+0000","last_peered":"2024-04-25T15:52:42.252997+0000","last_clean":"2024-04-25T15:52:42.252997+0000","last_became_active":"2024-04-25T15:51:47.264571+0000","last_became_peered":"2024-04-25T15:51:47.264571+0000","last_unstale":"2024-04-25T15:52:42.252997+0000","last_undegraded":"2024-04-25T15:52:42.252997+0000","last_fullsized":"2024-04-25T15:52:42.252997+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:39.343444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":46,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207492+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:52:42.207492+0000","last_peered":"2024-04-25T15:52:42.207492+0000","last_clean":"2024-04-25T15:52:42.207492+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:52:42.207492+0000","last_undegraded":"2024-04-25T15:52:42.207492+0000","last_fullsized":"2024-04-25T15:52:42.207492+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341857+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:52:42.341857+0000","last_peered":"2024-04-25T15:52:42.341857+0000","last_clean":"2024-04-25T15:52:42.341857+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:52:42.341857+0000","last_undegraded":"2024-04-25T15:52:42.341857+0000","last_fullsized":"2024-04-25T15:52:42.341857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":89,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258039+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:52:42.258039+0000","last_peered":"2024-04-25T15:52:42.258039+0000","last_clean":"2024-04-25T15:52:42.258039+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:52:42.258039+0000","last_undegraded":"2024-04-25T15:52:42.258039+0000","last_fullsized":"2024-04-25T15:52:42.258039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341506+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:52:42.341506+0000","last_peered":"2024-04-25T15:52:42.341506+0000","last_clean":"2024-04-25T15:52:42.341506+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:52:42.341506+0000","last_undegraded":"2024-04-25T15:52:42.341506+0000","last_fullsized":"2024-04-25T15:52:42.341506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":89,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257854+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:52:42.257854+0000","last_peered":"2024-04-25T15:52:42.257854+0000","last_clean":"2024-04-25T15:52:42.257854+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:52:42.257854+0000","last_undegraded":"2024-04-25T15:52:42.257854+0000","last_fullsized":"2024-04-25T15:52:42.257854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.337848+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:52:42.337848+0000","last_peered":"2024-04-25T15:52:42.337848+0000","last_clean":"2024-04-25T15:52:42.337848+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:52:42.337848+0000","last_undegraded":"2024-04-25T15:52:42.337848+0000","last_fullsized":"2024-04-25T15:52:42.337848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207655+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:52:42.207655+0000","last_peered":"2024-04-25T15:52:42.207655+0000","last_clean":"2024-04-25T15:52:42.207655+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:52:42.207655+0000","last_undegraded":"2024-04-25T15:52:42.207655+0000","last_fullsized":"2024-04-25T15:52:42.207655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257230+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:52:42.257230+0000","last_peered":"2024-04-25T15:52:42.257230+0000","last_clean":"2024-04-25T15:52:42.257230+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:52:42.257230+0000","last_undegraded":"2024-04-25T15:52:42.257230+0000","last_fullsized":"2024-04-25T15:52:42.257230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257042+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:52:42.257042+0000","last_peered":"2024-04-25T15:52:42.257042+0000","last_clean":"2024-04-25T15:52:42.257042+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:52:42.257042+0000","last_undegraded":"2024-04-25T15:52:42.257042+0000","last_fullsized":"2024-04-25T15:52:42.257042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"59'1","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257053+0000","last_change":"2024-04-25T15:52:32.125058+0000","last_active":"2024-04-25T15:52:42.257053+0000","last_peered":"2024-04-25T15:52:42.257053+0000","last_clean":"2024-04-25T15:52:42.257053+0000","last_became_active":"2024-04-25T15:52:32.124575+0000","last_became_peered":"2024-04-25T15:52:32.124575+0000","last_unstale":"2024-04-25T15:52:42.257053+0000","last_undegraded":"2024-04-25T15:52:42.257053+0000","last_fullsized":"2024-04-25T15:52:42.257053+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:24:57.698573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"59'1","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338512+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:52:42.338512+0000","last_peered":"2024-04-25T15:52:42.338512+0000","last_clean":"2024-04-25T15:52:42.338512+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:52:42.338512+0000","last_undegraded":"2024-04-25T15:52:42.338512+0000","last_fullsized":"2024-04-25T15:52:42.338512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":93,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252743+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:52:42.252743+0000","last_peered":"2024-04-25T15:52:42.252743+0000","last_clean":"2024-04-25T15:52:42.252743+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:52:42.252743+0000","last_undegraded":"2024-04-25T15:52:42.252743+0000","last_fullsized":"2024-04-25T15:52:42.252743+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":61,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252564+0000","last_change":"2024-04-25T15:52:32.129421+0000","last_active":"2024-04-25T15:52:42.252564+0000","last_peered":"2024-04-25T15:52:42.252564+0000","last_clean":"2024-04-25T15:52:42.252564+0000","last_became_active":"2024-04-25T15:52:32.128747+0000","last_became_peered":"2024-04-25T15:52:32.128747+0000","last_unstale":"2024-04-25T15:52:42.252564+0000","last_undegraded":"2024-04-25T15:52:42.252564+0000","last_fullsized":"2024-04-25T15:52:42.252564+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:44.731855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"59'1","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341356+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:52:42.341356+0000","last_peered":"2024-04-25T15:52:42.341356+0000","last_clean":"2024-04-25T15:52:42.341356+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:52:42.341356+0000","last_undegraded":"2024-04-25T15:52:42.341356+0000","last_fullsized":"2024-04-25T15:52:42.341356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341154+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:52:42.341154+0000","last_peered":"2024-04-25T15:52:42.341154+0000","last_clean":"2024-04-25T15:52:42.341154+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:52:42.341154+0000","last_undegraded":"2024-04-25T15:52:42.341154+0000","last_fullsized":"2024-04-25T15:52:42.341154+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":89,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207676+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:52:42.207676+0000","last_peered":"2024-04-25T15:52:42.207676+0000","last_clean":"2024-04-25T15:52:42.207676+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:52:42.207676+0000","last_undegraded":"2024-04-25T15:52:42.207676+0000","last_fullsized":"2024-04-25T15:52:42.207676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":89,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258353+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:52:42.258353+0000","last_peered":"2024-04-25T15:52:42.258353+0000","last_clean":"2024-04-25T15:52:42.258353+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:52:42.258353+0000","last_undegraded":"2024-04-25T15:52:42.258353+0000","last_fullsized":"2024-04-25T15:52:42.258353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"59'1","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257103+0000","last_change":"2024-04-25T15:52:32.120016+0000","last_active":"2024-04-25T15:52:42.257103+0000","last_peered":"2024-04-25T15:52:42.257103+0000","last_clean":"2024-04-25T15:52:42.257103+0000","last_became_active":"2024-04-25T15:52:32.119425+0000","last_became_peered":"2024-04-25T15:52:32.119425+0000","last_unstale":"2024-04-25T15:52:42.257103+0000","last_undegraded":"2024-04-25T15:52:42.257103+0000","last_fullsized":"2024-04-25T15:52:42.257103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:42:15.118421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":46,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258031+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:52:42.258031+0000","last_peered":"2024-04-25T15:52:42.258031+0000","last_clean":"2024-04-25T15:52:42.258031+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:52:42.258031+0000","last_undegraded":"2024-04-25T15:52:42.258031+0000","last_fullsized":"2024-04-25T15:52:42.258031+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"58'1","reported_seq":84,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207447+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:52:42.207447+0000","last_peered":"2024-04-25T15:52:42.207447+0000","last_clean":"2024-04-25T15:52:42.207447+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:52:42.207447+0000","last_undegraded":"2024-04-25T15:52:42.207447+0000","last_fullsized":"2024-04-25T15:52:42.207447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":95,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338114+0000","last_change":"2024-04-25T15:52:32.123939+0000","last_active":"2024-04-25T15:52:42.338114+0000","last_peered":"2024-04-25T15:52:42.338114+0000","last_clean":"2024-04-25T15:52:42.338114+0000","last_became_active":"2024-04-25T15:52:32.122698+0000","last_became_peered":"2024-04-25T15:52:32.122698+0000","last_unstale":"2024-04-25T15:52:42.338114+0000","last_undegraded":"2024-04-25T15:52:42.338114+0000","last_fullsized":"2024-04-25T15:52:42.338114+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:09.597867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341485+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:52:42.341485+0000","last_peered":"2024-04-25T15:52:42.341485+0000","last_clean":"2024-04-25T15:52:42.341485+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:52:42.341485+0000","last_undegraded":"2024-04-25T15:52:42.341485+0000","last_fullsized":"2024-04-25T15:52:42.341485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.339019+0000","last_change":"2024-04-25T15:52:32.120389+0000","last_active":"2024-04-25T15:52:42.339019+0000","last_peered":"2024-04-25T15:52:42.339019+0000","last_clean":"2024-04-25T15:52:42.339019+0000","last_became_active":"2024-04-25T15:52:32.120052+0000","last_became_peered":"2024-04-25T15:52:32.120052+0000","last_unstale":"2024-04-25T15:52:42.339019+0000","last_undegraded":"2024-04-25T15:52:42.339019+0000","last_fullsized":"2024-04-25T15:52:42.339019+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:45.982199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257371+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:52:42.257371+0000","last_peered":"2024-04-25T15:52:42.257371+0000","last_clean":"2024-04-25T15:52:42.257371+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:52:42.257371+0000","last_undegraded":"2024-04-25T15:52:42.257371+0000","last_fullsized":"2024-04-25T15:52:42.257371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":56,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252933+0000","last_change":"2024-04-25T15:51:47.271333+0000","last_active":"2024-04-25T15:52:42.252933+0000","last_peered":"2024-04-25T15:52:42.252933+0000","last_clean":"2024-04-25T15:52:42.252933+0000","last_became_active":"2024-04-25T15:51:47.270971+0000","last_became_peered":"2024-04-25T15:51:47.270971+0000","last_unstale":"2024-04-25T15:52:42.252933+0000","last_undegraded":"2024-04-25T15:52:42.252933+0000","last_fullsized":"2024-04-25T15:52:42.252933+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:23.860435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338391+0000","last_change":"2024-04-25T15:52:32.125522+0000","last_active":"2024-04-25T15:52:42.338391+0000","last_peered":"2024-04-25T15:52:42.338391+0000","last_clean":"2024-04-25T15:52:42.338391+0000","last_became_active":"2024-04-25T15:52:32.125187+0000","last_became_peered":"2024-04-25T15:52:32.125187+0000","last_unstale":"2024-04-25T15:52:42.338391+0000","last_undegraded":"2024-04-25T15:52:42.338391+0000","last_fullsized":"2024-04-25T15:52:42.338391+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.964571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":47,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338265+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:52:42.338265+0000","last_peered":"2024-04-25T15:52:42.338265+0000","last_clean":"2024-04-25T15:52:42.338265+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:52:42.338265+0000","last_undegraded":"2024-04-25T15:52:42.338265+0000","last_fullsized":"2024-04-25T15:52:42.338265+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":36,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252606+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:52:42.252606+0000","last_peered":"2024-04-25T15:52:42.252606+0000","last_clean":"2024-04-25T15:52:42.252606+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:52:42.252606+0000","last_undegraded":"2024-04-25T15:52:42.252606+0000","last_fullsized":"2024-04-25T15:52:42.252606+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"59'2","reported_seq":102,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.564172+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:52:42.564172+0000","last_peered":"2024-04-25T15:52:42.564172+0000","last_clean":"2024-04-25T15:52:42.564172+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:52:42.564172+0000","last_undegraded":"2024-04-25T15:52:42.564172+0000","last_fullsized":"2024-04-25T15:52:42.564172+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.256207+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:52:42.256207+0000","last_peered":"2024-04-25T15:52:42.256207+0000","last_clean":"2024-04-25T15:52:42.256207+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:52:42.256207+0000","last_undegraded":"2024-04-25T15:52:42.256207+0000","last_fullsized":"2024-04-25T15:52:42.256207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":65,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252708+0000","last_change":"2024-04-25T15:52:32.123541+0000","last_active":"2024-04-25T15:52:42.252708+0000","last_peered":"2024-04-25T15:52:42.252708+0000","last_clean":"2024-04-25T15:52:42.252708+0000","last_became_active":"2024-04-25T15:52:32.123158+0000","last_became_peered":"2024-04-25T15:52:32.123158+0000","last_unstale":"2024-04-25T15:52:42.252708+0000","last_undegraded":"2024-04-25T15:52:42.252708+0000","last_fullsized":"2024-04-25T15:52:42.252708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.858171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":98,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338600+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:52:42.338600+0000","last_peered":"2024-04-25T15:52:42.338600+0000","last_clean":"2024-04-25T15:52:42.338600+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:52:42.338600+0000","last_undegraded":"2024-04-25T15:52:42.338600+0000","last_fullsized":"2024-04-25T15:52:42.338600+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257867+0000","last_change":"2024-04-25T15:52:32.131021+0000","last_active":"2024-04-25T15:52:42.257867+0000","last_peered":"2024-04-25T15:52:42.257867+0000","last_clean":"2024-04-25T15:52:42.257867+0000","last_became_active":"2024-04-25T15:52:32.129920+0000","last_became_peered":"2024-04-25T15:52:32.129920+0000","last_unstale":"2024-04-25T15:52:42.257867+0000","last_undegraded":"2024-04-25T15:52:42.257867+0000","last_fullsized":"2024-04-25T15:52:42.257867+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:00.291863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":112,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.256578+0000","last_change":"2024-04-25T15:52:32.152478+0000","last_active":"2024-04-25T15:52:42.256578+0000","last_peered":"2024-04-25T15:52:42.256578+0000","last_clean":"2024-04-25T15:52:42.256578+0000","last_became_active":"2024-04-25T15:52:32.137069+0000","last_became_peered":"2024-04-25T15:52:32.137069+0000","last_unstale":"2024-04-25T15:52:42.256578+0000","last_undegraded":"2024-04-25T15:52:42.256578+0000","last_fullsized":"2024-04-25T15:52:42.256578+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:26.946219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":95,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338136+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:52:42.338136+0000","last_peered":"2024-04-25T15:52:42.338136+0000","last_clean":"2024-04-25T15:52:42.338136+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:52:42.338136+0000","last_undegraded":"2024-04-25T15:52:42.338136+0000","last_fullsized":"2024-04-25T15:52:42.338136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"58'1","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.337934+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:52:42.337934+0000","last_peered":"2024-04-25T15:52:42.337934+0000","last_clean":"2024-04-25T15:52:42.337934+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:52:42.337934+0000","last_undegraded":"2024-04-25T15:52:42.337934+0000","last_fullsized":"2024-04-25T15:52:42.337934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341435+0000","last_change":"2024-04-25T15:52:32.145033+0000","last_active":"2024-04-25T15:52:42.341435+0000","last_peered":"2024-04-25T15:52:42.341435+0000","last_clean":"2024-04-25T15:52:42.341435+0000","last_became_active":"2024-04-25T15:52:32.136902+0000","last_became_peered":"2024-04-25T15:52:32.136902+0000","last_unstale":"2024-04-25T15:52:42.341435+0000","last_undegraded":"2024-04-25T15:52:42.341435+0000","last_fullsized":"2024-04-25T15:52:42.341435+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:59.330286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":56,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252699+0000","last_change":"2024-04-25T15:51:47.271089+0000","last_active":"2024-04-25T15:52:42.252699+0000","last_peered":"2024-04-25T15:52:42.252699+0000","last_clean":"2024-04-25T15:52:42.252699+0000","last_became_active":"2024-04-25T15:51:47.270705+0000","last_became_peered":"2024-04-25T15:51:47.270705+0000","last_unstale":"2024-04-25T15:52:42.252699+0000","last_undegraded":"2024-04-25T15:52:42.252699+0000","last_fullsized":"2024-04-25T15:52:42.252699+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:08.380566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":97,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257558+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:52:42.257558+0000","last_peered":"2024-04-25T15:52:42.257558+0000","last_clean":"2024-04-25T15:52:42.257558+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:52:42.257558+0000","last_undegraded":"2024-04-25T15:52:42.257558+0000","last_fullsized":"2024-04-25T15:52:42.257558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341887+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:52:42.341887+0000","last_peered":"2024-04-25T15:52:42.341887+0000","last_clean":"2024-04-25T15:52:42.341887+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:52:42.341887+0000","last_undegraded":"2024-04-25T15:52:42.341887+0000","last_fullsized":"2024-04-25T15:52:42.341887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.342351+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:52:42.342351+0000","last_peered":"2024-04-25T15:52:42.342351+0000","last_clean":"2024-04-25T15:52:42.342351+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:52:42.342351+0000","last_undegraded":"2024-04-25T15:52:42.342351+0000","last_fullsized":"2024-04-25T15:52:42.342351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":47,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341994+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:52:42.341994+0000","last_peered":"2024-04-25T15:52:42.341994+0000","last_clean":"2024-04-25T15:52:42.341994+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:52:42.341994+0000","last_undegraded":"2024-04-25T15:52:42.341994+0000","last_fullsized":"2024-04-25T15:52:42.341994+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":66,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252772+0000","last_change":"2024-04-25T15:51:47.302272+0000","last_active":"2024-04-25T15:52:42.252772+0000","last_peered":"2024-04-25T15:52:42.252772+0000","last_clean":"2024-04-25T15:52:42.252772+0000","last_became_active":"2024-04-25T15:51:47.266158+0000","last_became_peered":"2024-04-25T15:51:47.266158+0000","last_unstale":"2024-04-25T15:52:42.252772+0000","last_undegraded":"2024-04-25T15:52:42.252772+0000","last_fullsized":"2024-04-25T15:52:42.252772+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:28.386086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":672,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"58'1","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258303+0000","last_change":"2024-04-25T15:52:32.129340+0000","last_active":"2024-04-25T15:52:42.258303+0000","last_peered":"2024-04-25T15:52:42.258303+0000","last_clean":"2024-04-25T15:52:42.258303+0000","last_became_active":"2024-04-25T15:52:32.128859+0000","last_became_peered":"2024-04-25T15:52:32.128859+0000","last_unstale":"2024-04-25T15:52:42.258303+0000","last_undegraded":"2024-04-25T15:52:42.258303+0000","last_fullsized":"2024-04-25T15:52:42.258303+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:59:42.834204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"59'23","reported_seq":115,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.405786+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:52:42.405786+0000","last_peered":"2024-04-25T15:52:42.405786+0000","last_clean":"2024-04-25T15:52:42.405786+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:52:42.405786+0000","last_undegraded":"2024-04-25T15:52:42.405786+0000","last_fullsized":"2024-04-25T15:52:42.405786+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28452,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":89,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258165+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:52:42.258165+0000","last_peered":"2024-04-25T15:52:42.258165+0000","last_clean":"2024-04-25T15:52:42.258165+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:52:42.258165+0000","last_undegraded":"2024-04-25T15:52:42.258165+0000","last_fullsized":"2024-04-25T15:52:42.258165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":32,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338111+0000","last_change":"2024-04-25T15:52:32.150681+0000","last_active":"2024-04-25T15:52:42.338111+0000","last_peered":"2024-04-25T15:52:42.338111+0000","last_clean":"2024-04-25T15:52:42.338111+0000","last_became_active":"2024-04-25T15:52:32.125799+0000","last_became_peered":"2024-04-25T15:52:32.125799+0000","last_unstale":"2024-04-25T15:52:42.338111+0000","last_undegraded":"2024-04-25T15:52:42.338111+0000","last_fullsized":"2024-04-25T15:52:42.338111+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.858701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"58'2","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257913+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:52:42.257913+0000","last_peered":"2024-04-25T15:52:42.257913+0000","last_clean":"2024-04-25T15:52:42.257913+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:52:42.257913+0000","last_undegraded":"2024-04-25T15:52:42.257913+0000","last_fullsized":"2024-04-25T15:52:42.257913+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":97,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341612+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:52:42.341612+0000","last_peered":"2024-04-25T15:52:42.341612+0000","last_clean":"2024-04-25T15:52:42.341612+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:52:42.341612+0000","last_undegraded":"2024-04-25T15:52:42.341612+0000","last_fullsized":"2024-04-25T15:52:42.341612+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258149+0000","last_change":"2024-04-25T15:52:32.135585+0000","last_active":"2024-04-25T15:52:42.258149+0000","last_peered":"2024-04-25T15:52:42.258149+0000","last_clean":"2024-04-25T15:52:42.258149+0000","last_became_active":"2024-04-25T15:52:32.134857+0000","last_became_peered":"2024-04-25T15:52:32.134857+0000","last_unstale":"2024-04-25T15:52:42.258149+0000","last_undegraded":"2024-04-25T15:52:42.258149+0000","last_fullsized":"2024-04-25T15:52:42.258149+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:58.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":111,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.256388+0000","last_change":"2024-04-25T15:52:32.144332+0000","last_active":"2024-04-25T15:52:42.256388+0000","last_peered":"2024-04-25T15:52:42.256388+0000","last_clean":"2024-04-25T15:52:42.256388+0000","last_became_active":"2024-04-25T15:52:32.124713+0000","last_became_peered":"2024-04-25T15:52:32.124713+0000","last_unstale":"2024-04-25T15:52:42.256388+0000","last_undegraded":"2024-04-25T15:52:42.256388+0000","last_fullsized":"2024-04-25T15:52:42.256388+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:15.176691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1029,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"59'1","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.256410+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:52:42.256410+0000","last_peered":"2024-04-25T15:52:42.256410+0000","last_clean":"2024-04-25T15:52:42.256410+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:52:42.256410+0000","last_undegraded":"2024-04-25T15:52:42.256410+0000","last_fullsized":"2024-04-25T15:52:42.256410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":154,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207078+0000","last_change":"2024-04-25T15:51:47.287562+0000","last_active":"2024-04-25T15:52:42.207078+0000","last_peered":"2024-04-25T15:52:42.207078+0000","last_clean":"2024-04-25T15:52:42.207078+0000","last_became_active":"2024-04-25T15:51:47.264513+0000","last_became_peered":"2024-04-25T15:51:47.264513+0000","last_unstale":"2024-04-25T15:52:42.207078+0000","last_undegraded":"2024-04-25T15:52:42.207078+0000","last_fullsized":"2024-04-25T15:52:42.207078+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:51:11.555766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257852+0000","last_change":"2024-04-25T15:52:32.130800+0000","last_active":"2024-04-25T15:52:42.257852+0000","last_peered":"2024-04-25T15:52:42.257852+0000","last_clean":"2024-04-25T15:52:42.257852+0000","last_became_active":"2024-04-25T15:52:32.129969+0000","last_became_peered":"2024-04-25T15:52:32.129969+0000","last_unstale":"2024-04-25T15:52:42.257852+0000","last_undegraded":"2024-04-25T15:52:42.257852+0000","last_fullsized":"2024-04-25T15:52:42.257852+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:45.521306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":24,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258536+0000","last_change":"2024-04-25T15:52:32.155801+0000","last_active":"2024-04-25T15:52:42.258536+0000","last_peered":"2024-04-25T15:52:42.258536+0000","last_clean":"2024-04-25T15:52:42.258536+0000","last_became_active":"2024-04-25T15:52:32.129444+0000","last_became_peered":"2024-04-25T15:52:32.129444+0000","last_unstale":"2024-04-25T15:52:42.258536+0000","last_undegraded":"2024-04-25T15:52:42.258536+0000","last_fullsized":"2024-04-25T15:52:42.258536+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:46.358507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258520+0000","last_change":"2024-04-25T15:52:32.129065+0000","last_active":"2024-04-25T15:52:42.258520+0000","last_peered":"2024-04-25T15:52:42.258520+0000","last_clean":"2024-04-25T15:52:42.258520+0000","last_became_active":"2024-04-25T15:52:32.128801+0000","last_became_peered":"2024-04-25T15:52:32.128801+0000","last_unstale":"2024-04-25T15:52:42.258520+0000","last_undegraded":"2024-04-25T15:52:42.258520+0000","last_fullsized":"2024-04-25T15:52:42.258520+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:09.896373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207779+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:52:42.207779+0000","last_peered":"2024-04-25T15:52:42.207779+0000","last_clean":"2024-04-25T15:52:42.207779+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:52:42.207779+0000","last_undegraded":"2024-04-25T15:52:42.207779+0000","last_fullsized":"2024-04-25T15:52:42.207779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258795+0000","last_change":"2024-04-25T15:52:32.129750+0000","last_active":"2024-04-25T15:52:42.258795+0000","last_peered":"2024-04-25T15:52:42.258795+0000","last_clean":"2024-04-25T15:52:42.258795+0000","last_became_active":"2024-04-25T15:52:32.129402+0000","last_became_peered":"2024-04-25T15:52:32.129402+0000","last_unstale":"2024-04-25T15:52:42.258795+0000","last_undegraded":"2024-04-25T15:52:42.258795+0000","last_fullsized":"2024-04-25T15:52:42.258795+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:48.203567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":69,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252911+0000","last_change":"2024-04-25T15:52:32.135358+0000","last_active":"2024-04-25T15:52:42.252911+0000","last_peered":"2024-04-25T15:52:42.252911+0000","last_clean":"2024-04-25T15:52:42.252911+0000","last_became_active":"2024-04-25T15:52:32.129558+0000","last_became_peered":"2024-04-25T15:52:32.129558+0000","last_unstale":"2024-04-25T15:52:42.252911+0000","last_undegraded":"2024-04-25T15:52:42.252911+0000","last_fullsized":"2024-04-25T15:52:42.252911+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:49.940252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.342250+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:52:42.342250+0000","last_peered":"2024-04-25T15:52:42.342250+0000","last_clean":"2024-04-25T15:52:42.342250+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:52:42.342250+0000","last_undegraded":"2024-04-25T15:52:42.342250+0000","last_fullsized":"2024-04-25T15:52:42.342250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":97,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207450+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:52:42.207450+0000","last_peered":"2024-04-25T15:52:42.207450+0000","last_clean":"2024-04-25T15:52:42.207450+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:52:42.207450+0000","last_undegraded":"2024-04-25T15:52:42.207450+0000","last_fullsized":"2024-04-25T15:52:42.207450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"59'1","reported_seq":90,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258258+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:52:42.258258+0000","last_peered":"2024-04-25T15:52:42.258258+0000","last_clean":"2024-04-25T15:52:42.258258+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:52:42.258258+0000","last_undegraded":"2024-04-25T15:52:42.258258+0000","last_fullsized":"2024-04-25T15:52:42.258258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":188,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257840+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:52:42.257840+0000","last_peered":"2024-04-25T15:52:42.257840+0000","last_clean":"2024-04-25T15:52:42.257840+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:52:42.257840+0000","last_undegraded":"2024-04-25T15:52:42.257840+0000","last_fullsized":"2024-04-25T15:52:42.257840+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":99,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341604+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:52:42.341604+0000","last_peered":"2024-04-25T15:52:42.341604+0000","last_clean":"2024-04-25T15:52:42.341604+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:52:42.341604+0000","last_undegraded":"2024-04-25T15:52:42.341604+0000","last_fullsized":"2024-04-25T15:52:42.341604+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":92,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338458+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:52:42.338458+0000","last_peered":"2024-04-25T15:52:42.338458+0000","last_clean":"2024-04-25T15:52:42.338458+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:52:42.338458+0000","last_undegraded":"2024-04-25T15:52:42.338458+0000","last_fullsized":"2024-04-25T15:52:42.338458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":114,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257725+0000","last_change":"2024-04-25T15:51:47.292263+0000","last_active":"2024-04-25T15:52:42.257725+0000","last_peered":"2024-04-25T15:52:42.257725+0000","last_clean":"2024-04-25T15:52:42.257725+0000","last_became_active":"2024-04-25T15:51:47.264780+0000","last_became_peered":"2024-04-25T15:51:47.264780+0000","last_unstale":"2024-04-25T15:52:42.257725+0000","last_undegraded":"2024-04-25T15:52:42.257725+0000","last_fullsized":"2024-04-25T15:52:42.257725+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:40.269146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.337872+0000","last_change":"2024-04-25T15:52:32.125579+0000","last_active":"2024-04-25T15:52:42.337872+0000","last_peered":"2024-04-25T15:52:42.337872+0000","last_clean":"2024-04-25T15:52:42.337872+0000","last_became_active":"2024-04-25T15:52:32.125247+0000","last_became_peered":"2024-04-25T15:52:32.125247+0000","last_unstale":"2024-04-25T15:52:42.337872+0000","last_undegraded":"2024-04-25T15:52:42.337872+0000","last_fullsized":"2024-04-25T15:52:42.337872+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:10.258830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":98,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207330+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:52:42.207330+0000","last_peered":"2024-04-25T15:52:42.207330+0000","last_clean":"2024-04-25T15:52:42.207330+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:52:42.207330+0000","last_undegraded":"2024-04-25T15:52:42.207330+0000","last_fullsized":"2024-04-25T15:52:42.207330+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":56,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252533+0000","last_change":"2024-04-25T15:51:47.255396+0000","last_active":"2024-04-25T15:52:42.252533+0000","last_peered":"2024-04-25T15:52:42.252533+0000","last_clean":"2024-04-25T15:52:42.252533+0000","last_became_active":"2024-04-25T15:51:47.254596+0000","last_became_peered":"2024-04-25T15:51:47.254596+0000","last_unstale":"2024-04-25T15:52:42.252533+0000","last_undegraded":"2024-04-25T15:52:42.252533+0000","last_fullsized":"2024-04-25T15:52:42.252533+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:06:53.573154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":120,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341665+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:52:42.341665+0000","last_peered":"2024-04-25T15:52:42.341665+0000","last_clean":"2024-04-25T15:52:42.341665+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:52:42.341665+0000","last_undegraded":"2024-04-25T15:52:42.341665+0000","last_fullsized":"2024-04-25T15:52:42.341665+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.256749+0000","last_change":"2024-04-25T15:52:32.118757+0000","last_active":"2024-04-25T15:52:42.256749+0000","last_peered":"2024-04-25T15:52:42.256749+0000","last_clean":"2024-04-25T15:52:42.256749+0000","last_became_active":"2024-04-25T15:52:32.118380+0000","last_became_peered":"2024-04-25T15:52:32.118380+0000","last_unstale":"2024-04-25T15:52:42.256749+0000","last_undegraded":"2024-04-25T15:52:42.256749+0000","last_fullsized":"2024-04-25T15:52:42.256749+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:18.978899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":56,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252718+0000","last_change":"2024-04-25T15:51:47.263905+0000","last_active":"2024-04-25T15:52:42.252718+0000","last_peered":"2024-04-25T15:52:42.252718+0000","last_clean":"2024-04-25T15:52:42.252718+0000","last_became_active":"2024-04-25T15:51:47.263546+0000","last_became_peered":"2024-04-25T15:51:47.263546+0000","last_unstale":"2024-04-25T15:52:42.252718+0000","last_undegraded":"2024-04-25T15:52:42.252718+0000","last_fullsized":"2024-04-25T15:52:42.252718+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:14.832346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"58'1","reported_seq":92,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.342065+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:52:42.342065+0000","last_peered":"2024-04-25T15:52:42.342065+0000","last_clean":"2024-04-25T15:52:42.342065+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:52:42.342065+0000","last_undegraded":"2024-04-25T15:52:42.342065+0000","last_fullsized":"2024-04-25T15:52:42.342065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258455+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:52:42.258455+0000","last_peered":"2024-04-25T15:52:42.258455+0000","last_clean":"2024-04-25T15:52:42.258455+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:52:42.258455+0000","last_undegraded":"2024-04-25T15:52:42.258455+0000","last_fullsized":"2024-04-25T15:52:42.258455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":56,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252741+0000","last_change":"2024-04-25T15:51:47.263756+0000","last_active":"2024-04-25T15:52:42.252741+0000","last_peered":"2024-04-25T15:52:42.252741+0000","last_clean":"2024-04-25T15:52:42.252741+0000","last_became_active":"2024-04-25T15:51:47.263130+0000","last_became_peered":"2024-04-25T15:51:47.263130+0000","last_unstale":"2024-04-25T15:52:42.252741+0000","last_undegraded":"2024-04-25T15:52:42.252741+0000","last_fullsized":"2024-04-25T15:52:42.252741+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:38.024802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"59'10","reported_seq":67,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.253124+0000","last_change":"2024-04-25T15:51:47.314396+0000","last_active":"2024-04-25T15:52:42.253124+0000","last_peered":"2024-04-25T15:52:42.253124+0000","last_clean":"2024-04-25T15:52:42.253124+0000","last_became_active":"2024-04-25T15:51:47.274984+0000","last_became_peered":"2024-04-25T15:51:47.274984+0000","last_unstale":"2024-04-25T15:52:42.253124+0000","last_undegraded":"2024-04-25T15:52:42.253124+0000","last_fullsized":"2024-04-25T15:52:42.253124+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:53.936857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"58'1","reported_seq":92,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.339029+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:52:42.339029+0000","last_peered":"2024-04-25T15:52:42.339029+0000","last_clean":"2024-04-25T15:52:42.339029+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:52:42.339029+0000","last_undegraded":"2024-04-25T15:52:42.339029+0000","last_fullsized":"2024-04-25T15:52:42.339029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":100,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258051+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:52:42.258051+0000","last_peered":"2024-04-25T15:52:42.258051+0000","last_clean":"2024-04-25T15:52:42.258051+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:52:42.258051+0000","last_undegraded":"2024-04-25T15:52:42.258051+0000","last_fullsized":"2024-04-25T15:52:42.258051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":93,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252942+0000","last_change":"2024-04-25T15:51:59.936821+0000","last_active":"2024-04-25T15:52:42.252942+0000","last_peered":"2024-04-25T15:52:42.252942+0000","last_clean":"2024-04-25T15:52:42.252942+0000","last_became_active":"2024-04-25T15:51:59.936474+0000","last_became_peered":"2024-04-25T15:51:59.936474+0000","last_unstale":"2024-04-25T15:52:42.252942+0000","last_undegraded":"2024-04-25T15:52:42.252942+0000","last_fullsized":"2024-04-25T15:52:42.252942+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:05.147722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":94,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.257730+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:52:42.257730+0000","last_peered":"2024-04-25T15:52:42.257730+0000","last_clean":"2024-04-25T15:52:42.257730+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:52:42.257730+0000","last_undegraded":"2024-04-25T15:52:42.257730+0000","last_fullsized":"2024-04-25T15:52:42.257730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207073+0000","last_change":"2024-04-25T15:52:32.131460+0000","last_active":"2024-04-25T15:52:42.207073+0000","last_peered":"2024-04-25T15:52:42.207073+0000","last_clean":"2024-04-25T15:52:42.207073+0000","last_became_active":"2024-04-25T15:52:32.131087+0000","last_became_peered":"2024-04-25T15:52:32.131087+0000","last_unstale":"2024-04-25T15:52:42.207073+0000","last_undegraded":"2024-04-25T15:52:42.207073+0000","last_fullsized":"2024-04-25T15:52:42.207073+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:14.840887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":66,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252651+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:52:42.252651+0000","last_peered":"2024-04-25T15:52:42.252651+0000","last_clean":"2024-04-25T15:52:42.252651+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:52:42.252651+0000","last_undegraded":"2024-04-25T15:52:42.252651+0000","last_fullsized":"2024-04-25T15:52:42.252651+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"59'2","reported_seq":103,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.795968+0000","last_change":"2024-04-25T15:51:47.270024+0000","last_active":"2024-04-25T15:52:42.795968+0000","last_peered":"2024-04-25T15:52:42.795968+0000","last_clean":"2024-04-25T15:52:42.795968+0000","last_became_active":"2024-04-25T15:51:47.269733+0000","last_became_peered":"2024-04-25T15:51:47.269733+0000","last_unstale":"2024-04-25T15:52:42.795968+0000","last_undegraded":"2024-04-25T15:52:42.795968+0000","last_fullsized":"2024-04-25T15:52:42.795968+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:23.505183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":62,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.252855+0000","last_change":"2024-04-25T15:51:47.286937+0000","last_active":"2024-04-25T15:52:42.252855+0000","last_peered":"2024-04-25T15:52:42.252855+0000","last_clean":"2024-04-25T15:52:42.252855+0000","last_became_active":"2024-04-25T15:51:47.266040+0000","last_became_peered":"2024-04-25T15:51:47.266040+0000","last_unstale":"2024-04-25T15:52:42.252855+0000","last_undegraded":"2024-04-25T15:52:42.252855+0000","last_fullsized":"2024-04-25T15:52:42.252855+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:48.792187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.207847+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:52:42.207847+0000","last_peered":"2024-04-25T15:52:42.207847+0000","last_clean":"2024-04-25T15:52:42.207847+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:52:42.207847+0000","last_undegraded":"2024-04-25T15:52:42.207847+0000","last_fullsized":"2024-04-25T15:52:42.207847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":106,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258255+0000","last_change":"2024-04-25T15:52:32.159964+0000","last_active":"2024-04-25T15:52:42.258255+0000","last_peered":"2024-04-25T15:52:42.258255+0000","last_clean":"2024-04-25T15:52:42.258255+0000","last_became_active":"2024-04-25T15:52:32.134918+0000","last_became_peered":"2024-04-25T15:52:32.134918+0000","last_unstale":"2024-04-25T15:52:42.258255+0000","last_undegraded":"2024-04-25T15:52:42.258255+0000","last_fullsized":"2024-04-25T15:52:42.258255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:26:55.807079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.256496+0000","last_change":"2024-04-25T15:52:32.126151+0000","last_active":"2024-04-25T15:52:42.256496+0000","last_peered":"2024-04-25T15:52:42.256496+0000","last_clean":"2024-04-25T15:52:42.256496+0000","last_became_active":"2024-04-25T15:52:32.125517+0000","last_became_peered":"2024-04-25T15:52:32.125517+0000","last_unstale":"2024-04-25T15:52:42.256496+0000","last_undegraded":"2024-04-25T15:52:42.256496+0000","last_fullsized":"2024-04-25T15:52:42.256496+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:13.443149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"58'1","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.341895+0000","last_change":"2024-04-25T15:52:32.129077+0000","last_active":"2024-04-25T15:52:42.341895+0000","last_peered":"2024-04-25T15:52:42.341895+0000","last_clean":"2024-04-25T15:52:42.341895+0000","last_became_active":"2024-04-25T15:52:32.128629+0000","last_became_peered":"2024-04-25T15:52:32.128629+0000","last_unstale":"2024-04-25T15:52:42.341895+0000","last_undegraded":"2024-04-25T15:52:42.341895+0000","last_fullsized":"2024-04-25T15:52:42.341895+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:05:51.807066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258122+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:52:42.258122+0000","last_peered":"2024-04-25T15:52:42.258122+0000","last_clean":"2024-04-25T15:52:42.258122+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:52:42.258122+0000","last_undegraded":"2024-04-25T15:52:42.258122+0000","last_fullsized":"2024-04-25T15:52:42.258122+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"58'1","reported_seq":96,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.342002+0000","last_change":"2024-04-25T15:52:32.132711+0000","last_active":"2024-04-25T15:52:42.342002+0000","last_peered":"2024-04-25T15:52:42.342002+0000","last_clean":"2024-04-25T15:52:42.342002+0000","last_became_active":"2024-04-25T15:52:32.132406+0000","last_became_peered":"2024-04-25T15:52:32.132406+0000","last_unstale":"2024-04-25T15:52:42.342002+0000","last_undegraded":"2024-04-25T15:52:42.342002+0000","last_fullsized":"2024-04-25T15:52:42.342002+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:30.616697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":103,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258359+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:52:42.258359+0000","last_peered":"2024-04-25T15:52:42.258359+0000","last_clean":"2024-04-25T15:52:42.258359+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:52:42.258359+0000","last_undegraded":"2024-04-25T15:52:42.258359+0000","last_fullsized":"2024-04-25T15:52:42.258359+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"59'1","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.342006+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:52:42.342006+0000","last_peered":"2024-04-25T15:52:42.342006+0000","last_clean":"2024-04-25T15:52:42.342006+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:52:42.342006+0000","last_undegraded":"2024-04-25T15:52:42.342006+0000","last_fullsized":"2024-04-25T15:52:42.342006+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":95,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338892+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:52:42.338892+0000","last_peered":"2024-04-25T15:52:42.338892+0000","last_clean":"2024-04-25T15:52:42.338892+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:52:42.338892+0000","last_undegraded":"2024-04-25T15:52:42.338892+0000","last_fullsized":"2024-04-25T15:52:42.338892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":91,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.338881+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:52:42.338881+0000","last_peered":"2024-04-25T15:52:42.338881+0000","last_clean":"2024-04-25T15:52:42.338881+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:52:42.338881+0000","last_undegraded":"2024-04-25T15:52:42.338881+0000","last_fullsized":"2024-04-25T15:52:42.338881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":70,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-25T15:52:42.258822+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:52:42.258822+0000","last_peered":"2024-04-25T15:52:42.258822+0000","last_clean":"2024-04-25T15:52:42.258822+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:52:42.258822+0000","last_undegraded":"2024-04-25T15:52:42.258822+0000","last_fullsized":"2024-04-25T15:52:42.258822+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167772160,"data_stored":167772160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":36197,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":229,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4003,"num_keys_recovered":22,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":74165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":200,"ondisk_log_size":200,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40584,"kb_used_data":13568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331256,"statfs":{"total":96636764160,"available":96595206144,"internally_reserved":0,"allocated":13893632,"data_stored":13478975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":321007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56408,"kb_used_data":29388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94315432,"statfs":{"total":96636764160,"available":96579002368,"internally_reserved":0,"allocated":30093312,"data_stored":29673116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574881,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52328,"kb_used_data":25308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94319512,"statfs":{"total":96636764160,"available":96583180288,"internally_reserved":0,"allocated":25915392,"data_stored":25500900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40008,"kb_used_data":12988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331832,"statfs":{"total":96636764160,"available":96595795968,"internally_reserved":0,"allocated":13299712,"data_stored":12893430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35916,"kb_used_data":8892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335924,"statfs":{"total":96636764160,"available":96599986176,"internally_reserved":0,"allocated":9105408,"data_stored":8699784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36544,"kb_used_data":5360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335296,"statfs":{"total":96636764160,"available":96599343104,"internally_reserved":0,"allocated":5488640,"data_stored":5088096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":29855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":29683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29360128,"data_stored":29360128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29360128,"data_stored":29360128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37748736,"data_stored":37748736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25165824,"data_stored":25165824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20971520,"data_stored":20971520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:52:48.187 INFO:tasks.thrashosds.thrasher:Setting 3.18 to [1, 4] 2024-04-25T15:52:48.187 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.18', '1', '4'] 2024-04-25T15:52:48.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.18 1 4 2024-04-25T15:52:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.178+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec152570 con 0x7fe3ec02c9e0 2024-04-25T15:52:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.182+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec152570 con 0x7fe3ec02c9e0 2024-04-25T15:52:48.189 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.182+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe3ec02d840 0x7fe3ec20c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.182+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec152570 con 0x7fe3ec02d840 2024-04-25T15:52:48.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.182+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe3ec02d840 0x7fe3ec20c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.186+0000 7fe3eb7fe640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe3ec02d840 0x7fe3ec20c610 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.186+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec152570 con 0x7fe3ec02d840 2024-04-25T15:52:48.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.206+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec152570 con 0x7fe3ec02d840 2024-04-25T15:52:48.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.206+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec152570 con 0x7fe3ec02d840 2024-04-25T15:52:48.212 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.206+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe3ec02dc50 0x7fe3ec20f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.206+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec211ef0 con 0x7fe3ec02dc50 2024-04-25T15:52:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.206+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe3ec02dc50 0x7fe3ec20f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.210+0000 7fe3f0f5a640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe3ec02dc50 0x7fe3ec20f070 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.210+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec211ef0 con 0x7fe3ec02dc50 2024-04-25T15:52:48.217 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:48.210+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.226+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec20f880 con 0x7fe3ec02dc50 2024-04-25T15:52:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.226+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec20f880 con 0x7fe3ec02dc50 2024-04-25T15:52:48.233 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:48.233 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.226+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe3ec20ffa0 0x7fe3ec2103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.226+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe3ec215340 con 0x7fe3ec20ffa0 2024-04-25T15:52:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.230+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe3ec20ffa0 0x7fe3ec2103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.230+0000 7fe3ebfff640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe3ec20ffa0 0x7fe3ec2103b0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.230+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3ec215340 con 0x7fe3ec20ffa0 2024-04-25T15:52:48.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.250+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3ec214140 con 0x7fe3ec20ffa0 2024-04-25T15:52:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3e8ff9640 1 -- 172.21.15.67:0/2073643489 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3ec214140 con 0x7fe3ec20ffa0 2024-04-25T15:52:48.258 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe3ec155680 msgr2=0x7fe3ec1487a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe3ec155680 0x7fe3ec1487a0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe3ec1f9280 msgr2=0x7fe3ec1f9690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe3ec1f9280 0x7fe3ec1f9690 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe3ec063090 msgr2=0x7fe3ec0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe3ec063090 0x7fe3ec0634a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe3ec151710 msgr2=0x7fe3ec151b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe3ec151710 0x7fe3ec151b20 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f0f5a640 1 -- 172.21.15.67:0/2073643489 reap_dead start 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe3ec02c9e0 msgr2=0x7fe3ec02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe3ec02c9e0 0x7fe3ec02cdf0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f0f5a640 1 -- 172.21.15.67:0/2073643489 reap_dead start 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe3ec02d840 msgr2=0x7fe3ec20c610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe3ec02d840 0x7fe3ec20c610 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe3ec02dc50 msgr2=0x7fe3ec20f070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe3ec02dc50 0x7fe3ec20f070 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe3ec20ffa0 msgr2=0x7fe3ec2103b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe3ec20ffa0 0x7fe3ec2103b0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe3d806f400 msgr2=0x7fe3d8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe3d806f400 0x7fe3d8071910 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0ae000 tx=0x7fe3dc008040 comp rx=0 tx=0).stop 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec15fe00 msgr2=0x7fe3ec15eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 --2- 172.21.15.67:0/2073643489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ec15fe00 0x7fe3ec15eed0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4007ef0 tx=0x7fe3e400d4a0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f0f5a640 1 -- 172.21.15.67:0/2073643489 reap_dead start 2024-04-25T15:52:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 shutdown_connections 2024-04-25T15:52:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 >> 172.21.15.67:0/2073643489 conn(0x7fe3ec06e420 msgr2=0x7fe3ec0cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 shutdown_connections 2024-04-25T15:52:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.254+0000 7fe3f29bd640 1 -- 172.21.15.67:0/2073643489 wait complete. 2024-04-25T15:52:48.325 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:48.322+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6d528c640 1 Processor -- start 2024-04-25T15:52:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6d528c640 1 -- start start 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6d528c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 0x7fa6d00cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6d528c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d015e510 con 0x7fa6d00ccb40 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6d528c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d00ca8a0 con 0x7fa6d0154ac0 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6d528c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d00cab80 con 0x7fa6d00cbe90 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6cf7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa6d00ccb40 0x7fa6d0165190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44676/0 (socket says 172.21.15.67:44676) 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/66640266 learned_addr learned my addr 172.21.15.67:0/66640266 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6c7fff640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 0x7fa6d00cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1275015457 0 0) 0x7fa6d015e510 con 0x7fa6d00ccb40 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6bc003530 con 0x7fa6d00ccb40 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3812093595 0 0) 0x7fa6bc003530 con 0x7fa6d00ccb40 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 msgr2=0x7fa6d00cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 0x7fa6d00cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 >> v1:172.21.15.176:6789/0 conn(0x7fa6d0154ac0 legacy=0x7fa6d015de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d015a920 con 0x7fa6d00ccb40 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fa6c8001a00 con 0x7fa6d00ccb40 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 >> v1:172.21.15.67:6789/0 conn(0x7fa6d00ccb40 legacy=0x7fa6d0165190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 0x7fa6bc0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6bc003f70 0x7fa6bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6bc004d50 0x7fa6bc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d00cab80 con 0x7fa6bc004d50 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa6bc00bc10 con 0x7fa6bc003f70 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa6bc00be10 con 0x7fa6d00cbe90 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.346+0000 7fa6cf7fe640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6bc004d50 0x7fa6bc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6ceffd640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 0x7fa6bc0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6c7fff640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6bc003f70 0x7fa6bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 msgr2=0x7fa6bc0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6cf7fe640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00cbe90 0x7fa6bc0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6bc003f70 msgr2=0x7fa6bc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:48.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6cf7fe640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6bc003f70 0x7fa6bc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/66640266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6bc003530 con 0x7fa6bc004d50 2024-04-25T15:52:48.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6cf7fe640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6bc004d50 0x7fa6bc005160 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00cda40 tx=0x7fa6c801fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ce18363bf954c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800a480 con 0x7fa6bc004d50 2024-04-25T15:52:48.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c80aa6b0 con 0x7fa6bc004d50 2024-04-25T15:52:48.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6ce7fc640 1 -- 172.21.15.67:0/66640266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80aab20 con 0x7fa6bc004d50 2024-04-25T15:52:48.357 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:48.357 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:48.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:18 2.14 2:2f55791f:::606.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 132.142582506, 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013142638999999999, 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:48.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210634+0000", 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210641+0000", 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999999e-06 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210657+0000", 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5630000000000001e-05 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210829+0000", 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017241399999999999 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210850+0000", 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0309000000000002e-05 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211131+0000", 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028102699999999999 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211418+0000", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028696199999999998 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211691+0000", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027375799999999998 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218973+0000", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072812789999999999 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223653+0000", 2024-04-25T15:52:48.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046802399999999996 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223701+0000", 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8157e-05 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223777+0000", 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.362 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:19 2.14 2:2f55791f:::606.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 132.14255324199999, 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013295688, 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210663+0000", 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210664+0000", 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3499999999999995e-07 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210670+0000", 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6450000000000002e-06 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210916+0000", 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024579000000000002 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210930+0000", 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4285e-05 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212370+0000", 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001440439 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212542+0000", 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017121200000000001 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212818+0000", 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027659400000000002 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218979+0000", 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061603500000000002 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223854+0000", 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048753010000000003 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223912+0000", 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8260999999999997e-05 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223959+0000", 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7076000000000001e-05 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:25 2.7 2:e0b41d06:::609.00000000:head [create,setxattr parent (62) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 132.14253637799999, 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692059, 2024-04-25T15:52:48.367 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210680+0000", 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210681+0000", 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8500000000000001e-07 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210687+0000", 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.755e-06 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211584+0000", 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089709299999999996 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211596+0000", 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2532e-05 2024-04-25T15:52:48.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212680+0000", 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010839109999999999 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212802+0000", 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012158 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.212942+0000", 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140016 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.215584+0000", 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002642227 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223210+0000", 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076260520000000004 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223304+0000", 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4077000000000001e-05 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.223372+0000", 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8131000000000002e-05 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1 2.14 2:292cf221:::200.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 132.142527745, 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01448253, 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210689+0000", 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210727+0000", 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8346000000000002e-05 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210735+0000", 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6170000000000001e-06 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211652+0000", 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00091682999999999999 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211665+0000", 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3586e-05 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213308+0000", 2024-04-25T15:52:48.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00164266 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213550+0000", 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024244499999999999 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.213808+0000", 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025752799999999998 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218983+0000", 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051748239999999997 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225041+0000", 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060579989999999997 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225096+0000", 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5519000000000002e-05 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225172+0000", 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5176e-05 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.376 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:3 2.14 2:287034c4:::1.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 132.142477912, 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014645415, 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210739+0000", 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210740+0000", 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1099999999999995e-07 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210744+0000", 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.763e-06 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211689+0000", 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094511200000000001 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211700+0000", 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0181e-05 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217043+0000", 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005342996 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217377+0000", 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033410699999999999 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.217611+0000", 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023468600000000001 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.220697+0000", 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030851229999999999 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225258+0000", 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045617909999999999 2024-04-25T15:52:48.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225316+0000", 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7926000000000001e-05 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225384+0000", 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8019e-05 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:32 2.14 2:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:48.381 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 132.14247003400001, 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014868523999999999, 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210747+0000", 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2600000000000002e-07 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.210770+0000", 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2792000000000001e-05 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211722+0000", 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095205500000000005 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.211733+0000", 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0694e-05 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218331+0000", 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065981240000000003 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.218536+0000", 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020543300000000001 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:36.225615+0000", 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070787999999999997 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.1.10:1 2.17 2:ea85266a:::201.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 126.105959764, 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012270133000000001, 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247257+0000", 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247258+0000", 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0299999999999998e-07 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.247263+0000", 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4269999999999999e-06 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248075+0000", 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00081147299999999995 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.248103+0000", 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7881999999999998e-05 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251834+0000", 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003731584 2024-04-25T15:52:48.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.251988+0000", 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.252187+0000", 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199359 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.256968+0000", 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047802180000000001 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259440+0000", 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024725720000000001 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259478+0000", 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7790000000000002e-05 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:42.259527+0000", 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9066000000000001e-05 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:40 3.2e e59/18 3:75eea580:::10000000002.00000024:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.754004+0000", 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.5992131450000002, 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016142111000000001, 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.754004+0000", 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.754004+0000", 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.754004+0000", 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8999999999999995e-07 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.758726+0000", 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047215269999999997 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.758757+0000", 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0923000000000002e-05 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.758781+0000", 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3604e-05 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.758886+0000", 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105227 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.758920+0000", 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4567000000000003e-05 2024-04-25T15:52:48.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.770067+0000", 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011146445 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.770146+0000", 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8928000000000001e-05 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.393 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:43 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.4512003059999996, 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024229051000000001, 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902019+0000", 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9459999999999998e-06 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907768+0000", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005748525 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907831+0000", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2842999999999994e-05 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907857+0000", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6636e-05 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908056+0000", 2024-04-25T15:52:48.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198295 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908464+0000", 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040813600000000002 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908853+0000", 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038892400000000002 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925332+0000", 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016479074 2024-04-25T15:52:48.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925882+0000", 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000550232 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926050+0000", 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016822199999999999 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926245+0000", 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019521799999999999 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:53 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.9750181449999999, 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020793931000000002, 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.399 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 53 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378202+0000", 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1669999999999999e-06 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382531+0000", 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043289130000000002 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382593+0000", 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2143000000000005e-05 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382618+0000", 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382814+0000", 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195964 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383075+0000", 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026081800000000001 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:48.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383324+0000", 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024972200000000002 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.392258+0000", 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089338549999999992 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398665+0000", 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064066649999999998 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398841+0000", 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017607899999999999 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398993+0000", 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015144799999999999 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:55 3.10 3:09288c7b:::10000000002.00000033:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.8814575690000002, 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018106206999999999, 2024-04-25T15:52:48.404 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 55 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471763+0000", 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.794e-06 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476322+0000", 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045595009999999997 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476383+0000", 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0936000000000001e-05 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476410+0000", 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6313e-05 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476613+0000", 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020343499999999999 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476897+0000", 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028380400000000002 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.477152+0000", 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521799999999998 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.484642+0000", 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074893970000000001 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489677+0000", 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050351010000000002 2024-04-25T15:52:48.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489779+0000", 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102592 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489865+0000", 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6116e-05 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:57 3.3 3:c0bc6424:::10000000002.00000035:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.409 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.7951834450000002, 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017860478999999999, 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 57 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:48.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558036+0000", 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9799999999999998e-06 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562071+0000", 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040351400000000004 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562126+0000", 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4249000000000002e-05 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562149+0000", 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3674000000000001e-05 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562327+0000", 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177498 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562581+0000", 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025446799999999999 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562836+0000", 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000254751 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.572670+0000", 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0098337700000000004 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575703+0000", 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030327980000000002 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575802+0000", 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9278999999999996e-05 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575894+0000", 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1872000000000004e-05 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:59 3.20 3:072133bc:::10000000002.00000037:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.719847074, 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016873692999999999, 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 59 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633373+0000", 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2430000000000001e-06 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637297+0000", 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039241060000000001 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637354+0000", 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7357e-05 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637380+0000", 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5165999999999999e-05 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637564+0000", 2024-04-25T15:52:48.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000184332 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637837+0000", 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027320399999999998 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.638090+0000", 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025292100000000003 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.647362+0000", 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092716989999999996 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650052+0000", 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026907599999999999 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650151+0000", 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8603000000000004e-05 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650243+0000", 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2301999999999998e-05 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:62 3.11 e60/54 3:8969f477:::10000000002.0000003a:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.5915332520000001, 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019322214000000001, 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761684+0000", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3399999999999998e-07 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765759+0000", 2024-04-25T15:52:48.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040747159999999999 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765801+0000", 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1675999999999999e-05 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765835+0000", 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4518999999999999e-05 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765978+0000", 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014263899999999999 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.766025+0000", 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6842999999999999e-05 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780902+0000", 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014877118 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781006+0000", 2024-04-25T15:52:48.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010396899999999999 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:73 3.1d e60/54 3:b94578cb:::10000000002.00000045:head v 60'2, mlcod=0'0)", 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.0898078519999999, 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019180661000000002, 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263412+0000", 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9900000000000002e-06 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268799+0000", 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053871980000000002 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268844+0000", 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5062999999999999e-05 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268870+0000", 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5633e-05 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269028+0000", 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000158135 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269078+0000", 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0463000000000001e-05 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282434+0000", 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013355156999999999 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282590+0000", 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015602199999999999 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:75 3.35 3:ad096b5b:::10000000002.00000047:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.0197076049999998, 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022276672000000001, 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333513+0000", 2024-04-25T15:52:48.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339006+0000", 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054929599999999999 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339071+0000", 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5545999999999999e-05 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339098+0000", 2024-04-25T15:52:48.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6647000000000001e-05 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339299+0000", 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201351 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339703+0000", 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040396399999999998 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.340160+0000", 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045697899999999999 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.352668+0000", 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012508137000000001 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355605+0000", 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002937013 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355701+0000", 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5681999999999996e-05 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355786+0000", 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4908999999999996e-05 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:77 3.3f e60/18 3:ff5d5af4:::10000000002.00000049:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.415128+0000", 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 2.938088327, 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014176997, 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.433 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.415128+0000", 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.415128+0000", 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.415131+0000", 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.17e-06 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.418863+0000", 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037323819999999998 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.418902+0000", 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8615999999999998e-05 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.418929+0000", 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7393e-05 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.419085+0000", 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015598899999999999 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.419130+0000", 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4869e-05 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.429201+0000", 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010071645000000001 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.429305+0000", 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103933 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:79 3.3 3:c0361aff:::10000000002.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 2.8439375099999999, 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022495096999999999, 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.437 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 79 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509282+0000", 2024-04-25T15:52:48.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2200000000000001e-06 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517219+0000", 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079362270000000006 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517284+0000", 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5052e-05 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517310+0000", 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6072999999999999e-05 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517506+0000", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019606199999999999 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517779+0000", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027302799999999998 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.518047+0000", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000268454 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.528734+0000", 2024-04-25T15:52:48.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01068712 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531586+0000", 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028516589999999999 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531689+0000", 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010260900000000001 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531774+0000", 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5593000000000005e-05 2024-04-25T15:52:48.441 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:80 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 2.8013718760000001, 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018714178000000001, 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.442 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 80 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551848+0000", 2024-04-25T15:52:48.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0230000000000001e-06 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556855+0000", 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050073180000000002 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556907+0000", 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1446000000000002e-05 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556929+0000", 2024-04-25T15:52:48.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2405999999999999e-05 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557100+0000", 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000171003 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557387+0000", 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028688699999999999 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557640+0000", 2024-04-25T15:52:48.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025292499999999998 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.566164+0000", 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085239770000000003 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570375+0000", 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004211345 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570471+0000", 2024-04-25T15:52:48.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5933000000000005e-05 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570559+0000", 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7915000000000001e-05 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:82 3.34 3:2c8ffcca:::10000000002.0000004e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 2.7135165489999999, 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017224060999999999, 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:48.447 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 82 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:48.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639703+0000", 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7070000000000001e-06 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644603+0000", 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048998549999999998 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644658+0000", 2024-04-25T15:52:48.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4987000000000001e-05 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644682+0000", 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4280000000000001e-05 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644876+0000", 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000193678 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.645141+0000", 2024-04-25T15:52:48.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000265321 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.645394+0000", 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025291099999999998 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.655180+0000", 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0097862139999999997 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656739+0000", 2024-04-25T15:52:48.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015587800000000001 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656840+0000", 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101172 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656924+0000", 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4155999999999999e-05 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:48.452 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:48.454 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:48.455 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:48.451+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6d528c640 1 -- 172.21.15.67:0/66640266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6bc004d50 msgr2=0x7fa6bc005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.350+0000 7fa6d528c640 1 --2- 172.21.15.67:0/66640266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6bc004d50 0x7fa6bc005160 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00cda40 tx=0x7fa6c801fa80 comp rx=0 tx=0).stop 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.354+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/66640266 reap_dead start 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.354+0000 7fa6d528c640 1 -- 172.21.15.67:0/66640266 shutdown_connections 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.354+0000 7fa6d528c640 1 -- 172.21.15.67:0/66640266 >> 172.21.15.67:0/66640266 conn(0x7fa6d0068c70 msgr2=0x7fa6d0071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.354+0000 7fa6d528c640 1 -- 172.21.15.67:0/66640266 shutdown_connections 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.354+0000 7fa6d528c640 1 -- 172.21.15.67:0/66640266 wait complete. 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.354+0000 7fa6d528c640 1 Processor -- start 2024-04-25T15:52:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.354+0000 7fa6d528c640 1 -- start start 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6d00ce620 0x7fa6d014eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00c54a0 0x7fa6d0150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d01f61e0 0x7fa6d01f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d015b1c0 con 0x7fa6d01f61e0 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d015aee0 con 0x7fa6d00ce620 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d0154ed0 con 0x7fa6d00c54a0 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d01f61e0 0x7fa6d01f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d01f61e0 0x7fa6d01f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49168/0 (socket says 172.21.15.67:49168) 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/227718093 learned_addr learned my addr 172.21.15.67:0/227718093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c7fff640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00c54a0 0x7fa6d0150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6ceffd640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6d00ce620 0x7fa6d014eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00c54a0 msgr2=0x7fa6d0150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00c54a0 0x7fa6d0150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6d00ce620 msgr2=0x7fa6d014eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6d00ce620 0x7fa6d014eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 -- 172.21.15.67:0/227718093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d006d8f0 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c7fff640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00c54a0 0x7fa6d0150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6ceffd640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6d00ce620 0x7fa6d014eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6cf7fe640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d01f61e0 0x7fa6d01f65f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8020320 tx=0x7fa6bc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c77fe640 1 -- 172.21.15.67:0/227718093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800a480 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c77fe640 1 -- 172.21.15.67:0/227718093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c80aa4b0 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d006f2c0 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c77fe640 1 -- 172.21.15.67:0/227718093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8001a00 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d006ea90 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6d006a480 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c77fe640 1 -- 172.21.15.67:0/227718093 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa6c80ba030 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c77fe640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa6b006f400 0x7fa6b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.358+0000 7fa6c77fe640 1 -- 172.21.15.67:0/227718093 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5857+0+0 (secure 0 0 0) 0x7fa6c812bf30 con 0x7fa6d01f61e0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.362+0000 7fa6ceffd640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa6b006f400 0x7fa6b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.362+0000 7fa6ceffd640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa6b006f400 0x7fa6b0071910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c10 tx=0x7fa6c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.366+0000 7fa6c77fe640 1 -- 172.21.15.67:0/227718093 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa6c80ae030 con 0x7fa6d01f61e0 2024-04-25T15:52:48.554 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:48.551+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:48.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:48.586+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "3.18", "id": [1, 4]} v 0) -- 0x7fa6d00cb260 con 0x7fa6d01f61e0 2024-04-25T15:52:48.652 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:48.652 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:48.653 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:48.653 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:48.655 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:48.655+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:48.675 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:48.767 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:48.767+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:48.857 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:48.850+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:48.875 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:48.875 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:48.875 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:48.875 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:48.875 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:48.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:48.987 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:48.987+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:49.059 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:49.054+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:49.102 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:49.102 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:49.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:49.102 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:49.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 150.00025020999999, 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.108 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.40921747499999, 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019412740000000001, 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 48 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689497+0000", 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689498+0000", 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5999999999999991e-07 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689505+0000", 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1339999999999998e-06 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689885+0000", 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000380553 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689894+0000", 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0660000000000004e-06 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703735+0000", 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013841509 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.703880+0000", 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014450899999999999 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704121+0000", 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024184799999999999 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708286+0000", 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041643369999999997 2024-04-25T15:52:49.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708530+0000", 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024477400000000001 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708710+0000", 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179715 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708910+0000", 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199335 2024-04-25T15:52:49.113 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.40920546500001, 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019597107999999999, 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.114 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 49 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689509+0000", 2024-04-25T15:52:49.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4700000000000001e-07 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689515+0000", 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3770000000000004e-06 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689911+0000", 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039616199999999998 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689920+0000", 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2939999999999994e-06 2024-04-25T15:52:49.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704840+0000", 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014919412999999999 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.704990+0000", 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150388 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.705238+0000", 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000248392 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708485+0000", 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032464740000000001 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.708936+0000", 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045072599999999998 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709037+0000", 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101482 2024-04-25T15:52:49.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709106+0000", 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8752999999999997e-05 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.40919590799999, 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.119 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:52:49.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.409047286, 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.124 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:49.129 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.40903620700001, 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:52:49.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.10191978500001, 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019380720000000001, 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.135 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 71 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996794+0000", 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996796+0000", 2024-04-25T15:52:49.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.063e-06 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996810+0000", 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4100000000000001e-05 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997061+0000", 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000251797 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997076+0000", 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4970000000000001e-05 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003008+0000", 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005931487 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003148+0000", 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140097 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.003371+0000", 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022273400000000001 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009117+0000", 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057466929999999998 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016106+0000", 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069880919999999996 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016144+0000", 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8686000000000002e-05 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016175+0000", 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1000999999999997e-05 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.101897119, 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019495743999999999, 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.140 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996817+0000", 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996818+0000", 2024-04-25T15:52:49.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.046e-06 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.996944+0000", 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012572199999999999 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997139+0000", 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019529899999999999 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997151+0000", 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2116000000000001e-05 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004407+0000", 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072560519999999998 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004474+0000", 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6810000000000006e-05 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.004576+0000", 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101431 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009121+0000", 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045451099999999998 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016239+0000", 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071178359999999998 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016280+0000", 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0914000000000002e-05 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.016313+0000", 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3408000000000003e-05 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.10149057300001, 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020191183000000001, 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.145 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997224+0000", 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0999999999999998e-07 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997376+0000", 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015132899999999999 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997528+0000", 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152275 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997538+0000", 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0137e-05 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007159+0000", 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096209090000000004 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007248+0000", 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9078000000000003e-05 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.007394+0000", 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014618699999999999 2024-04-25T15:52:49.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009133+0000", 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001739008 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017311+0000", 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081775159999999993 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017360+0000", 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9202000000000001e-05 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.017415+0000", 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4932000000000002e-05 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 149.101333168, 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.150 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:37 3.3b 3:ddac2f4e:::10000000002.00000021:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.606740+0000", 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.4919746370000002, 2024-04-25T15:52:49.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020101346999999999, 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 37 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.606740+0000", 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.606740+0000", 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.606742+0000", 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4590000000000001e-06 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611280+0000", 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00453792 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611315+0000", 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5033000000000002e-05 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611329+0000", 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3481e-05 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611482+0000", 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000153857 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611751+0000", 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026831799999999997 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.611967+0000", 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021583799999999999 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.622715+0000", 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010748186 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.626645+0000", 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039301199999999996 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.626751+0000", 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010613200000000001 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.626841+0000", 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0002999999999999e-05 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:41 3.f 3:f045ad74:::10000000002.00000025:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.2892489779999998, 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022153520999999999, 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 41 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809469+0000", 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7670000000000001e-06 2024-04-25T15:52:49.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814177+0000", 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047077969999999997 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814211+0000", 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4173e-05 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814226+0000", 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5444e-05 2024-04-25T15:52:49.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814394+0000", 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016725400000000001 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814776+0000", 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038195999999999998 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.815181+0000", 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040559299999999999 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.824339+0000", 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0091577129999999996 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831521+0000", 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071817010000000004 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831568+0000", 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7250000000000003e-05 2024-04-25T15:52:49.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831619+0000", 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0868999999999997e-05 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:47 3.12 3:489b4446:::10000000002.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.9981904410000002, 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022914499000000001, 2024-04-25T15:52:49.166 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100528+0000", 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3030000000000001e-06 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105104+0000", 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045757710000000002 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105167+0000", 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2936999999999997e-05 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105194+0000", 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7606e-05 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105424+0000", 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229998 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105705+0000", 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000280408 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105966+0000", 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261258 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.118921+0000", 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012954482999999999 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123247+0000", 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043263199999999998 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123350+0000", 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103423 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123438+0000", 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7991999999999994e-05 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:52 3.12 3:499536aa:::10000000002.00000030:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.330550+0000", 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.7681640160000001, 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019570648, 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.330550+0000", 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.330550+0000", 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.330554+0000", 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5329999999999999e-06 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.335008+0000", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044545260000000003 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.335072+0000", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3985999999999994e-05 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.335106+0000", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3880000000000001e-05 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.335284+0000", 2024-04-25T15:52:49.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017799800000000001 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.335571+0000", 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028668200000000002 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.335838+0000", 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026703700000000003 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.348862+0000", 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013023853 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.349873+0000", 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010111790000000001 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.350019+0000", 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146262 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.350121+0000", 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101712 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:62 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3430613149999999, 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025889728000000001, 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 62 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755657+0000", 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9439999999999998e-06 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760593+0000", 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049360740000000004 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760651+0000", 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8e-05 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760676+0000", 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5219999999999999e-05 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760847+0000", 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017057199999999999 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761123+0000", 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027578100000000001 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761365+0000", 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024248100000000001 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780642+0000", 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019277327 2024-04-25T15:52:49.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781358+0000", 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000715611 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781456+0000", 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7946e-05 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781543+0000", 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6772000000000002e-05 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:72 3.39 3:9df4b1b7:::10000000002.00000044:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.8829878989999997, 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022151042999999999, 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:52:49.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215730+0000", 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:52:49.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221532+0000", 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058028009999999998 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221606+0000", 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3842000000000002e-05 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221642+0000", 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6232999999999997e-05 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221851+0000", 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020811199999999999 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.222277+0000", 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042676200000000002 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.222696+0000", 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041885400000000003 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.236603+0000", 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013906992 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237694+0000", 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001091091 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237791+0000", 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6455999999999999e-05 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237877+0000", 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6749999999999994e-05 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:77 3.3f 3:ff5d5af4:::10000000002.00000049:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.408320+0000", 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.6903942139999999, 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021608914, 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 77 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.408320+0000", 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.408320+0000", 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.408324+0000", 2024-04-25T15:52:49.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7270000000000001e-06 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.414158+0000", 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005834321 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.414221+0000", 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2671999999999999e-05 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.414247+0000", 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6353e-05 2024-04-25T15:52:49.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.414410+0000", 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016277300000000001 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.414661+0000", 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000251518 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.414891+0000", 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023006099999999999 2024-04-25T15:52:49.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.425215+0000", 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010323845 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.429741+0000", 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045257220000000003 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.429837+0000", 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5703000000000002e-05 2024-04-25T15:52:49.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.429929+0000", 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2219000000000006e-05 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:150 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.44754238299999999, 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024331635000000001, 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.192 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 150 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:49.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651176+0000", 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8770000000000003e-06 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659329+0000", 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081530219999999994 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659399+0000", 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0376000000000005e-05 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659428+0000", 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8952999999999999e-05 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659580+0000", 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015188199999999999 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659842+0000", 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026179399999999998 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.660120+0000", 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027824799999999998 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.673813+0000", 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013693145 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675307+0000", 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014934600000000001 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675412+0000", 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105233 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675504+0000", 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1644999999999994e-05 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:152 3.39 3:9e24e8b4:::10000000002.00000094:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.36767435599999998, 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022233425000000001, 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 152 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731043+0000", 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.735937+0000", 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048940520000000003 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736003+0000", 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6112000000000007e-05 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736030+0000", 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6744e-05 2024-04-25T15:52:49.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736177+0000", 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014675800000000001 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736344+0000", 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016679100000000001 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736481+0000", 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000137112 2024-04-25T15:52:49.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.744634+0000", 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081533030000000006 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753167+0000", 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085328769999999995 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753222+0000", 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5167999999999998e-05 2024-04-25T15:52:49.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753273+0000", 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1304999999999998e-05 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:155 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.22539109800000001, 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025243293999999999, 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:49.203 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 155 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:49.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873327+0000", 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7699999999999999e-06 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879173+0000", 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005846141 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:49.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879239+0000", 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6315999999999993e-05 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879267+0000", 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7137000000000001e-05 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879415+0000", 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000148715 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879693+0000", 2024-04-25T15:52:49.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000278108 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879951+0000", 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025793700000000002 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.896019+0000", 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016067814999999999 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898153+0000", 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021339280000000002 2024-04-25T15:52:49.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898411+0000", 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025760799999999999 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898566+0000", 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015581899999999999 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:49.208 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:49.209 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:49.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:49.210 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:49.206+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.238+0000 7fa6c77fe640 1 -- 172.21.15.67:0/227718093 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [1, 4]}]=0 set 3.18 pg_upmap mapping to [1,4] v62) ==== 124+0+0 (secure 0 0 0) 0x7fa6c8019790 con 0x7fa6d01f61e0 2024-04-25T15:52:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.18 pg_upmap mapping to [1,4] 2024-04-25T15:52:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa6b006f400 msgr2=0x7fa6b0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa6b006f400 0x7fa6b0071910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c10 tx=0x7fa6c0009040 comp rx=0 tx=0).stop 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d01f61e0 msgr2=0x7fa6d01f65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d01f61e0 0x7fa6d01f65f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8020320 tx=0x7fa6bc005840 comp rx=0 tx=0).stop 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 shutdown_connections 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa6b006f400 0x7fa6b0071910 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d01f61e0 0x7fa6d01f65f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6d00c54a0 0x7fa6d0150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 --2- 172.21.15.67:0/227718093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6d00ce620 0x7fa6d014eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 >> 172.21.15.67:0/227718093 conn(0x7fa6d0068c70 msgr2=0x7fa6d0071550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 shutdown_connections 2024-04-25T15:52:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.242+0000 7fa6d528c640 1 -- 172.21.15.67:0/227718093 wait complete. 2024-04-25T15:52:49.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:52:49.311 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:49.311+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:49.385 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:49.386 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:49.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.414+0000 7f59a13a8640 1 Processor -- start 2024-04-25T15:52:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.414+0000 7f59a13a8640 1 -- start start 2024-04-25T15:52:49.419 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:49.414+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.414+0000 7f59a13a8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f599c0c49f0 0x7f599c0c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.414+0000 7f59a13a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c161620 0x7f599c163a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.414+0000 7f59a13a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f599c06fa10 con 0x7f599c156330 2024-04-25T15:52:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.414+0000 7f59a13a8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f599c0ca6c0 con 0x7f599c0c49f0 2024-04-25T15:52:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.414+0000 7f59a13a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f599c0c43d0 con 0x7f599c161620 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c161620 0x7f599c163a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599a7fc640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f599c0c49f0 0x7f599c0c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c161620 0x7f599c163a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:45796/0 (socket says 172.21.15.67:45796) 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 -- 172.21.15.67:0/3448491942 learned_addr learned my addr 172.21.15.67:0/3448491942 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 -- 172.21.15.67:0/3448491942 >> v2:172.21.15.176:3300/0 conn(0x7f599c0c49f0 msgr2=0x7f599c0c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 --2- 172.21.15.67:0/3448491942 >> v2:172.21.15.176:3300/0 conn(0x7f599c0c49f0 0x7f599c0c4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 -- 172.21.15.67:0/3448491942 >> v1:172.21.15.176:6789/0 conn(0x7f599c156330 legacy=0x7f599c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 -- 172.21.15.67:0/3448491942 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c0ca8f0 con 0x7f599c161620 2024-04-25T15:52:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f599b7fe640 1 --2- 172.21.15.67:0/3448491942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c161620 0x7f599c163a40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5994009e50 tx=0x7f599401f210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcf46a3671040a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f5999ffb640 1 -- 172.21.15.67:0/3448491942 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59940b4070 con 0x7f599c161620 2024-04-25T15:52:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f5999ffb640 1 -- 172.21.15.67:0/3448491942 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59940af400 con 0x7f599c161620 2024-04-25T15:52:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.418+0000 7f5999ffb640 1 -- 172.21.15.67:0/3448491942 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59940af7d0 con 0x7f599c161620 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 -- 172.21.15.67:0/3448491942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c161620 msgr2=0x7f599c163a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3448491942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c161620 0x7f599c163a40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5994009e50 tx=0x7f599401f210 comp rx=0 tx=0).stop 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 -- 172.21.15.67:0/3448491942 shutdown_connections 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3448491942 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c161620 0x7f599c163a40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3448491942 >> v2:172.21.15.176:3300/0 conn(0x7f599c0c49f0 0x7f599c0c4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 -- 172.21.15.67:0/3448491942 >> 172.21.15.67:0/3448491942 conn(0x7f599c0ce1c0 msgr2=0x7f599c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 -- 172.21.15.67:0/3448491942 shutdown_connections 2024-04-25T15:52:49.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 -- 172.21.15.67:0/3448491942 wait complete. 2024-04-25T15:52:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.422+0000 7f59a13a8640 1 Processor -- start 2024-04-25T15:52:49.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 -- start start 2024-04-25T15:52:49.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f599c0c49f0 0x7f599c158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:49.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c156330 0x7f599c1f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:49.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c161620 0x7f599c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:49.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f599c0c4060 con 0x7f599c161620 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f599c0c3d80 con 0x7f599c0c49f0 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f599c156740 con 0x7f599c156330 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c161620 0x7f599c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c161620 0x7f599c0cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49172/0 (socket says 172.21.15.67:49172) 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 -- 172.21.15.67:0/3409787433 learned_addr learned my addr 172.21.15.67:0/3409787433 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599affd640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f599c0c49f0 0x7f599c158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599a7fc640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c156330 0x7f599c1f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 -- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c156330 msgr2=0x7f599c1f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c156330 0x7f599c1f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 -- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f599c0c49f0 msgr2=0x7f599c158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f599c0c49f0 0x7f599c158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 -- 172.21.15.67:0/3409787433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f599c1f67d0 con 0x7f599c161620 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599affd640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f599c0c49f0 0x7f599c158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:49.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f599b7fe640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c161620 0x7f599c0cada0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f599401d8c0 tx=0x7f599401bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:49.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59837fe640 1 -- 172.21.15.67:0/3409787433 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59940c0070 con 0x7f599c161620 2024-04-25T15:52:49.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59837fe640 1 -- 172.21.15.67:0/3409787433 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59940afcc0 con 0x7f599c161620 2024-04-25T15:52:49.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59837fe640 1 -- 172.21.15.67:0/3409787433 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59940b4040 con 0x7f599c161620 2024-04-25T15:52:49.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f599c157010 con 0x7f599c161620 2024-04-25T15:52:49.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.426+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f599c1575d0 con 0x7f599c161620 2024-04-25T15:52:49.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.430+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f599c06a480 con 0x7f599c161620 2024-04-25T15:52:49.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.430+0000 7f59837fe640 1 -- 172.21.15.67:0/3409787433 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f59940a8d10 con 0x7f599c161620 2024-04-25T15:52:49.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.430+0000 7f59837fe640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f598806f430 0x7f5988071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:49.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.430+0000 7f59837fe640 1 -- 172.21.15.67:0/3409787433 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5886+0+0 (secure 0 0 0) 0x7f599412c590 con 0x7f599c161620 2024-04-25T15:52:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.430+0000 7f599affd640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f598806f430 0x7f5988071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:49.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.430+0000 7f599affd640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f598806f430 0x7f5988071940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f598c000f70 tx=0x7f598c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:49.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.438+0000 7f59837fe640 1 -- 172.21.15.67:0/3409787433 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5994018d70 con 0x7f599c161620 2024-04-25T15:52:49.512 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:49.511+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:49.608 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:49.608 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:49.608 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:49.608 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:49.609 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:49.613 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:49.606+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:49.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:49.714 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:49.711+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:49.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:49.714+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f599c0ca4a0 con 0x7f599c161620 2024-04-25T15:52:49.815 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:49.810+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:49.834 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 149.831305555, 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:49.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 133.622417912, 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:49.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 133.62237052699999, 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:49.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:49.849 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 133.622358445, 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:49.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:49.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.854 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 133.622338273, 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:49.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:49.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:49.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 133.62209993499999, 2024-04-25T15:52:49.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:49.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.862 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:36 3.1b e59/54 3:d88a10eb:::10000000002.00000020:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2683679809999999, 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013408461999999999, 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565548+0000", 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5620000000000002e-06 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572005+0000", 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064569370000000003 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572034+0000", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8586999999999999e-05 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572056+0000", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2481e-05 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572135+0000", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9264999999999994e-05 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572159+0000", 2024-04-25T15:52:49.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3275e-05 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.578912+0000", 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067533030000000004 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.578954+0000", 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2052000000000003e-05 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0181482109999997, 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:52:49.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.7273500679999998, 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:52 3.12 e60/40 3:499536aa:::10000000002.00000030:head v 60'5, mlcod=59'3)", 2024-04-25T15:52:49.874 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4973390220000002, 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01319507, 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.336575+0000", 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4200000000000002e-07 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.342934+0000", 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006359211 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.342953+0000", 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9256999999999999e-05 2024-04-25T15:52:49.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.342968+0000", 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.49e-05 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.343057+0000", 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9151000000000001e-05 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.343087+0000", 2024-04-25T15:52:49.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9241999999999999e-05 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.349681+0000", 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065946440000000002 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.349770+0000", 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8323000000000001e-05 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:53 3.20 e60/18 3:068d0381:::10000000002.00000031:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4498925839999997, 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014486163, 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389720+0000", 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056985300000000003 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389742+0000", 2024-04-25T15:52:49.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1838999999999999e-05 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389758+0000", 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6058000000000001e-05 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389844+0000", 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6293999999999997e-05 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389866+0000", 2024-04-25T15:52:49.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2033000000000001e-05 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398429+0000", 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085631660000000005 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398507+0000", 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7869999999999998e-05 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:66 3.b e60/21 3:d381a89f:::10000000002.0000003e:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8886999270000002, 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015316048000000001, 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945216+0000", 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9e-06 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948749+0000", 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035338119999999999 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948777+0000", 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7915999999999998e-05 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948796+0000", 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.844e-05 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948895+0000", 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9704000000000006e-05 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948916+0000", 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.107e-05 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960445+0000", 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011528722999999999 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960530+0000", 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4482999999999997e-05 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:72 3.39 e60/18 3:9df4b1b7:::10000000002.00000044:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:45.223349+0000", 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.6105648800000001, 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013205985, 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.223349+0000", 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.223349+0000", 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.223351+0000", 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5009999999999999e-06 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.229270+0000", 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059184099999999998 2024-04-25T15:52:49.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.229295+0000", 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5017000000000001e-05 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.229327+0000", 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2431999999999997e-05 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.229396+0000", 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9227000000000007e-05 2024-04-25T15:52:49.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.229415+0000", 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8821000000000002e-05 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.236484+0000", 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007068548 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:45.236555+0000", 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1029e-05 2024-04-25T15:52:49.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:147 3.b e61/21 3:d1738051:::10000000002.0000008f:head v 61'4, mlcod=60'2)", 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.560320+0000", 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.2735936749999999, 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013203866, 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.560320+0000", 2024-04-25T15:52:49.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.560320+0000", 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.560322+0000", 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1720000000000001e-06 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.561824+0000", 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015021699999999999 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.561846+0000", 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1784e-05 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.561859+0000", 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3015e-05 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.561958+0000", 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8640000000000004e-05 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.561992+0000", 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3970999999999997e-05 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573411+0000", 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011419135 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573524+0000", 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112979 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:152 3.39 e61/18 3:9e24e8b4:::10000000002.00000094:head v 61'2, mlcod=0'0)", 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.0968298489999999, 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016125006000000001, 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737086+0000", 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744295+0000", 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072082320000000002 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744341+0000", 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6072000000000002e-05 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744376+0000", 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4903000000000001e-05 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744533+0000", 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015736299999999999 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744590+0000", 2024-04-25T15:52:49.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6771000000000002e-05 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753101+0000", 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085115490000000002 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753209+0000", 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107352 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.898 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:153 3.1 e61/54 3:814e3e07:::10000000002.00000095:head v 61'3, mlcod=59'1)", 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.0395724200000001, 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014546699999999999, 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:49.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794344+0000", 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796364+0000", 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002019968 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796414+0000", 2024-04-25T15:52:49.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0065999999999998e-05 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796445+0000", 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0783e-05 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796600+0000", 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154685 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796670+0000", 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0605999999999995e-05 2024-04-25T15:52:49.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.808781+0000", 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012110368999999999 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.808888+0000", 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107353 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.902 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:156 3.3e e61/35 3:7d47d675:::10000000002.00000098:head v 61'1, mlcod=0'0)", 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.89967637600000006, 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016362634000000001, 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934240+0000", 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3149999999999999e-06 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939270+0000", 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050303869999999999 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939308+0000", 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8047999999999999e-05 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939335+0000", 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7251999999999999e-05 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939432+0000", 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6637000000000003e-05 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939459+0000", 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7651999999999999e-05 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950534+0000", 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011074993999999999 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950600+0000", 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5349000000000001e-05 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:158 3.12 e61/40 3:4b703321:::10000000002.0000009a:head v 61'6, mlcod=59'4)", 2024-04-25T15:52:49.906 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.81671624899999995, 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017102098, 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017200+0000", 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3199999999999998e-06 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024143+0000", 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069429569999999996 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024169+0000", 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6007e-05 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024190+0000", 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1203999999999999e-05 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024272+0000", 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1833000000000001e-05 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024301+0000", 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9475000000000002e-05 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034243+0000", 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099420589999999996 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034299+0000", 2024-04-25T15:52:49.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6242999999999997e-05 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:162 3.2b e61/40 3:d691b431:::10000000002.0000009e:head v 61'3, mlcod=60'1)", 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.60837223799999995, 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014572351000000001, 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225544+0000", 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7199999999999998e-06 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230207+0000", 2024-04-25T15:52:49.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046626719999999997 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230239+0000", 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2548000000000001e-05 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230266+0000", 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6599999999999999e-05 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230382+0000", 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115667 2024-04-25T15:52:49.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230417+0000", 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5800999999999998e-05 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240029+0000", 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096115349999999992 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240114+0000", 2024-04-25T15:52:49.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4808000000000005e-05 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:164 3.30 e62/18 3:0efeaece:::10000000002.000000a0:head v 62'5, mlcod=60'3)", 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.49631679899999998, 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013652381, 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8999999999999997e-07 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343547+0000", 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059493669999999997 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343576+0000", 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9850000000000001e-05 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343597+0000", 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0330999999999999e-05 2024-04-25T15:52:49.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343687+0000", 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0278999999999994e-05 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343710+0000", 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2796e-05 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.351154+0000", 2024-04-25T15:52:49.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074439739999999999 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.351249+0000", 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5293999999999999e-05 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:49.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:49.921 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:49.919+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:50.021 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:50.014+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:50.122 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:50.119+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:50.138 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:50.138 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:50.138 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:50.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:50.223 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:50.219+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:50.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.258+0000 7f59837fe640 1 -- 172.21.15.67:0/3409787433 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v63) ==== 101+0+0 (secure 0 0 0) 0x7f59940f9490 con 0x7f599c161620 2024-04-25T15:52:50.265 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f598806f430 msgr2=0x7f5988071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f598806f430 0x7f5988071940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f598c000f70 tx=0x7f598c009040 comp rx=0 tx=0).stop 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c161620 msgr2=0x7f599c0cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c161620 0x7f599c0cada0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f599401d8c0 tx=0x7f599401bfb0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 shutdown_connections 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f598806f430 0x7f5988071940 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f599c161620 0x7f599c0cada0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f599c156330 0x7f599c1f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 --2- 172.21.15.67:0/3409787433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f599c0c49f0 0x7f599c158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 >> 172.21.15.67:0/3409787433 conn(0x7f599c0ce1c0 msgr2=0x7f599c0c9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 shutdown_connections 2024-04-25T15:52:50.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.262+0000 7f59a13a8640 1 -- 172.21.15.67:0/3409787433 wait complete. 2024-04-25T15:52:50.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:50.323 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:50.323+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:50.353 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:50.353 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:50.354 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:50.354 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:50.354 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:50.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:50.424 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:50.423+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:50.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc2e6e640 1 Processor -- start 2024-04-25T15:52:50.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc2e6e640 1 -- start start 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc2e6e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fadbc06d460 0x7fadbc0736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc2e6e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fadbc1547e0 0x7fadbc06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc2e6e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadbc0caa60 con 0x7fadbc06cb80 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc2e6e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fadbc068590 con 0x7fadbc1547e0 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc2e6e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fadbc15a5f0 con 0x7fadbc06d460 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadbbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fadbc1547e0 0x7fadbc06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc140b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fadbc06cb80 0x7fadbc15aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44690/0 (socket says 172.21.15.67:44690) 2024-04-25T15:52:50.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc140b640 1 -- 172.21.15.67:0/1462422479 learned_addr learned my addr 172.21.15.67:0/1462422479 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:50.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/1462422479 >> v2:172.21.15.176:3300/0 conn(0x7fadbc06d460 0x7fadbc0736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadbbfff640 1 -- 172.21.15.67:0/1462422479 >> v2:172.21.15.176:3300/0 conn(0x7fadbc06d460 msgr2=0x7fadbc0736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:50.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadbbfff640 1 --2- 172.21.15.67:0/1462422479 >> v2:172.21.15.176:3300/0 conn(0x7fadbc06d460 0x7fadbc0736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadbbfff640 1 -- 172.21.15.67:0/1462422479 >> v1:172.21.15.67:6789/0 conn(0x7fadbc06cb80 legacy=0x7fadbc15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadbb7fe640 1 -- 172.21.15.67:0/1462422479 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401921116 0 0) 0x7fadbc0caa60 con 0x7fadbc06cb80 2024-04-25T15:52:50.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.442+0000 7fadbbfff640 1 -- 172.21.15.67:0/1462422479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadbc0ca100 con 0x7fadbc1547e0 2024-04-25T15:52:50.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadbbfff640 1 --2- 172.21.15.67:0/1462422479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc1547e0 0x7fadbc06c640 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0cbd80 tx=0x7fada801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1b430413301f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadbb7fe640 1 -- 172.21.15.67:0/1462422479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada80b0070 con 0x7fadbc1547e0 2024-04-25T15:52:50.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadbb7fe640 1 -- 172.21.15.67:0/1462422479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada8004230 con 0x7fadbc1547e0 2024-04-25T15:52:50.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadbb7fe640 1 -- 172.21.15.67:0/1462422479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada8003620 con 0x7fadbc1547e0 2024-04-25T15:52:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 -- 172.21.15.67:0/1462422479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc1547e0 msgr2=0x7fadbc06c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/1462422479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc1547e0 0x7fadbc06c640 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0cbd80 tx=0x7fada801c460 comp rx=0 tx=0).stop 2024-04-25T15:52:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 -- 172.21.15.67:0/1462422479 shutdown_connections 2024-04-25T15:52:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/1462422479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc1547e0 0x7fadbc06c640 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/1462422479 >> v2:172.21.15.176:3300/0 conn(0x7fadbc06d460 0x7fadbc0736d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 -- 172.21.15.67:0/1462422479 >> 172.21.15.67:0/1462422479 conn(0x7fadbc0ce1c0 msgr2=0x7fadbc0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:50.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 -- 172.21.15.67:0/1462422479 shutdown_connections 2024-04-25T15:52:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.446+0000 7fadc2e6e640 1 -- 172.21.15.67:0/1462422479 wait complete. 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 Processor -- start 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 -- start start 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc06cb80 0x7fadbc1f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadbc06d460 0x7fadbc14b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadbc1547e0 0x7fadbc0b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fadbc154d70 con 0x7fadbc06cb80 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fadbc154bf0 con 0x7fadbc06d460 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc2e6e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fadbc167790 con 0x7fadbc1547e0 2024-04-25T15:52:50.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc06cb80 0x7fadbc1f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc06cb80 0x7fadbc1f4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49194/0 (socket says 172.21.15.67:49194) 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 -- 172.21.15.67:0/4072535846 learned_addr learned my addr 172.21.15.67:0/4072535846 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadbbfff640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadbc06d460 0x7fadbc14b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc140b640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadbc1547e0 0x7fadbc0b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadbc1547e0 msgr2=0x7fadbc0b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fadbc1547e0 0x7fadbc0b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadbc06d460 msgr2=0x7fadbc14b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fadbc06d460 0x7fadbc14b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadbc1f4980 con 0x7fadbc06cb80 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.450+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc06cb80 0x7fadbc1f4440 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fadb000d620 tx=0x7fadb000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb000b620 con 0x7fadbc06cb80 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb0019070 con 0x7fadbc06cb80 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadbc14b7a0 con 0x7fadbc06cb80 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadbc1f4b90 con 0x7fadbc06cb80 2024-04-25T15:52:50.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb000b9e0 con 0x7fadbc06cb80 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fadb000bc00 con 0x7fadbc06cb80 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fadbc0cc940 con 0x7fadbc06cb80 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadb97fa640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fada006f400 0x7fada0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5886+0+0 (secure 0 0 0) 0x7fadb008efd0 con 0x7fadbc06cb80 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v63) ==== 54+0+16 (secure 0 0 0) 0x7fadbc0cc940 con 0x7fadbc06cb80 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fadbc0cb260 0x7fadbc150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fadbc06a480 con 0x7fadbc0cb260 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.454+0000 7fadc140b640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fadbc0cb260 0x7fadbc150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.458+0000 7fadbbfff640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fada006f400 0x7fada0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.458+0000 7fadc140b640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fadbc0cb260 0x7fadbc150440 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.458+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc06a480 con 0x7fadbc0cb260 2024-04-25T15:52:50.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.458+0000 7fadbbfff640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fada006f400 0x7fada0071910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fada80ad000 tx=0x7fada80a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.482+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc0b4460 con 0x7fadbc0cb260 2024-04-25T15:52:50.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.482+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc0b4460 con 0x7fadbc0cb260 2024-04-25T15:52:50.489 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.482+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fadbc139dc0 0x7fadbc1f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.482+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fadbc153940 con 0x7fadbc139dc0 2024-04-25T15:52:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.482+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fadbc139dc0 0x7fadbc1f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.486+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fadbc139dc0 0x7fadbc1f71d0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.486+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc153940 con 0x7fadbc139dc0 2024-04-25T15:52:50.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.506+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc0b1b20 con 0x7fadbc139dc0 2024-04-25T15:52:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.506+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc0b1b20 con 0x7fadbc139dc0 2024-04-25T15:52:50.513 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.510+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fadbc1f7c20 0x7fadbc063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.510+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fadbc1f8030 con 0x7fadbc1f7c20 2024-04-25T15:52:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.510+0000 7fadbbfff640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fadbc1f7c20 0x7fadbc063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.510+0000 7fadbbfff640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fadbc1f7c20 0x7fadbc063090 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.510+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc1f8030 con 0x7fadbc1f7c20 2024-04-25T15:52:50.525 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:50.518+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.530+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc1f8030 con 0x7fadbc1f7c20 2024-04-25T15:52:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.534+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc1f8030 con 0x7fadbc1f7c20 2024-04-25T15:52:50.538 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.534+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fadbc0cccf0 0x7fadbc0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.534+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fadbc1f8030 con 0x7fadbc0cccf0 2024-04-25T15:52:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.534+0000 7fadc140b640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fadbc0cccf0 0x7fadbc0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.534+0000 7fadc140b640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fadbc0cccf0 0x7fadbc0cd100 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.534+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc1f8030 con 0x7fadbc0cccf0 2024-04-25T15:52:50.543 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:50.543 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:50.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.550+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc1f8030 con 0x7fadbc0cccf0 2024-04-25T15:52:50.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.554+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc1f8030 con 0x7fadbc0cccf0 2024-04-25T15:52:50.559 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:50.559 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.554+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fadbc0cd970 0x7fadbc02cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.554+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fadbc072db0 con 0x7fadbc0cd970 2024-04-25T15:52:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.554+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fadbc0cd970 0x7fadbc02cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.554+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fadbc0cd970 0x7fadbc02cef0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.558+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc072db0 con 0x7fadbc0cd970 2024-04-25T15:52:50.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:50.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.578+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc072db0 con 0x7fadbc0cd970 2024-04-25T15:52:50.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.582+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc072db0 con 0x7fadbc0cd970 2024-04-25T15:52:50.588 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:50.589 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:50.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.582+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fadbc02d940 0x7fadbc20a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.582+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fadbc02dd50 con 0x7fadbc02d940 2024-04-25T15:52:50.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.582+0000 7fadbbfff640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fadbc02d940 0x7fadbc20a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.586+0000 7fadbbfff640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fadbc02d940 0x7fadbc20a560 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.586+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc02dd50 con 0x7fadbc02d940 2024-04-25T15:52:50.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.602+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc02dd50 con 0x7fadbc02d940 2024-04-25T15:52:50.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.606+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc02dd50 con 0x7fadbc02d940 2024-04-25T15:52:50.612 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:50.612 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.606+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fadbc20afb0 0x7fadbc20d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.606+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fadbc02df40 con 0x7fadbc20afb0 2024-04-25T15:52:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.606+0000 7fadc140b640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fadbc20afb0 0x7fadbc20d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.610+0000 7fadc140b640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fadbc20afb0 0x7fadbc20d3d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.610+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc02df40 con 0x7fadbc20afb0 2024-04-25T15:52:50.626 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:50.623+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:50.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.634+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc02df40 con 0x7fadbc20afb0 2024-04-25T15:52:50.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.634+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc02df40 con 0x7fadbc20afb0 2024-04-25T15:52:50.640 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:50.641 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:50.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.634+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fadbc20de80 0x7fadbc20e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.634+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fadbc213360 con 0x7fadbc20de80 2024-04-25T15:52:50.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.638+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fadbc20de80 0x7fadbc20e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.638+0000 7fadc0c0a640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fadbc20de80 0x7fadbc20e290 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.638+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadbc213360 con 0x7fadbc20de80 2024-04-25T15:52:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.666+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fadbc2121e0 con 0x7fadbc20de80 2024-04-25T15:52:50.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadb97fa640 1 -- 172.21.15.67:0/4072535846 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fadbc2121e0 con 0x7fadbc20de80 2024-04-25T15:52:50.672 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fadbc0cb260 msgr2=0x7fadbc150440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fadbc0cb260 0x7fadbc150440 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fadbc139dc0 msgr2=0x7fadbc1f71d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fadbc139dc0 0x7fadbc1f71d0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fadbc1f7c20 msgr2=0x7fadbc063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fadbc1f7c20 0x7fadbc063090 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fadbc0cccf0 msgr2=0x7fadbc0cd100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fadbc0cccf0 0x7fadbc0cd100 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fadbc0cd970 msgr2=0x7fadbc02cef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fadbc0cd970 0x7fadbc02cef0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fadbc02d940 msgr2=0x7fadbc20a560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fadbc02d940 0x7fadbc20a560 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fadbc20afb0 msgr2=0x7fadbc20d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc140b640 1 -- 172.21.15.67:0/4072535846 reap_dead start 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fadbc20afb0 0x7fadbc20d3d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc140b640 1 -- 172.21.15.67:0/4072535846 reap_dead start 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fadbc20de80 msgr2=0x7fadbc20e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fadbc20de80 0x7fadbc20e290 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc140b640 1 -- 172.21.15.67:0/4072535846 reap_dead start 2024-04-25T15:52:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fada006f400 msgr2=0x7fada0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fada006f400 0x7fada0071910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fada80ad000 tx=0x7fada80a6000 comp rx=0 tx=0).stop 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc06cb80 msgr2=0x7fadbc1f4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc06cb80 0x7fadbc1f4440 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fadb000d620 tx=0x7fadb000dae0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 shutdown_connections 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fada006f400 0x7fada0071910 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 --2- 172.21.15.67:0/4072535846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadbc06cb80 0x7fadbc1f4440 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 >> 172.21.15.67:0/4072535846 conn(0x7fadbc0ce1c0 msgr2=0x7fadbc0ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 shutdown_connections 2024-04-25T15:52:50.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:50.670+0000 7fadc2e6e640 1 -- 172.21.15.67:0/4072535846 wait complete. 2024-04-25T15:52:50.741 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:50.741 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:50.741 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:50.742+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:50.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:50.828 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:50.827+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:50.929 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:50.926+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:50.958 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:50.958 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:50.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:51.061 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:51.062+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:51.131 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:51.130+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:51.166 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:51.166 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:51.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:51.232 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:51.231+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:51.332 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:51.330+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:51.386 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:51.386 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:51.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:51.433 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:51.431+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:51.534 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:51.534+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:51.601 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:51.602 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:51.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:51.635 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:51.635+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:51.735 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:51.734+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:51.819 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:51.819 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:51.835 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:51.837 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:51.835+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:51.938 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:51.938+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:51.992 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:51.992 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:52.010 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:52.055 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:52.055+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:52.139 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:52.138+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:52.159 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:52.159 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:52.177 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:52.267 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:52.267+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:52.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:52:52.340 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:52.339+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:52.374 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:52.374 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:52.392 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd0850e6640 1 Processor -- start 2024-04-25T15:52:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd0850e6640 1 -- start start 2024-04-25T15:52:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd0850e6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0800c8910 0x7fd080156c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd0850e6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd08015cc60 con 0x7fd0800c9bc0 2024-04-25T15:52:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd0850e6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd08015cf40 con 0x7fd0800c6c10 2024-04-25T15:52:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd0850e6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd08015d220 con 0x7fd0800c8910 2024-04-25T15:52:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd07f59e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0800c8910 0x7fd080156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd07e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd0800c9bc0 0x7fd0800c8200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44700/0 (socket says 172.21.15.67:44700) 2024-04-25T15:52:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd07e59c640 1 -- 172.21.15.67:0/1583005404 learned_addr learned my addr 172.21.15.67:0/1583005404 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd07f59e640 1 -- 172.21.15.67:0/1583005404 >> v1:172.21.15.176:6789/0 conn(0x7fd0800c6c10 legacy=0x7fd0800d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd07f59e640 1 -- 172.21.15.67:0/1583005404 >> v1:172.21.15.67:6789/0 conn(0x7fd0800c9bc0 legacy=0x7fd0800c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd07f59e640 1 -- 172.21.15.67:0/1583005404 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08015d500 con 0x7fd0800c8910 2024-04-25T15:52:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.454+0000 7fd07dd9b640 1 -- 172.21.15.67:0/1583005404 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202192834 0 0) 0x7fd08015cc60 con 0x7fd0800c9bc0 2024-04-25T15:52:52.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd07f59e640 1 --2- 172.21.15.67:0/1583005404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c8910 0x7fd080156c60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd078009d90 tx=0x7fd07801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b983493ace6f2789 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd07dd9b640 1 -- 172.21.15.67:0/1583005404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0780b4070 con 0x7fd0800c8910 2024-04-25T15:52:52.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd07dd9b640 1 -- 172.21.15.67:0/1583005404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0780af600 con 0x7fd0800c8910 2024-04-25T15:52:52.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd07dd9b640 1 -- 172.21.15.67:0/1583005404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0780af9d0 con 0x7fd0800c8910 2024-04-25T15:52:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd0850e6640 1 -- 172.21.15.67:0/1583005404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c8910 msgr2=0x7fd080156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd0850e6640 1 --2- 172.21.15.67:0/1583005404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c8910 0x7fd080156c60 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd078009d90 tx=0x7fd07801f150 comp rx=0 tx=0).stop 2024-04-25T15:52:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd0850e6640 1 -- 172.21.15.67:0/1583005404 shutdown_connections 2024-04-25T15:52:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd0850e6640 1 --2- 172.21.15.67:0/1583005404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c8910 0x7fd080156c60 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd0850e6640 1 -- 172.21.15.67:0/1583005404 >> 172.21.15.67:0/1583005404 conn(0x7fd080068760 msgr2=0x7fd080068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd0850e6640 1 -- 172.21.15.67:0/1583005404 shutdown_connections 2024-04-25T15:52:52.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.458+0000 7fd0850e6640 1 -- 172.21.15.67:0/1583005404 wait complete. 2024-04-25T15:52:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 Processor -- start 2024-04-25T15:52:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 -- start start 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c6c10 0x7fd08014c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0800c8910 0x7fd080159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0800c9bc0 0x7fd0801f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd08015c8b0 con 0x7fd0800c6c10 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd0800c7ce0 con 0x7fd0800c8910 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd0801f3390 con 0x7fd0800c9bc0 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c6c10 0x7fd08014c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c6c10 0x7fd08014c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49210/0 (socket says 172.21.15.67:49210) 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 -- 172.21.15.67:0/3897013021 learned_addr learned my addr 172.21.15.67:0/3897013021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07e59c640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0800c8910 0x7fd080159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07f59e640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0800c9bc0 0x7fd0801f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 -- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0800c9bc0 msgr2=0x7fd0801f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0800c9bc0 0x7fd0801f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 -- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0800c8910 msgr2=0x7fd080159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0800c8910 0x7fd080159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 -- 172.21.15.67:0/3897013021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08014cee0 con 0x7fd0800c6c10 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd07ed9d640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c6c10 0x7fd08014c9a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c3f0 tx=0x7fd07000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0677fe640 1 -- 172.21.15.67:0/3897013021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070017070 con 0x7fd0800c6c10 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0677fe640 1 -- 172.21.15.67:0/3897013021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0700052d0 con 0x7fd0800c6c10 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd080151a10 con 0x7fd0800c6c10 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0677fe640 1 -- 172.21.15.67:0/3897013021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070003840 con 0x7fd0800c6c10 2024-04-25T15:52:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.462+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd080157910 con 0x7fd0800c6c10 2024-04-25T15:52:52.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.466+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd08006a480 con 0x7fd0800c6c10 2024-04-25T15:52:52.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.466+0000 7fd0677fe640 1 -- 172.21.15.67:0/3897013021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd07001a070 con 0x7fd0800c6c10 2024-04-25T15:52:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.466+0000 7fd0677fe640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06c06f400 0x7fd06c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.466+0000 7fd0677fe640 1 -- 172.21.15.67:0/3897013021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5886+0+0 (secure 0 0 0) 0x7fd07008ea50 con 0x7fd0800c6c10 2024-04-25T15:52:52.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.466+0000 7fd07e59c640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06c06f400 0x7fd06c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.466+0000 7fd07e59c640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06c06f400 0x7fd06c071910 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd074007bc0 tx=0x7fd074007b10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.470+0000 7fd0677fe640 1 -- 172.21.15.67:0/3897013021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd070021020 con 0x7fd0800c6c10 2024-04-25T15:52:52.483 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:52.483+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:52.542 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:52.542+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:52.584 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:52.584 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:52.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:52.644 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:52.642+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:52.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.698+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fd080160cb0 con 0x7fd0800c6c10 2024-04-25T15:52:52.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:52.745 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:52.746+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:52.797 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:52.797 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:52.817 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:52.845 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:52.846+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.846+0000 7fde41bed640 1 Processor -- start 2024-04-25T15:52:52.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.846+0000 7fde41bed640 1 -- start start 2024-04-25T15:52:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde41bed640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c0caf60 0x7fde3c15d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde41bed640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde3c15d790 0x7fde3c06d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde41bed640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3c0c77a0 con 0x7fde3c073bf0 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde41bed640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fde3c0cab60 con 0x7fde3c15d790 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde41bed640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde3c159620 con 0x7fde3c0caf60 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde3c15d790 0x7fde3c06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fde3c073bf0 0x7fde3c072bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42460/0 (socket says 172.21.15.67:42460) 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde3c15d790 0x7fde3c06d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60450/0 (socket says 172.21.15.67:60450) 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3bfff640 1 -- 172.21.15.67:0/1549264955 learned_addr learned my addr 172.21.15.67:0/1549264955 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/1549264955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c0caf60 0x7fde3c15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3affd640 1 -- 172.21.15.67:0/1549264955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c0caf60 msgr2=0x7fde3c15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3affd640 1 --2- 172.21.15.67:0/1549264955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c0caf60 0x7fde3c15d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3affd640 1 -- 172.21.15.67:0/1549264955 >> v1:172.21.15.67:6789/0 conn(0x7fde3c073bf0 legacy=0x7fde3c072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3a7fc640 1 -- 172.21.15.67:0/1549264955 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561097605 0 0) 0x7fde3c0c77a0 con 0x7fde3c073bf0 2024-04-25T15:52:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3affd640 1 -- 172.21.15.67:0/1549264955 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c0c7a60 con 0x7fde3c15d790 2024-04-25T15:52:52.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3affd640 1 --2- 172.21.15.67:0/1549264955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d790 0x7fde3c06d750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fde300097c0 tx=0x7fde3001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde663cf49aed876 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3a7fc640 1 -- 172.21.15.67:0/1549264955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde300b0070 con 0x7fde3c15d790 2024-04-25T15:52:52.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3a7fc640 1 -- 172.21.15.67:0/1549264955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde300ab600 con 0x7fde3c15d790 2024-04-25T15:52:52.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.850+0000 7fde3a7fc640 1 -- 172.21.15.67:0/1549264955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde300ab9d0 con 0x7fde3c15d790 2024-04-25T15:52:52.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- 172.21.15.67:0/1549264955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d790 msgr2=0x7fde3c06d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 --2- 172.21.15.67:0/1549264955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d790 0x7fde3c06d750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fde300097c0 tx=0x7fde3001c4a0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- 172.21.15.67:0/1549264955 shutdown_connections 2024-04-25T15:52:52.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 --2- 172.21.15.67:0/1549264955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c15d790 0x7fde3c06d750 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 --2- 172.21.15.67:0/1549264955 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c0caf60 0x7fde3c15d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- 172.21.15.67:0/1549264955 >> 172.21.15.67:0/1549264955 conn(0x7fde3c070db0 msgr2=0x7fde3c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:52.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- 172.21.15.67:0/1549264955 shutdown_connections 2024-04-25T15:52:52.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- 172.21.15.67:0/1549264955 wait complete. 2024-04-25T15:52:52.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 Processor -- start 2024-04-25T15:52:52.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- start start 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde3c073bf0 0x7fde3c163780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c0caf60 0x7fde3c1f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c15d790 0x7fde3c14d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c167060 con 0x7fde3c0caf60 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c166ee0 con 0x7fde3c073bf0 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde41bed640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde3c1671e0 con 0x7fde3c15d790 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde3affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c0caf60 0x7fde3c1f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde3affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c0caf60 0x7fde3c1f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60460/0 (socket says 172.21.15.67:60460) 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde3affd640 1 -- 172.21.15.67:0/362633932 learned_addr learned my addr 172.21.15.67:0/362633932 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.854+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde3c073bf0 0x7fde3c163780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c15d790 0x7fde3c14d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3affd640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c15d790 msgr2=0x7fde3c14d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c15d790 0x7fde3c14d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3affd640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde3c073bf0 msgr2=0x7fde3c163780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde3c073bf0 0x7fde3c163780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3affd640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c14d690 con 0x7fde3c0caf60 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde3c15d790 0x7fde3c14d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c0caf60 0x7fde3c1f70d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fde3001a0d0 tx=0x7fde30019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde300ba070 con 0x7fde3c0caf60 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde300b43f0 con 0x7fde3c0caf60 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c14cc70 con 0x7fde3c0caf60 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde300b0040 con 0x7fde3c0caf60 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c1644d0 con 0x7fde3c0caf60 2024-04-25T15:52:52.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fde3c06a480 con 0x7fde3c0caf60 2024-04-25T15:52:52.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde300bd070 con 0x7fde3c0caf60 2024-04-25T15:52:52.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde23fff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde2806f400 0x7fde28071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5886+0+0 (secure 0 0 0) 0x7fde301281f0 con 0x7fde3c0caf60 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde2806f400 0x7fde28071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v63) ==== 54+0+16 (secure 0 0 0) 0x7fde30016090 con 0x7fde3c0caf60 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde3c0ce940 0x7fde3c162210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde3c15cb10 con 0x7fde3c0ce940 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.858+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde3c0ce940 0x7fde3c162210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.862+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde2806f400 0x7fde28071910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009ac0 tx=0x7fde2c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.862+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde3c0ce940 0x7fde3c162210 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.862+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c15cb10 con 0x7fde3c0ce940 2024-04-25T15:52:52.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.878+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c0c8a40 con 0x7fde3c0ce940 2024-04-25T15:52:52.881 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:52.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.878+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c0c8a40 con 0x7fde3c0ce940 2024-04-25T15:52:52.882 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:52.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.878+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde3c145420 0x7fde3c1f9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.878+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fde3c145830 con 0x7fde3c145420 2024-04-25T15:52:52.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.878+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde3c145420 0x7fde3c1f9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.882+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde3c145420 0x7fde3c1f9bc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.882+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c145830 con 0x7fde3c145420 2024-04-25T15:52:52.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.894+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c06d6f0 con 0x7fde3c145420 2024-04-25T15:52:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.898+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c06d6f0 con 0x7fde3c145420 2024-04-25T15:52:52.898 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:52.898 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.898+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde3c1fa610 0x7fde3c063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.898+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fde3c1faa20 con 0x7fde3c1fa610 2024-04-25T15:52:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.898+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde3c1fa610 0x7fde3c063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.898+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde3c1fa610 0x7fde3c063090 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.898+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c1faa20 con 0x7fde3c1fa610 2024-04-25T15:52:52.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.914+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c15bf60 con 0x7fde3c1fa610 2024-04-25T15:52:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.914+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c15bf60 con 0x7fde3c1fa610 2024-04-25T15:52:52.916 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:52.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:52.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.914+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde3c157490 0x7fde3c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.914+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fde3c163f10 con 0x7fde3c157490 2024-04-25T15:52:52.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.914+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde3c157490 0x7fde3c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.914+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde3c157490 0x7fde3c1578a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.918+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c163f10 con 0x7fde3c157490 2024-04-25T15:52:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.938+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c163f10 con 0x7fde3c157490 2024-04-25T15:52:52.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.942+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c163f10 con 0x7fde3c157490 2024-04-25T15:52:52.942 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.942+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde3c02c9e0 0x7fde3c02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.942+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fde3c163f10 con 0x7fde3c02c9e0 2024-04-25T15:52:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.942+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde3c02c9e0 0x7fde3c02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.942+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde3c02c9e0 0x7fde3c02cdf0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:52.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.946+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c163f10 con 0x7fde3c02c9e0 2024-04-25T15:52:52.946 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:52.946+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:52.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.966+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c163f10 con 0x7fde3c02c9e0 2024-04-25T15:52:52.968 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:52.968 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c163f10 con 0x7fde3c02c9e0 2024-04-25T15:52:52.971 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 0x7fde3c20cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fde3c063ea0 con 0x7fde3c02d840 2024-04-25T15:52:52.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 0x7fde3c20cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:52.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde3affd640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 msgr2=0x7fde3c20cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 26 2024-04-25T15:52:52.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde3affd640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 msgr2=0x7fde3c20cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:52:52.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 0x7fde3c20cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:52:52.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:52.970+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 0x7fde3c20cf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:52:52.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:53.047 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:53.047+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:53.137 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:53.138 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:53.148 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:53.147+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:53.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.174+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 0x7fde3c20cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.174+0000 7fde3affd640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 0x7fde3c20cf50 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:53.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.174+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c063ea0 con 0x7fde3c02d840 2024-04-25T15:52:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.190+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c063ea0 con 0x7fde3c02d840 2024-04-25T15:52:53.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.194+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c063ea0 con 0x7fde3c02d840 2024-04-25T15:52:53.195 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:53.197 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:53.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.194+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde3c02dc50 0x7fde3c20f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:53.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.194+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fde3c212780 con 0x7fde3c02dc50 2024-04-25T15:52:53.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.194+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde3c02dc50 0x7fde3c20f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:53.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.194+0000 7fde3bfff640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde3c02dc50 0x7fde3c20f9b0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:53.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.194+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c212780 con 0x7fde3c02dc50 2024-04-25T15:52:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.222+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c20fef0 con 0x7fde3c02dc50 2024-04-25T15:52:53.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.226+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c20fef0 con 0x7fde3c02dc50 2024-04-25T15:52:53.227 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.226+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde3c210640 0x7fde3c212a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.226+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fde3c2159a0 con 0x7fde3c210640 2024-04-25T15:52:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.226+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde3c210640 0x7fde3c212a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.226+0000 7fde3b7fe640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde3c210640 0x7fde3c212a60 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:53.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.230+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde3c2159a0 con 0x7fde3c210640 2024-04-25T15:52:53.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.246+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde3c215970 con 0x7fde3c210640 2024-04-25T15:52:53.249 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:53.246+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:53.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde23fff640 1 -- 172.21.15.67:0/362633932 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde3c215970 con 0x7fde3c210640 2024-04-25T15:52:53.250 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:53.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde3c0ce940 msgr2=0x7fde3c162210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde3c0ce940 0x7fde3c162210 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde3c145420 msgr2=0x7fde3c1f9bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde3c145420 0x7fde3c1f9bc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde3c1fa610 msgr2=0x7fde3c063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde3c1fa610 0x7fde3c063090 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde3c157490 msgr2=0x7fde3c1578a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde3c157490 0x7fde3c1578a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde3bfff640 1 -- 172.21.15.67:0/362633932 reap_dead start 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde3c02c9e0 msgr2=0x7fde3c02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde3c02c9e0 0x7fde3c02cdf0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde3bfff640 1 -- 172.21.15.67:0/362633932 reap_dead start 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 msgr2=0x7fde3c20cf50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde3c02d840 0x7fde3c20cf50 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde3c02dc50 msgr2=0x7fde3c20f9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde3c02dc50 0x7fde3c20f9b0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde3c210640 msgr2=0x7fde3c212a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde3c210640 0x7fde3c212a60 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde2806f400 msgr2=0x7fde28071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde2806f400 0x7fde28071910 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009ac0 tx=0x7fde2c009190 comp rx=0 tx=0).stop 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c0caf60 msgr2=0x7fde3c1f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde3bfff640 1 -- 172.21.15.67:0/362633932 reap_dead start 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c0caf60 0x7fde3c1f70d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fde3001a0d0 tx=0x7fde30019240 comp rx=0 tx=0).stop 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 shutdown_connections 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 --2- 172.21.15.67:0/362633932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde3c0caf60 0x7fde3c1f70d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 >> 172.21.15.67:0/362633932 conn(0x7fde3c070db0 msgr2=0x7fde3c0c8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 shutdown_connections 2024-04-25T15:52:53.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.250+0000 7fde41bed640 1 -- 172.21.15.67:0/362633932 wait complete. 2024-04-25T15:52:53.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0677fe640 1 -- 172.21.15.67:0/3897013021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v64) ==== 87+0+0 (secure 0 0 0) 0x7fd070005480 con 0x7fd0800c6c10 2024-04-25T15:52:53.294 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:52:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06c06f400 msgr2=0x7fd06c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06c06f400 0x7fd06c071910 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd074007bc0 tx=0x7fd074007b10 comp rx=0 tx=0).stop 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c6c10 msgr2=0x7fd08014c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c6c10 0x7fd08014c9a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c3f0 tx=0x7fd07000c8b0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 shutdown_connections 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06c06f400 0x7fd06c071910 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0800c9bc0 0x7fd0801f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0800c8910 0x7fd080159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 --2- 172.21.15.67:0/3897013021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0800c6c10 0x7fd08014c9a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 >> 172.21.15.67:0/3897013021 conn(0x7fd080068760 msgr2=0x7fd0800cb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 shutdown_connections 2024-04-25T15:52:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:53.294+0000 7fd0850e6640 1 -- 172.21.15.67:0/3897013021 wait complete. 2024-04-25T15:52:53.350 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:53.347+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:53.368 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:53.368 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:53.369 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:53.369 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:53.369 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:53.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:53.471 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:53.471+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:53.503 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:52:53.503 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 initial writers = 88890.96 kB/sec 2024-04-25T15:52:53.503 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 initial writers = 88886.12 kB/sec 2024-04-25T15:52:53.504 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 88890.96 kB/sec 2024-04-25T15:52:53.504 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 88890.96 kB/sec 2024-04-25T15:52:53.504 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 88890.96 kB/sec 2024-04-25T15:52:53.504 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:52:53.551 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:53.551+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:53.585 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:53.585 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:53.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:53.585 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:53.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.585 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:43 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6794627339999995, 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024229051000000001, 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902019+0000", 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9459999999999998e-06 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907768+0000", 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005748525 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907831+0000", 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2842999999999994e-05 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907857+0000", 2024-04-25T15:52:53.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6636e-05 2024-04-25T15:52:53.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908056+0000", 2024-04-25T15:52:53.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198295 2024-04-25T15:52:53.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908464+0000", 2024-04-25T15:52:53.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040813600000000002 2024-04-25T15:52:53.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908853+0000", 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038892400000000002 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925332+0000", 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016479074 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925882+0000", 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000550232 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926050+0000", 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016822199999999999 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926245+0000", 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019521799999999999 2024-04-25T15:52:53.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:53 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.2032805730000007, 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020793931000000002, 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.593 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 53 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378202+0000", 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1669999999999999e-06 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382531+0000", 2024-04-25T15:52:53.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043289130000000002 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382593+0000", 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2143000000000005e-05 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382618+0000", 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382814+0000", 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195964 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383075+0000", 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026081800000000001 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383324+0000", 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024972200000000002 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.392258+0000", 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089338549999999992 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398665+0000", 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064066649999999998 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398841+0000", 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017607899999999999 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398993+0000", 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015144799999999999 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:55 3.10 3:09288c7b:::10000000002.00000033:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.1097199969999991, 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018106206999999999, 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.599 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 55 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:53.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471763+0000", 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.794e-06 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476322+0000", 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045595009999999997 2024-04-25T15:52:53.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476383+0000", 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0936000000000001e-05 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476410+0000", 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6313e-05 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476613+0000", 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020343499999999999 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476897+0000", 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028380400000000002 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.477152+0000", 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521799999999998 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.484642+0000", 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074893970000000001 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489677+0000", 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050351010000000002 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489779+0000", 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102592 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489865+0000", 2024-04-25T15:52:53.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6116e-05 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:57 3.3 3:c0bc6424:::10000000002.00000035:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.023445873, 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017860478999999999, 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.605 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 57 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558036+0000", 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9799999999999998e-06 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562071+0000", 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040351400000000004 2024-04-25T15:52:53.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562126+0000", 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4249000000000002e-05 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562149+0000", 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3674000000000001e-05 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562327+0000", 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177498 2024-04-25T15:52:53.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562581+0000", 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025446799999999999 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562836+0000", 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000254751 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.572670+0000", 2024-04-25T15:52:53.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0098337700000000004 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575703+0000", 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030327980000000002 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575802+0000", 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9278999999999996e-05 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575894+0000", 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1872000000000004e-05 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:59 3.20 3:072133bc:::10000000002.00000037:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.9481095019999994, 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016873692999999999, 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.611 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 59 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633373+0000", 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2430000000000001e-06 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637297+0000", 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039241060000000001 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637354+0000", 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7357e-05 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637380+0000", 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5165999999999999e-05 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637564+0000", 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000184332 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637837+0000", 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027320399999999998 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.638090+0000", 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025292100000000003 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.647362+0000", 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092716989999999996 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650052+0000", 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026907599999999999 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650151+0000", 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8603000000000004e-05 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650243+0000", 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2301999999999998e-05 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:62 3.11 e60/54 3:8969f477:::10000000002.0000003a:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:53.616 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.8197956800000004, 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019322214000000001, 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:53.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761684+0000", 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3399999999999998e-07 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765759+0000", 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040747159999999999 2024-04-25T15:52:53.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765801+0000", 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1675999999999999e-05 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765835+0000", 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4518999999999999e-05 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765978+0000", 2024-04-25T15:52:53.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014263899999999999 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.766025+0000", 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6842999999999999e-05 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780902+0000", 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014877118 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781006+0000", 2024-04-25T15:52:53.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010396899999999999 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:73 3.1d e60/54 3:b94578cb:::10000000002.00000045:head v 60'2, mlcod=0'0)", 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.3180702800000006, 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019180661000000002, 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263412+0000", 2024-04-25T15:52:53.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9900000000000002e-06 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268799+0000", 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053871980000000002 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268844+0000", 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5062999999999999e-05 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268870+0000", 2024-04-25T15:52:53.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5633e-05 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269028+0000", 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000158135 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269078+0000", 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0463000000000001e-05 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282434+0000", 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013355156999999999 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282590+0000", 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015602199999999999 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:75 3.35 3:ad096b5b:::10000000002.00000047:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:53.625 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.2479700329999996, 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022276672000000001, 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333513+0000", 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339006+0000", 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054929599999999999 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339071+0000", 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5545999999999999e-05 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339098+0000", 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6647000000000001e-05 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339299+0000", 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201351 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339703+0000", 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040396399999999998 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.340160+0000", 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045697899999999999 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.352668+0000", 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012508137000000001 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355605+0000", 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002937013 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355701+0000", 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5681999999999996e-05 2024-04-25T15:52:53.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355786+0000", 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4908999999999996e-05 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:79 3.3 3:c0361aff:::10000000002.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.0721999380000007, 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022495096999999999, 2024-04-25T15:52:53.631 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 79 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:53.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509282+0000", 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2200000000000001e-06 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517219+0000", 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079362270000000006 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517284+0000", 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5052e-05 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517310+0000", 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6072999999999999e-05 2024-04-25T15:52:53.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517506+0000", 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019606199999999999 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517779+0000", 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027302799999999998 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.518047+0000", 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000268454 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.528734+0000", 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01068712 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531586+0000", 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028516589999999999 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531689+0000", 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010260900000000001 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531774+0000", 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5593000000000005e-05 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:80 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.029634304, 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018714178000000001, 2024-04-25T15:52:53.637 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 80 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551848+0000", 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0230000000000001e-06 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556855+0000", 2024-04-25T15:52:53.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050073180000000002 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556907+0000", 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1446000000000002e-05 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556929+0000", 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2405999999999999e-05 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557100+0000", 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000171003 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557387+0000", 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028688699999999999 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557640+0000", 2024-04-25T15:52:53.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025292499999999998 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.566164+0000", 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085239770000000003 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570375+0000", 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004211345 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570471+0000", 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5933000000000005e-05 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570559+0000", 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7915000000000001e-05 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:82 3.34 3:2c8ffcca:::10000000002.0000004e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 7.9417789770000002, 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017224060999999999, 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 82 2024-04-25T15:52:53.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639703+0000", 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7070000000000001e-06 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644603+0000", 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048998549999999998 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644658+0000", 2024-04-25T15:52:53.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4987000000000001e-05 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644682+0000", 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4280000000000001e-05 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644876+0000", 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000193678 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.645141+0000", 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000265321 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.645394+0000", 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025291099999999998 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.655180+0000", 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0097862139999999997 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656739+0000", 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015587800000000001 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656840+0000", 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101172 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656924+0000", 2024-04-25T15:52:53.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4155999999999999e-05 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:141 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.240113+0000", 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.3413665630000002, 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016842197, 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 141 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.240113+0000", 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.240113+0000", 2024-04-25T15:52:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.240116+0000", 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.411e-06 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244602+0000", 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044857550000000001 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244648+0000", 2024-04-25T15:52:53.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.613e-05 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244666+0000", 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8003e-05 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244759+0000", 2024-04-25T15:52:53.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3203999999999997e-05 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244904+0000", 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014452000000000001 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.245015+0000", 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011147800000000001 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.253876+0000", 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088605030000000005 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.256853+0000", 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029777689999999999 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.256910+0000", 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6419999999999999e-05 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.256955+0000", 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5003999999999998e-05 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:144 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.2132924909999998, 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019320924999999999, 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 144 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368190+0000", 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4170000000000001e-06 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373302+0000", 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005111857 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373366+0000", 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3570000000000003e-05 2024-04-25T15:52:53.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373413+0000", 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7679000000000002e-05 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373579+0000", 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000165422 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373836+0000", 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025765800000000002 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.374080+0000", 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024384499999999999 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.386500+0000", 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012419688999999999 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387295+0000", 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00079550599999999997 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387403+0000", 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010791899999999999 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387508+0000", 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104363 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:154 3.34 3:2cf0bbf2:::10000000002.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.834003+0000", 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.747476153, 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018116257, 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 154 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.834003+0000", 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.834003+0000", 2024-04-25T15:52:53.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.834007+0000", 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6330000000000002e-06 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.838978+0000", 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049714049999999999 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839038+0000", 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0133000000000002e-05 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839064+0000", 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5962999999999999e-05 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839256+0000", 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019204199999999999 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839527+0000", 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027048099999999999 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839790+0000", 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000263114 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.849864+0000", 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010074471999999999 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.851907+0000", 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020428260000000002 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.852017+0000", 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110402 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.852119+0000", 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101786 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:165 3.11 e62/54 3:88ab7984:::10000000002.000000a1:head v 62'7, mlcod=61'5)", 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.2004053109999999, 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020252645999999999, 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381075+0000", 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9299999999999997e-07 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386313+0000", 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052379799999999997 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386350+0000", 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7469000000000003e-05 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386384+0000", 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3862999999999997e-05 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386546+0000", 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016226899999999999 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386587+0000", 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1026000000000003e-05 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401215+0000", 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01462777 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401327+0000", 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111576 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:166 3.2c 3:3723e56f:::10000000002.000000a2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.1674569290000001, 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019979538000000002, 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.672 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 166 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414025+0000", 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.92e-06 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.419994+0000", 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059684200000000003 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420068+0000", 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4878999999999999e-05 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420093+0000", 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4553e-05 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420239+0000", 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014577500000000001 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420512+0000", 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027295800000000001 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420761+0000", 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024945499999999999 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.431910+0000", 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011148739 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.433806+0000", 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018964909999999999 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.433907+0000", 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100179 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.434002+0000", 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5168999999999996e-05 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:176 3.26 3:66bf65cf:::10000000002.000000ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.7546500790000001, 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020587095999999999, 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.678 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 176 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826832+0000", 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1200000000000002e-06 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831165+0000", 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043325289999999999 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831220+0000", 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5686999999999997e-05 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831245+0000", 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4105999999999999e-05 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831400+0000", 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015513899999999999 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831669+0000", 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026955099999999998 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831947+0000", 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027750399999999998 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.842230+0000", 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010283537000000001 2024-04-25T15:52:53.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847229+0000", 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049990939999999999 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847326+0000", 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7106000000000002e-05 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847416+0000", 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9722999999999995e-05 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.683 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:177 3.26 3:66aef6e9:::10000000002.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.704791181, 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026823691, 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.684 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876691+0000", 2024-04-25T15:52:53.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4939999999999999e-06 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880736+0000", 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040441660000000001 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880793+0000", 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7581999999999999e-05 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880817+0000", 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881039+0000", 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022171800000000001 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881305+0000", 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026580799999999997 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:53.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881558+0000", 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025290099999999998 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.893182+0000", 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011624242 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903322+0000", 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010139754000000001 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903425+0000", 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102706 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903512+0000", 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7093000000000001e-05 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:179 3.1e e62/18 3:789394ea:::10000000002.000000af:head v 62'2, mlcod=0'0)", 2024-04-25T15:52:53.689 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.963928+0000", 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.6175508399999998, 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018290402000000001, 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.963928+0000", 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.963928+0000", 2024-04-25T15:52:53.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.963929+0000", 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969719+0000", 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057901879999999999 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969755+0000", 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6019000000000002e-05 2024-04-25T15:52:53.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969779+0000", 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3266999999999999e-05 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969939+0000", 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016062199999999999 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969992+0000", 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2519000000000003e-05 2024-04-25T15:52:53.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.982088+0000", 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012096129000000001 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.982219+0000", 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013094499999999999 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:182 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:53.693 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.4917580250000002, 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018848521, 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 182 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089724+0000", 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2270000000000002e-06 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095098+0000", 2024-04-25T15:52:53.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053738709999999997 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095165+0000", 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6618999999999994e-05 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095192+0000", 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6783000000000001e-05 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095330+0000", 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013835900000000001 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095586+0000", 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025566400000000001 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095830+0000", 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024478199999999997 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.104485+0000", 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086547110000000007 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108421+0000", 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039358010000000001 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108497+0000", 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5799999999999999e-05 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:53.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108570+0000", 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2904000000000005e-05 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:53.699 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:53.701 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:53.701 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:53.698+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:53.801 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:53.799+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:53.898 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:53.898 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:53.898 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:53.898 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:53.902 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:53.899+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:53.915 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:54.007 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:54.007+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:54.105 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:54.103+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:54.110 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:54.110 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:54.111 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:54.111 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:54.111 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:54.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:54.205 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:54.206+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:54.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:52:54.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:52:54.277 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T15:52:54.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:52:54.306 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:54.303+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 155.223892019, 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.326 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:52:54.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 154.632837717, 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:52:54.332 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:52:54.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:52:54.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 154.63268909499999, 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.338 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:54.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:52:54.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:52:54.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.344 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.344 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.344 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 154.632678016, 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:54.345 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:52:54.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 154.32497497700001, 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.351 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:52:54.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:52:54.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:41 3.f 3:f045ad74:::10000000002.00000025:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.512890787, 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022153520999999999, 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 41 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.357 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809469+0000", 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7670000000000001e-06 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814177+0000", 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047077969999999997 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814211+0000", 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4173e-05 2024-04-25T15:52:54.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814226+0000", 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5444e-05 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814394+0000", 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016725400000000001 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814776+0000", 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038195999999999998 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.815181+0000", 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040559299999999999 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.824339+0000", 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0091577129999999996 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831521+0000", 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071817010000000004 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831568+0000", 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7250000000000003e-05 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831619+0000", 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0868999999999997e-05 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:47 3.12 3:489b4446:::10000000002.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.22183225, 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022914499000000001, 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:52:54.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100528+0000", 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3030000000000001e-06 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105104+0000", 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045757710000000002 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105167+0000", 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2936999999999997e-05 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105194+0000", 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7606e-05 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105424+0000", 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229998 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105705+0000", 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000280408 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105966+0000", 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261258 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.118921+0000", 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012954482999999999 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123247+0000", 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043263199999999998 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123350+0000", 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103423 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123438+0000", 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7991999999999994e-05 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:62 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.566703124, 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025889728000000001, 2024-04-25T15:52:54.369 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 62 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755657+0000", 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9439999999999998e-06 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760593+0000", 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049360740000000004 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760651+0000", 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8e-05 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760676+0000", 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5219999999999999e-05 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760847+0000", 2024-04-25T15:52:54.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017057199999999999 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761123+0000", 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027578100000000001 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761365+0000", 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024248100000000001 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780642+0000", 2024-04-25T15:52:54.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019277327 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781358+0000", 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000715611 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781456+0000", 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7946e-05 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781543+0000", 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6772000000000002e-05 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:72 3.39 3:9df4b1b7:::10000000002.00000044:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.1066297079999998, 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022151042999999999, 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.375 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215730+0000", 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221532+0000", 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058028009999999998 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221606+0000", 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3842000000000002e-05 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221642+0000", 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6232999999999997e-05 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221851+0000", 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020811199999999999 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.222277+0000", 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042676200000000002 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.222696+0000", 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041885400000000003 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.236603+0000", 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013906992 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237694+0000", 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001091091 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237791+0000", 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6455999999999999e-05 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237877+0000", 2024-04-25T15:52:54.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6749999999999994e-05 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:150 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6711841920000001, 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024331635000000001, 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.381 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 150 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651176+0000", 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8770000000000003e-06 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659329+0000", 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081530219999999994 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659399+0000", 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0376000000000005e-05 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659428+0000", 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8952999999999999e-05 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659580+0000", 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015188199999999999 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659842+0000", 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026179399999999998 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:54.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.660120+0000", 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027824799999999998 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.673813+0000", 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013693145 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675307+0000", 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014934600000000001 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675412+0000", 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105233 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675504+0000", 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1644999999999994e-05 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:152 3.39 3:9e24e8b4:::10000000002.00000094:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5913161650000003, 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022233425000000001, 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 152 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731043+0000", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.735937+0000", 2024-04-25T15:52:54.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048940520000000003 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736003+0000", 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6112000000000007e-05 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736030+0000", 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6744e-05 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736177+0000", 2024-04-25T15:52:54.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014675800000000001 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736344+0000", 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016679100000000001 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736481+0000", 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000137112 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.744634+0000", 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081533030000000006 2024-04-25T15:52:54.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753167+0000", 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085328769999999995 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753222+0000", 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5167999999999998e-05 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753273+0000", 2024-04-25T15:52:54.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1304999999999998e-05 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:155 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.4490329070000003, 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025243293999999999, 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.392 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 155 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873327+0000", 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7699999999999999e-06 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879173+0000", 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005846141 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879239+0000", 2024-04-25T15:52:54.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6315999999999993e-05 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879267+0000", 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7137000000000001e-05 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879415+0000", 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000148715 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879693+0000", 2024-04-25T15:52:54.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000278108 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879951+0000", 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025793700000000002 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.896019+0000", 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016067814999999999 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898153+0000", 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021339280000000002 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898411+0000", 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025760799999999999 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898566+0000", 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015581899999999999 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:158 3.12 3:4b703321:::10000000002.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.3121351920000004, 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024089508999999999, 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 158 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010224+0000", 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3969999999999998e-06 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016098+0000", 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058735000000000002 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016148+0000", 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0640999999999998e-05 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016177+0000", 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.813e-05 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016313+0000", 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136052 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016470+0000", 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000157637 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016601+0000", 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013075599999999999 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.023425+0000", 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068237799999999998 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034203+0000", 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01077811 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034260+0000", 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6702e-05 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034310+0000", 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0803999999999997e-05 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:162 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.218611+0000", 2024-04-25T15:52:54.403 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.1037452679999999, 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02174914, 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 162 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.218611+0000", 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.218611+0000", 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.218615+0000", 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9870000000000001e-06 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224073+0000", 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054577610000000002 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224135+0000", 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2496000000000006e-05 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224162+0000", 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6800000000000001e-05 2024-04-25T15:52:54.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224348+0000", 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018565399999999999 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224622+0000", 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027439199999999998 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224874+0000", 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025226799999999999 2024-04-25T15:52:54.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.232926+0000", 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080514980000000007 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.240158+0000", 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072328189999999997 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.240264+0000", 2024-04-25T15:52:54.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010596599999999999 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.240360+0000", 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5499000000000002e-05 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:165 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.9530623699999996, 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032658418000000002, 2024-04-25T15:52:54.409 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 165 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369298+0000", 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3429999999999998e-06 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374706+0000", 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054080250000000003 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374776+0000", 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9972e-05 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374803+0000", 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7384000000000001e-05 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375010+0000", 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206431 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375037+0000", 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7206999999999999e-05 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380232+0000", 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051947859999999998 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380474+0000", 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000241864 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380709+0000", 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023560200000000001 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.392962+0000", 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012252219999999999 2024-04-25T15:52:54.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401765+0000", 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088034089999999999 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401866+0000", 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101099 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401952+0000", 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6075999999999994e-05 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.415 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:173 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.6130458890000003, 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022867683, 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.416 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 173 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709313+0000", 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8159999999999998e-06 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716389+0000", 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070763739999999999 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716451+0000", 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1958999999999994e-05 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716476+0000", 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4246000000000001e-05 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716689+0000", 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000212926 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716951+0000", 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026266700000000001 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.717199+0000", 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024812800000000002 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.726072+0000", 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088724990000000007 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.731628+0000", 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055558229999999997 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.732003+0000", 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00037564100000000002 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.732178+0000", 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000174604 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:184 3.1b 3:db82751e:::10000000002.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.174460+0000", 2024-04-25T15:52:54.421 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.1478956990000002, 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022559854000000001, 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 184 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.174460+0000", 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.174460+0000", 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.174464+0000", 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.14e-06 2024-04-25T15:52:54.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181591+0000", 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071270270000000002 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181652+0000", 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1177000000000001e-05 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181678+0000", 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6276000000000001e-05 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181881+0000", 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020344199999999999 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.182292+0000", 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041062299999999999 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.182742+0000", 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044982600000000001 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.193295+0000", 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010553294 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.196703+0000", 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034081409999999999 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.196879+0000", 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017578900000000001 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.197020+0000", 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.427 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:191 3.39 3:9c9620cb:::10000000002.000000bb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.498577+0000", 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.8237787880000003, 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022002436, 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 191 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.428 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.498577+0000", 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.498577+0000", 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.498579+0000", 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9539999999999998e-06 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503161+0000", 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045813019999999998 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503203+0000", 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1953999999999998e-05 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503221+0000", 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8457e-05 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503384+0000", 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162867 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503640+0000", 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025651800000000001 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503879+0000", 2024-04-25T15:52:54.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023874200000000001 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.512692+0000", 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088129860000000001 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.520260+0000", 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007567898 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.520442+0000", 2024-04-25T15:52:54.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181609 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.520580+0000", 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013814900000000001 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.1046515379999999, 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.433 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:52:54.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.010017486, 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:54.439 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:52:54.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:54.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.444 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.445 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.445 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:54.445 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:54.445 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:54.446 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:54.443+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:54.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:54.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.446+0000 7fba5ecaa640 1 Processor -- start 2024-04-25T15:52:54.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.446+0000 7fba5ecaa640 1 -- start start 2024-04-25T15:52:54.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5ecaa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba58074790 0x7fba5815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5ecaa640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fba5815e020 0x7fba580723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5ecaa640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba58072dc0 con 0x7fba580728e0 2024-04-25T15:52:54.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5ecaa640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fba5815b430 con 0x7fba5815e020 2024-04-25T15:52:54.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5ecaa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fba5815a670 con 0x7fba58074790 2024-04-25T15:52:54.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5d247640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba580728e0 0x7fba580c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42474/0 (socket says 172.21.15.67:42474) 2024-04-25T15:52:54.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5d247640 1 -- 172.21.15.67:0/3675489175 learned_addr learned my addr 172.21.15.67:0/3675489175 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:54.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5ca46640 1 --2- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba58074790 0x7fba5815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba57fff640 1 --2- 172.21.15.67:0/3675489175 >> v2:172.21.15.176:3300/0 conn(0x7fba5815e020 0x7fba580723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba577fe640 1 -- 172.21.15.67:0/3675489175 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2473496258 0 0) 0x7fba58072dc0 con 0x7fba580728e0 2024-04-25T15:52:54.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba577fe640 1 -- 172.21.15.67:0/3675489175 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba48003530 con 0x7fba580728e0 2024-04-25T15:52:54.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba57fff640 1 -- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba58074790 msgr2=0x7fba5815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:54.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba577fe640 1 -- 172.21.15.67:0/3675489175 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2662496704 0 0) 0x7fba48003530 con 0x7fba580728e0 2024-04-25T15:52:54.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba57fff640 1 --2- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba58074790 0x7fba5815dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba57fff640 1 -- 172.21.15.67:0/3675489175 >> v1:172.21.15.67:6789/0 conn(0x7fba580728e0 legacy=0x7fba580c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba57fff640 1 -- 172.21.15.67:0/3675489175 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5815a950 con 0x7fba5815e020 2024-04-25T15:52:54.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.450+0000 7fba5ca46640 1 --2- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba58074790 0x7fba5815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba57fff640 1 --2- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 0x7fba580723a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba440097c0 tx=0x7fba4401ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4231e9baad0b158 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba577fe640 1 -- 172.21.15.67:0/3675489175 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba440b5070 con 0x7fba5815e020 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba577fe640 1 -- 172.21.15.67:0/3675489175 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba440a85f0 con 0x7fba5815e020 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba577fe640 1 -- 172.21.15.67:0/3675489175 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50001a00 con 0x7fba5815e020 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 -- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 msgr2=0x7fba580723a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 0x7fba580723a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba440097c0 tx=0x7fba4401ebb0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 -- 172.21.15.67:0/3675489175 shutdown_connections 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 0x7fba580723a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/3675489175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba58074790 0x7fba5815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 -- 172.21.15.67:0/3675489175 >> 172.21.15.67:0/3675489175 conn(0x7fba5806fe70 msgr2=0x7fba580cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 -- 172.21.15.67:0/3675489175 shutdown_connections 2024-04-25T15:52:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.454+0000 7fba5ecaa640 1 -- 172.21.15.67:0/3675489175 wait complete. 2024-04-25T15:52:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 Processor -- start 2024-04-25T15:52:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 -- start start 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba580728e0 0x7fba58148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba58074790 0x7fba58153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 0x7fba581f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba58073080 con 0x7fba58074790 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fba5815af10 con 0x7fba5815e020 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fba58074ba0 con 0x7fba580728e0 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba58074790 0x7fba58153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba58074790 0x7fba58153a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60470/0 (socket says 172.21.15.67:60470) 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 -- 172.21.15.67:0/2219243745 learned_addr learned my addr 172.21.15.67:0/2219243745 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:54.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ca46640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba580728e0 0x7fba58148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 -- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba580728e0 msgr2=0x7fba58148350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba580728e0 0x7fba58148350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 -- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 msgr2=0x7fba581f0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 0x7fba581f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 -- 172.21.15.67:0/2219243745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba58153fb0 con 0x7fba58074790 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba57fff640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba58074790 0x7fba58153a70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fba4401d260 tx=0x7fba4401b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba557fa640 1 -- 172.21.15.67:0/2219243745 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba440bf070 con 0x7fba58074790 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba557fa640 1 -- 172.21.15.67:0/2219243745 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba440a89a0 con 0x7fba58074790 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5815cab0 con 0x7fba58074790 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba557fa640 1 -- 172.21.15.67:0/2219243745 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba440b5040 con 0x7fba58074790 2024-04-25T15:52:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba5815d550 con 0x7fba58074790 2024-04-25T15:52:54.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.458+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba581541c0 con 0x7fba58074790 2024-04-25T15:52:54.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.462+0000 7fba557fa640 1 -- 172.21.15.67:0/2219243745 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fba44003620 con 0x7fba58074790 2024-04-25T15:52:54.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.462+0000 7fba557fa640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fba3c06f400 0x7fba3c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.462+0000 7fba557fa640 1 -- 172.21.15.67:0/2219243745 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5886+0+0 (secure 0 0 0) 0x7fba4412bb60 con 0x7fba58074790 2024-04-25T15:52:54.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.462+0000 7fba5ca46640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fba3c06f400 0x7fba3c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.462+0000 7fba5ca46640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fba3c06f400 0x7fba3c071910 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009580 tx=0x7fba4c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:54.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.466+0000 7fba557fa640 1 -- 172.21.15.67:0/2219243745 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fba44018710 con 0x7fba58074790 2024-04-25T15:52:54.547 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:54.546+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:54.587 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:54.588 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:54.588 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:54.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.638+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fba5806a480 con 0x7fba58074790 2024-04-25T15:52:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.642+0000 7fba557fa640 1 -- 172.21.15.67:0/2219243745 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v64) ==== 81+0+35136 (secure 0 0 0) 0x7fba440f8470 con 0x7fba58074790 2024-04-25T15:52:54.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 64, 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:52:53.282383+0000", 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:52:31.099760+0000", 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5801984, 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-04-25T15:52:54.645 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:52:54.646 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:52:54.647 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:54.648 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:54.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:54.650 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.651 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:52:54.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:54.653 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:54.654 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:52:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:54.656 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:54.657 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:52:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.75, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.75, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.75, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.75, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:54.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:52:54.661 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:52:54.662 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:52:54.663 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:52:54.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:52:54.665 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.5299999713897705, 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.5299999713897705, 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1, 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-25T15:52:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:54.667 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.674 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.675 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.678 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.680 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.682 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:54.683 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.685 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:52:54.687 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 0, 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0, 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:54.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 40, 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:52:54.692 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.693 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.696 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:52:54.697 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.698 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:52:54.702 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:52:54.703 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 54, 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:52:54.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:52:54.706 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:52:54.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.708 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.709 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.711 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 65536, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:52:54.713 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.18", 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: 4 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.714 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:52:54.715 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:52:54.716 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:52:54.717 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:52:54.718 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:52:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:52:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:52:54.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:54.719 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fba3c06f400 msgr2=0x7fba3c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fba3c06f400 0x7fba3c071910 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009580 tx=0x7fba4c009190 comp rx=0 tx=0).stop 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba58074790 msgr2=0x7fba58153a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba58074790 0x7fba58153a70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fba4401d260 tx=0x7fba4401b950 comp rx=0 tx=0).stop 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 shutdown_connections 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fba3c06f400 0x7fba3c071910 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba5815e020 0x7fba581f0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba58074790 0x7fba58153a70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 --2- 172.21.15.67:0/2219243745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba580728e0 0x7fba58148350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 >> 172.21.15.67:0/2219243745 conn(0x7fba5806fe70 msgr2=0x7fba58075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 shutdown_connections 2024-04-25T15:52:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.646+0000 7fba5ecaa640 1 -- 172.21.15.67:0/2219243745 wait complete. 2024-04-25T15:52:54.722 INFO:tasks.thrashosds.thrasher:j is {'epoch': 64, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:52:53.282383+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:52:31.099760+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [1, 4]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:52:54.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:52:54.723 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:54.718+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:54.800 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:54.800 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:54.800 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:54.800 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:54.801 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:54.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:54.820 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:54.819+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:54.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.906+0000 7fc4f09d8640 1 Processor -- start 2024-04-25T15:52:54.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.906+0000 7fc4f09d8640 1 -- start start 2024-04-25T15:52:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.906+0000 7fc4f09d8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc4ec0caf60 0x7fc4ec15d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.906+0000 7fc4f09d8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4ec0c77a0 con 0x7fc4ec15d790 2024-04-25T15:52:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.906+0000 7fc4f09d8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4ec0c7a60 con 0x7fc4ec073bf0 2024-04-25T15:52:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.906+0000 7fc4f09d8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc4ec0cab60 con 0x7fc4ec0caf60 2024-04-25T15:52:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.906+0000 7fc4e9d9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc4ec15d790 0x7fc4ec06d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42482/0 (socket says 172.21.15.67:42482) 2024-04-25T15:52:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4e9d9b640 1 -- 172.21.15.67:0/3904415079 learned_addr learned my addr 172.21.15.67:0/3904415079 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4ea59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc4ec0caf60 0x7fc4ec15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4e959a640 1 -- 172.21.15.67:0/3904415079 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323361300 0 0) 0x7fc4ec0c77a0 con 0x7fc4ec15d790 2024-04-25T15:52:54.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4e959a640 1 -- 172.21.15.67:0/3904415079 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4d4003530 con 0x7fc4ec15d790 2024-04-25T15:52:54.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4ea59c640 1 -- 172.21.15.67:0/3904415079 >> v1:172.21.15.176:6789/0 conn(0x7fc4ec073bf0 legacy=0x7fc4ec163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:54.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4ea59c640 1 -- 172.21.15.67:0/3904415079 >> v1:172.21.15.67:6789/0 conn(0x7fc4ec15d790 legacy=0x7fc4ec06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4ea59c640 1 -- 172.21.15.67:0/3904415079 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec0c7d40 con 0x7fc4ec0caf60 2024-04-25T15:52:54.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4ea59c640 1 --2- 172.21.15.67:0/3904415079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec15d250 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0097c0 tx=0x7fc4dc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1516566686d8634 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:54.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4e959a640 1 -- 172.21.15.67:0/3904415079 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0b0070 con 0x7fc4ec0caf60 2024-04-25T15:52:54.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4e959a640 1 -- 172.21.15.67:0/3904415079 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4dc0ab600 con 0x7fc4ec0caf60 2024-04-25T15:52:54.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4e959a640 1 -- 172.21.15.67:0/3904415079 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0ab9d0 con 0x7fc4ec0caf60 2024-04-25T15:52:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3904415079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 msgr2=0x7fc4ec15d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3904415079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec15d250 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0097c0 tx=0x7fc4dc01c4a0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3904415079 shutdown_connections 2024-04-25T15:52:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3904415079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec15d250 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3904415079 >> 172.21.15.67:0/3904415079 conn(0x7fc4ec070db0 msgr2=0x7fc4ec0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3904415079 shutdown_connections 2024-04-25T15:52:54.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.910+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3904415079 wait complete. 2024-04-25T15:52:54.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 Processor -- start 2024-04-25T15:52:54.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 -- start start 2024-04-25T15:52:54.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4ec073bf0 0x7fc4ec15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4ec15d790 0x7fc4ec1531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec167e00 con 0x7fc4ec0caf60 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec159620 con 0x7fc4ec073bf0 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4f09d8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ec167f80 con 0x7fc4ec15d790 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec1521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60482/0 (socket says 172.21.15.67:60482) 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 -- 172.21.15.67:0/3615669354 learned_addr learned my addr 172.21.15.67:0/3615669354 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4ea59c640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4ec073bf0 0x7fc4ec15dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4ead9d640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4ec15d790 0x7fc4ec1531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 -- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4ec15d790 msgr2=0x7fc4ec1531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4ec15d790 0x7fc4ec1531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 -- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4ec073bf0 msgr2=0x7fc4ec15dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4ec073bf0 0x7fc4ec15dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 -- 172.21.15.67:0/3615669354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec0cd520 con 0x7fc4ec0caf60 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4ead9d640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4ec15d790 0x7fc4ec1531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.914+0000 7fc4e9d9b640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec1521a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000cbe0 tx=0x7fc4e000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4daffd640 1 -- 172.21.15.67:0/3615669354 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0017070 con 0x7fc4ec0caf60 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4daffd640 1 -- 172.21.15.67:0/3615669354 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e000aca0 con 0x7fc4ec0caf60 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ec1526e0 con 0x7fc4ec0caf60 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4daffd640 1 -- 172.21.15.67:0/3615669354 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0012600 con 0x7fc4ec0caf60 2024-04-25T15:52:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ec1513e0 con 0x7fc4ec0caf60 2024-04-25T15:52:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ec06a480 con 0x7fc4ec0caf60 2024-04-25T15:52:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4daffd640 1 -- 172.21.15.67:0/3615669354 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc4e001a070 con 0x7fc4ec0caf60 2024-04-25T15:52:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4daffd640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4cc06f400 0x7fc4cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4daffd640 1 -- 172.21.15.67:0/3615669354 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5886+0+0 (secure 0 0 0) 0x7fc4e008ef30 con 0x7fc4ec0caf60 2024-04-25T15:52:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4ea59c640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4cc06f400 0x7fc4cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:54.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.918+0000 7fc4ea59c640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4cc06f400 0x7fc4cc071910 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0ad000 tx=0x7fc4dc0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:54.921 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:54.922+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:54.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:54.922+0000 7fc4daffd640 1 -- 172.21.15.67:0/3615669354 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc4e005b840 con 0x7fc4ec0caf60 2024-04-25T15:52:55.022 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:55.022+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:55.025 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:55.025 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:55.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:55.025 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 155.02093581299999, 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 138.81204817, 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:55.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:55.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:55.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:55.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:55.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 138.81200078500001, 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:55.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:55.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 138.811988703, 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:55.040 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:55.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:55.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.045 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 138.81196853099999, 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:55.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:55.049 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 138.81173019299999, 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:55.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:36 3.1b e59/54 3:d88a10eb:::10000000002.00000020:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.457998239, 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013408461999999999, 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565546+0000", 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.565548+0000", 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5620000000000002e-06 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572005+0000", 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064569370000000003 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572034+0000", 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8586999999999999e-05 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572056+0000", 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2481e-05 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572135+0000", 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9264999999999994e-05 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.572159+0000", 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3275e-05 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.578912+0000", 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067533030000000004 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.578954+0000", 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2052000000000003e-05 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.207778469000001, 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:55.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:52:55.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:52:55.061 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.916980326000001, 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:52:55.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:53 3.20 e60/18 3:068d0381:::10000000002.00000031:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.639522842, 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014486163, 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.066 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:55.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389720+0000", 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056985300000000003 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389742+0000", 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1838999999999999e-05 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389758+0000", 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6058000000000001e-05 2024-04-25T15:52:55.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389844+0000", 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6293999999999997e-05 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389866+0000", 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2033000000000001e-05 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398429+0000", 2024-04-25T15:52:55.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085631660000000005 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398507+0000", 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7869999999999998e-05 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:66 3.b e60/21 3:d381a89f:::10000000002.0000003e:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.078330185, 2024-04-25T15:52:55.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015316048000000001, 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945216+0000", 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9e-06 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948749+0000", 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035338119999999999 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948777+0000", 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7915999999999998e-05 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948796+0000", 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.844e-05 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948895+0000", 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9704000000000006e-05 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948916+0000", 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.107e-05 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960445+0000", 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011528722999999999 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960530+0000", 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4482999999999997e-05 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:152 3.39 e61/18 3:9e24e8b4:::10000000002.00000094:head v 61'2, mlcod=0'0)", 2024-04-25T15:52:55.074 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2864601069999999, 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016125006000000001, 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:55.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737086+0000", 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744295+0000", 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072082320000000002 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744341+0000", 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6072000000000002e-05 2024-04-25T15:52:55.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744376+0000", 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4903000000000001e-05 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744533+0000", 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015736299999999999 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744590+0000", 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6771000000000002e-05 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753101+0000", 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085115490000000002 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753209+0000", 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107352 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:153 3.1 e61/54 3:814e3e07:::10000000002.00000095:head v 61'3, mlcod=59'1)", 2024-04-25T15:52:55.078 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.229202678, 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014546699999999999, 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:55.079 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794344+0000", 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796364+0000", 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002019968 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796414+0000", 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0065999999999998e-05 2024-04-25T15:52:55.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796445+0000", 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0783e-05 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796600+0000", 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154685 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796670+0000", 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0605999999999995e-05 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.808781+0000", 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012110368999999999 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.808888+0000", 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107353 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.082 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:156 3.3e e61/35 3:7d47d675:::10000000002.00000098:head v 61'1, mlcod=0'0)", 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0893066339999997, 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016362634000000001, 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.083 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934240+0000", 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3149999999999999e-06 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939270+0000", 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050303869999999999 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.084 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939308+0000", 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8047999999999999e-05 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939335+0000", 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7251999999999999e-05 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939432+0000", 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6637000000000003e-05 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939459+0000", 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7651999999999999e-05 2024-04-25T15:52:55.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950534+0000", 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011074993999999999 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950600+0000", 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5349000000000001e-05 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.086 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:158 3.12 e61/40 3:4b703321:::10000000002.0000009a:head v 61'6, mlcod=59'4)", 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.0063465069999999, 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017102098, 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.087 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017200+0000", 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3199999999999998e-06 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024143+0000", 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069429569999999996 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.088 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024169+0000", 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6007e-05 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024190+0000", 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1203999999999999e-05 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024272+0000", 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1833000000000001e-05 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024301+0000", 2024-04-25T15:52:55.089 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9475000000000002e-05 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034243+0000", 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099420589999999996 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034299+0000", 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6242999999999997e-05 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:162 3.2b e61/40 3:d691b431:::10000000002.0000009e:head v 61'3, mlcod=60'1)", 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.7980024959999996, 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014572351000000001, 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225544+0000", 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7199999999999998e-06 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230207+0000", 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046626719999999997 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.092 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230239+0000", 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2548000000000001e-05 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230266+0000", 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6599999999999999e-05 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230382+0000", 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115667 2024-04-25T15:52:55.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230417+0000", 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5800999999999998e-05 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240029+0000", 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096115349999999992 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240114+0000", 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4808000000000005e-05 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.094 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:164 3.30 e62/18 3:0efeaece:::10000000002.000000a0:head v 62'5, mlcod=60'3)", 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6859470569999999, 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013652381, 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:55.095 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8999999999999997e-07 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343547+0000", 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059493669999999997 2024-04-25T15:52:55.096 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343576+0000", 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9850000000000001e-05 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343597+0000", 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0330999999999999e-05 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343687+0000", 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0278999999999994e-05 2024-04-25T15:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343710+0000", 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2796e-05 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.351154+0000", 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074439739999999999 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.351249+0000", 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5293999999999999e-05 2024-04-25T15:52:55.098 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:173 3.2b e62/40 3:d5480d6b:::10000000002.000000a9:head v 62'5, mlcod=61'3)", 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.717902+0000", 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.3056421719999998, 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013644187, 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.099 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.717902+0000", 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.717902+0000", 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.717904+0000", 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6989999999999999e-06 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722820+0000", 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049151619999999998 2024-04-25T15:52:55.100 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722842+0000", 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2408999999999998e-05 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722855+0000", 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3485999999999999e-05 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722943+0000", 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7981999999999999e-05 2024-04-25T15:52:55.101 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722976+0000", 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3096999999999998e-05 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.731460+0000", 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084837539999999996 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.731546+0000", 2024-04-25T15:52:55.102 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5598000000000003e-05 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:181 3.1f e62/40 3:f8dc3ee2:::10000000002.000000b1:head v 62'2, mlcod=0'0)", 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.9733319009999999, 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015250434, 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.103 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050214+0000", 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.013e-06 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.104 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054240+0000", 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040256609999999998 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054262+0000", 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2691e-05 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054277+0000", 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4464000000000001e-05 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.105 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054365+0000", 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8657000000000002e-05 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054400+0000", 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4505999999999998e-05 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065391+0000", 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991287000000001 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.106 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065462+0000", 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1155000000000006e-05 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:249 3.3e e63/35 3:7d99c36c:::10000000002.000000f5:head v 63'3, mlcod=61'1)", 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:53.086243+0000", 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.9373008839999999, 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014234955000000001, 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:55.107 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.086243+0000", 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.086243+0000", 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.086245+0000", 2024-04-25T15:52:55.108 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6130000000000001e-06 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088030+0000", 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017854629999999999 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088058+0000", 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8279000000000001e-05 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:55.109 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088075+0000", 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6711999999999999e-05 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088156+0000", 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1457000000000003e-05 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088184+0000", 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7073999999999999e-05 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.110 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.100414+0000", 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012230636 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.100478+0000", 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3720999999999995e-05 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:55.111 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:55.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:55.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.086+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4ec0c8a40 con 0x7fc4cc06f400 2024-04-25T15:52:55.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.106+0000 7fc4daffd640 1 -- 172.21.15.67:0/3615669354 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253040 (secure 0 0 0) 0x7fc4ec0c8a40 con 0x7fc4cc06f400 2024-04-25T15:52:55.115 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:52:55.116 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:52:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4cc06f400 msgr2=0x7fc4cc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4cc06f400 0x7fc4cc071910 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0ad000 tx=0x7fc4dc0a6000 comp rx=0 tx=0).stop 2024-04-25T15:52:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 msgr2=0x7fc4ec1521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec1521a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc4e000cbe0 tx=0x7fc4e000a060 comp rx=0 tx=0).stop 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 shutdown_connections 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4cc06f400 0x7fc4cc071910 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4ec15d790 0x7fc4ec1531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ec0caf60 0x7fc4ec1521a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 --2- 172.21.15.67:0/3615669354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4ec073bf0 0x7fc4ec15dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 >> 172.21.15.67:0/3615669354 conn(0x7fc4ec070db0 msgr2=0x7fc4ec074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 shutdown_connections 2024-04-25T15:52:55.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.110+0000 7fc4f09d8640 1 -- 172.21.15.67:0/3615669354 wait complete. 2024-04-25T15:52:55.129 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-04-25T15:52:53.848703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1003086138,"num_objects":339,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":576,"num_write_kb":980697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4003,"num_keys_recovered":22,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":519,"ondisk_log_size":519,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":2088108,"kb_used_data":1890676,"kb_used_omap":10,"kb_used_meta":197365,"kb_avail":658514772,"statfs":{"total":676457349120,"available":674319126528,"internally_reserved":0,"allocated":1936052224,"data_stored":1933001920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":202101867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":918573493,"num_objects":219,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":226,"num_write_kb":897048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002292"},"pg_stats":[{"pgid":"3.3f","version":"63'3","reported_seq":101,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.277487+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:52:50.277487+0000","last_peered":"2024-04-25T15:52:50.277487+0000","last_clean":"2024-04-25T15:52:50.277487+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:52:50.277487+0000","last_undegraded":"2024-04-25T15:52:50.277487+0000","last_fullsized":"2024-04-25T15:52:50.277487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"63'2","reported_seq":66,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401813+0000","last_change":"2024-04-25T15:51:47.264956+0000","last_active":"2024-04-25T15:52:50.401813+0000","last_peered":"2024-04-25T15:52:50.401813+0000","last_clean":"2024-04-25T15:52:50.401813+0000","last_became_active":"2024-04-25T15:51:47.264571+0000","last_became_peered":"2024-04-25T15:51:47.264571+0000","last_unstale":"2024-04-25T15:52:50.401813+0000","last_undegraded":"2024-04-25T15:52:50.401813+0000","last_fullsized":"2024-04-25T15:52:50.401813+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:39.343444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"63'3","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.385103+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:52:51.385103+0000","last_peered":"2024-04-25T15:52:51.385103+0000","last_clean":"2024-04-25T15:52:51.385103+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:52:51.385103+0000","last_undegraded":"2024-04-25T15:52:51.385103+0000","last_fullsized":"2024-04-25T15:52:51.385103+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"63'3","reported_seq":102,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478132+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:52:50.478132+0000","last_peered":"2024-04-25T15:52:50.478132+0000","last_clean":"2024-04-25T15:52:50.478132+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:52:50.478132+0000","last_undegraded":"2024-04-25T15:52:50.478132+0000","last_fullsized":"2024-04-25T15:52:50.478132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"63'5","reported_seq":102,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.343309+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:52:51.343309+0000","last_peered":"2024-04-25T15:52:51.343309+0000","last_clean":"2024-04-25T15:52:51.343309+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:52:51.343309+0000","last_undegraded":"2024-04-25T15:52:51.343309+0000","last_fullsized":"2024-04-25T15:52:51.343309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"62'2","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.477685+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:52:50.477685+0000","last_peered":"2024-04-25T15:52:50.477685+0000","last_clean":"2024-04-25T15:52:50.477685+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:52:50.477685+0000","last_undegraded":"2024-04-25T15:52:50.477685+0000","last_fullsized":"2024-04-25T15:52:50.477685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"63'7","reported_seq":104,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.817457+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:52:51.817457+0000","last_peered":"2024-04-25T15:52:51.817457+0000","last_clean":"2024-04-25T15:52:51.817457+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:52:51.817457+0000","last_undegraded":"2024-04-25T15:52:51.817457+0000","last_fullsized":"2024-04-25T15:52:51.817457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"62'1","reported_seq":99,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474268+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:52:50.474268+0000","last_peered":"2024-04-25T15:52:50.474268+0000","last_clean":"2024-04-25T15:52:50.474268+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:52:50.474268+0000","last_undegraded":"2024-04-25T15:52:50.474268+0000","last_fullsized":"2024-04-25T15:52:50.474268+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"63'4","reported_seq":95,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.408066+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:52:50.408066+0000","last_peered":"2024-04-25T15:52:50.408066+0000","last_clean":"2024-04-25T15:52:50.408066+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:52:50.408066+0000","last_undegraded":"2024-04-25T15:52:50.408066+0000","last_fullsized":"2024-04-25T15:52:50.408066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"61'5","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.274946+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:52:50.274946+0000","last_peered":"2024-04-25T15:52:50.274946+0000","last_clean":"2024-04-25T15:52:50.274946+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:52:50.274946+0000","last_undegraded":"2024-04-25T15:52:50.274946+0000","last_fullsized":"2024-04-25T15:52:50.274946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"63'2","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.275526+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:52:50.275526+0000","last_peered":"2024-04-25T15:52:50.275526+0000","last_clean":"2024-04-25T15:52:50.275526+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:52:50.275526+0000","last_undegraded":"2024-04-25T15:52:50.275526+0000","last_fullsized":"2024-04-25T15:52:50.275526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"63'7","reported_seq":34,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.273126+0000","last_change":"2024-04-25T15:52:32.125058+0000","last_active":"2024-04-25T15:52:50.273126+0000","last_peered":"2024-04-25T15:52:50.273126+0000","last_clean":"2024-04-25T15:52:50.273126+0000","last_became_active":"2024-04-25T15:52:32.124575+0000","last_became_peered":"2024-04-25T15:52:32.124575+0000","last_unstale":"2024-04-25T15:52:50.273126+0000","last_undegraded":"2024-04-25T15:52:50.273126+0000","last_fullsized":"2024-04-25T15:52:50.273126+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:24:57.698573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.33","version":"63'2","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.475177+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:52:50.475177+0000","last_peered":"2024-04-25T15:52:50.475177+0000","last_clean":"2024-04-25T15:52:50.475177+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:52:50.475177+0000","last_undegraded":"2024-04-25T15:52:50.475177+0000","last_fullsized":"2024-04-25T15:52:50.475177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"63'5","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.601311+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:52:50.601311+0000","last_peered":"2024-04-25T15:52:50.601311+0000","last_clean":"2024-04-25T15:52:50.601311+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:52:50.601311+0000","last_undegraded":"2024-04-25T15:52:50.601311+0000","last_fullsized":"2024-04-25T15:52:50.601311+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"63'4","reported_seq":73,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401022+0000","last_change":"2024-04-25T15:52:32.129421+0000","last_active":"2024-04-25T15:52:50.401022+0000","last_peered":"2024-04-25T15:52:50.401022+0000","last_clean":"2024-04-25T15:52:50.401022+0000","last_became_active":"2024-04-25T15:52:32.128747+0000","last_became_peered":"2024-04-25T15:52:32.128747+0000","last_unstale":"2024-04-25T15:52:50.401022+0000","last_undegraded":"2024-04-25T15:52:50.401022+0000","last_fullsized":"2024-04-25T15:52:50.401022+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:44.731855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"63'6","reported_seq":105,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:52.816632+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:52:52.816632+0000","last_peered":"2024-04-25T15:52:52.816632+0000","last_clean":"2024-04-25T15:52:52.816632+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:52:52.816632+0000","last_undegraded":"2024-04-25T15:52:52.816632+0000","last_fullsized":"2024-04-25T15:52:52.816632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"60'3","reported_seq":35,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.477231+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:52:50.477231+0000","last_peered":"2024-04-25T15:52:50.477231+0000","last_clean":"2024-04-25T15:52:50.477231+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:52:50.477231+0000","last_undegraded":"2024-04-25T15:52:50.477231+0000","last_fullsized":"2024-04-25T15:52:50.477231+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"61'2","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407835+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:52:50.407835+0000","last_peered":"2024-04-25T15:52:50.407835+0000","last_clean":"2024-04-25T15:52:50.407835+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:52:50.407835+0000","last_undegraded":"2024-04-25T15:52:50.407835+0000","last_fullsized":"2024-04-25T15:52:50.407835+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"63'1","reported_seq":99,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276730+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:52:50.276730+0000","last_peered":"2024-04-25T15:52:50.276730+0000","last_clean":"2024-04-25T15:52:50.276730+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:52:50.276730+0000","last_undegraded":"2024-04-25T15:52:50.276730+0000","last_fullsized":"2024-04-25T15:52:50.276730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"62'2","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.273476+0000","last_change":"2024-04-25T15:52:32.120016+0000","last_active":"2024-04-25T15:52:50.273476+0000","last_peered":"2024-04-25T15:52:50.273476+0000","last_clean":"2024-04-25T15:52:50.273476+0000","last_became_active":"2024-04-25T15:52:32.119425+0000","last_became_peered":"2024-04-25T15:52:32.119425+0000","last_unstale":"2024-04-25T15:52:50.273476+0000","last_undegraded":"2024-04-25T15:52:50.273476+0000","last_fullsized":"2024-04-25T15:52:50.273476+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:42:15.118421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"62'5","reported_seq":59,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.275917+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:52:50.275917+0000","last_peered":"2024-04-25T15:52:50.275917+0000","last_clean":"2024-04-25T15:52:50.275917+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:52:50.275917+0000","last_undegraded":"2024-04-25T15:52:50.275917+0000","last_fullsized":"2024-04-25T15:52:50.275917+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"61'5","reported_seq":96,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407662+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:52:50.407662+0000","last_peered":"2024-04-25T15:52:50.407662+0000","last_clean":"2024-04-25T15:52:50.407662+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:52:50.407662+0000","last_undegraded":"2024-04-25T15:52:50.407662+0000","last_fullsized":"2024-04-25T15:52:50.407662+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"60'1","reported_seq":105,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474619+0000","last_change":"2024-04-25T15:52:32.123939+0000","last_active":"2024-04-25T15:52:50.474619+0000","last_peered":"2024-04-25T15:52:50.474619+0000","last_clean":"2024-04-25T15:52:50.474619+0000","last_became_active":"2024-04-25T15:52:32.122698+0000","last_became_peered":"2024-04-25T15:52:32.122698+0000","last_unstale":"2024-04-25T15:52:50.474619+0000","last_undegraded":"2024-04-25T15:52:50.474619+0000","last_fullsized":"2024-04-25T15:52:50.474619+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:09.597867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"63'4","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.655449+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:52:50.655449+0000","last_peered":"2024-04-25T15:52:50.655449+0000","last_clean":"2024-04-25T15:52:50.655449+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:52:50.655449+0000","last_undegraded":"2024-04-25T15:52:50.655449+0000","last_fullsized":"2024-04-25T15:52:50.655449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"63'2","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474665+0000","last_change":"2024-04-25T15:52:32.120389+0000","last_active":"2024-04-25T15:52:50.474665+0000","last_peered":"2024-04-25T15:52:50.474665+0000","last_clean":"2024-04-25T15:52:50.474665+0000","last_became_active":"2024-04-25T15:52:32.120052+0000","last_became_peered":"2024-04-25T15:52:32.120052+0000","last_unstale":"2024-04-25T15:52:50.474665+0000","last_undegraded":"2024-04-25T15:52:50.474665+0000","last_fullsized":"2024-04-25T15:52:50.474665+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:45.982199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"63'9","reported_seq":108,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.253818+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:52:51.253818+0000","last_peered":"2024-04-25T15:52:51.253818+0000","last_clean":"2024-04-25T15:52:51.253818+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:52:51.253818+0000","last_undegraded":"2024-04-25T15:52:51.253818+0000","last_fullsized":"2024-04-25T15:52:51.253818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"63'3","reported_seq":67,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401579+0000","last_change":"2024-04-25T15:51:47.271333+0000","last_active":"2024-04-25T15:52:50.401579+0000","last_peered":"2024-04-25T15:52:50.401579+0000","last_clean":"2024-04-25T15:52:50.401579+0000","last_became_active":"2024-04-25T15:51:47.270971+0000","last_became_peered":"2024-04-25T15:51:47.270971+0000","last_unstale":"2024-04-25T15:52:50.401579+0000","last_undegraded":"2024-04-25T15:52:50.401579+0000","last_fullsized":"2024-04-25T15:52:50.401579+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:55:23.860435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"63'3","reported_seq":107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.475269+0000","last_change":"2024-04-25T15:52:32.125522+0000","last_active":"2024-04-25T15:52:50.475269+0000","last_peered":"2024-04-25T15:52:50.475269+0000","last_clean":"2024-04-25T15:52:50.475269+0000","last_became_active":"2024-04-25T15:52:32.125187+0000","last_became_peered":"2024-04-25T15:52:32.125187+0000","last_unstale":"2024-04-25T15:52:50.475269+0000","last_undegraded":"2024-04-25T15:52:50.475269+0000","last_fullsized":"2024-04-25T15:52:50.475269+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.964571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"62'3","reported_seq":58,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.475040+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:52:50.475040+0000","last_peered":"2024-04-25T15:52:50.475040+0000","last_clean":"2024-04-25T15:52:50.475040+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:52:50.475040+0000","last_undegraded":"2024-04-25T15:52:50.475040+0000","last_fullsized":"2024-04-25T15:52:50.475040+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":44,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.400826+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:52:50.400826+0000","last_peered":"2024-04-25T15:52:50.400826+0000","last_clean":"2024-04-25T15:52:50.400826+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:52:50.400826+0000","last_undegraded":"2024-04-25T15:52:50.400826+0000","last_fullsized":"2024-04-25T15:52:50.400826+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"61'4","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407593+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:52:50.407593+0000","last_peered":"2024-04-25T15:52:50.407593+0000","last_clean":"2024-04-25T15:52:50.407593+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:52:50.407593+0000","last_undegraded":"2024-04-25T15:52:50.407593+0000","last_fullsized":"2024-04-25T15:52:50.407593+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"63'7","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.272649+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:52:50.272649+0000","last_peered":"2024-04-25T15:52:50.272649+0000","last_clean":"2024-04-25T15:52:50.272649+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:52:50.272649+0000","last_undegraded":"2024-04-25T15:52:50.272649+0000","last_fullsized":"2024-04-25T15:52:50.272649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"63'4","reported_seq":77,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401313+0000","last_change":"2024-04-25T15:52:32.123541+0000","last_active":"2024-04-25T15:52:50.401313+0000","last_peered":"2024-04-25T15:52:50.401313+0000","last_clean":"2024-04-25T15:52:50.401313+0000","last_became_active":"2024-04-25T15:52:32.123158+0000","last_became_peered":"2024-04-25T15:52:32.123158+0000","last_unstale":"2024-04-25T15:52:50.401313+0000","last_undegraded":"2024-04-25T15:52:50.401313+0000","last_fullsized":"2024-04-25T15:52:50.401313+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.858171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.475210+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:52:50.475210+0000","last_peered":"2024-04-25T15:52:50.475210+0000","last_clean":"2024-04-25T15:52:50.475210+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:52:50.475210+0000","last_undegraded":"2024-04-25T15:52:50.475210+0000","last_fullsized":"2024-04-25T15:52:50.475210+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"63'3","reported_seq":29,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.275953+0000","last_change":"2024-04-25T15:52:32.131021+0000","last_active":"2024-04-25T15:52:50.275953+0000","last_peered":"2024-04-25T15:52:50.275953+0000","last_clean":"2024-04-25T15:52:50.275953+0000","last_became_active":"2024-04-25T15:52:32.129920+0000","last_became_peered":"2024-04-25T15:52:32.129920+0000","last_unstale":"2024-04-25T15:52:50.275953+0000","last_undegraded":"2024-04-25T15:52:50.275953+0000","last_fullsized":"2024-04-25T15:52:50.275953+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:00.291863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":120,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.272877+0000","last_change":"2024-04-25T15:52:32.152478+0000","last_active":"2024-04-25T15:52:50.272877+0000","last_peered":"2024-04-25T15:52:50.272877+0000","last_clean":"2024-04-25T15:52:50.272877+0000","last_became_active":"2024-04-25T15:52:32.137069+0000","last_became_peered":"2024-04-25T15:52:32.137069+0000","last_unstale":"2024-04-25T15:52:50.272877+0000","last_undegraded":"2024-04-25T15:52:50.272877+0000","last_fullsized":"2024-04-25T15:52:50.272877+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:26.946219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474749+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:52:50.474749+0000","last_peered":"2024-04-25T15:52:50.474749+0000","last_clean":"2024-04-25T15:52:50.474749+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:52:50.474749+0000","last_undegraded":"2024-04-25T15:52:50.474749+0000","last_fullsized":"2024-04-25T15:52:50.474749+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"63'3","reported_seq":101,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474419+0000","last_change":"2024-04-25T15:50:31.194328+0000","last_active":"2024-04-25T15:52:50.474419+0000","last_peered":"2024-04-25T15:52:50.474419+0000","last_clean":"2024-04-25T15:52:50.474419+0000","last_became_active":"2024-04-25T15:50:31.193466+0000","last_became_peered":"2024-04-25T15:50:31.193466+0000","last_unstale":"2024-04-25T15:52:50.474419+0000","last_undegraded":"2024-04-25T15:52:50.474419+0000","last_fullsized":"2024-04-25T15:52:50.474419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:05.565514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.477505+0000","last_change":"2024-04-25T15:52:32.145033+0000","last_active":"2024-04-25T15:52:50.477505+0000","last_peered":"2024-04-25T15:52:50.477505+0000","last_clean":"2024-04-25T15:52:50.477505+0000","last_became_active":"2024-04-25T15:52:32.136902+0000","last_became_peered":"2024-04-25T15:52:32.136902+0000","last_unstale":"2024-04-25T15:52:50.477505+0000","last_undegraded":"2024-04-25T15:52:50.477505+0000","last_fullsized":"2024-04-25T15:52:50.477505+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:59.330286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"63'2","reported_seq":66,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401093+0000","last_change":"2024-04-25T15:51:47.271089+0000","last_active":"2024-04-25T15:52:50.401093+0000","last_peered":"2024-04-25T15:52:50.401093+0000","last_clean":"2024-04-25T15:52:50.401093+0000","last_became_active":"2024-04-25T15:51:47.270705+0000","last_became_peered":"2024-04-25T15:51:47.270705+0000","last_unstale":"2024-04-25T15:52:50.401093+0000","last_undegraded":"2024-04-25T15:52:50.401093+0000","last_fullsized":"2024-04-25T15:52:50.401093+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:08.380566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":105,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.274641+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:52:50.274641+0000","last_peered":"2024-04-25T15:52:50.274641+0000","last_clean":"2024-04-25T15:52:50.274641+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:52:50.274641+0000","last_undegraded":"2024-04-25T15:52:50.274641+0000","last_fullsized":"2024-04-25T15:52:50.274641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"61'2","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478089+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:52:50.478089+0000","last_peered":"2024-04-25T15:52:50.478089+0000","last_clean":"2024-04-25T15:52:50.478089+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:52:50.478089+0000","last_undegraded":"2024-04-25T15:52:50.478089+0000","last_fullsized":"2024-04-25T15:52:50.478089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":104,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478240+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:52:50.478240+0000","last_peered":"2024-04-25T15:52:50.478240+0000","last_clean":"2024-04-25T15:52:50.478240+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:52:50.478240+0000","last_undegraded":"2024-04-25T15:52:50.478240+0000","last_fullsized":"2024-04-25T15:52:50.478240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"61'5","reported_seq":60,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478476+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:52:50.478476+0000","last_peered":"2024-04-25T15:52:50.478476+0000","last_clean":"2024-04-25T15:52:50.478476+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:52:50.478476+0000","last_undegraded":"2024-04-25T15:52:50.478476+0000","last_fullsized":"2024-04-25T15:52:50.478476+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":74,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401271+0000","last_change":"2024-04-25T15:51:47.302272+0000","last_active":"2024-04-25T15:52:50.401271+0000","last_peered":"2024-04-25T15:52:50.401271+0000","last_clean":"2024-04-25T15:52:50.401271+0000","last_became_active":"2024-04-25T15:51:47.266158+0000","last_became_peered":"2024-04-25T15:51:47.266158+0000","last_unstale":"2024-04-25T15:52:50.401271+0000","last_undegraded":"2024-04-25T15:52:50.401271+0000","last_fullsized":"2024-04-25T15:52:50.401271+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:28.386086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":672,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"63'5","reported_seq":32,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.861817+0000","last_change":"2024-04-25T15:52:32.129340+0000","last_active":"2024-04-25T15:52:51.861817+0000","last_peered":"2024-04-25T15:52:51.861817+0000","last_clean":"2024-04-25T15:52:51.861817+0000","last_became_active":"2024-04-25T15:52:32.128859+0000","last_became_peered":"2024-04-25T15:52:32.128859+0000","last_unstale":"2024-04-25T15:52:51.861817+0000","last_undegraded":"2024-04-25T15:52:51.861817+0000","last_fullsized":"2024-04-25T15:52:51.861817+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:59:42.834204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"63'30","reported_seq":131,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:52.627188+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:52:52.627188+0000","last_peered":"2024-04-25T15:52:52.627188+0000","last_clean":"2024-04-25T15:52:52.627188+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:52:52.627188+0000","last_undegraded":"2024-04-25T15:52:52.627188+0000","last_fullsized":"2024-04-25T15:52:52.627188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49369,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"61'3","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276128+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:52:50.276128+0000","last_peered":"2024-04-25T15:52:50.276128+0000","last_clean":"2024-04-25T15:52:50.276128+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:52:50.276128+0000","last_undegraded":"2024-04-25T15:52:50.276128+0000","last_fullsized":"2024-04-25T15:52:50.276128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":40,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474922+0000","last_change":"2024-04-25T15:52:32.150681+0000","last_active":"2024-04-25T15:52:50.474922+0000","last_peered":"2024-04-25T15:52:50.474922+0000","last_clean":"2024-04-25T15:52:50.474922+0000","last_became_active":"2024-04-25T15:52:32.125799+0000","last_became_peered":"2024-04-25T15:52:32.125799+0000","last_unstale":"2024-04-25T15:52:50.474922+0000","last_undegraded":"2024-04-25T15:52:50.474922+0000","last_fullsized":"2024-04-25T15:52:50.474922+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.858701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"62'7","reported_seq":61,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.275761+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:52:50.275761+0000","last_peered":"2024-04-25T15:52:50.275761+0000","last_clean":"2024-04-25T15:52:50.275761+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:52:50.275761+0000","last_undegraded":"2024-04-25T15:52:50.275761+0000","last_fullsized":"2024-04-25T15:52:50.275761+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.477883+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:52:50.477883+0000","last_peered":"2024-04-25T15:52:50.477883+0000","last_clean":"2024-04-25T15:52:50.477883+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:52:50.477883+0000","last_undegraded":"2024-04-25T15:52:50.477883+0000","last_fullsized":"2024-04-25T15:52:50.477883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"63'3","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.781554+0000","last_change":"2024-04-25T15:52:32.135585+0000","last_active":"2024-04-25T15:52:51.781554+0000","last_peered":"2024-04-25T15:52:51.781554+0000","last_clean":"2024-04-25T15:52:51.781554+0000","last_became_active":"2024-04-25T15:52:32.134857+0000","last_became_peered":"2024-04-25T15:52:32.134857+0000","last_unstale":"2024-04-25T15:52:51.781554+0000","last_undegraded":"2024-04-25T15:52:51.781554+0000","last_fullsized":"2024-04-25T15:52:51.781554+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:58.829902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":120,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.272900+0000","last_change":"2024-04-25T15:52:32.144332+0000","last_active":"2024-04-25T15:52:50.272900+0000","last_peered":"2024-04-25T15:52:50.272900+0000","last_clean":"2024-04-25T15:52:50.272900+0000","last_became_active":"2024-04-25T15:52:32.124713+0000","last_became_peered":"2024-04-25T15:52:32.124713+0000","last_unstale":"2024-04-25T15:52:50.272900+0000","last_undegraded":"2024-04-25T15:52:50.272900+0000","last_fullsized":"2024-04-25T15:52:50.272900+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:15.176691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1029,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"60'3","reported_seq":102,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.272650+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:52:50.272650+0000","last_peered":"2024-04-25T15:52:50.272650+0000","last_clean":"2024-04-25T15:52:50.272650+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:52:50.272650+0000","last_undegraded":"2024-04-25T15:52:50.272650+0000","last_fullsized":"2024-04-25T15:52:50.272650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":163,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.408978+0000","last_change":"2024-04-25T15:51:47.287562+0000","last_active":"2024-04-25T15:52:50.408978+0000","last_peered":"2024-04-25T15:52:50.408978+0000","last_clean":"2024-04-25T15:52:50.408978+0000","last_became_active":"2024-04-25T15:51:47.264513+0000","last_became_peered":"2024-04-25T15:51:47.264513+0000","last_unstale":"2024-04-25T15:52:50.408978+0000","last_undegraded":"2024-04-25T15:52:50.408978+0000","last_fullsized":"2024-04-25T15:52:50.408978+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:51:11.555766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"63'8","reported_seq":111,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.299375+0000","last_change":"2024-04-25T15:52:32.130800+0000","last_active":"2024-04-25T15:52:51.299375+0000","last_peered":"2024-04-25T15:52:51.299375+0000","last_clean":"2024-04-25T15:52:51.299375+0000","last_became_active":"2024-04-25T15:52:32.129969+0000","last_became_peered":"2024-04-25T15:52:32.129969+0000","last_unstale":"2024-04-25T15:52:51.299375+0000","last_undegraded":"2024-04-25T15:52:51.299375+0000","last_fullsized":"2024-04-25T15:52:51.299375+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:05:45.521306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":32,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276815+0000","last_change":"2024-04-25T15:52:32.155801+0000","last_active":"2024-04-25T15:52:50.276815+0000","last_peered":"2024-04-25T15:52:50.276815+0000","last_clean":"2024-04-25T15:52:50.276815+0000","last_became_active":"2024-04-25T15:52:32.129444+0000","last_became_peered":"2024-04-25T15:52:32.129444+0000","last_unstale":"2024-04-25T15:52:50.276815+0000","last_undegraded":"2024-04-25T15:52:50.276815+0000","last_fullsized":"2024-04-25T15:52:50.276815+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:46.358507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":105,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276756+0000","last_change":"2024-04-25T15:52:32.129065+0000","last_active":"2024-04-25T15:52:50.276756+0000","last_peered":"2024-04-25T15:52:50.276756+0000","last_clean":"2024-04-25T15:52:50.276756+0000","last_became_active":"2024-04-25T15:52:32.128801+0000","last_became_peered":"2024-04-25T15:52:32.128801+0000","last_unstale":"2024-04-25T15:52:50.276756+0000","last_undegraded":"2024-04-25T15:52:50.276756+0000","last_fullsized":"2024-04-25T15:52:50.276756+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:09.896373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":105,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407914+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:52:50.407914+0000","last_peered":"2024-04-25T15:52:50.407914+0000","last_clean":"2024-04-25T15:52:50.407914+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:52:50.407914+0000","last_undegraded":"2024-04-25T15:52:50.407914+0000","last_fullsized":"2024-04-25T15:52:50.407914+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"61'4","reported_seq":107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.277574+0000","last_change":"2024-04-25T15:52:32.129750+0000","last_active":"2024-04-25T15:52:50.277574+0000","last_peered":"2024-04-25T15:52:50.277574+0000","last_clean":"2024-04-25T15:52:50.277574+0000","last_became_active":"2024-04-25T15:52:32.129402+0000","last_became_peered":"2024-04-25T15:52:32.129402+0000","last_unstale":"2024-04-25T15:52:50.277574+0000","last_undegraded":"2024-04-25T15:52:50.277574+0000","last_fullsized":"2024-04-25T15:52:50.277574+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:48.203567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":77,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401390+0000","last_change":"2024-04-25T15:52:32.135358+0000","last_active":"2024-04-25T15:52:50.401390+0000","last_peered":"2024-04-25T15:52:50.401390+0000","last_clean":"2024-04-25T15:52:50.401390+0000","last_became_active":"2024-04-25T15:52:32.129558+0000","last_became_peered":"2024-04-25T15:52:32.129558+0000","last_unstale":"2024-04-25T15:52:50.401390+0000","last_undegraded":"2024-04-25T15:52:50.401390+0000","last_fullsized":"2024-04-25T15:52:50.401390+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:49.940252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"63'4","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478452+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:52:50.478452+0000","last_peered":"2024-04-25T15:52:50.478452+0000","last_clean":"2024-04-25T15:52:50.478452+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:52:50.478452+0000","last_undegraded":"2024-04-25T15:52:50.478452+0000","last_fullsized":"2024-04-25T15:52:50.478452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407750+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:52:50.407750+0000","last_peered":"2024-04-25T15:52:50.407750+0000","last_clean":"2024-04-25T15:52:50.407750+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:52:50.407750+0000","last_undegraded":"2024-04-25T15:52:50.407750+0000","last_fullsized":"2024-04-25T15:52:50.407750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"60'2","reported_seq":100,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276189+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:52:50.276189+0000","last_peered":"2024-04-25T15:52:50.276189+0000","last_clean":"2024-04-25T15:52:50.276189+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:52:50.276189+0000","last_undegraded":"2024-04-25T15:52:50.276189+0000","last_fullsized":"2024-04-25T15:52:50.276189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":196,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.274363+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:52:50.274363+0000","last_peered":"2024-04-25T15:52:50.274363+0000","last_clean":"2024-04-25T15:52:50.274363+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:52:50.274363+0000","last_undegraded":"2024-04-25T15:52:50.274363+0000","last_fullsized":"2024-04-25T15:52:50.274363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":107,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478235+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:52:50.478235+0000","last_peered":"2024-04-25T15:52:50.478235+0000","last_clean":"2024-04-25T15:52:50.478235+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:52:50.478235+0000","last_undegraded":"2024-04-25T15:52:50.478235+0000","last_fullsized":"2024-04-25T15:52:50.478235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"63'4","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474844+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:52:50.474844+0000","last_peered":"2024-04-25T15:52:50.474844+0000","last_clean":"2024-04-25T15:52:50.474844+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:52:50.474844+0000","last_undegraded":"2024-04-25T15:52:50.474844+0000","last_fullsized":"2024-04-25T15:52:50.474844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":122,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.274251+0000","last_change":"2024-04-25T15:51:47.292263+0000","last_active":"2024-04-25T15:52:50.274251+0000","last_peered":"2024-04-25T15:52:50.274251+0000","last_clean":"2024-04-25T15:52:50.274251+0000","last_became_active":"2024-04-25T15:51:47.264780+0000","last_became_peered":"2024-04-25T15:51:47.264780+0000","last_unstale":"2024-04-25T15:52:50.274251+0000","last_undegraded":"2024-04-25T15:52:50.274251+0000","last_fullsized":"2024-04-25T15:52:50.274251+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:40.269146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"62'5","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474313+0000","last_change":"2024-04-25T15:52:32.125579+0000","last_active":"2024-04-25T15:52:50.474313+0000","last_peered":"2024-04-25T15:52:50.474313+0000","last_clean":"2024-04-25T15:52:50.474313+0000","last_became_active":"2024-04-25T15:52:32.125247+0000","last_became_peered":"2024-04-25T15:52:32.125247+0000","last_unstale":"2024-04-25T15:52:50.474313+0000","last_undegraded":"2024-04-25T15:52:50.474313+0000","last_fullsized":"2024-04-25T15:52:50.474313+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:06:10.258830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407927+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:52:50.407927+0000","last_peered":"2024-04-25T15:52:50.407927+0000","last_clean":"2024-04-25T15:52:50.407927+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:52:50.407927+0000","last_undegraded":"2024-04-25T15:52:50.407927+0000","last_fullsized":"2024-04-25T15:52:50.407927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"63'4","reported_seq":68,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.400845+0000","last_change":"2024-04-25T15:51:47.255396+0000","last_active":"2024-04-25T15:52:50.400845+0000","last_peered":"2024-04-25T15:52:50.400845+0000","last_clean":"2024-04-25T15:52:50.400845+0000","last_became_active":"2024-04-25T15:51:47.254596+0000","last_became_peered":"2024-04-25T15:51:47.254596+0000","last_unstale":"2024-04-25T15:52:50.400845+0000","last_undegraded":"2024-04-25T15:52:50.400845+0000","last_fullsized":"2024-04-25T15:52:50.400845+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:06:53.573154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":128,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.477958+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:52:50.477958+0000","last_peered":"2024-04-25T15:52:50.477958+0000","last_clean":"2024-04-25T15:52:50.477958+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:52:50.477958+0000","last_undegraded":"2024-04-25T15:52:50.477958+0000","last_fullsized":"2024-04-25T15:52:50.477958+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"60'3","reported_seq":29,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.272960+0000","last_change":"2024-04-25T15:52:32.118757+0000","last_active":"2024-04-25T15:52:50.272960+0000","last_peered":"2024-04-25T15:52:50.272960+0000","last_clean":"2024-04-25T15:52:50.272960+0000","last_became_active":"2024-04-25T15:52:32.118380+0000","last_became_peered":"2024-04-25T15:52:32.118380+0000","last_unstale":"2024-04-25T15:52:50.272960+0000","last_undegraded":"2024-04-25T15:52:50.272960+0000","last_fullsized":"2024-04-25T15:52:50.272960+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:18.978899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":64,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401292+0000","last_change":"2024-04-25T15:51:47.263905+0000","last_active":"2024-04-25T15:52:50.401292+0000","last_peered":"2024-04-25T15:52:50.401292+0000","last_clean":"2024-04-25T15:52:50.401292+0000","last_became_active":"2024-04-25T15:51:47.263546+0000","last_became_peered":"2024-04-25T15:51:47.263546+0000","last_unstale":"2024-04-25T15:52:50.401292+0000","last_undegraded":"2024-04-25T15:52:50.401292+0000","last_fullsized":"2024-04-25T15:52:50.401292+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:14.832346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"63'4","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478186+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:52:50.478186+0000","last_peered":"2024-04-25T15:52:50.478186+0000","last_clean":"2024-04-25T15:52:50.478186+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:52:50.478186+0000","last_undegraded":"2024-04-25T15:52:50.478186+0000","last_fullsized":"2024-04-25T15:52:50.478186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":104,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276677+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:52:50.276677+0000","last_peered":"2024-04-25T15:52:50.276677+0000","last_clean":"2024-04-25T15:52:50.276677+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:52:50.276677+0000","last_undegraded":"2024-04-25T15:52:50.276677+0000","last_fullsized":"2024-04-25T15:52:50.276677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"62'2","reported_seq":66,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401434+0000","last_change":"2024-04-25T15:51:47.263756+0000","last_active":"2024-04-25T15:52:50.401434+0000","last_peered":"2024-04-25T15:52:50.401434+0000","last_clean":"2024-04-25T15:52:50.401434+0000","last_became_active":"2024-04-25T15:51:47.263130+0000","last_became_peered":"2024-04-25T15:51:47.263130+0000","last_unstale":"2024-04-25T15:52:50.401434+0000","last_undegraded":"2024-04-25T15:52:50.401434+0000","last_fullsized":"2024-04-25T15:52:50.401434+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:38.024802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"59'10","reported_seq":75,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401731+0000","last_change":"2024-04-25T15:51:47.314396+0000","last_active":"2024-04-25T15:52:50.401731+0000","last_peered":"2024-04-25T15:52:50.401731+0000","last_clean":"2024-04-25T15:52:50.401731+0000","last_became_active":"2024-04-25T15:51:47.274984+0000","last_became_peered":"2024-04-25T15:51:47.274984+0000","last_unstale":"2024-04-25T15:52:50.401731+0000","last_undegraded":"2024-04-25T15:52:50.401731+0000","last_fullsized":"2024-04-25T15:52:50.401731+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:53.936857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'7","reported_seq":106,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474665+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:52:50.474665+0000","last_peered":"2024-04-25T15:52:50.474665+0000","last_clean":"2024-04-25T15:52:50.474665+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:52:50.474665+0000","last_undegraded":"2024-04-25T15:52:50.474665+0000","last_fullsized":"2024-04-25T15:52:50.474665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":108,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.275147+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:52:50.275147+0000","last_peered":"2024-04-25T15:52:50.275147+0000","last_clean":"2024-04-25T15:52:50.275147+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:52:50.275147+0000","last_undegraded":"2024-04-25T15:52:50.275147+0000","last_fullsized":"2024-04-25T15:52:50.275147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"63'2","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401433+0000","last_change":"2024-04-25T15:51:59.936821+0000","last_active":"2024-04-25T15:52:50.401433+0000","last_peered":"2024-04-25T15:52:50.401433+0000","last_clean":"2024-04-25T15:52:50.401433+0000","last_became_active":"2024-04-25T15:51:59.936474+0000","last_became_peered":"2024-04-25T15:51:59.936474+0000","last_unstale":"2024-04-25T15:52:50.401433+0000","last_undegraded":"2024-04-25T15:52:50.401433+0000","last_fullsized":"2024-04-25T15:52:50.401433+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:05.147722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":102,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.274119+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:52:50.274119+0000","last_peered":"2024-04-25T15:52:50.274119+0000","last_clean":"2024-04-25T15:52:50.274119+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:52:50.274119+0000","last_undegraded":"2024-04-25T15:52:50.274119+0000","last_fullsized":"2024-04-25T15:52:50.274119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"61'4","reported_seq":31,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407593+0000","last_change":"2024-04-25T15:52:32.131460+0000","last_active":"2024-04-25T15:52:50.407593+0000","last_peered":"2024-04-25T15:52:50.407593+0000","last_clean":"2024-04-25T15:52:50.407593+0000","last_became_active":"2024-04-25T15:52:32.131087+0000","last_became_peered":"2024-04-25T15:52:32.131087+0000","last_unstale":"2024-04-25T15:52:50.407593+0000","last_undegraded":"2024-04-25T15:52:50.407593+0000","last_fullsized":"2024-04-25T15:52:50.407593+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:14.840887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":74,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.400906+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:52:50.400906+0000","last_peered":"2024-04-25T15:52:50.400906+0000","last_clean":"2024-04-25T15:52:50.400906+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:52:50.400906+0000","last_undegraded":"2024-04-25T15:52:50.400906+0000","last_fullsized":"2024-04-25T15:52:50.400906+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"63'7","reported_seq":116,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474464+0000","last_change":"2024-04-25T15:51:47.270024+0000","last_active":"2024-04-25T15:52:50.474464+0000","last_peered":"2024-04-25T15:52:50.474464+0000","last_clean":"2024-04-25T15:52:50.474464+0000","last_became_active":"2024-04-25T15:51:47.269733+0000","last_became_peered":"2024-04-25T15:51:47.269733+0000","last_unstale":"2024-04-25T15:52:50.474464+0000","last_undegraded":"2024-04-25T15:52:50.474464+0000","last_fullsized":"2024-04-25T15:52:50.474464+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:23.505183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":70,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.401523+0000","last_change":"2024-04-25T15:51:47.286937+0000","last_active":"2024-04-25T15:52:50.401523+0000","last_peered":"2024-04-25T15:52:50.401523+0000","last_clean":"2024-04-25T15:52:50.401523+0000","last_became_active":"2024-04-25T15:51:47.266040+0000","last_became_peered":"2024-04-25T15:51:47.266040+0000","last_unstale":"2024-04-25T15:52:50.401523+0000","last_undegraded":"2024-04-25T15:52:50.401523+0000","last_fullsized":"2024-04-25T15:52:50.401523+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:48.792187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"61'4","reported_seq":95,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.407764+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:52:50.407764+0000","last_peered":"2024-04-25T15:52:50.407764+0000","last_clean":"2024-04-25T15:52:50.407764+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:52:50.407764+0000","last_undegraded":"2024-04-25T15:52:50.407764+0000","last_fullsized":"2024-04-25T15:52:50.407764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":115,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276407+0000","last_change":"2024-04-25T15:52:32.159964+0000","last_active":"2024-04-25T15:52:50.276407+0000","last_peered":"2024-04-25T15:52:50.276407+0000","last_clean":"2024-04-25T15:52:50.276407+0000","last_became_active":"2024-04-25T15:52:32.134918+0000","last_became_peered":"2024-04-25T15:52:32.134918+0000","last_unstale":"2024-04-25T15:52:50.276407+0000","last_undegraded":"2024-04-25T15:52:50.276407+0000","last_fullsized":"2024-04-25T15:52:50.276407+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:26:55.807079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"61'1","reported_seq":105,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.272604+0000","last_change":"2024-04-25T15:52:32.126151+0000","last_active":"2024-04-25T15:52:50.272604+0000","last_peered":"2024-04-25T15:52:50.272604+0000","last_clean":"2024-04-25T15:52:50.272604+0000","last_became_active":"2024-04-25T15:52:32.125517+0000","last_became_peered":"2024-04-25T15:52:32.125517+0000","last_unstale":"2024-04-25T15:52:50.272604+0000","last_undegraded":"2024-04-25T15:52:50.272604+0000","last_fullsized":"2024-04-25T15:52:50.272604+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:13.443149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"63'6","reported_seq":109,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:51.528182+0000","last_change":"2024-04-25T15:52:32.129077+0000","last_active":"2024-04-25T15:52:51.528182+0000","last_peered":"2024-04-25T15:52:51.528182+0000","last_clean":"2024-04-25T15:52:51.528182+0000","last_became_active":"2024-04-25T15:52:32.128629+0000","last_became_peered":"2024-04-25T15:52:32.128629+0000","last_unstale":"2024-04-25T15:52:51.528182+0000","last_undegraded":"2024-04-25T15:52:51.528182+0000","last_fullsized":"2024-04-25T15:52:51.528182+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:05:51.807066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":104,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276131+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:52:50.276131+0000","last_peered":"2024-04-25T15:52:50.276131+0000","last_clean":"2024-04-25T15:52:50.276131+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:52:50.276131+0000","last_undegraded":"2024-04-25T15:52:50.276131+0000","last_fullsized":"2024-04-25T15:52:50.276131+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"62'6","reported_seq":109,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478173+0000","last_change":"2024-04-25T15:52:32.132711+0000","last_active":"2024-04-25T15:52:50.478173+0000","last_peered":"2024-04-25T15:52:50.478173+0000","last_clean":"2024-04-25T15:52:50.478173+0000","last_became_active":"2024-04-25T15:52:32.132406+0000","last_became_peered":"2024-04-25T15:52:32.132406+0000","last_unstale":"2024-04-25T15:52:50.478173+0000","last_undegraded":"2024-04-25T15:52:50.478173+0000","last_fullsized":"2024-04-25T15:52:50.478173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:30.616697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":111,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.276544+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:52:50.276544+0000","last_peered":"2024-04-25T15:52:50.276544+0000","last_clean":"2024-04-25T15:52:50.276544+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:52:50.276544+0000","last_undegraded":"2024-04-25T15:52:50.276544+0000","last_fullsized":"2024-04-25T15:52:50.276544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"62'2","reported_seq":57,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.478362+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:52:50.478362+0000","last_peered":"2024-04-25T15:52:50.478362+0000","last_clean":"2024-04-25T15:52:50.478362+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:52:50.478362+0000","last_undegraded":"2024-04-25T15:52:50.478362+0000","last_fullsized":"2024-04-25T15:52:50.478362+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.474764+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:52:50.474764+0000","last_peered":"2024-04-25T15:52:50.474764+0000","last_clean":"2024-04-25T15:52:50.474764+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:52:50.474764+0000","last_undegraded":"2024-04-25T15:52:50.474764+0000","last_fullsized":"2024-04-25T15:52:50.474764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'4","reported_seq":103,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:52.036968+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:52:52.036968+0000","last_peered":"2024-04-25T15:52:52.036968+0000","last_clean":"2024-04-25T15:52:52.036968+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:52:52.036968+0000","last_undegraded":"2024-04-25T15:52:52.036968+0000","last_fullsized":"2024-04-25T15:52:52.036968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-25T15:52:50.277728+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:52:50.277728+0000","last_peered":"2024-04-25T15:52:50.277728+0000","last_clean":"2024-04-25T15:52:50.277728+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:52:50.277728+0000","last_undegraded":"2024-04-25T15:52:50.277728+0000","last_fullsized":"2024-04-25T15:52:50.277728+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1002438656,"num_objects":239,"num_object_clones":0,"num_object_copies":478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":239,"num_write_kb":978944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2004877312,"data_stored":2004877312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":57114,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":236,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4003,"num_keys_recovered":22,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":115999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376580,"kb_used_data":345404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93995260,"statfs":{"total":96636764160,"available":96251146240,"internally_reserved":0,"allocated":353693696,"data_stored":353247979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285852,"kb_used_data":258836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94085988,"statfs":{"total":96636764160,"available":96344051712,"internally_reserved":0,"allocated":265048064,"data_stored":264621288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314500,"kb_used_data":287484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94057340,"statfs":{"total":96636764160,"available":96314716160,"internally_reserved":0,"allocated":294383616,"data_stored":293938583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289964,"kb_used_data":262948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94081876,"statfs":{"total":96636764160,"available":96339841024,"internally_reserved":0,"allocated":269258752,"data_stored":268819158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220276,"kb_used_data":193260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94151564,"statfs":{"total":96636764160,"available":96411201536,"internally_reserved":0,"allocated":197898240,"data_stored":197467121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212084,"kb_used_data":185068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94159756,"statfs":{"total":96636764160,"available":96419590144,"internally_reserved":0,"allocated":189509632,"data_stored":189079171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388852,"kb_used_data":357676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93982988,"statfs":{"total":96636764160,"available":96238579712,"internally_reserved":0,"allocated":366260224,"data_stored":365828620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":50772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":50600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385875968,"data_stored":385875968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":188743680,"data_stored":188743680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":197132288,"data_stored":197132288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272629760,"data_stored":272629760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":306184192,"data_stored":306184192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":289406976,"data_stored":289406976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":364904448,"data_stored":364904448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:52:55.133 INFO:tasks.thrashosds.thrasher:Setting 3.31 to [2, 3, 1, 0] 2024-04-25T15:52:55.133 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.31', '2', '3', '1', '0'] 2024-04-25T15:52:55.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.31 2 3 1 0 2024-04-25T15:52:55.134 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:55.130+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:55.225 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:55.226+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:55.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:52:55.275 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:55.276 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:55.276 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:55.276 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:55.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:52:55.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:52:55.325 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:55.323+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e04066640 1 Processor -- start 2024-04-25T15:52:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e04066640 1 -- start start 2024-04-25T15:52:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e04066640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc0cb1a0 0x7f0dfc160500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e04066640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dfc160a40 0x7f0dfc156530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e04066640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc06fe40 con 0x7f0dfc0cad90 2024-04-25T15:52:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e04066640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc15fd90 con 0x7f0dfc160a40 2024-04-25T15:52:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e04066640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc06f550 con 0x7f0dfc0cb1a0 2024-04-25T15:52:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e02603640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dfc160a40 0x7f0dfc156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e01e02640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0dfc0cad90 0x7f0dfc06ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42488/0 (socket says 172.21.15.67:42488) 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e01e02640 1 -- 172.21.15.67:0/2164904840 learned_addr learned my addr 172.21.15.67:0/2164904840 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e01601640 1 --2- 172.21.15.67:0/2164904840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc0cb1a0 0x7f0dfc160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e02603640 1 -- 172.21.15.67:0/2164904840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc0cb1a0 msgr2=0x7f0dfc160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e02603640 1 --2- 172.21.15.67:0/2164904840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc0cb1a0 0x7f0dfc160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e02603640 1 -- 172.21.15.67:0/2164904840 >> v1:172.21.15.67:6789/0 conn(0x7f0dfc0cad90 legacy=0x7f0dfc06ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e00e00640 1 -- 172.21.15.67:0/2164904840 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2391705737 0 0) 0x7f0df0001120 con 0x7f0dfc0cad90 2024-04-25T15:52:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.330+0000 7f0e02603640 1 -- 172.21.15.67:0/2164904840 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc06c7c0 con 0x7f0dfc160a40 2024-04-25T15:52:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e01601640 1 --2- 172.21.15.67:0/2164904840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc0cb1a0 0x7f0dfc160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e02603640 1 --2- 172.21.15.67:0/2164904840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc160a40 0x7f0dfc156530 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009d90 tx=0x7f0df401c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8223c079e9fea52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e00e00640 1 -- 172.21.15.67:0/2164904840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df40b2070 con 0x7f0dfc160a40 2024-04-25T15:52:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e00e00640 1 -- 172.21.15.67:0/2164904840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df40a5940 con 0x7f0dfc160a40 2024-04-25T15:52:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e00e00640 1 -- 172.21.15.67:0/2164904840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df40a5d10 con 0x7f0dfc160a40 2024-04-25T15:52:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e04066640 1 -- 172.21.15.67:0/2164904840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc160a40 msgr2=0x7f0dfc156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e04066640 1 --2- 172.21.15.67:0/2164904840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc160a40 0x7f0dfc156530 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009d90 tx=0x7f0df401c990 comp rx=0 tx=0).stop 2024-04-25T15:52:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e04066640 1 -- 172.21.15.67:0/2164904840 shutdown_connections 2024-04-25T15:52:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e04066640 1 --2- 172.21.15.67:0/2164904840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc160a40 0x7f0dfc156530 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e04066640 1 --2- 172.21.15.67:0/2164904840 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc0cb1a0 0x7f0dfc160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.334+0000 7f0e04066640 1 -- 172.21.15.67:0/2164904840 >> 172.21.15.67:0/2164904840 conn(0x7f0dfc0cebf0 msgr2=0x7f0dfc0cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 -- 172.21.15.67:0/2164904840 shutdown_connections 2024-04-25T15:52:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 -- 172.21.15.67:0/2164904840 wait complete. 2024-04-25T15:52:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 Processor -- start 2024-04-25T15:52:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 -- start start 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc0cad90 0x7f0dfc1f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dfc0cb1a0 0x7f0dfc14d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc160a40 0x7f0dfc0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc156e80 con 0x7f0dfc0cad90 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc15ffe0 con 0x7f0dfc0cb1a0 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.338+0000 7f0e04066640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc15c7a0 con 0x7f0dfc160a40 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc0cad90 0x7f0dfc1f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc0cad90 0x7f0dfc1f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60510/0 (socket says 172.21.15.67:60510) 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 -- 172.21.15.67:0/3422516927 learned_addr learned my addr 172.21.15.67:0/3422516927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e02603640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc160a40 0x7f0dfc0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01601640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dfc0cb1a0 0x7f0dfc14d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 -- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc160a40 msgr2=0x7f0dfc0b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc160a40 0x7f0dfc0b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 -- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dfc0cb1a0 msgr2=0x7f0dfc14d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dfc0cb1a0 0x7f0dfc14d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 -- 172.21.15.67:0/3422516927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc1f5c90 con 0x7f0dfc0cad90 2024-04-25T15:52:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e01e02640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc0cad90 0x7f0dfc1f6750 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c4b0 tx=0x7f0df000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0dee7fc640 1 -- 172.21.15.67:0/3422516927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0018070 con 0x7f0dfc0cad90 2024-04-25T15:52:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0dee7fc640 1 -- 172.21.15.67:0/3422516927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df0007500 con 0x7f0dfc0cad90 2024-04-25T15:52:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e04066640 1 -- 172.21.15.67:0/3422516927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc0b1090 con 0x7f0dfc0cad90 2024-04-25T15:52:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0e04066640 1 -- 172.21.15.67:0/3422516927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc1f5f50 con 0x7f0dfc0cad90 2024-04-25T15:52:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0dee7fc640 1 -- 172.21.15.67:0/3422516927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0004a30 con 0x7f0dfc0cad90 2024-04-25T15:52:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd80052a0 con 0x7f0dfc0cad90 2024-04-25T15:52:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.342+0000 7f0dee7fc640 1 -- 172.21.15.67:0/3422516927 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0df001b070 con 0x7f0dfc0cad90 2024-04-25T15:52:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.346+0000 7f0dee7fc640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0de006f430 0x7f0de0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.346+0000 7f0dee7fc640 1 -- 172.21.15.67:0/3422516927 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5886+0+0 (secure 0 0 0) 0x7f0df008e5c0 con 0x7f0dfc0cad90 2024-04-25T15:52:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.346+0000 7f0e01601640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0de006f430 0x7f0de0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.346+0000 7f0e01601640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0de006f430 0x7f0de0071940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0b2370 tx=0x7f0df8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.350+0000 7f0dee7fc640 1 -- 172.21.15.67:0/3422516927 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0dfc06fe40 con 0x7f0dfc0cad90 2024-04-25T15:52:55.425 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:55.426+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962f164640 1 Processor -- start 2024-04-25T15:52:55.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962f164640 1 -- start start 2024-04-25T15:52:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962f164640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9628153a00 0x7f96280c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962f164640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962815a0b0 0x7f96280746a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962f164640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96280c3bc0 con 0x7f96280725c0 2024-04-25T15:52:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962f164640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9628153e10 con 0x7f9628153a00 2024-04-25T15:52:55.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962f164640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9628068c40 con 0x7f962815a0b0 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f9627fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96280725c0 0x7f9628073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42498/0 (socket says 172.21.15.67:42498) 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f9627fff640 1 -- 172.21.15.67:0/1775214363 learned_addr learned my addr 172.21.15.67:0/1775214363 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962cf00640 1 --2- 172.21.15.67:0/1775214363 >> v2:172.21.15.67:3300/0 conn(0x7f9628153a00 0x7f96280c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962d701640 1 --2- 172.21.15.67:0/1775214363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962815a0b0 0x7f96280746a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962cf00640 1 -- 172.21.15.67:0/1775214363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962815a0b0 msgr2=0x7f96280746a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962cf00640 1 --2- 172.21.15.67:0/1775214363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962815a0b0 0x7f96280746a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962cf00640 1 -- 172.21.15.67:0/1775214363 >> v1:172.21.15.67:6789/0 conn(0x7f96280725c0 legacy=0x7f9628073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f96277fe640 1 -- 172.21.15.67:0/1775214363 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1681284584 0 0) 0x7f96280c3bc0 con 0x7f96280725c0 2024-04-25T15:52:55.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.430+0000 7f962cf00640 1 -- 172.21.15.67:0/1775214363 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96280c53a0 con 0x7f9628153a00 2024-04-25T15:52:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962cf00640 1 --2- 172.21.15.67:0/1775214363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9628153a00 0x7f96280c4010 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f961c0097c0 tx=0x7f961c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175c938c2043a415 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f96277fe640 1 -- 172.21.15.67:0/1775214363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c0b0070 con 0x7f9628153a00 2024-04-25T15:52:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f96277fe640 1 -- 172.21.15.67:0/1775214363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961c0ab600 con 0x7f9628153a00 2024-04-25T15:52:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f96277fe640 1 -- 172.21.15.67:0/1775214363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c0ab9d0 con 0x7f9628153a00 2024-04-25T15:52:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 -- 172.21.15.67:0/1775214363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9628153a00 msgr2=0x7f96280c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 --2- 172.21.15.67:0/1775214363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9628153a00 0x7f96280c4010 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f961c0097c0 tx=0x7f961c01c410 comp rx=0 tx=0).stop 2024-04-25T15:52:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 -- 172.21.15.67:0/1775214363 shutdown_connections 2024-04-25T15:52:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 --2- 172.21.15.67:0/1775214363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f962815a0b0 0x7f96280746a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 --2- 172.21.15.67:0/1775214363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9628153a00 0x7f96280c4010 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 -- 172.21.15.67:0/1775214363 >> 172.21.15.67:0/1775214363 conn(0x7f96280ce1c0 msgr2=0x7f96280ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 -- 172.21.15.67:0/1775214363 shutdown_connections 2024-04-25T15:52:55.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.434+0000 7f962f164640 1 -- 172.21.15.67:0/1775214363 wait complete. 2024-04-25T15:52:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.438+0000 7f962f164640 1 Processor -- start 2024-04-25T15:52:55.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.438+0000 7f962f164640 1 -- start start 2024-04-25T15:52:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96280725c0 0x7f962815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9628153a00 0x7f9628146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962815a0b0 0x7f9628144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9628074be0 con 0x7f962815a0b0 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f96280686c0 con 0x7f96280725c0 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96280729d0 con 0x7f9628153a00 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962815a0b0 0x7f9628144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962815a0b0 0x7f9628144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60538/0 (socket says 172.21.15.67:60538) 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 -- 172.21.15.67:0/2412720999 learned_addr learned my addr 172.21.15.67:0/2412720999 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f9627fff640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9628153a00 0x7f9628146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96280725c0 0x7f962815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9628153a00 msgr2=0x7f9628146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9628153a00 0x7f9628146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96280725c0 msgr2=0x7f962815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96280725c0 0x7f962815c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962815cc10 con 0x7f962815a0b0 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96280725c0 0x7f962815c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962d701640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962815a0b0 0x7f9628144960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9620009cb0 tx=0x7f962000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620018020 con 0x7f962815a0b0 2024-04-25T15:52:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9620014480 con 0x7f962815a0b0 2024-04-25T15:52:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9628146d40 con 0x7f962815a0b0 2024-04-25T15:52:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620014870 con 0x7f962815a0b0 2024-04-25T15:52:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f962815a830 con 0x7f962815a0b0 2024-04-25T15:52:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.442+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f962806a480 con 0x7f962815a0b0 2024-04-25T15:52:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f962001b020 con 0x7f962815a0b0 2024-04-25T15:52:55.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f96257fa640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f960c06f430 0x7f960c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5886+0+0 (secure 0 0 0) 0x7f962008ff80 con 0x7f962815a0b0 2024-04-25T15:52:55.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v64) ==== 54+0+16 (secure 0 0 0) 0x7f9620022030 con 0x7f962815a0b0 2024-04-25T15:52:55.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f960c06f430 0x7f960c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f962815cde0 0x7f9628145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96280c32b0 con 0x7f962815cde0 2024-04-25T15:52:55.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f9627fff640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f962815cde0 0x7f9628145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f9627fff640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f962815cde0 0x7f9628145910 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f960c06f430 0x7f960c071940 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f961c0ad000 tx=0x7f961c0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.446+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96280c32b0 con 0x7f962815cde0 2024-04-25T15:52:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.466+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9628159020 con 0x7f962815cde0 2024-04-25T15:52:55.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.470+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9628159020 con 0x7f962815cde0 2024-04-25T15:52:55.471 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.470+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f96281467d0 0x7f96281f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.470+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9628158a60 con 0x7f96281467d0 2024-04-25T15:52:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.470+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f96281467d0 0x7f96281f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.470+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f96281467d0 0x7f96281f7010 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.470+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9628158a60 con 0x7f96281467d0 2024-04-25T15:52:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f9a9c640 1 Processor -- start 2024-04-25T15:52:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f9a9c640 1 -- start start 2024-04-25T15:52:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f9a9c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f60f40727b0 0x7f60f4165e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f9a9c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f4072dc0 con 0x7f60f40723a0 2024-04-25T15:52:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f9a9c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f415b430 con 0x7f60f4074790 2024-04-25T15:52:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f9a9c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f60f415a670 con 0x7f60f40727b0 2024-04-25T15:52:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f60f40723a0 0x7f60f40c6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42500/0 (socket says 172.21.15.67:42500) 2024-04-25T15:52:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f2ffd640 1 -- 172.21.15.67:0/545447809 learned_addr learned my addr 172.21.15.67:0/545447809 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f3fff640 1 --2- 172.21.15.67:0/545447809 >> v2:172.21.15.176:3300/0 conn(0x7f60f40727b0 0x7f60f4165e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2282128214 0 0) 0x7f60f4072dc0 con 0x7f60f40723a0 2024-04-25T15:52:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60dc003530 con 0x7f60f40723a0 2024-04-25T15:52:55.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2914405082 0 0) 0x7f60dc003530 con 0x7f60f40723a0 2024-04-25T15:52:55.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 >> v2:172.21.15.176:3300/0 conn(0x7f60f40727b0 msgr2=0x7f60f4165e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:55.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 --2- 172.21.15.67:0/545447809 >> v2:172.21.15.176:3300/0 conn(0x7f60f40727b0 0x7f60f4165e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 >> v1:172.21.15.176:6789/0 conn(0x7f60f4074790 legacy=0x7f60f415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f415a950 con 0x7f60f40723a0 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f60e80016a0 con 0x7f60f40723a0 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 >> v1:172.21.15.67:6789/0 conn(0x7f60f40723a0 legacy=0x7f60f40c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc0039c0 0x7f60dc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60dc004810 0x7f60dc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60dc0051d0 0x7f60dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60f415a670 con 0x7f60dc004810 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc00c160 con 0x7f60dc0039c0 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.482+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc00c360 con 0x7f60dc0051d0 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f2ffd640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60dc004810 0x7f60dc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f37fe640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc0039c0 0x7f60dc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f2ffd640 1 -- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60dc0051d0 msgr2=0x7f60dc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f2ffd640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60dc0051d0 0x7f60dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f2ffd640 1 -- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc0039c0 msgr2=0x7f60dc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:55.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f2ffd640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc0039c0 0x7f60dc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f2ffd640 1 -- 172.21.15.67:0/545447809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc003530 con 0x7f60dc004810 2024-04-25T15:52:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f3fff640 1 -- 172.21.15.67:0/545447809 reap_dead start 2024-04-25T15:52:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f37fe640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc0039c0 0x7f60dc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f2ffd640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60dc004810 0x7f60dc004c20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c210 tx=0x7f60e801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d067d7f961ae18e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8007d40 con 0x7f60dc004810 2024-04-25T15:52:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e80a9070 con 0x7f60dc004810 2024-04-25T15:52:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f60f27fc640 1 -- 172.21.15.67:0/545447809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e80a5d80 con 0x7f60dc004810 2024-04-25T15:52:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.486+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f96281578b0 con 0x7f96281467d0 2024-04-25T15:52:55.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- 172.21.15.67:0/545447809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60dc004810 msgr2=0x7f60dc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60dc004810 0x7f60dc004c20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f60e800c210 tx=0x7f60e801cc20 comp rx=0 tx=0).stop 2024-04-25T15:52:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- 172.21.15.67:0/545447809 shutdown_connections 2024-04-25T15:52:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/545447809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60dc004810 0x7f60dc004c20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- 172.21.15.67:0/545447809 >> 172.21.15.67:0/545447809 conn(0x7f60f406fe70 msgr2=0x7f60f40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- 172.21.15.67:0/545447809 shutdown_connections 2024-04-25T15:52:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- 172.21.15.67:0/545447809 wait complete. 2024-04-25T15:52:55.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96281578b0 con 0x7f96281467d0 2024-04-25T15:52:55.491 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f96281f7a60 0x7f9628063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f96281f7e70 con 0x7f96281f7a60 2024-04-25T15:52:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f9627fff640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f96281f7a60 0x7f9628063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f9627fff640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f96281f7a60 0x7f9628063090 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 Processor -- start 2024-04-25T15:52:55.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- start start 2024-04-25T15:52:55.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96281f7e70 con 0x7f96281f7a60 2024-04-25T15:52:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc004810 0x7f60f41e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f4146c20 0x7f60f41e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f414a990 0x7f60f414ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60f4073080 con 0x7f60f4146c20 2024-04-25T15:52:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f60f415af10 con 0x7f60dc004810 2024-04-25T15:52:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f9a9c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f60f4074ba0 con 0x7f60f414a990 2024-04-25T15:52:55.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f4146c20 0x7f60f41e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f4146c20 0x7f60f41e6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60562/0 (socket says 172.21.15.67:60562) 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.490+0000 7f60f2ffd640 1 -- 172.21.15.67:0/4088737409 learned_addr learned my addr 172.21.15.67:0/4088737409 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f37fe640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc004810 0x7f60f41e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f3fff640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f414a990 0x7f60f414ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f2ffd640 1 -- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f414a990 msgr2=0x7f60f414ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f2ffd640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f414a990 0x7f60f414ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f2ffd640 1 -- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc004810 msgr2=0x7f60f41e5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f2ffd640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc004810 0x7f60f41e5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f2ffd640 1 -- 172.21.15.67:0/4088737409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f41e6eb0 con 0x7f60f4146c20 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f3fff640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f414a990 0x7f60f414ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f2ffd640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f4146c20 0x7f60f41e6970 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f60e801ca50 tx=0x7f60dc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60dbfff640 1 -- 172.21.15.67:0/4088737409 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e80bc070 con 0x7f60f4146c20 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f41e5c20 con 0x7f60f4146c20 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60dbfff640 1 -- 172.21.15.67:0/4088737409 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e8000ee0 con 0x7f60f4146c20 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60dbfff640 1 -- 172.21.15.67:0/4088737409 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8007d40 con 0x7f60f4146c20 2024-04-25T15:52:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f41f6500 con 0x7f60f4146c20 2024-04-25T15:52:55.496 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:55.496 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f406a480 con 0x7f60f4146c20 2024-04-25T15:52:55.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60dbfff640 1 -- 172.21.15.67:0/4088737409 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f60e80bf070 con 0x7f60f4146c20 2024-04-25T15:52:55.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60dbfff640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f60e006f430 0x7f60e0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.494+0000 7f60dbfff640 1 -- 172.21.15.67:0/4088737409 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5886+0+0 (secure 0 0 0) 0x7f60e8128930 con 0x7f60f4146c20 2024-04-25T15:52:55.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.498+0000 7f60f37fe640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f60e006f430 0x7f60e0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.498+0000 7f60f37fe640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f60e006f430 0x7f60e0071940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f60f406a1f0 tx=0x7f60e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.502+0000 7f60dbfff640 1 -- 172.21.15.67:0/4088737409 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f60e8016930 con 0x7f60f4146c20 2024-04-25T15:52:55.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:52:55.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.510+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f96281534b0 con 0x7f96281f7a60 2024-04-25T15:52:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.514+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96281534b0 con 0x7f96281f7a60 2024-04-25T15:52:55.515 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.514+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f96280cbfc0 0x7f96280cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.514+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9628153660 con 0x7f96280cbfc0 2024-04-25T15:52:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.514+0000 7f962d701640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f96280cbfc0 0x7f96280cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.514+0000 7f962d701640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f96280cbfc0 0x7f96280cc3d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.518+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9628153660 con 0x7f96280cbfc0 2024-04-25T15:52:55.527 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:55.527+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.542+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9628153660 con 0x7f96280cbfc0 2024-04-25T15:52:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.546+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9628153660 con 0x7f96280cbfc0 2024-04-25T15:52:55.547 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.546+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f962802c9e0 0x7f962802cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.546+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f96280cce20 con 0x7f962802c9e0 2024-04-25T15:52:55.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.546+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f962802c9e0 0x7f962802cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.546+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f962802c9e0 0x7f962802cdf0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.550+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96280cce20 con 0x7f962802c9e0 2024-04-25T15:52:55.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.566+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.31", "id": [2, 3, 1, 0]} v 0) -- 0x7f0dd8005860 con 0x7f0dfc0cad90 2024-04-25T15:52:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.570+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f96280cce20 con 0x7f962802c9e0 2024-04-25T15:52:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.578+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96280cce20 con 0x7f962802c9e0 2024-04-25T15:52:55.578 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:55.578 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:55.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.578+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f962802d840 0x7f962820a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.578+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9628064010 con 0x7f962802d840 2024-04-25T15:52:55.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.578+0000 7f9627fff640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f962802d840 0x7f962820a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.578+0000 7f9627fff640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f962802d840 0x7f962820a3a0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.582+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9628064010 con 0x7f962802d840 2024-04-25T15:52:55.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.594+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9628064010 con 0x7f962802d840 2024-04-25T15:52:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.598+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9628064010 con 0x7f962802d840 2024-04-25T15:52:55.599 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.598+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f962802dc50 0x7f962820ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.598+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f962820fd10 con 0x7f962802dc50 2024-04-25T15:52:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.598+0000 7f962d701640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f962802dc50 0x7f962820ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.598+0000 7f962d701640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f962802dc50 0x7f962820ce00 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.602+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f962820fd10 con 0x7f962802dc50 2024-04-25T15:52:55.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.614+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f962820d340 con 0x7f962802dc50 2024-04-25T15:52:55.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.618+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f962820d340 con 0x7f962802dc50 2024-04-25T15:52:55.621 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:55.622 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:55.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.618+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f962820da60 0x7f962820de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:55.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.622+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f96282130e0 con 0x7f962820da60 2024-04-25T15:52:55.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.622+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f962820da60 0x7f962820de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:55.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.622+0000 7f962cf00640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f962820da60 0x7f962820de70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:55.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.622+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96282130e0 con 0x7f962820da60 2024-04-25T15:52:55.627 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:55.627+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:55.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.638+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9628211f60 con 0x7f962820da60 2024-04-25T15:52:55.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f96257fa640 1 -- 172.21.15.67:0/2412720999 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9628211f60 con 0x7f962820da60 2024-04-25T15:52:55.643 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f962815cde0 msgr2=0x7f9628145910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f962815cde0 0x7f9628145910 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f96281467d0 msgr2=0x7f96281f7010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f96281467d0 0x7f96281f7010 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f96281f7a60 msgr2=0x7f9628063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f96281f7a60 0x7f9628063090 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962d701640 1 -- 172.21.15.67:0/2412720999 reap_dead start 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f96280cbfc0 msgr2=0x7f96280cc3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f96280cbfc0 0x7f96280cc3d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f962802c9e0 msgr2=0x7f962802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f962802c9e0 0x7f962802cdf0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f962802d840 msgr2=0x7f962820a3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f962802d840 0x7f962820a3a0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f962802dc50 msgr2=0x7f962820ce00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f962802dc50 0x7f962820ce00 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f962820da60 msgr2=0x7f962820de70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f962820da60 0x7f962820de70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962d701640 1 -- 172.21.15.67:0/2412720999 reap_dead start 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f960c06f430 msgr2=0x7f960c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f960c06f430 0x7f960c071940 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f961c0ad000 tx=0x7f961c0a6000 comp rx=0 tx=0).stop 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962815a0b0 msgr2=0x7f9628144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:55.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962815a0b0 0x7f9628144960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9620009cb0 tx=0x7f962000c490 comp rx=0 tx=0).stop 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 shutdown_connections 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f960c06f430 0x7f960c071940 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 --2- 172.21.15.67:0/2412720999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962815a0b0 0x7f9628144960 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 >> 172.21.15.67:0/2412720999 conn(0x7f96280ce1c0 msgr2=0x7f962814ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 shutdown_connections 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.642+0000 7f962f164640 1 -- 172.21.15.67:0/2412720999 wait complete. 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:55.646 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:55.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:52:55.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:55.694+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f60f40c8e90 con 0x7f60f4146c20 2024-04-25T15:52:55.741 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:55.742+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:55.829 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:55.827+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:55.831 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:55.831 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:55.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:52:55.933 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:55.934+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:56.031 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:56.031+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:56.040 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:56.040 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:56.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:52:56.145 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:56.146+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:56.233 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:56.230+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:56.247 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:56.248 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:56.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:52:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f0dee7fc640 1 -- 172.21.15.67:0/3422516927 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.31", "id": [2, 3, 1, 0]}]=0 set 3.31 pg_upmap_items mapping to [2->3,1->0] v65) ==== 148+0+0 (secure 0 0 0) 0x7f0df005ae40 con 0x7f0dfc0cad90 2024-04-25T15:52:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60dbfff640 1 -- 172.21.15.67:0/4088737409 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v65) ==== 97+0+0 (secure 0 0 0) 0x7f60e80f5800 con 0x7f60f4146c20 2024-04-25T15:52:56.311 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.31 pg_upmap_items mapping to [2->3,1->0] 2024-04-25T15:52:56.311 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:52:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f60e006f430 msgr2=0x7f60e0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f60e006f430 0x7f60e0071940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f60f406a1f0 tx=0x7f60e4009040 comp rx=0 tx=0).stop 2024-04-25T15:52:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f4146c20 msgr2=0x7f60f41e6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f4146c20 0x7f60f41e6970 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f60e801ca50 tx=0x7f60dc005840 comp rx=0 tx=0).stop 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 shutdown_connections 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f60e006f430 0x7f60e0071940 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f414a990 0x7f60f414ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.310+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f4146c20 0x7f60f41e6970 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f60f9a9c640 1 --2- 172.21.15.67:0/4088737409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60dc004810 0x7f60f41e5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 >> 172.21.15.67:0/4088737409 conn(0x7f60f406fe70 msgr2=0x7f60f40ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 shutdown_connections 2024-04-25T15:52:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f60f9a9c640 1 -- 172.21.15.67:0/4088737409 wait complete. 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0de006f430 msgr2=0x7f0de0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0de006f430 0x7f0de0071940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0b2370 tx=0x7f0df8008040 comp rx=0 tx=0).stop 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc0cad90 msgr2=0x7f0dfc1f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc0cad90 0x7f0dfc1f6750 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c4b0 tx=0x7f0df000c970 comp rx=0 tx=0).stop 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 shutdown_connections 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0de006f430 0x7f0de0071940 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0dfc160a40 0x7f0dfc0b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0dfc0cb1a0 0x7f0dfc14d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 --2- 172.21.15.67:0/3422516927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dfc0cad90 0x7f0dfc1f6750 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 >> 172.21.15.67:0/3422516927 conn(0x7f0dfc0cebf0 msgr2=0x7f0dfc15a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 shutdown_connections 2024-04-25T15:52:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:56.314+0000 7f0ddffff640 1 -- 172.21.15.67:0/3422516927 wait complete. 2024-04-25T15:52:56.334 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:56.334+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:56.435 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:56.435+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:56.455 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:56.455 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:56.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:52:56.536 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:56.535+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:56.626 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:56.626 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:56.638 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:56.638+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:56.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:52:56.737 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:56.735+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:56.838 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:56.835+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:56.845 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:56.846 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:56.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:52:56.939 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:56.939+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:57.019 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:57.020 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:57.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:52:57.079 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:57.079+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:57.141 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:57.139+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:57.228 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:57.228 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:57.242 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:57.242+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:57.246 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:52:57.342 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:57.342+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:57.434 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:57.434 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:57.443 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:57.442+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:57.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:52:57.544 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:57.543+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:57.599 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:57.599 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:57.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:52:57.659 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:57.659+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:57.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:52:57.746 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:57.743+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:57.810 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:57.810 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:57.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe226da2640 1 Processor -- start 2024-04-25T15:52:57.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe226da2640 1 -- start start 2024-04-25T15:52:57.828 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe226da2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 0x7fe2200736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe226da2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe22015aaf0 0x7fe22015af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe226da2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2200caa60 con 0x7fe2201547e0 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe226da2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe220068590 con 0x7fe22015aaf0 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe226da2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe22015a5f0 con 0x7fe22006d460 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe21ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe2201547e0 0x7fe22006c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42512/0 (socket says 172.21.15.67:42512) 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe21ffff640 1 -- 172.21.15.67:0/183713063 learned_addr learned my addr 172.21.15.67:0/183713063 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe224b3e640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 0x7fe2200736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe22533f640 1 --2- 172.21.15.67:0/183713063 >> v2:172.21.15.176:3300/0 conn(0x7fe22015aaf0 0x7fe22015af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288934609 0 0) 0x7fe2200caa60 con 0x7fe2201547e0 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.826+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe20c003530 con 0x7fe2201547e0 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1426632303 0 0) 0x7fe20c003530 con 0x7fe2201547e0 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 msgr2=0x7fe2200736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 0x7fe2200736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:57.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 >> v2:172.21.15.176:3300/0 conn(0x7fe22015aaf0 msgr2=0x7fe22015af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 --2- 172.21.15.67:0/183713063 >> v2:172.21.15.176:3300/0 conn(0x7fe22015aaf0 0x7fe22015af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2200ca100 con 0x7fe2201547e0 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fe208001800 con 0x7fe2201547e0 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 >> v1:172.21.15.67:6789/0 conn(0x7fe2201547e0 legacy=0x7fe22006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 0x7fe20c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe20c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe220068590 con 0x7fe20c003f40 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe22015a5f0 con 0x7fe20c004d50 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe20c00ba20 con 0x7fe22006d460 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21ffff640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe20c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe224b3e640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 0x7fe20c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe22533f640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21ffff640 1 -- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 msgr2=0x7fe20c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21ffff640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe22006d460 0x7fe20c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21ffff640 1 -- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe20c004d50 msgr2=0x7fe20c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21ffff640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21ffff640 1 -- 172.21.15.67:0/183713063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c003530 con 0x7fe20c003f40 2024-04-25T15:52:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe22533f640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe20c004d50 0x7fe20c005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21ffff640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe20c004810 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe2080011e0 tx=0x7fe20801f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ca518aca3879d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208007d50 con 0x7fe20c003f40 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe208007290 con 0x7fe20c003f40 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe21f7fe640 1 -- 172.21.15.67:0/183713063 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2080057b0 con 0x7fe20c003f40 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe226da2640 1 -- 172.21.15.67:0/183713063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 msgr2=0x7fe20c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe226da2640 1 --2- 172.21.15.67:0/183713063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe20c004810 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe2080011e0 tx=0x7fe20801f390 comp rx=0 tx=0).stop 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe22533f640 1 -- 172.21.15.67:0/183713063 reap_dead start 2024-04-25T15:52:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe226da2640 1 -- 172.21.15.67:0/183713063 shutdown_connections 2024-04-25T15:52:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.830+0000 7fe226da2640 1 -- 172.21.15.67:0/183713063 >> 172.21.15.67:0/183713063 conn(0x7fe2200ce1c0 msgr2=0x7fe2200ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 -- 172.21.15.67:0/183713063 shutdown_connections 2024-04-25T15:52:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 -- 172.21.15.67:0/183713063 wait complete. 2024-04-25T15:52:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 Processor -- start 2024-04-25T15:52:57.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 -- start start 2024-04-25T15:52:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe2200ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2201f3db0 0x7fe22006c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2200c7640 0x7fe220155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe220068ac0 con 0x7fe20c003f40 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe2200687e0 con 0x7fe2201f3db0 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe226da2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe22015b440 con 0x7fe2200c7640 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe2200ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe2200ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60582/0 (socket says 172.21.15.67:60582) 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 -- 172.21.15.67:0/771168986 learned_addr learned my addr 172.21.15.67:0/771168986 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2201f3db0 0x7fe22006c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2200c7640 0x7fe220155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2200c7640 msgr2=0x7fe220155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2200c7640 0x7fe220155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2201f3db0 msgr2=0x7fe22006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2201f3db0 0x7fe22006c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2200c60e0 con 0x7fe20c003f40 2024-04-25T15:52:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2200c7640 0x7fe220155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2201f3db0 0x7fe22006c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:52:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.834+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe2200ad1b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe210011e30 tx=0x7fe21000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210015630 con 0x7fe20c003f40 2024-04-25T15:52:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2100157d0 con 0x7fe20c003f40 2024-04-25T15:52:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2200c6680 con 0x7fe20c003f40 2024-04-25T15:52:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210015ba0 con 0x7fe20c003f40 2024-04-25T15:52:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2200c58e0 con 0x7fe20c003f40 2024-04-25T15:52:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe22006a480 con 0x7fe20c003f40 2024-04-25T15:52:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe21001d400 con 0x7fe20c003f40 2024-04-25T15:52:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe20006f400 0x7fe200071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5915+0+0 (secure 0 0 0) 0x7fe210090d80 con 0x7fe20c003f40 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v66) ==== 54+0+16 (secure 0 0 0) 0x7fe210019030 con 0x7fe20c003f40 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe20006f400 0x7fe200071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe2201f42b0 0x7fe2201564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2200cb160 con 0x7fe2201f42b0 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe2201f42b0 0x7fe2201564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe2201f42b0 0x7fe2201564b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe20006f400 0x7fe200071910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe2080ae000 tx=0x7fe208001170 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.838+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe2200cb160 con 0x7fe2201f42b0 2024-04-25T15:52:57.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.854+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe2200734c0 con 0x7fe2201f42b0 2024-04-25T15:52:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.858+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe2200734c0 con 0x7fe2201f42b0 2024-04-25T15:52:57.858 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:52:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:52:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.858+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe2201515e0 0x7fe2201519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.858+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe2200b4440 con 0x7fe2201515e0 2024-04-25T15:52:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.858+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe2201515e0 0x7fe2201519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.858+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe2201515e0 0x7fe2201519f0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.858+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe2200b4440 con 0x7fe2201515e0 2024-04-25T15:52:57.871 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:57.871+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:57.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe2200c8770 con 0x7fe2201515e0 2024-04-25T15:52:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe2200c8770 con 0x7fe2201515e0 2024-04-25T15:52:57.875 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:52:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:52:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 0x7fe2200634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe220152440 con 0x7fe220063090 2024-04-25T15:52:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 0x7fe2200634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe21ffff640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 msgr2=0x7fe2200634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-04-25T15:52:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 0x7fe2200634a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2024-04-25T15:52:57.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:57.874+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 0x7fe2200634a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:52:57.948 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:57.946+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:58.024 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:58.024 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:58.041 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:52:58.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.078+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 0x7fe2200634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.078+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 0x7fe2200634a0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.082+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe220152440 con 0x7fe220063090 2024-04-25T15:52:58.087 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:52:58.087+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:52:58.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.102+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe2200639e0 con 0x7fe220063090 2024-04-25T15:52:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.106+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe2200639e0 con 0x7fe220063090 2024-04-25T15:52:58.107 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:52:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:52:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.106+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe220158ad0 0x7fe220158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.106+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe2200639e0 con 0x7fe220158ad0 2024-04-25T15:52:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.106+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe220158ad0 0x7fe220158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.106+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe220158ad0 0x7fe220158ee0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.110+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe2200639e0 con 0x7fe220158ad0 2024-04-25T15:52:58.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.138+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe2200639e0 con 0x7fe220158ad0 2024-04-25T15:52:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.142+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe2200639e0 con 0x7fe220158ad0 2024-04-25T15:52:58.143 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:52:58.143 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:52:58.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.142+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe22002cef0 0x7fe22002d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.142+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe2200639e0 con 0x7fe22002cef0 2024-04-25T15:52:58.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.142+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe22002cef0 0x7fe22002d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.142+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe22002cef0 0x7fe22002d300 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.146+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe2200639e0 con 0x7fe22002cef0 2024-04-25T15:52:58.149 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:52:58.147+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:52:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.178+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe22002d840 con 0x7fe22002cef0 2024-04-25T15:52:58.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.178+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe22002d840 con 0x7fe22002cef0 2024-04-25T15:52:58.180 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:52:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:52:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.178+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe2202053b0 0x7fe2202077d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.178+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe22020a5a0 con 0x7fe2202053b0 2024-04-25T15:52:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.182+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe2202053b0 0x7fe2202077d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.182+0000 7fe21ffff640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe2202053b0 0x7fe2202077d0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.182+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe22020a5a0 con 0x7fe2202053b0 2024-04-25T15:52:58.193 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:52:58.193 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:52:58.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.206+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe220207d10 con 0x7fe2202053b0 2024-04-25T15:52:58.212 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:52:58.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.210+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe220207d10 con 0x7fe2202053b0 2024-04-25T15:52:58.213 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:52:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:52:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.210+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe220208220 0x7fe22020a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.210+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe22020d740 con 0x7fe220208220 2024-04-25T15:52:58.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.210+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe220208220 0x7fe22020a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.214+0000 7fe22533f640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe220208220 0x7fe22020a6a0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.214+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe22020d740 con 0x7fe220208220 2024-04-25T15:52:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.234+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe22020c460 con 0x7fe220208220 2024-04-25T15:52:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.238+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe22020c460 con 0x7fe220208220 2024-04-25T15:52:58.241 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:52:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:52:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.238+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe22020cb80 0x7fe22020cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.238+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe220210b10 con 0x7fe22020cb80 2024-04-25T15:52:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.242+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe22020cb80 0x7fe22020cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.242+0000 7fe224b3e640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe22020cb80 0x7fe22020cf90 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.242+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe220210b10 con 0x7fe22020cb80 2024-04-25T15:52:58.251 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:58.250+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:58.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.262+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe22020f990 con 0x7fe22020cb80 2024-04-25T15:52:58.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe21d7fa640 1 -- 172.21.15.67:0/771168986 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe22020f990 con 0x7fe22020cb80 2024-04-25T15:52:58.266 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe2201f42b0 msgr2=0x7fe2201564b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe2201f42b0 0x7fe2201564b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe2201515e0 msgr2=0x7fe2201519f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe2201515e0 0x7fe2201519f0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 msgr2=0x7fe2200634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe220063090 0x7fe2200634a0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe220158ad0 msgr2=0x7fe220158ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe220158ad0 0x7fe220158ee0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe22002cef0 msgr2=0x7fe22002d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe22002cef0 0x7fe22002d300 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe2202053b0 msgr2=0x7fe2202077d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe2202053b0 0x7fe2202077d0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe220208220 msgr2=0x7fe22020a6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe220208220 0x7fe22020a6a0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe22020cb80 msgr2=0x7fe22020cf90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe22020cb80 0x7fe22020cf90 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe22533f640 1 -- 172.21.15.67:0/771168986 reap_dead start 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe22533f640 1 -- 172.21.15.67:0/771168986 reap_dead start 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe22533f640 1 -- 172.21.15.67:0/771168986 reap_dead start 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe22533f640 1 -- 172.21.15.67:0/771168986 reap_dead start 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe20006f400 msgr2=0x7fe200071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe20006f400 0x7fe200071910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe2080ae000 tx=0x7fe208001170 comp rx=0 tx=0).stop 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe22533f640 1 -- 172.21.15.67:0/771168986 reap_dead start 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 msgr2=0x7fe2200ad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe2200ad1b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe210011e30 tx=0x7fe21000f580 comp rx=0 tx=0).stop 2024-04-25T15:52:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 shutdown_connections 2024-04-25T15:52:58.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe20006f400 0x7fe200071910 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 --2- 172.21.15.67:0/771168986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe20c003f40 0x7fe2200ad1b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 >> 172.21.15.67:0/771168986 conn(0x7fe2200ce1c0 msgr2=0x7fe220159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:58.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 shutdown_connections 2024-04-25T15:52:58.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.266+0000 7fe226da2640 1 -- 172.21.15.67:0/771168986 wait complete. 2024-04-25T15:52:58.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:52:58.386 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:58.386+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:58.400 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:58.400 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:58.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:52:58.452 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:58.450+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:58.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.502+0000 7f2e66d27640 1 Processor -- start 2024-04-25T15:52:58.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.502+0000 7f2e66d27640 1 -- start start 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e66d27640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e600caf60 0x7f2e6015d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e66d27640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e6015d790 0x7f2e6006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e66d27640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e600c77a0 con 0x7f2e60073bf0 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e66d27640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2e600cab60 con 0x7f2e6015d790 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e66d27640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2e60159620 con 0x7f2e600caf60 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e6015d790 0x7f2e6006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e6015d790 0x7f2e6006d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60584/0 (socket says 172.21.15.67:60584) 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e652c4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2e60073bf0 0x7f2e60072bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42516/0 (socket says 172.21.15.67:42516) 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 -- 172.21.15.67:0/3154554178 learned_addr learned my addr 172.21.15.67:0/3154554178 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e64ac3640 1 --2- 172.21.15.67:0/3154554178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e600caf60 0x7f2e6015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 -- 172.21.15.67:0/3154554178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e600caf60 msgr2=0x7f2e6015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 --2- 172.21.15.67:0/3154554178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e600caf60 0x7f2e6015d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 -- 172.21.15.67:0/3154554178 >> v1:172.21.15.67:6789/0 conn(0x7f2e60073bf0 legacy=0x7f2e60072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 -- 172.21.15.67:0/3154554178 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e600c7a60 con 0x7f2e6015d790 2024-04-25T15:52:58.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e577fe640 1 -- 172.21.15.67:0/3154554178 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345964399 0 0) 0x7f2e600c77a0 con 0x7f2e60073bf0 2024-04-25T15:52:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e57fff640 1 --2- 172.21.15.67:0/3154554178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6015d790 0x7f2e6006d750 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009870 tx=0x7f2e5001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b6cef6eac0eec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e577fe640 1 -- 172.21.15.67:0/3154554178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e500b2070 con 0x7f2e6015d790 2024-04-25T15:52:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e577fe640 1 -- 172.21.15.67:0/3154554178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e500a9070 con 0x7f2e6015d790 2024-04-25T15:52:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.506+0000 7f2e577fe640 1 -- 172.21.15.67:0/3154554178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e500ad880 con 0x7f2e6015d790 2024-04-25T15:52:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- 172.21.15.67:0/3154554178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6015d790 msgr2=0x7f2e6006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3154554178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6015d790 0x7f2e6006d750 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009870 tx=0x7f2e5001c560 comp rx=0 tx=0).stop 2024-04-25T15:52:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- 172.21.15.67:0/3154554178 shutdown_connections 2024-04-25T15:52:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3154554178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6015d790 0x7f2e6006d750 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3154554178 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e600caf60 0x7f2e6015d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- 172.21.15.67:0/3154554178 >> 172.21.15.67:0/3154554178 conn(0x7f2e60070db0 msgr2=0x7f2e600ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- 172.21.15.67:0/3154554178 shutdown_connections 2024-04-25T15:52:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- 172.21.15.67:0/3154554178 wait complete. 2024-04-25T15:52:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 Processor -- start 2024-04-25T15:52:58.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- start start 2024-04-25T15:52:58.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e60073bf0 0x7f2e601f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e600caf60 0x7f2e6015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e6015d790 0x7f2e60074330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e60165050 con 0x7f2e600caf60 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2e60164ed0 con 0x7f2e6015d790 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e66d27640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2e601651d0 con 0x7f2e60073bf0 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.510+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e600caf60 0x7f2e6015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e600caf60 0x7f2e6015c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60596/0 (socket says 172.21.15.67:60596) 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 -- 172.21.15.67:0/3683575199 learned_addr learned my addr 172.21.15.67:0/3683575199 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e64ac3640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e60073bf0 0x7f2e601f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e652c4640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e6015d790 0x7f2e60074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 -- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e60073bf0 msgr2=0x7f2e601f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e60073bf0 0x7f2e601f7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 -- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e6015d790 msgr2=0x7f2e60074330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e6015d790 0x7f2e60074330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 -- 172.21.15.67:0/3683575199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6015c9a0 con 0x7f2e600caf60 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e57fff640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e600caf60 0x7f2e6015c460 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2e5001ac10 tx=0x7f2e5001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e557fa640 1 -- 172.21.15.67:0/3683575199 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e500bc070 con 0x7f2e600caf60 2024-04-25T15:52:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e557fa640 1 -- 172.21.15.67:0/3683575199 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e500ade50 con 0x7f2e600caf60 2024-04-25T15:52:58.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e601f7990 con 0x7f2e600caf60 2024-04-25T15:52:58.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e557fa640 1 -- 172.21.15.67:0/3683575199 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e500b2040 con 0x7f2e600caf60 2024-04-25T15:52:58.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e6015bc60 con 0x7f2e600caf60 2024-04-25T15:52:58.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e6006a480 con 0x7f2e600caf60 2024-04-25T15:52:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e557fa640 1 -- 172.21.15.67:0/3683575199 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2e500ab070 con 0x7f2e600caf60 2024-04-25T15:52:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e557fa640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2e4c06f400 0x7f2e4c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e557fa640 1 -- 172.21.15.67:0/3683575199 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5915+0+0 (secure 0 0 0) 0x7f2e50128ff0 con 0x7f2e600caf60 2024-04-25T15:52:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.514+0000 7f2e64ac3640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2e4c06f400 0x7f2e4c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.518+0000 7f2e64ac3640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2e4c06f400 0x7f2e4c071910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00b930 tx=0x7f2e5c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.518+0000 7f2e557fa640 1 -- 172.21.15.67:0/3683575199 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2e500c7060 con 0x7f2e600caf60 2024-04-25T15:52:58.553 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:58.551+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:52:58.559 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:58.559 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:58.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:58.559 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:58.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:43 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.653894032, 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024229051000000001, 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902019+0000", 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9459999999999998e-06 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907768+0000", 2024-04-25T15:52:58.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005748525 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907831+0000", 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2842999999999994e-05 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907857+0000", 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6636e-05 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908056+0000", 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198295 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908464+0000", 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040813600000000002 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908853+0000", 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038892400000000002 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925332+0000", 2024-04-25T15:52:58.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016479074 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925882+0000", 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000550232 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926050+0000", 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016822199999999999 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926245+0000", 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019521799999999999 2024-04-25T15:52:58.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:53 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.177711871, 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020793931000000002, 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 53 2024-04-25T15:52:58.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378202+0000", 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1669999999999999e-06 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382531+0000", 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043289130000000002 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382593+0000", 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2143000000000005e-05 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382618+0000", 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382814+0000", 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195964 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383075+0000", 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026081800000000001 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383324+0000", 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024972200000000002 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.392258+0000", 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089338549999999992 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398665+0000", 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064066649999999998 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398841+0000", 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017607899999999999 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398993+0000", 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015144799999999999 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:55 3.10 3:09288c7b:::10000000002.00000033:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.084151295, 2024-04-25T15:52:58.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018106206999999999, 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 55 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471759+0000", 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.471763+0000", 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.794e-06 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476322+0000", 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045595009999999997 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476383+0000", 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0936000000000001e-05 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476410+0000", 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6313e-05 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476613+0000", 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020343499999999999 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.476897+0000", 2024-04-25T15:52:58.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028380400000000002 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.477152+0000", 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025521799999999998 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.484642+0000", 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074893970000000001 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489677+0000", 2024-04-25T15:52:58.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050351010000000002 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489779+0000", 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102592 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.489865+0000", 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6116e-05 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:57 3.3 3:c0bc6424:::10000000002.00000035:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.575 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.997877171000001, 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017860478999999999, 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 57 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558033+0000", 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.558036+0000", 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9799999999999998e-06 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562071+0000", 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040351400000000004 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562126+0000", 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4249000000000002e-05 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562149+0000", 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3674000000000001e-05 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562327+0000", 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177498 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562581+0000", 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025446799999999999 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.562836+0000", 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000254751 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.572670+0000", 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0098337700000000004 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575703+0000", 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030327980000000002 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575802+0000", 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9278999999999996e-05 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.575894+0000", 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1872000000000004e-05 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:59 3.20 3:072133bc:::10000000002.00000037:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.9225408, 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016873692999999999, 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 59 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633370+0000", 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.633373+0000", 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2430000000000001e-06 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637297+0000", 2024-04-25T15:52:58.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039241060000000001 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637354+0000", 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7357e-05 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637380+0000", 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5165999999999999e-05 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637564+0000", 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000184332 2024-04-25T15:52:58.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.637837+0000", 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027320399999999998 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.638090+0000", 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025292100000000003 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.647362+0000", 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092716989999999996 2024-04-25T15:52:58.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650052+0000", 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026907599999999999 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650151+0000", 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8603000000000004e-05 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.650243+0000", 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2301999999999998e-05 2024-04-25T15:52:58.585 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:62 3.11 e60/54 3:8969f477:::10000000002.0000003a:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.794226977999999, 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019322214000000001, 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:58.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761684+0000", 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3399999999999998e-07 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765759+0000", 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040747159999999999 2024-04-25T15:52:58.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765801+0000", 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1675999999999999e-05 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765835+0000", 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4518999999999999e-05 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765978+0000", 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014263899999999999 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.766025+0000", 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6842999999999999e-05 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780902+0000", 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014877118 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781006+0000", 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010396899999999999 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.589 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:73 3.1d e60/54 3:b94578cb:::10000000002.00000045:head v 60'2, mlcod=0'0)", 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.292501578, 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019180661000000002, 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:58.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263412+0000", 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9900000000000002e-06 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268799+0000", 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053871980000000002 2024-04-25T15:52:58.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268844+0000", 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5062999999999999e-05 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268870+0000", 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5633e-05 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269028+0000", 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000158135 2024-04-25T15:52:58.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269078+0000", 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0463000000000001e-05 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282434+0000", 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013355156999999999 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282590+0000", 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015602199999999999 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.593 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:75 3.35 3:ad096b5b:::10000000002.00000047:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.222401331, 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022276672000000001, 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.594 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333513+0000", 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:52:58.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339006+0000", 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054929599999999999 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339071+0000", 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5545999999999999e-05 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339098+0000", 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6647000000000001e-05 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339299+0000", 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201351 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339703+0000", 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040396399999999998 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.340160+0000", 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045697899999999999 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.352668+0000", 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012508137000000001 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355605+0000", 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002937013 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355701+0000", 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5681999999999996e-05 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355786+0000", 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4908999999999996e-05 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:79 3.3 3:c0361aff:::10000000002.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.046631236, 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022495096999999999, 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.599 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 79 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509282+0000", 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2200000000000001e-06 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517219+0000", 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079362270000000006 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517284+0000", 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5052e-05 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517310+0000", 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6072999999999999e-05 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517506+0000", 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019606199999999999 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517779+0000", 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027302799999999998 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.518047+0000", 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000268454 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.528734+0000", 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01068712 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531586+0000", 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028516589999999999 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531689+0000", 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010260900000000001 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531774+0000", 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5593000000000005e-05 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:80 3.20 3:0533b28f:::10000000002.0000004c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:58.604 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.004065602000001, 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018714178000000001, 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 80 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551845+0000", 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.551848+0000", 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0230000000000001e-06 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556855+0000", 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050073180000000002 2024-04-25T15:52:58.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556907+0000", 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1446000000000002e-05 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.556929+0000", 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2405999999999999e-05 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557100+0000", 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000171003 2024-04-25T15:52:58.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557387+0000", 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028688699999999999 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.557640+0000", 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025292499999999998 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.566164+0000", 2024-04-25T15:52:58.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085239770000000003 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570375+0000", 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004211345 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570471+0000", 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5933000000000005e-05 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.570559+0000", 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7915000000000001e-05 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:82 3.34 3:2c8ffcca:::10000000002.0000004e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.916210274999999, 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017224060999999999, 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.610 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 82 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639700+0000", 2024-04-25T15:52:58.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.639703+0000", 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7070000000000001e-06 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644603+0000", 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048998549999999998 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644658+0000", 2024-04-25T15:52:58.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4987000000000001e-05 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644682+0000", 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4280000000000001e-05 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.644876+0000", 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000193678 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.645141+0000", 2024-04-25T15:52:58.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000265321 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.645394+0000", 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025291099999999998 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.655180+0000", 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0097862139999999997 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656739+0000", 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015587800000000001 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656840+0000", 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101172 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.656924+0000", 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4155999999999999e-05 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:141 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.240113+0000", 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.315797861, 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016842197, 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 141 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.240113+0000", 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.240113+0000", 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.240116+0000", 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.411e-06 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244602+0000", 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044857550000000001 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244648+0000", 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.613e-05 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244666+0000", 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8003e-05 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244759+0000", 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3203999999999997e-05 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.244904+0000", 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014452000000000001 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.245015+0000", 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011147800000000001 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.253876+0000", 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088605030000000005 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.256853+0000", 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029777689999999999 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.256910+0000", 2024-04-25T15:52:58.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6419999999999999e-05 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.256955+0000", 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5003999999999998e-05 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:144 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.187723789, 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019320924999999999, 2024-04-25T15:52:58.621 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 144 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368190+0000", 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4170000000000001e-06 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373302+0000", 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005111857 2024-04-25T15:52:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373366+0000", 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3570000000000003e-05 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373413+0000", 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7679000000000002e-05 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373579+0000", 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000165422 2024-04-25T15:52:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373836+0000", 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025765800000000002 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.374080+0000", 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024384499999999999 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.386500+0000", 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012419688999999999 2024-04-25T15:52:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387295+0000", 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00079550599999999997 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387403+0000", 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010791899999999999 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387508+0000", 2024-04-25T15:52:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104363 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:154 3.34 3:2cf0bbf2:::10000000002.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.834003+0000", 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7219074509999999, 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018116257, 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 154 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.834003+0000", 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.834003+0000", 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.834007+0000", 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6330000000000002e-06 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.838978+0000", 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049714049999999999 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839038+0000", 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0133000000000002e-05 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839064+0000", 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5962999999999999e-05 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839256+0000", 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019204199999999999 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839527+0000", 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027048099999999999 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.839790+0000", 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000263114 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.849864+0000", 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010074471999999999 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.851907+0000", 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020428260000000002 2024-04-25T15:52:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.852017+0000", 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110402 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.852119+0000", 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101786 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:165 3.11 e62/54 3:88ab7984:::10000000002.000000a1:head v 62'7, mlcod=61'5)", 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.1748366089999998, 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020252645999999999, 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381075+0000", 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9299999999999997e-07 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386313+0000", 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052379799999999997 2024-04-25T15:52:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386350+0000", 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7469000000000003e-05 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386384+0000", 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3862999999999997e-05 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386546+0000", 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016226899999999999 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386587+0000", 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1026000000000003e-05 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401215+0000", 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01462777 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401327+0000", 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111576 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:166 3.2c 3:3723e56f:::10000000002.000000a2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.1418882270000008, 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019979538000000002, 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 166 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414025+0000", 2024-04-25T15:52:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.92e-06 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.419994+0000", 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059684200000000003 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420068+0000", 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4878999999999999e-05 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420093+0000", 2024-04-25T15:52:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4553e-05 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420239+0000", 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014577500000000001 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420512+0000", 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027295800000000001 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420761+0000", 2024-04-25T15:52:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024945499999999999 2024-04-25T15:52:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.431910+0000", 2024-04-25T15:52:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011148739 2024-04-25T15:52:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.433806+0000", 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018964909999999999 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.433907+0000", 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100179 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.434002+0000", 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5168999999999996e-05 2024-04-25T15:52:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:176 3.26 3:66bf65cf:::10000000002.000000ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.729081377, 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020587095999999999, 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 176 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826832+0000", 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1200000000000002e-06 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831165+0000", 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043325289999999999 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831220+0000", 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5686999999999997e-05 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831245+0000", 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4105999999999999e-05 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831400+0000", 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015513899999999999 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831669+0000", 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026955099999999998 2024-04-25T15:52:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831947+0000", 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027750399999999998 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.842230+0000", 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010283537000000001 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847229+0000", 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049990939999999999 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847326+0000", 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7106000000000002e-05 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847416+0000", 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9722999999999995e-05 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:177 3.26 3:66aef6e9:::10000000002.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.6792224789999999, 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026823691, 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876691+0000", 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4939999999999999e-06 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880736+0000", 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040441660000000001 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880793+0000", 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7581999999999999e-05 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880817+0000", 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881039+0000", 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022171800000000001 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881305+0000", 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026580799999999997 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881558+0000", 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025290099999999998 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.893182+0000", 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011624242 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903322+0000", 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010139754000000001 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903425+0000", 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102706 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903512+0000", 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7093000000000001e-05 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:179 3.1e e62/18 3:789394ea:::10000000002.000000af:head v 62'2, mlcod=0'0)", 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.963928+0000", 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.5919821380000005, 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018290402000000001, 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.963928+0000", 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.963928+0000", 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.963929+0000", 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1299999999999999e-07 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969719+0000", 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057901879999999999 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969755+0000", 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6019000000000002e-05 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969779+0000", 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3266999999999999e-05 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969939+0000", 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016062199999999999 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.969992+0000", 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2519000000000003e-05 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.982088+0000", 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012096129000000001 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.982219+0000", 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013094499999999999 2024-04-25T15:52:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:182 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.466189323, 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018848521, 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 182 2024-04-25T15:52:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089724+0000", 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2270000000000002e-06 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095098+0000", 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053738709999999997 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095165+0000", 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6618999999999994e-05 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095192+0000", 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6783000000000001e-05 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095330+0000", 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013835900000000001 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095586+0000", 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025566400000000001 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095830+0000", 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024478199999999997 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.104485+0000", 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086547110000000007 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108421+0000", 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039358010000000001 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108497+0000", 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5799999999999999e-05 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108570+0000", 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2904000000000005e-05 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:58.664 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:58.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:52:58.666 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:52:58.666+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:52:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:58.690+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f2e600c8a40 con 0x7f2e600caf60 2024-04-25T15:52:58.766 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:58.766+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:58.853 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:58.853 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:58.853 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:52:58.853 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:58.867 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:52:58.866+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:52:58.868 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:52:58.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:52:58.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:52:58.968 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:58.966+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:59.001 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:52:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:52:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:52:59.001 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:59.016 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:52:59.068 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:59.066+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:59.153 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:52:59.153 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:52:59.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:52:59.153 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 160.05142673200001, 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:52:59.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 159.46037243000001, 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021682439000000001, 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.160 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 50 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689518+0000", 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689519+0000", 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.13e-07 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689661+0000", 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014232900000000001 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689984+0000", 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032253000000000002 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689992+0000", 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5639999999999997e-06 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706437+0000", 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016444754999999998 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.706687+0000", 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024987499999999999 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.707123+0000", 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000436243 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.710963+0000", 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038392859999999999 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711006+0000", 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3551999999999999e-05 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711119+0000", 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011282599999999999 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711201+0000", 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1866000000000005e-05 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 159.46022380799999, 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:52:59.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:52:59.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:52:59.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 159.46021272900001, 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.171 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 159.15250968999999, 2024-04-25T15:52:59.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:52:59.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:52:59.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:41 3.f 3:f045ad74:::10000000002.00000025:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.3404255, 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022153520999999999, 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 41 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809465+0000", 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.809469+0000", 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7670000000000001e-06 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814177+0000", 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047077969999999997 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814211+0000", 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4173e-05 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814226+0000", 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5444e-05 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814394+0000", 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016725400000000001 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.814776+0000", 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038195999999999998 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.815181+0000", 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040559299999999999 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.824339+0000", 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0091577129999999996 2024-04-25T15:52:59.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831521+0000", 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071817010000000004 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831568+0000", 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7250000000000003e-05 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.831619+0000", 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0868999999999997e-05 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.186 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:47 3.12 3:489b4446:::10000000002.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.049366963000001, 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022914499000000001, 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.187 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100528+0000", 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3030000000000001e-06 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105104+0000", 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045757710000000002 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105167+0000", 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2936999999999997e-05 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105194+0000", 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7606e-05 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105424+0000", 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229998 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105705+0000", 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000280408 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105966+0000", 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261258 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.118921+0000", 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012954482999999999 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123247+0000", 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043263199999999998 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123350+0000", 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103423 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123438+0000", 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7991999999999994e-05 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:62 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.394237837, 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025889728000000001, 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.192 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 62 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:52:59.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755657+0000", 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9439999999999998e-06 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760593+0000", 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049360740000000004 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760651+0000", 2024-04-25T15:52:59.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8e-05 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760676+0000", 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5219999999999999e-05 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760847+0000", 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017057199999999999 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761123+0000", 2024-04-25T15:52:59.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027578100000000001 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761365+0000", 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024248100000000001 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780642+0000", 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019277327 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781358+0000", 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000715611 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781456+0000", 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7946e-05 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781543+0000", 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6772000000000002e-05 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.197 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:72 3.39 3:9df4b1b7:::10000000002.00000044:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.934164421, 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022151042999999999, 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 72 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215726+0000", 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.215730+0000", 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221532+0000", 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058028009999999998 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221606+0000", 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3842000000000002e-05 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221642+0000", 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6232999999999997e-05 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.221851+0000", 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020811199999999999 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.222277+0000", 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042676200000000002 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.222696+0000", 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041885400000000003 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.236603+0000", 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013906992 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237694+0000", 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001091091 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237791+0000", 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6455999999999999e-05 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.237877+0000", 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6749999999999994e-05 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:150 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.498718905, 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024331635000000001, 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.203 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 150 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651176+0000", 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8770000000000003e-06 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659329+0000", 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081530219999999994 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659399+0000", 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0376000000000005e-05 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659428+0000", 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8952999999999999e-05 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659580+0000", 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015188199999999999 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659842+0000", 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026179399999999998 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.660120+0000", 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027824799999999998 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.673813+0000", 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013693145 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675307+0000", 2024-04-25T15:52:59.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014934600000000001 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675412+0000", 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105233 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675504+0000", 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1644999999999994e-05 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:152 3.39 3:9e24e8b4:::10000000002.00000094:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.418850878000001, 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022233425000000001, 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 152 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731040+0000", 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.731043+0000", 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.735937+0000", 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048940520000000003 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736003+0000", 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6112000000000007e-05 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736030+0000", 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6744e-05 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736177+0000", 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014675800000000001 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736344+0000", 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016679100000000001 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.736481+0000", 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000137112 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.744634+0000", 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081533030000000006 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753167+0000", 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085328769999999995 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753222+0000", 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5167999999999998e-05 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.753273+0000", 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1304999999999998e-05 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:155 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.27656762, 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025243293999999999, 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.214 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 155 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873327+0000", 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7699999999999999e-06 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879173+0000", 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005846141 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879239+0000", 2024-04-25T15:52:59.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6315999999999993e-05 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879267+0000", 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7137000000000001e-05 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879415+0000", 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000148715 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879693+0000", 2024-04-25T15:52:59.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000278108 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879951+0000", 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025793700000000002 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.896019+0000", 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016067814999999999 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898153+0000", 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021339280000000002 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898411+0000", 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025760799999999999 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898566+0000", 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015581899999999999 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:158 3.12 3:4b703321:::10000000002.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.139669905, 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024089508999999999, 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 158 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010224+0000", 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3969999999999998e-06 2024-04-25T15:52:59.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016098+0000", 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058735000000000002 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016148+0000", 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0640999999999998e-05 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016177+0000", 2024-04-25T15:52:59.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.813e-05 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016313+0000", 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136052 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016470+0000", 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000157637 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016601+0000", 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013075599999999999 2024-04-25T15:52:59.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.023425+0000", 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068237799999999998 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034203+0000", 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01077811 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034260+0000", 2024-04-25T15:52:59.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6702e-05 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034310+0000", 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0803999999999997e-05 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:162 3.2b 3:d691b431:::10000000002.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.218611+0000", 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.9312799809999994, 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02174914, 2024-04-25T15:52:59.225 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 162 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.218611+0000", 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.218611+0000", 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.218615+0000", 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9870000000000001e-06 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224073+0000", 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054577610000000002 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224135+0000", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2496000000000006e-05 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224162+0000", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6800000000000001e-05 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224348+0000", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018565399999999999 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224622+0000", 2024-04-25T15:52:59.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027439199999999998 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.224874+0000", 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025226799999999999 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.232926+0000", 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080514980000000007 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.240158+0000", 2024-04-25T15:52:59.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072328189999999997 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.240264+0000", 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010596599999999999 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.240360+0000", 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5499000000000002e-05 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:165 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.780597083, 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032658418000000002, 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 165 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369298+0000", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3429999999999998e-06 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374706+0000", 2024-04-25T15:52:59.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054080250000000003 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374776+0000", 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9972e-05 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374803+0000", 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7384000000000001e-05 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375010+0000", 2024-04-25T15:52:59.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206431 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375037+0000", 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7206999999999999e-05 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380232+0000", 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051947859999999998 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380474+0000", 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000241864 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380709+0000", 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023560200000000001 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.392962+0000", 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012252219999999999 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401765+0000", 2024-04-25T15:52:59.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088034089999999999 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401866+0000", 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101099 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401952+0000", 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6075999999999994e-05 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:173 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.4405806020000007, 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022867683, 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 173 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.237 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709313+0000", 2024-04-25T15:52:59.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8159999999999998e-06 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716389+0000", 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070763739999999999 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716451+0000", 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1958999999999994e-05 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716476+0000", 2024-04-25T15:52:59.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4246000000000001e-05 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716689+0000", 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000212926 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716951+0000", 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026266700000000001 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.717199+0000", 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024812800000000002 2024-04-25T15:52:59.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.726072+0000", 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088724990000000007 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.731628+0000", 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055558229999999997 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.732003+0000", 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00037564100000000002 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.732178+0000", 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000174604 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:184 3.1b 3:db82751e:::10000000002.000000b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.174460+0000", 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.9754304119999997, 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022559854000000001, 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.242 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 184 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.174460+0000", 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.174460+0000", 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.174464+0000", 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.14e-06 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181591+0000", 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071270270000000002 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181652+0000", 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1177000000000001e-05 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181678+0000", 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6276000000000001e-05 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.181881+0000", 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020344199999999999 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.182292+0000", 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041062299999999999 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.182742+0000", 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044982600000000001 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.193295+0000", 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010553294 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.196703+0000", 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034081409999999999 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.196879+0000", 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017578900000000001 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.197020+0000", 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014111899999999999 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:191 3.39 3:9c9620cb:::10000000002.000000bb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:52:59.247 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.498577+0000", 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.6513135010000006, 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022002436, 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 191 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.498577+0000", 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.498577+0000", 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.498579+0000", 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9539999999999998e-06 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503161+0000", 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045813019999999998 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503203+0000", 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1953999999999998e-05 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503221+0000", 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8457e-05 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503384+0000", 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162867 2024-04-25T15:52:59.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503640+0000", 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025651800000000001 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.503879+0000", 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023874200000000001 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.512692+0000", 2024-04-25T15:52:59.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088129860000000001 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.520260+0000", 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007567898 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.520442+0000", 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181609 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.520580+0000", 2024-04-25T15:52:59.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013814900000000001 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.9321862510000001, 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.8375521990000001, 2024-04-25T15:52:59.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:52:59.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:52:59.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:52:59.263 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:52:59.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:52:59.265 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:59.263+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:59.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.334+0000 7f2e557fa640 1 -- 172.21.15.67:0/3683575199 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v67) ==== 91+0+0 (secure 0 0 0) 0x7f2e50016150 con 0x7f2e600caf60 2024-04-25T15:52:59.336 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:52:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2e4c06f400 msgr2=0x7f2e4c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2e4c06f400 0x7f2e4c071910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c00b930 tx=0x7f2e5c008040 comp rx=0 tx=0).stop 2024-04-25T15:52:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e600caf60 msgr2=0x7f2e6015c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e600caf60 0x7f2e6015c460 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2e5001ac10 tx=0x7f2e5001ada0 comp rx=0 tx=0).stop 2024-04-25T15:52:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 shutdown_connections 2024-04-25T15:52:59.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2e4c06f400 0x7f2e4c071910 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e6015d790 0x7f2e60074330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e600caf60 0x7f2e6015c460 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 --2- 172.21.15.67:0/3683575199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e60073bf0 0x7f2e601f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 >> 172.21.15.67:0/3683575199 conn(0x7f2e60070db0 msgr2=0x7f2e60072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 shutdown_connections 2024-04-25T15:52:59.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:52:59.338+0000 7f2e66d27640 1 -- 172.21.15.67:0/3683575199 wait complete. 2024-04-25T15:52:59.366 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:59.366+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:59.435 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:59.435 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:59.435 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:52:59.435 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:59.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:52:59.467 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:52:59.467+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:52:59.568 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:52:59.566+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:52:59.612 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:52:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:52:59.613 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:52:59.613 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:59.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:52:59.677 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:59.678+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:59.769 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:52:59.770+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4201.0:69 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'47, mlcod=15'40)", 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:59.857 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 159.85368121400001, 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013829869, 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002608+0000", 2024-04-25T15:52:59.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002610+0000", 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.973e-06 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002675+0000", 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5063999999999999e-05 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002699+0000", 2024-04-25T15:52:59.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3618e-05 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.002742+0000", 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004117+0000", 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013750889999999999 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.004159+0000", 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1993000000000003e-05 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016386+0000", 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012226969000000001 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:20.016438+0000", 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1816e-05 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 143.64479357100001, 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:52:59.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:52:59.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:52:59.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 143.64474618599999, 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.867 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:52:59.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:52:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:52:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:28 2.13 2:c91d4a1d:::mds0_inotable:head [writefull 0~34] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:59.872 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 143.64473410400001, 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013577562, 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211555+0000", 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211557+0000", 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5230000000000001e-06 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211563+0000", 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8399999999999997e-06 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212067+0000", 2024-04-25T15:52:59.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00050345900000000005 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212084+0000", 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6552e-05 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214199+0000", 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002115792 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214564+0000", 2024-04-25T15:52:59.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00036446800000000001 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214950+0000", 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038574399999999998 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.222418+0000", 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074680320000000003 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.224860+0000", 2024-04-25T15:52:59.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024423489999999999 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225009+0000", 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014922199999999999 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225133+0000", 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012358099999999999 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 143.644713932, 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:52:59.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:52:59.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.881 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:2 2.f e20/17 2:f1f69726:::400.00000000:head v 20'1, mlcod=0'0)", 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 143.644475594, 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01415665, 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211814+0000", 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211816+0000", 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0839999999999999e-06 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211873+0000", 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7460000000000002e-05 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211899+0000", 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6251999999999999e-05 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211919+0000", 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9743000000000001e-05 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213579+0000", 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016602800000000001 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.213635+0000", 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5488000000000001e-05 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225833+0000", 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012197716000000001 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225970+0000", 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137627 2024-04-25T15:52:59.885 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.040523870000001, 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:52:59.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.749725727, 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:53 3.20 e60/18 3:068d0381:::10000000002.00000031:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.472268243, 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014486163, 2024-04-25T15:52:59.894 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.384021+0000", 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7300000000000002e-07 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389720+0000", 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056985300000000003 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389742+0000", 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1838999999999999e-05 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389758+0000", 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6058000000000001e-05 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389844+0000", 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6293999999999997e-05 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.389866+0000", 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2033000000000001e-05 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398429+0000", 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085631660000000005 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.398507+0000", 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7869999999999998e-05 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:66 3.b e60/21 3:d381a89f:::10000000002.0000003e:head v 60'1, mlcod=0'0)", 2024-04-25T15:52:59.898 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.911075586000001, 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015316048000000001, 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:52:59.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945216+0000", 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9e-06 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948749+0000", 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035338119999999999 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948777+0000", 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7915999999999998e-05 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948796+0000", 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.844e-05 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948895+0000", 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9704000000000006e-05 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948916+0000", 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.107e-05 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960445+0000", 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011528722999999999 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960530+0000", 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4482999999999997e-05 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:152 3.39 e61/18 3:9e24e8b4:::10000000002.00000094:head v 61'2, mlcod=0'0)", 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.119205508, 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016125006000000001, 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737086+0000", 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744295+0000", 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072082320000000002 2024-04-25T15:52:59.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744341+0000", 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6072000000000002e-05 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744376+0000", 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4903000000000001e-05 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744533+0000", 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015736299999999999 2024-04-25T15:52:59.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744590+0000", 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6771000000000002e-05 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753101+0000", 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085115490000000002 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753209+0000", 2024-04-25T15:52:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107352 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:153 3.1 e61/54 3:814e3e07:::10000000002.00000095:head v 61'3, mlcod=59'1)", 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.061948079, 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014546699999999999, 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794341+0000", 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.794344+0000", 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.908 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796364+0000", 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002019968 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796414+0000", 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0065999999999998e-05 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796445+0000", 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0783e-05 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796600+0000", 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000154685 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.796670+0000", 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0605999999999995e-05 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.808781+0000", 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012110368999999999 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.808888+0000", 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107353 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:156 3.3e e61/35 3:7d47d675:::10000000002.00000098:head v 61'1, mlcod=0'0)", 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.922052035, 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016362634000000001, 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934240+0000", 2024-04-25T15:52:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3149999999999999e-06 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939270+0000", 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050303869999999999 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939308+0000", 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8047999999999999e-05 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939335+0000", 2024-04-25T15:52:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7251999999999999e-05 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939432+0000", 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6637000000000003e-05 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939459+0000", 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7651999999999999e-05 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950534+0000", 2024-04-25T15:52:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011074993999999999 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950600+0000", 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5349000000000001e-05 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:158 3.12 e61/40 3:4b703321:::10000000002.0000009a:head v 61'6, mlcod=59'4)", 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.839091908, 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017102098, 2024-04-25T15:52:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017200+0000", 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3199999999999998e-06 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024143+0000", 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069429569999999996 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024169+0000", 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6007e-05 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024190+0000", 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1203999999999999e-05 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024272+0000", 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1833000000000001e-05 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024301+0000", 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9475000000000002e-05 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034243+0000", 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099420589999999996 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034299+0000", 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6242999999999997e-05 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:162 3.2b e61/40 3:d691b431:::10000000002.0000009e:head v 61'3, mlcod=60'1)", 2024-04-25T15:52:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.630747896999999, 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014572351000000001, 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225544+0000", 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7199999999999998e-06 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230207+0000", 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046626719999999997 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230239+0000", 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2548000000000001e-05 2024-04-25T15:52:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230266+0000", 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6599999999999999e-05 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230382+0000", 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115667 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230417+0000", 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5800999999999998e-05 2024-04-25T15:52:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240029+0000", 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096115349999999992 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240114+0000", 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4808000000000005e-05 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:164 3.30 e62/18 3:0efeaece:::10000000002.000000a0:head v 62'5, mlcod=60'3)", 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.518692458, 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013652381, 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.337597+0000", 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8999999999999997e-07 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343547+0000", 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0059493669999999997 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343576+0000", 2024-04-25T15:52:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9850000000000001e-05 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343597+0000", 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0330999999999999e-05 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343687+0000", 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0278999999999994e-05 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.343710+0000", 2024-04-25T15:52:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2796e-05 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.351154+0000", 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074439739999999999 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.351249+0000", 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5293999999999999e-05 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:173 3.2b e62/40 3:d5480d6b:::10000000002.000000a9:head v 62'5, mlcod=61'3)", 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.717902+0000", 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.138387572999999, 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013644187, 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.717902+0000", 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.717902+0000", 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.717904+0000", 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6989999999999999e-06 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722820+0000", 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049151619999999998 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722842+0000", 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2408999999999998e-05 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722855+0000", 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3485999999999999e-05 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722943+0000", 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7981999999999999e-05 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.722976+0000", 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3096999999999998e-05 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.731460+0000", 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084837539999999996 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.731546+0000", 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5598000000000003e-05 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:181 3.1f e62/40 3:f8dc3ee2:::10000000002.000000b1:head v 62'2, mlcod=0'0)", 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8060773020000003, 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015250434, 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:52:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050214+0000", 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.013e-06 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054240+0000", 2024-04-25T15:52:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040256609999999998 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054262+0000", 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2691e-05 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054277+0000", 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4464000000000001e-05 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054365+0000", 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8657000000000002e-05 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054400+0000", 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4505999999999998e-05 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065391+0000", 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991287000000001 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065462+0000", 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1155000000000006e-05 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:249 3.3e e63/35 3:7d99c36c:::10000000002.000000f5:head v 63'3, mlcod=61'1)", 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:53.086243+0000", 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.7700462850000003, 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014234955000000001, 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.086243+0000", 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.086243+0000", 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.086245+0000", 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6130000000000001e-06 2024-04-25T15:52:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088030+0000", 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017854629999999999 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088058+0000", 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8279000000000001e-05 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088075+0000", 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6711999999999999e-05 2024-04-25T15:52:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088156+0000", 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1457000000000003e-05 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.088184+0000", 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7073999999999999e-05 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.100414+0000", 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012230636 2024-04-25T15:52:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:53.100478+0000", 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3720999999999995e-05 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:260 3.12 e66/40 3:4bec3886:::10000000002.00000000:head v 66'8, mlcod=61'6)", 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.157822253, 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019802660999999999, 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:52:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:52:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698470+0000", 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704135+0000", 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056657490000000003 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704159+0000", 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3903000000000001e-05 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:52:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704176+0000", 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6461e-05 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704290+0000", 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011446999999999999 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704336+0000", 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5558e-05 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:52:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718190+0000", 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013854577999999999 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718270+0000", 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9377999999999998e-05 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:52:59.944 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:52:59.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:52:59.946 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:52:59.943+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:00.046 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:00.046+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:00.146 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:00.146+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:00.167 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:00.167 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:00.168 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:00.168 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:00.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:00.247 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:00.247+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:00.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:00.348 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:00.347+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:00.399 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:00.400 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:00.400 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:00.400 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:00.400 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:00.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.462+0000 7fee34ff3640 1 Processor -- start 2024-04-25T15:53:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.462+0000 7fee34ff3640 1 -- start start 2024-04-25T15:53:00.464 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:00.462+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee34ff3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee30068830 0x7fee3006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee34ff3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fee3006cb80 0x7fee30167190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee34ff3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee30068c40 con 0x7fee3015dd00 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee34ff3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fee3015e110 con 0x7fee30068830 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee34ff3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fee30160990 con 0x7fee3006cb80 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2dd9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee30068830 0x7fee3006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2dd9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee30068830 0x7fee3006c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60604/0 (socket says 172.21.15.67:60604) 2024-04-25T15:53:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2dd9b640 1 -- 172.21.15.67:0/2230826107 learned_addr learned my addr 172.21.15.67:0/2230826107 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:00.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/2230826107 >> v2:172.21.15.176:3300/0 conn(0x7fee3006cb80 0x7fee30167190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2dd9b640 1 -- 172.21.15.67:0/2230826107 >> v2:172.21.15.176:3300/0 conn(0x7fee3006cb80 msgr2=0x7fee30167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:00.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/2230826107 >> v2:172.21.15.176:3300/0 conn(0x7fee3006cb80 0x7fee30167190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2dd9b640 1 -- 172.21.15.67:0/2230826107 >> v1:172.21.15.176:6789/0 conn(0x7fee3015dd00 legacy=0x7fee301569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.466+0000 7fee2dd9b640 1 -- 172.21.15.67:0/2230826107 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee30160c70 con 0x7fee30068830 2024-04-25T15:53:00.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/2230826107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30068830 0x7fee3006c640 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fee300cf1e0 tx=0x7fee2401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a82f02bcad8fc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee2d59a640 1 -- 172.21.15.67:0/2230826107 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee240b0070 con 0x7fee30068830 2024-04-25T15:53:00.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee2d59a640 1 -- 172.21.15.67:0/2230826107 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee240ab600 con 0x7fee30068830 2024-04-25T15:53:00.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee2d59a640 1 -- 172.21.15.67:0/2230826107 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee240ab9d0 con 0x7fee30068830 2024-04-25T15:53:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 -- 172.21.15.67:0/2230826107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30068830 msgr2=0x7fee3006c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 --2- 172.21.15.67:0/2230826107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30068830 0x7fee3006c640 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fee300cf1e0 tx=0x7fee2401c560 comp rx=0 tx=0).stop 2024-04-25T15:53:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 -- 172.21.15.67:0/2230826107 shutdown_connections 2024-04-25T15:53:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 --2- 172.21.15.67:0/2230826107 >> v2:172.21.15.176:3300/0 conn(0x7fee3006cb80 0x7fee30167190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 --2- 172.21.15.67:0/2230826107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30068830 0x7fee3006c640 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 -- 172.21.15.67:0/2230826107 >> 172.21.15.67:0/2230826107 conn(0x7fee300ce1c0 msgr2=0x7fee300ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 -- 172.21.15.67:0/2230826107 shutdown_connections 2024-04-25T15:53:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.470+0000 7fee34ff3640 1 -- 172.21.15.67:0/2230826107 wait complete. 2024-04-25T15:53:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 Processor -- start 2024-04-25T15:53:00.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 -- start start 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee3006cb80 0x7fee301f9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee3015dd00 0x7fee300c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30147890 0x7fee30147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fee30074a80 con 0x7fee30147890 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30147890 0x7fee30147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30147890 0x7fee30147ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60616/0 (socket says 172.21.15.67:60616) 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 -- 172.21.15.67:0/1685288154 learned_addr learned my addr 172.21.15.67:0/1685288154 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fee30074900 con 0x7fee3006cb80 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee3006d660 con 0x7fee3015dd00 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee3015dd00 0x7fee300c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee3006cb80 0x7fee301f9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee3015dd00 msgr2=0x7fee300c6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee3015dd00 0x7fee300c6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee3006cb80 msgr2=0x7fee301f9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee3006cb80 0x7fee301f9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.474+0000 7fee2ed9d640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee301f9a90 con 0x7fee30147890 2024-04-25T15:53:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30147890 0x7fee30147ca0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fee2800efc0 tx=0x7fee2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28018020 con 0x7fee30147890 2024-04-25T15:53:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee28014480 con 0x7fee30147890 2024-04-25T15:53:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28009e60 con 0x7fee30147890 2024-04-25T15:53:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee300cb990 con 0x7fee30147890 2024-04-25T15:53:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee300c83f0 con 0x7fee30147890 2024-04-25T15:53:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fee3006a480 con 0x7fee30147890 2024-04-25T15:53:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fee2801b020 con 0x7fee30147890 2024-04-25T15:53:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee1effd640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1006f400 0x7fee10071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5915+0+0 (secure 0 0 0) 0x7fee2808feb0 con 0x7fee30147890 2024-04-25T15:53:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v67) ==== 54+0+16 (secure 0 0 0) 0x7fee28022030 con 0x7fee30147890 2024-04-25T15:53:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1006f400 0x7fee10071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.478+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fee301f9c60 0x7fee300ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.482+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee300c7ad0 con 0x7fee301f9c60 2024-04-25T15:53:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.482+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fee301f9c60 0x7fee300ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.482+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1006f400 0x7fee10071910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee20001770 tx=0x7fee20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.482+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fee301f9c60 0x7fee300ca900 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.482+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee300c7ad0 con 0x7fee301f9c60 2024-04-25T15:53:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.502+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee3014a760 con 0x7fee301f9c60 2024-04-25T15:53:00.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.502+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee3014a760 con 0x7fee301f9c60 2024-04-25T15:53:00.505 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:00.505 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:00.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.506+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fee300c55c0 0x7fee300c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.506+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fee30155280 con 0x7fee300c55c0 2024-04-25T15:53:00.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.506+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fee300c55c0 0x7fee300c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.506+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fee300c55c0 0x7fee300c59d0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.506+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee30155280 con 0x7fee300c55c0 2024-04-25T15:53:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.526+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee301481e0 con 0x7fee300c55c0 2024-04-25T15:53:00.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.526+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee301481e0 con 0x7fee300c55c0 2024-04-25T15:53:00.529 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:00.530 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:00.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.530+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fee30063090 0x7fee300634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.530+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fee300c6420 con 0x7fee30063090 2024-04-25T15:53:00.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.530+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fee30063090 0x7fee300634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.530+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fee30063090 0x7fee300634a0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.530+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee300c6420 con 0x7fee30063090 2024-04-25T15:53:00.549 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:00.547+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:00.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.550+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee300cb4c0 con 0x7fee30063090 2024-04-25T15:53:00.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.550+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee300cb4c0 con 0x7fee30063090 2024-04-25T15:53:00.552 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:00.552 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:00.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.550+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fee30063da0 0x7fee300cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.550+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fee300ccf30 con 0x7fee30063da0 2024-04-25T15:53:00.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.550+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fee30063da0 0x7fee300cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.550+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fee30063da0 0x7fee300cc9f0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.554+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee300ccf30 con 0x7fee30063da0 2024-04-25T15:53:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.570+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee300cd440 con 0x7fee30063da0 2024-04-25T15:53:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.570+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee300cd440 con 0x7fee30063da0 2024-04-25T15:53:00.573 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:00.573 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.574+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fee3002cef0 0x7fee3002d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.574+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fee300cd3c0 con 0x7fee3002cef0 2024-04-25T15:53:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.574+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fee3002cef0 0x7fee3002d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.574+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fee3002cef0 0x7fee3002d300 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.574+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee300cd3c0 con 0x7fee3002cef0 2024-04-25T15:53:00.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.590+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee3002d840 con 0x7fee3002cef0 2024-04-25T15:53:00.593 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:00.594 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:00.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.594+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee3002d840 con 0x7fee3002cef0 2024-04-25T15:53:00.597 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:00.598 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:00.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.598+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fee3020a310 0x7fee3020c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.598+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fee3020f500 con 0x7fee3020a310 2024-04-25T15:53:00.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.598+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fee3020a310 0x7fee3020c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.598+0000 7fee2dd9b640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fee3020a310 0x7fee3020c730 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.598+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee3020f500 con 0x7fee3020a310 2024-04-25T15:53:00.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:00.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.618+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee3020cc70 con 0x7fee3020a310 2024-04-25T15:53:00.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.622+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee3020cc70 con 0x7fee3020a310 2024-04-25T15:53:00.622 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.622+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fee3020d2f0 0x7fee3020f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.622+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fee30212710 con 0x7fee3020d2f0 2024-04-25T15:53:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.622+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fee3020d2f0 0x7fee3020f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.622+0000 7fee2ed9d640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fee3020d2f0 0x7fee3020f710 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.626+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee30212710 con 0x7fee3020d2f0 2024-04-25T15:53:00.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.646+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee30211430 con 0x7fee3020d2f0 2024-04-25T15:53:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.650+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee30211430 con 0x7fee3020d2f0 2024-04-25T15:53:00.650 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.650+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fee30211b50 0x7fee30211f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.650+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fee30215ae0 con 0x7fee30211b50 2024-04-25T15:53:00.651 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:00.650+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.650+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fee30211b50 0x7fee30211f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.650+0000 7fee2e59c640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fee30211b50 0x7fee30211f60 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:00.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.654+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee30215ae0 con 0x7fee30211b50 2024-04-25T15:53:00.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.670+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee30214960 con 0x7fee30211b50 2024-04-25T15:53:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.674+0000 7fee1effd640 1 -- 172.21.15.67:0/1685288154 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee30214960 con 0x7fee30211b50 2024-04-25T15:53:00.677 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:00.678 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fee301f9c60 msgr2=0x7fee300ca900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fee301f9c60 0x7fee300ca900 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fee300c55c0 msgr2=0x7fee300c59d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fee300c55c0 0x7fee300c59d0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fee30063090 msgr2=0x7fee300634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fee30063090 0x7fee300634a0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fee30063da0 msgr2=0x7fee300cc9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fee30063da0 0x7fee300cc9f0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fee3002cef0 msgr2=0x7fee3002d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fee3002cef0 0x7fee3002d300 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee2ed9d640 1 -- 172.21.15.67:0/1685288154 reap_dead start 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee2ed9d640 1 -- 172.21.15.67:0/1685288154 reap_dead start 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fee3020a310 msgr2=0x7fee3020c730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fee3020a310 0x7fee3020c730 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fee3020d2f0 msgr2=0x7fee3020f710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fee3020d2f0 0x7fee3020f710 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fee30211b50 msgr2=0x7fee30211f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fee30211b50 0x7fee30211f60 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1006f400 msgr2=0x7fee10071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1006f400 0x7fee10071910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee20001770 tx=0x7fee20009040 comp rx=0 tx=0).stop 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30147890 msgr2=0x7fee30147ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 --2- 172.21.15.67:0/1685288154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee30147890 0x7fee30147ca0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fee2800efc0 tx=0x7fee2800c490 comp rx=0 tx=0).stop 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee2ed9d640 1 -- 172.21.15.67:0/1685288154 reap_dead start 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 shutdown_connections 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 >> 172.21.15.67:0/1685288154 conn(0x7fee300ce1c0 msgr2=0x7fee30151b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 shutdown_connections 2024-04-25T15:53:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:00.678+0000 7fee34ff3640 1 -- 172.21.15.67:0/1685288154 wait complete. 2024-04-25T15:53:00.751 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:00.750+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:00.783 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:00.783 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:00.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:00.853 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:00.851+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:00.953 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:00.954+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:01.009 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:01.009 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:01.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:01.077 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:01.078+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:01.155 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:01.154+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:01.244 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:01.244 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:01.257 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:01.255+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:01.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:01.342 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:01.342 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:01.342 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.295403250210301 2024-04-25T15:53:01.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.295403250210301 2024-04-25T15:53:01.358 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:01.355+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:01.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:53:01.459 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:01.458+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:01.480 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:01.480 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:01.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:01.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f9395cbb640 1 Processor -- start 2024-04-25T15:53:01.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f9395cbb640 1 -- start start 2024-04-25T15:53:01.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f9395cbb640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f93900c92d0 0x7f9390072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f9395cbb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f939006d8e0 con 0x7f93900c96e0 2024-04-25T15:53:01.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f9395cbb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9390073c70 con 0x7f9390160800 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f9395cbb640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f939006dbc0 con 0x7f93900c92d0 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f93900c96e0 0x7f9390167180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42518/0 (socket says 172.21.15.67:42518) 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938ffff640 1 -- 172.21.15.67:0/3253166331 learned_addr learned my addr 172.21.15.67:0/3253166331 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938effd640 1 --2- 172.21.15.67:0/3253166331 >> v2:172.21.15.176:3300/0 conn(0x7f93900c92d0 0x7f9390072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2451500514 0 0) 0x7f939006d8e0 con 0x7f93900c96e0 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9378003530 con 0x7f93900c96e0 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 26165405 0 0) 0x7f9378003530 con 0x7f93900c96e0 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 >> v2:172.21.15.176:3300/0 conn(0x7f93900c92d0 msgr2=0x7f9390072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 --2- 172.21.15.67:0/3253166331 >> v2:172.21.15.176:3300/0 conn(0x7f93900c92d0 0x7f9390072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 >> v1:172.21.15.176:6789/0 conn(0x7f9390160800 legacy=0x7f939006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9390073f50 con 0x7f93900c96e0 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f9388001a00 con 0x7f93900c96e0 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 >> v1:172.21.15.67:6789/0 conn(0x7f93900c96e0 legacy=0x7f9390167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93780039c0 0x7f9378003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9378004810 0x7f9378004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9378005260 0x7f937800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f939006dbc0 con 0x7f93780039c0 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f937800c1f0 con 0x7f9378005260 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f937800c3f0 con 0x7f9378004810 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938f7fe640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93780039c0 0x7f9378003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938ffff640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9378005260 0x7f937800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938effd640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9378004810 0x7f9378004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938f7fe640 1 -- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9378004810 msgr2=0x7f9378004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938f7fe640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9378004810 0x7f9378004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938f7fe640 1 -- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9378005260 msgr2=0x7f937800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938f7fe640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9378005260 0x7f937800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938f7fe640 1 -- 172.21.15.67:0/3253166331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9378003530 con 0x7f93780039c0 2024-04-25T15:53:01.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938ffff640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9378005260 0x7f937800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:01.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938ffff640 1 -- 172.21.15.67:0/3253166331 reap_dead start 2024-04-25T15:53:01.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.538+0000 7f938effd640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9378004810 0x7f9378004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f938f7fe640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93780039c0 0x7f9378003e30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9380001780 tx=0x7f938001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5555fa782a072cd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93800013a0 con 0x7f93780039c0 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93800052f0 con 0x7f93780039c0 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f938e7fc640 1 -- 172.21.15.67:0/3253166331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380003840 con 0x7f93780039c0 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f9395cbb640 1 -- 172.21.15.67:0/3253166331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93780039c0 msgr2=0x7f9378003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f9395cbb640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93780039c0 0x7f9378003e30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9380001780 tx=0x7f938001c400 comp rx=0 tx=0).stop 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f9395cbb640 1 -- 172.21.15.67:0/3253166331 shutdown_connections 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f9395cbb640 1 --2- 172.21.15.67:0/3253166331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93780039c0 0x7f9378003e30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f9395cbb640 1 -- 172.21.15.67:0/3253166331 >> 172.21.15.67:0/3253166331 conn(0x7f9390070db0 msgr2=0x7f93900ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f9395cbb640 1 -- 172.21.15.67:0/3253166331 shutdown_connections 2024-04-25T15:53:01.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.542+0000 7f9395cbb640 1 -- 172.21.15.67:0/3253166331 wait complete. 2024-04-25T15:53:01.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 Processor -- start 2024-04-25T15:53:01.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 -- start start 2024-04-25T15:53:01.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93780039c0 0x7f93901576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93901f5ab0 0x7f939014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939015a6f0 0x7f93901520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9390160d90 con 0x7f93901f5ab0 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9390160c10 con 0x7f939015a6f0 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f9395cbb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9390072dc0 con 0x7f93780039c0 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f938effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93901f5ab0 0x7f939014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f938effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93901f5ab0 0x7f939014f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60640/0 (socket says 172.21.15.67:60640) 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.546+0000 7f938effd640 1 -- 172.21.15.67:0/2433605544 learned_addr learned my addr 172.21.15.67:0/2433605544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938f7fe640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93780039c0 0x7f93901576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938ffff640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939015a6f0 0x7f93901520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938effd640 1 -- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93780039c0 msgr2=0x7f93901576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938effd640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93780039c0 0x7f93901576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938effd640 1 -- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939015a6f0 msgr2=0x7f93901520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938effd640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939015a6f0 0x7f93901520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938effd640 1 -- 172.21.15.67:0/2433605544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93901598f0 con 0x7f93901f5ab0 2024-04-25T15:53:01.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938ffff640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939015a6f0 0x7f93901520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:01.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f938effd640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93901f5ab0 0x7f939014f5f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9384010790 tx=0x7f9384010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:01.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f9377fff640 1 -- 172.21.15.67:0/2433605544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938400f040 con 0x7f93901f5ab0 2024-04-25T15:53:01.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f9377fff640 1 -- 172.21.15.67:0/2433605544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9384004220 con 0x7f93901f5ab0 2024-04-25T15:53:01.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9390159bd0 con 0x7f93901f5ab0 2024-04-25T15:53:01.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939014fb30 con 0x7f93901f5ab0 2024-04-25T15:53:01.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f939006a480 con 0x7f93901f5ab0 2024-04-25T15:53:01.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.550+0000 7f9377fff640 1 -- 172.21.15.67:0/2433605544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384010ea0 con 0x7f93901f5ab0 2024-04-25T15:53:01.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.554+0000 7f9377fff640 1 -- 172.21.15.67:0/2433605544 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f938400e960 con 0x7f93901f5ab0 2024-04-25T15:53:01.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.554+0000 7f9377fff640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f937c06f400 0x7f937c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.554+0000 7f9377fff640 1 -- 172.21.15.67:0/2433605544 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5915+0+0 (secure 0 0 0) 0x7f938408f9e0 con 0x7f93901f5ab0 2024-04-25T15:53:01.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.554+0000 7f938f7fe640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f937c06f400 0x7f937c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.554+0000 7f938f7fe640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f937c06f400 0x7f937c071910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9380001750 tx=0x7f93800011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.558+0000 7f9377fff640 1 -- 172.21.15.67:0/2433605544 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9384018030 con 0x7f93901f5ab0 2024-04-25T15:53:01.560 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:01.559+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:01.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef90354640 1 Processor -- start 2024-04-25T15:53:01.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef90354640 1 -- start start 2024-04-25T15:53:01.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef90354640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8806d460 0x7fef880736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef90354640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef880caa60 con 0x7fef881547e0 2024-04-25T15:53:01.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef90354640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef8815a5f0 con 0x7fef8815aaf0 2024-04-25T15:53:01.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef90354640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fef88068590 con 0x7fef8806d460 2024-04-25T15:53:01.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d8ef640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef881547e0 0x7fef8806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42520/0 (socket says 172.21.15.67:42520) 2024-04-25T15:53:01.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d8ef640 1 -- 172.21.15.67:0/970197055 learned_addr learned my addr 172.21.15.67:0/970197055 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:01.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8e0f0640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8806d460 0x7fef880736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1212821474 0 0) 0x7fef880caa60 con 0x7fef881547e0 2024-04-25T15:53:01.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef74003530 con 0x7fef881547e0 2024-04-25T15:53:01.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 404779330 0 0) 0x7fef74003530 con 0x7fef881547e0 2024-04-25T15:53:01.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8806d460 msgr2=0x7fef880736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:01.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8806d460 0x7fef880736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 >> v1:172.21.15.176:6789/0 conn(0x7fef8815aaf0 legacy=0x7fef8815af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef880ca100 con 0x7fef881547e0 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fef780016d0 con 0x7fef881547e0 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 >> v1:172.21.15.67:6789/0 conn(0x7fef881547e0 legacy=0x7fef8806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef740039f0 0x7fef74003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef740043a0 0x7fef740048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef740050d0 0x7fef7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef88068590 con 0x7fef740043a0 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fef7400c040 con 0x7fef740050d0 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.582+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fef7400c240 con 0x7fef740039f0 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d8ef640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef740043a0 0x7fef740048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8e8f1640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef740050d0 0x7fef7400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d8ef640 1 -- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef740039f0 msgr2=0x7fef74003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d8ef640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef740039f0 0x7fef74003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d8ef640 1 -- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef740050d0 msgr2=0x7fef7400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:01.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d8ef640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef740050d0 0x7fef7400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d8ef640 1 -- 172.21.15.67:0/970197055 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef74003530 con 0x7fef740043a0 2024-04-25T15:53:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d8ef640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef740043a0 0x7fef740048d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fef7800c160 tx=0x7fef7801cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf693148c8db637 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:01.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78007cb0 con 0x7fef740043a0 2024-04-25T15:53:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef780a9070 con 0x7fef740043a0 2024-04-25T15:53:01.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.586+0000 7fef8d0ee640 1 -- 172.21.15.67:0/970197055 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78005d60 con 0x7fef740043a0 2024-04-25T15:53:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 -- 172.21.15.67:0/970197055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef740043a0 msgr2=0x7fef740048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 --2- 172.21.15.67:0/970197055 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef740043a0 0x7fef740048d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fef7800c160 tx=0x7fef7801cbd0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef8e8f1640 1 -- 172.21.15.67:0/970197055 reap_dead start 2024-04-25T15:53:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 -- 172.21.15.67:0/970197055 shutdown_connections 2024-04-25T15:53:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 -- 172.21.15.67:0/970197055 >> 172.21.15.67:0/970197055 conn(0x7fef880ce1c0 msgr2=0x7fef880ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 -- 172.21.15.67:0/970197055 shutdown_connections 2024-04-25T15:53:01.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 -- 172.21.15.67:0/970197055 wait complete. 2024-04-25T15:53:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 Processor -- start 2024-04-25T15:53:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.590+0000 7fef90354640 1 -- start start 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef90354640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef880c31e0 0x7fef8814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef90354640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef880c4cb0 0x7fef88152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef90354640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8814cae0 0x7fef8814cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef90354640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef88154d70 con 0x7fef880c4cb0 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef90354640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fef88154bf0 con 0x7fef880c31e0 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef90354640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fef8806cd50 con 0x7fef8814cae0 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef880c4cb0 0x7fef88152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef880c4cb0 0x7fef88152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60660/0 (socket says 172.21.15.67:60660) 2024-04-25T15:53:01.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8e0f0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef880c31e0 0x7fef8814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 -- 172.21.15.67:0/3393758718 learned_addr learned my addr 172.21.15.67:0/3393758718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8e8f1640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8814cae0 0x7fef8814cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 -- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8814cae0 msgr2=0x7fef8814cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8814cae0 0x7fef8814cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 -- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef880c31e0 msgr2=0x7fef8814e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef880c31e0 0x7fef8814e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:01.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 -- 172.21.15.67:0/3393758718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8814fda0 con 0x7fef880c4cb0 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8e0f0640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef880c31e0 0x7fef8814e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.594+0000 7fef8d8ef640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef880c4cb0 0x7fef88152080 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fef7801d470 tx=0x7fef74005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.598+0000 7fef7effd640 1 -- 172.21.15.67:0/3393758718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78007cb0 con 0x7fef880c4cb0 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.598+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8814dc80 con 0x7fef880c4cb0 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.598+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef880c44b0 con 0x7fef880c4cb0 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.598+0000 7fef7effd640 1 -- 172.21.15.67:0/3393758718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef780047f0 con 0x7fef880c4cb0 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.598+0000 7fef7effd640 1 -- 172.21.15.67:0/3393758718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78004ba0 con 0x7fef880c4cb0 2024-04-25T15:53:01.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.598+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef700052a0 con 0x7fef880c4cb0 2024-04-25T15:53:01.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.598+0000 7fef7effd640 1 -- 172.21.15.67:0/3393758718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fef78007070 con 0x7fef880c4cb0 2024-04-25T15:53:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.602+0000 7fef7effd640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef6c06f400 0x7fef6c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.602+0000 7fef7effd640 1 -- 172.21.15.67:0/3393758718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5915+0+0 (secure 0 0 0) 0x7fef78128910 con 0x7fef880c4cb0 2024-04-25T15:53:01.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.602+0000 7fef8e0f0640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef6c06f400 0x7fef6c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:01.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.602+0000 7fef8e0f0640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef6c06f400 0x7fef6c071910 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef880b0890 tx=0x7fef84009410 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:01.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.606+0000 7fef7effd640 1 -- 172.21.15.67:0/3393758718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fef780bd030 con 0x7fef880c4cb0 2024-04-25T15:53:01.661 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:01.659+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:01.703 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:01.703 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:01.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:01.765 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:01.766+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:01.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.774+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 0.295403250210301} v 0) -- 0x7f9390073920 con 0x7f93901f5ab0 2024-04-25T15:53:01.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:01.822+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fef70005860 con 0x7fef880c4cb0 2024-04-25T15:53:01.862 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:01.862+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:01.927 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:01.927 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:01.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:01.989 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:01.990+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:02.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:02.062+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:02.109 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:02.109 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:02.126 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:02.165 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:02.162+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:02.265 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:02.266+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:02.279 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:02.279 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:02.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:02.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9377fff640 1 -- 172.21.15.67:0/2433605544 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 0.295403250210301}]=0 reweighted osd.1 to 0.295403 (4b9f) v68) ==== 133+0+0 (secure 0 0 0) 0x7f938405c320 con 0x7f93901f5ab0 2024-04-25T15:53:02.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef7effd640 1 -- 172.21.15.67:0/3393758718 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v68) ==== 101+0+0 (secure 0 0 0) 0x7fef78016910 con 0x7fef880c4cb0 2024-04-25T15:53:02.342 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 0.295403 (4b9f) 2024-04-25T15:53:02.343 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:53:02.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f937c06f400 msgr2=0x7f937c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f937c06f400 0x7f937c071910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9380001750 tx=0x7f93800011a0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93901f5ab0 msgr2=0x7f939014f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93901f5ab0 0x7f939014f5f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9384010790 tx=0x7f9384010c50 comp rx=0 tx=0).stop 2024-04-25T15:53:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef6c06f400 msgr2=0x7fef6c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef6c06f400 0x7fef6c071910 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef880b0890 tx=0x7fef84009410 comp rx=0 tx=0).stop 2024-04-25T15:53:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef880c4cb0 msgr2=0x7fef88152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef880c4cb0 0x7fef88152080 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fef7801d470 tx=0x7fef74005840 comp rx=0 tx=0).stop 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 shutdown_connections 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f937c06f400 0x7f937c071910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f939015a6f0 0x7f93901520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93901f5ab0 0x7f939014f5f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 --2- 172.21.15.67:0/2433605544 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93780039c0 0x7f93901576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 >> 172.21.15.67:0/2433605544 conn(0x7f9390070db0 msgr2=0x7f9390151080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 shutdown_connections 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 shutdown_connections 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef6c06f400 0x7fef6c071910 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef8814cae0 0x7fef8814cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef880c4cb0 0x7fef88152080 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 --2- 172.21.15.67:0/3393758718 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef880c31e0 0x7fef8814e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7f9395cbb640 1 -- 172.21.15.67:0/2433605544 wait complete. 2024-04-25T15:53:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 >> 172.21.15.67:0/3393758718 conn(0x7fef880ce1c0 msgr2=0x7fef880ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 shutdown_connections 2024-04-25T15:53:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.342+0000 7fef90354640 1 -- 172.21.15.67:0/3393758718 wait complete. 2024-04-25T15:53:02.366 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:02.366+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:02.459 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:02.459 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:02.467 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:02.463+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:02.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:02.567 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:02.566+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:02.668 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:02.667+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:02.673 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:02.673 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:02.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:02.701 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:02.769 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:02.766+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:02.869 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:02.870+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:02.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.874+0000 7f22ba391640 1 Processor -- start 2024-04-25T15:53:02.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.874+0000 7f22ba391640 1 -- start start 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.874+0000 7f22ba391640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f22b415aaf0 0x7f22b415af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.874+0000 7f22ba391640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b40caa60 con 0x7f22b406d460 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.874+0000 7f22ba391640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b415a5f0 con 0x7f22b40736d0 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.874+0000 7f22ba391640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f22b4068590 con 0x7f22b415aaf0 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f22b406d460 0x7f22b41547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40094/0 (socket says 172.21.15.67:40094) 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 -- 172.21.15.67:0/1397757003 learned_addr learned my addr 172.21.15.67:0/1397757003 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b892e640 1 --2- 172.21.15.67:0/1397757003 >> v2:172.21.15.176:3300/0 conn(0x7f22b415aaf0 0x7f22b415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998620708 0 0) 0x7f22b40caa60 con 0x7f22b406d460 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f229c003530 con 0x7f22b406d460 2024-04-25T15:53:02.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2665267446 0 0) 0x7f229c003530 con 0x7f22b406d460 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 >> v2:172.21.15.176:3300/0 conn(0x7f22b415aaf0 msgr2=0x7f22b415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 --2- 172.21.15.67:0/1397757003 >> v2:172.21.15.176:3300/0 conn(0x7f22b415aaf0 0x7f22b415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 >> v1:172.21.15.176:6789/0 conn(0x7f22b40736d0 legacy=0x7f22b406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b40ca100 con 0x7f22b406d460 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f22a4001680 con 0x7f22b406d460 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 >> v1:172.21.15.67:6789/0 conn(0x7f22b406d460 legacy=0x7f22b41547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f229c0039c0 0x7f229c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f229c004810 0x7f229c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0051d0 0x7f229c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f22b4068590 con 0x7f229c0039c0 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f229c00c170 con 0x7f229c0051d0 2024-04-25T15:53:02.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f229c00c370 con 0x7f229c004810 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f229c0039c0 0x7f229c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b37fe640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f229c004810 0x7f229c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 -- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f229c004810 msgr2=0x7f229c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b892e640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0051d0 0x7f229c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f229c004810 0x7f229c004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 -- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0051d0 msgr2=0x7f229c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0051d0 0x7f229c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 -- 172.21.15.67:0/1397757003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229c003530 con 0x7f229c0039c0 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b892e640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0051d0 0x7f229c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:02.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b892e640 1 -- 172.21.15.67:0/1397757003 reap_dead start 2024-04-25T15:53:02.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b3fff640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f229c0039c0 0x7f229c003e30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f22a400c210 tx=0x7f22a401cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e933e3e276a08851 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4007d20 con 0x7f229c0039c0 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a40a9070 con 0x7f229c0039c0 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.878+0000 7f22b2ffd640 1 -- 172.21.15.67:0/1397757003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4005d80 con 0x7f229c0039c0 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 -- 172.21.15.67:0/1397757003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f229c0039c0 msgr2=0x7f229c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f229c0039c0 0x7f229c003e30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f22a400c210 tx=0x7f22a401cb60 comp rx=0 tx=0).stop 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 -- 172.21.15.67:0/1397757003 shutdown_connections 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 --2- 172.21.15.67:0/1397757003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f229c0039c0 0x7f229c003e30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 -- 172.21.15.67:0/1397757003 >> 172.21.15.67:0/1397757003 conn(0x7f22b40ce1c0 msgr2=0x7f22b40ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 -- 172.21.15.67:0/1397757003 shutdown_connections 2024-04-25T15:53:02.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 -- 172.21.15.67:0/1397757003 wait complete. 2024-04-25T15:53:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 Processor -- start 2024-04-25T15:53:02.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.882+0000 7f22ba391640 1 -- start start 2024-04-25T15:53:02.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0039c0 0x7f22b415d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b41f4090 0x7f22b40b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22b4152ee0 0x7f22b4151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f22b4154bf0 con 0x7f22b41f4090 2024-04-25T15:53:02.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f22b406ce50 con 0x7f229c0039c0 2024-04-25T15:53:02.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f22b4154d70 con 0x7f22b4152ee0 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b41f4090 0x7f22b40b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b41f4090 0x7f22b40b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49186/0 (socket says 172.21.15.67:49186) 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 -- 172.21.15.67:0/2819520983 learned_addr learned my addr 172.21.15.67:0/2819520983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0039c0 0x7f22b415d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b892e640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22b4152ee0 0x7f22b4151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22b4152ee0 msgr2=0x7f22b4151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22b4152ee0 0x7f22b4151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0039c0 msgr2=0x7f22b415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f229c0039c0 0x7f22b415d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b40c5920 con 0x7f22b41f4090 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b37fe640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b41f4090 0x7f22b40b07f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f22a800d4d0 tx=0x7f22a800d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8016070 con 0x7f22b41f4090 2024-04-25T15:53:02.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a8012800 con 0x7f22b41f4090 2024-04-25T15:53:02.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b40c5c00 con 0x7f22b41f4090 2024-04-25T15:53:02.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8012bd0 con 0x7f22b41f4090 2024-04-25T15:53:02.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b40c61c0 con 0x7f22b41f4090 2024-04-25T15:53:02.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f22b406a480 con 0x7f22b41f4090 2024-04-25T15:53:02.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f22a8019070 con 0x7f22b41f4090 2024-04-25T15:53:02.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b0ff9640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22a006f400 0x7f22a0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.886+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5886+0+0 (secure 0 0 0) 0x7f22a808f600 con 0x7f22b41f4090 2024-04-25T15:53:02.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v68) ==== 54+0+16 (secure 0 0 0) 0x7f22a808fa90 con 0x7f22b41f4090 2024-04-25T15:53:02.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22a006f400 0x7f22a0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f22b41532f0 0x7f22b40cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22b40c5680 con 0x7f22b41532f0 2024-04-25T15:53:02.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22b892e640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f22b41532f0 0x7f22b40cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22a006f400 0x7f22a0071910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f22a40ab000 tx=0x7f22a40b0040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22b892e640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f22b41532f0 0x7f22b40cc890 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.891 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:02.891 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:02.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.890+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b40c5680 con 0x7f22b41532f0 2024-04-25T15:53:02.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.910+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b415a040 con 0x7f22b41532f0 2024-04-25T15:53:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.910+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b415a040 con 0x7f22b41532f0 2024-04-25T15:53:02.912 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:02.913 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:02.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.910+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f22b41f76e0 0x7f22b41f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.910+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f22b41f8030 con 0x7f22b41f76e0 2024-04-25T15:53:02.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.910+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f22b41f76e0 0x7f22b41f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.914+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f22b41f76e0 0x7f22b41f7af0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.914+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b41f8030 con 0x7f22b41f76e0 2024-04-25T15:53:02.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.934+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b40b3290 con 0x7f22b41f76e0 2024-04-25T15:53:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.938+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b40b3290 con 0x7f22b41f76e0 2024-04-25T15:53:02.938 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.938+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f22b40635a0 0x7f22b40639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.938+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f22b40b4530 con 0x7f22b40635a0 2024-04-25T15:53:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.938+0000 7f22b37fe640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f22b40635a0 0x7f22b40639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.938+0000 7f22b37fe640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f22b40635a0 0x7f22b40639b0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.938+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b40b4530 con 0x7f22b40635a0 2024-04-25T15:53:02.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.954+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b40b4530 con 0x7f22b40635a0 2024-04-25T15:53:02.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.958+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b40b4530 con 0x7f22b40635a0 2024-04-25T15:53:02.959 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.958+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f22b40b1910 0x7f22b40b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.958+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f22b40b4530 con 0x7f22b40b1910 2024-04-25T15:53:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.958+0000 7f22b892e640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f22b40b1910 0x7f22b40b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.958+0000 7f22b892e640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f22b40b1910 0x7f22b40b1d20 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.958+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b40b4530 con 0x7f22b40b1910 2024-04-25T15:53:02.970 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:02.970+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:02.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.982+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b40b4530 con 0x7f22b40b1910 2024-04-25T15:53:02.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.982+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b40b4530 con 0x7f22b40b1910 2024-04-25T15:53:02.985 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:02.985 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:02.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.982+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f22b402d2b0 0x7f22b402d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:02.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.982+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f22b40b4530 con 0x7f22b402d2b0 2024-04-25T15:53:02.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.986+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f22b402d2b0 0x7f22b402d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:02.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.986+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f22b402d2b0 0x7f22b402d6c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:02.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:02.986+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b40b4530 con 0x7f22b402d2b0 2024-04-25T15:53:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.006+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b40b4530 con 0x7f22b402d2b0 2024-04-25T15:53:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.010+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b40b4530 con 0x7f22b402d2b0 2024-04-25T15:53:03.013 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.010+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f22b402dc00 0x7f22b420aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.010+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f22b420d8a0 con 0x7f22b402dc00 2024-04-25T15:53:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.014+0000 7f22b37fe640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f22b402dc00 0x7f22b420aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:03.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.014+0000 7f22b37fe640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f22b402dc00 0x7f22b420aad0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.014+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b420d8a0 con 0x7f22b402dc00 2024-04-25T15:53:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.026+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b420b010 con 0x7f22b402dc00 2024-04-25T15:53:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.030+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b420b010 con 0x7f22b402dc00 2024-04-25T15:53:03.032 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.030+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f22b420b730 0x7f22b420db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.030+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f22b4210bd0 con 0x7f22b420b730 2024-04-25T15:53:03.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.030+0000 7f22b892e640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f22b420b730 0x7f22b420db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.034+0000 7f22b892e640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f22b420b730 0x7f22b420db90 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.034+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b4210bd0 con 0x7f22b420b730 2024-04-25T15:53:03.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.054+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b4210bd0 con 0x7f22b420b730 2024-04-25T15:53:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.058+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b4210bd0 con 0x7f22b420b730 2024-04-25T15:53:03.061 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.058+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f22b420fee0 0x7f22b42102f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.058+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f22b4213fa0 con 0x7f22b420fee0 2024-04-25T15:53:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.058+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f22b420fee0 0x7f22b42102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.062+0000 7f22b3fff640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f22b420fee0 0x7f22b42102f0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:03.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.062+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b4213fa0 con 0x7f22b420fee0 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:42 3.2f 3:f5f5d160:::10000000002.00000026:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.853971+0000", 2024-04-25T15:53:03.065 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.209530843, 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024061375999999999, 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 42 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.853971+0000", 2024-04-25T15:53:03.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.853971+0000", 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.853974+0000", 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4300000000000002e-06 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.859365+0000", 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053908910000000001 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.859397+0000", 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1334999999999997e-05 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.859410+0000", 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3585e-05 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.859596+0000", 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018555500000000001 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.860004+0000", 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000408379 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.860478+0000", 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00047381300000000001 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.873318+0000", 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012839972 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.877833+0000", 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045148380000000002 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.877938+0000", 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010511299999999999 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.878032+0000", 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4464999999999998e-05 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:46 3.37 e59/21 3:ec154225:::10000000002.0000002a:head v 59'1, mlcod=0'0)", 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.047872+0000", 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.015630388000002, 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021519554, 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.071 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.047872+0000", 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.047872+0000", 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.047874+0000", 2024-04-25T15:53:03.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3300000000000001e-06 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.053713+0000", 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058395829999999998 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.053753+0000", 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9730000000000001e-05 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.053778+0000", 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4493000000000001e-05 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.053960+0000", 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018199500000000001 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.054031+0000", 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1582999999999995e-05 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.069244+0000", 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015212335 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.069391+0000", 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147505 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:58 3.2f 3:f7c72662:::10000000002.00000036:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.595553+0000", 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.467949262000001, 2024-04-25T15:53:03.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023089618999999999, 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 58 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.595553+0000", 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.595553+0000", 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.595556+0000", 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2499999999999998e-06 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.599667+0000", 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004110842 2024-04-25T15:53:03.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.599722+0000", 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4910000000000001e-05 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.599751+0000", 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9082e-05 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.599914+0000", 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000163363 2024-04-25T15:53:03.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.600197+0000", 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028306400000000003 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.600449+0000", 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025230100000000002 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.608115+0000", 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007665959 2024-04-25T15:53:03.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.618452+0000", 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010336913 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.618556+0000", 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103639 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.618642+0000", 2024-04-25T15:53:03.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6296000000000002e-05 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:60 3.1c 3:394f8a26:::10000000002.00000038:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.673603+0000", 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.389898777999999, 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024555535999999999, 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.081 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 60 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.673603+0000", 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.673603+0000", 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.673605+0000", 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2000000000000001e-06 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.681065+0000", 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074594409999999998 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.681117+0000", 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2599000000000002e-05 2024-04-25T15:53:03.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.681144+0000", 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6427e-05 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.681317+0000", 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017289200000000001 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.681616+0000", 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029888400000000002 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.681878+0000", 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000262501 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.693691+0000", 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011813403 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.697963+0000", 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042714709999999998 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.698067+0000", 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103872 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.698159+0000", 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1846000000000001e-05 2024-04-25T15:53:03.086 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:63 3.c 3:31d8a57c:::10000000002.0000003b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.794030+0000", 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.269471944999999, 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021440168999999999, 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.087 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 63 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.794030+0000", 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.794030+0000", 2024-04-25T15:53:03.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.794034+0000", 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9199999999999997e-06 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.801817+0000", 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077829350000000004 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.801884+0000", 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6774999999999997e-05 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.801910+0000", 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6086999999999999e-05 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.802088+0000", 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178448 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.802352+0000", 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026367799999999999 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.802598+0000", 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024617500000000001 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.811512+0000", 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.008914468 2024-04-25T15:53:03.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.815149+0000", 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036369290000000001 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.815326+0000", 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176496 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.815470+0000", 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144258 2024-04-25T15:53:03.092 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:67 3.28 3:1772fc85:::10000000002.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.996236+0000", 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.067265524, 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025389129, 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.093 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 67 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.996236+0000", 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.996236+0000", 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.996240+0000", 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1829999999999998e-06 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005298+0000", 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.009058254 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005367+0000", 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9162e-05 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005392+0000", 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5510000000000001e-05 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005587+0000", 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000194792 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005874+0000", 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000286564 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.006129+0000", 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025482400000000001 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.015438+0000", 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093097019999999996 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.021435+0000", 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059962790000000002 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.021539+0000", 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010387000000000001 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.021626+0000", 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6989000000000003e-05 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:68 3.16 3:6a386151:::10000000002.00000040:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.098 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.038185+0000", 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.025316887999999, 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024089879000000002, 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 68 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.038185+0000", 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.038185+0000", 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.038188+0000", 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.903e-06 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.042919+0000", 2024-04-25T15:53:03.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047310529999999998 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.042978+0000", 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8573000000000003e-05 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.043002+0000", 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4015999999999998e-05 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.043206+0000", 2024-04-25T15:53:03.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020442799999999999 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.043454+0000", 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000247958 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.043720+0000", 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026612800000000002 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.054351+0000", 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010630550000000001 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.061939+0000", 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075888600000000002 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.062109+0000", 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169758 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.062275+0000", 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016565199999999999 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:73 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.257557+0000", 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 17.805945089000002, 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025594236999999999, 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.104 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 73 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.257557+0000", 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.257557+0000", 2024-04-25T15:53:03.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.257560+0000", 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.213e-06 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262293+0000", 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047326570000000004 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262353+0000", 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0009000000000001e-05 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262377+0000", 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4593e-05 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262599+0000", 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022189200000000001 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262870+0000", 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027080099999999998 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263123+0000", 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025334799999999997 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.272154+0000", 2024-04-25T15:53:03.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090310040000000005 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282961+0000", 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010806827999999999 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.283065+0000", 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010398 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.283151+0000", 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5912e-05 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:78 3.3c 3:3db0268f:::10000000002.0000004a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.456848+0000", 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 17.606654005999999, 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023752525, 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.110 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 78 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.456848+0000", 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.456848+0000", 2024-04-25T15:53:03.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.456851+0000", 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.236e-06 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.463407+0000", 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065559010000000003 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.463465+0000", 2024-04-25T15:53:03.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7766999999999998e-05 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.463490+0000", 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5646000000000001e-05 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.463693+0000", 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020228499999999999 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.464124+0000", 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043157999999999998 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.464535+0000", 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041020299999999999 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.475524+0000", 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01098946 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.480286+0000", 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004761904 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.480457+0000", 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017092799999999999 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.480600+0000", 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014361500000000001 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:146 3.16 3:6bf10b49:::10000000002.0000008e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.456179+0000", 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.607323239999999, 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022453794999999999, 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 146 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.456179+0000", 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.456179+0000", 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.456182+0000", 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1980000000000001e-06 2024-04-25T15:53:03.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.463948+0000", 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077665679999999997 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.464007+0000", 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9036999999999997e-05 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.464032+0000", 2024-04-25T15:53:03.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4885999999999999e-05 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.464202+0000", 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169428 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.464466+0000", 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000263902 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.464720+0000", 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000254651 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.474671+0000", 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0099503160000000007 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.478365+0000", 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036939249999999998 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.478470+0000", 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104969 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.478632+0000", 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016291499999999999 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:164 3.30 3:0efeaece:::10000000002.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:03.121 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.327592+0000", 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.735909418, 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023856829, 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 164 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.327592+0000", 2024-04-25T15:53:03.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.327592+0000", 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.327596+0000", 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7100000000000001e-06 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.335773+0000", 2024-04-25T15:53:03.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081769579999999998 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.335835+0000", 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2043000000000002e-05 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.335861+0000", 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5473000000000002e-05 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.336260+0000", 2024-04-25T15:53:03.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039901699999999999 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.336580+0000", 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032002000000000002 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.336892+0000", 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000312727 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.346124+0000", 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092320319999999994 2024-04-25T15:53:03.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.351255+0000", 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051309839999999999 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.351361+0000", 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105659 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.351449+0000", 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8206000000000002e-05 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:168 3.3a 3:5e20dd20:::10000000002.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.127 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.499594+0000", 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.563907763, 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025637967000000001, 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.128 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 168 2024-04-25T15:53:03.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.129 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.499594+0000", 2024-04-25T15:53:03.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.499594+0000", 2024-04-25T15:53:03.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.499598+0000", 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506582+0000", 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069845569999999997 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506649+0000", 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6500000000000004e-05 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506684+0000", 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4999999999999997e-05 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506896+0000", 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021213499999999999 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.507193+0000", 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029704699999999999 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.507463+0000", 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027019400000000002 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.520156+0000", 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692914 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.524914+0000", 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047575960000000002 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.525096+0000", 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018226599999999999 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.525232+0000", 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013627800000000001 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.134 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:171 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.617523+0000", 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.445979334, 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022243102000000001, 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 171 2024-04-25T15:53:03.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.617523+0000", 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.617523+0000", 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.617526+0000", 2024-04-25T15:53:03.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4699999999999998e-06 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.623940+0000", 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064136549999999999 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.624001+0000", 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0930000000000001e-05 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.624051+0000", 2024-04-25T15:53:03.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0278999999999998e-05 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.624249+0000", 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019806500000000001 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.624506+0000", 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025719099999999998 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.624760+0000", 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025363499999999999 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.634425+0000", 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0096648180000000004 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.639568+0000", 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051438270000000001 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.639680+0000", 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111956 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.639766+0000", 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5276000000000001e-05 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:180 3.2 e62/18 3:42ce5726:::10000000002.000000b0:head v 62'3, mlcod=59'1)", 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.003402+0000", 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.060100140999999, 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029043465000000001, 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.003402+0000", 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.003402+0000", 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.003405+0000", 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7000000000000002e-06 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.013858+0000", 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010452258000000001 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.013898+0000", 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0033000000000001e-05 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.013922+0000", 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4686e-05 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.014112+0000", 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018924499999999999 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.014192+0000", 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0742999999999996e-05 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.032267+0000", 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018074921000000001 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.032445+0000", 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017787900000000001 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.144 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:181 3.1f 3:f8dc3ee2:::10000000002.000000b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.040594+0000", 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.022907622, 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02520416, 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 181 2024-04-25T15:53:03.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.040594+0000", 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.040594+0000", 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.040597+0000", 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1489999999999998e-06 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048339+0000", 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077415790000000002 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048410+0000", 2024-04-25T15:53:03.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0545999999999998e-05 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048436+0000", 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6426000000000001e-05 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048640+0000", 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020419200000000001 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.049063+0000", 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000422626 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.049532+0000", 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00046956999999999999 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.064111+0000", 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014578420999999999 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.065438+0000", 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001327665 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.065621+0000", 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018292800000000001 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.065798+0000", 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017705800000000001 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.150 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:186 3.1c 3:3913eadd:::10000000002.000000b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.255851+0000", 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.807650405, 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024633971000000001, 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 186 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.151 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.255851+0000", 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.255851+0000", 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.255856+0000", 2024-04-25T15:53:03.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4000000000000002e-06 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.263964+0000", 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081081690000000001 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.264056+0000", 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1551999999999994e-05 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.264093+0000", 2024-04-25T15:53:03.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7259999999999999e-05 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.264211+0000", 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118064 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.264390+0000", 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178918 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:53:03.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.264569+0000", 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017906000000000001 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.270977+0000", 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064080700000000001 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.280350+0000", 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093727350000000001 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.280424+0000", 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3831000000000005e-05 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.280485+0000", 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1911999999999999e-05 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:255 3.1d 3:b88107a3:::10000000002.000000fb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.353358+0000", 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7101434219999998, 2024-04-25T15:53:03.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024862755, 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 255 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.353358+0000", 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.353358+0000", 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.353362+0000", 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.76e-06 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.359978+0000", 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066161509999999998 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.360065+0000", 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6528999999999998e-05 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.360092+0000", 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7253000000000001e-05 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.360240+0000", 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014767299999999999 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.360497+0000", 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000256968 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.360743+0000", 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024626699999999999 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.370101+0000", 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093574620000000004 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.378027+0000", 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079261829999999998 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.378129+0000", 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102519 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.378221+0000", 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1990000000000005e-05 2024-04-25T15:53:03.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:261 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.752398+0000", 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.3111037240000001, 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030613782999999999, 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.162 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 261 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.752398+0000", 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.752398+0000", 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.752402+0000", 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.783e-06 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.758763+0000", 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063605900000000002 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.758830+0000", 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7768999999999994e-05 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.758860+0000", 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0034000000000001e-05 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759094+0000", 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000233351 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759271+0000", 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017722899999999999 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759518+0000", 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024753399999999999 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.776213+0000", 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016694102999999998 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782820+0000", 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066076980000000004 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782930+0000", 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109633 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.783012+0000", 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2058999999999995e-05 2024-04-25T15:53:03.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:263 3.5 3:a0e740da:::10000000002.00000003:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.842785+0000", 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.2207172380000002, 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021429429999999999, 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.168 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 263 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.842785+0000", 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.842785+0000", 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.842789+0000", 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0199999999999996e-06 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.846784+0000", 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039954430000000004 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.846846+0000", 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1879999999999997e-05 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.846874+0000", 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7543e-05 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.847104+0000", 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023054799999999999 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.847287+0000", 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018250800000000001 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.847544+0000", 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025774199999999999 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.858994+0000", 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011449183999999999 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.863995+0000", 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005001821 2024-04-25T15:53:03.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.864124+0000", 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012893900000000001 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.864214+0000", 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9802000000000005e-05 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.173 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:275 3.30 3:0ca36612:::10000000002.0000000f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.369715+0000", 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 2.6937869340000002, 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023408410000000001, 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 275 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.369715+0000", 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.369715+0000", 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.369719+0000", 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8070000000000001e-06 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.376395+0000", 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066764299999999997 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.376461+0000", 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5949000000000002e-05 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.376489+0000", 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7883999999999999e-05 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.376652+0000", 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016307800000000001 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.376835+0000", 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018268200000000001 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.377095+0000", 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026047400000000001 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.389830+0000", 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01273473 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.392931+0000", 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00310137 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.393037+0000", 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105673 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.393123+0000", 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6333000000000002e-05 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.179 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.078+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f22b4212e20 con 0x7f22b420fee0 2024-04-25T15:53:03.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22b0ff9640 1 -- 172.21.15.67:0/2819520983 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f22b4212e20 con 0x7f22b420fee0 2024-04-25T15:53:03.180 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:03.181 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:03.182 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:03.179+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f22b41532f0 msgr2=0x7f22b40cc890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f22b41532f0 0x7f22b40cc890 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f22b41f76e0 msgr2=0x7f22b41f7af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f22b41f76e0 0x7f22b41f7af0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f22b40635a0 msgr2=0x7f22b40639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f22b40635a0 0x7f22b40639b0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f22b40b1910 msgr2=0x7f22b40b1d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f22b40b1910 0x7f22b40b1d20 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f22b402d2b0 msgr2=0x7f22b402d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f22b402d2b0 0x7f22b402d6c0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22b892e640 1 -- 172.21.15.67:0/2819520983 reap_dead start 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22b892e640 1 -- 172.21.15.67:0/2819520983 reap_dead start 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f22b402dc00 msgr2=0x7f22b420aad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f22b402dc00 0x7f22b420aad0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f22b420b730 msgr2=0x7f22b420db90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f22b420b730 0x7f22b420db90 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f22b420fee0 msgr2=0x7f22b42102f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f22b420fee0 0x7f22b42102f0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22a006f400 msgr2=0x7f22a0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22a006f400 0x7f22a0071910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f22a40ab000 tx=0x7f22a40b0040 comp rx=0 tx=0).stop 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b41f4090 msgr2=0x7f22b40b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 --2- 172.21.15.67:0/2819520983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22b41f4090 0x7f22b40b07f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f22a800d4d0 tx=0x7f22a800d990 comp rx=0 tx=0).stop 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22b892e640 1 -- 172.21.15.67:0/2819520983 reap_dead start 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 shutdown_connections 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 >> 172.21.15.67:0/2819520983 conn(0x7f22b40ce1c0 msgr2=0x7f22b40cbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 shutdown_connections 2024-04-25T15:53:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:03.086+0000 7f22ba391640 1 -- 172.21.15.67:0/2819520983 wait complete. 2024-04-25T15:53:03.281 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:03.279+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:03.382 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:03.382+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:03.401 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "MOSDPGPush(3.1c 69/68 [PushOp(3:3aa14ed1:::10000000002.00000059:head, version: 60'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3aa14ed1:::10000000002.00000059:head@60'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.400001+0000", 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.001512939, 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015830410000000001, 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400001+0000", 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400001+0000", 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400003+0000", 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.423e-06 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400561+0000", 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00055818700000000003 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400603+0000", 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1904999999999999e-05 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400637+0000", 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4468000000000003e-05 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400806+0000", 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016813200000000001 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.400825+0000", 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9281999999999999e-05 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 1 2024-04-25T15:53:03.405 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:03.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:03.482 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:03.482+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:03.583 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:03.582+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:03.643 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:03.643 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:03.644 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:03.644 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:03.644 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:03.668 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:03.711 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:03.711+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:03.784 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:03.783+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:03.830 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:03.830 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:03.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:03.830 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:03.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:43 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.926287974000001, 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024229051000000001, 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902019+0000", 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9459999999999998e-06 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907768+0000", 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005748525 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907831+0000", 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2842999999999994e-05 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907857+0000", 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6636e-05 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908056+0000", 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198295 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908464+0000", 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040813600000000002 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908853+0000", 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038892400000000002 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925332+0000", 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016479074 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925882+0000", 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000550232 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926050+0000", 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016822199999999999 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926245+0000", 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019521799999999999 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:53 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.450105813, 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020793931000000002, 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 53 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.837 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378202+0000", 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1669999999999999e-06 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382531+0000", 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043289130000000002 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382593+0000", 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2143000000000005e-05 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382618+0000", 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382814+0000", 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195964 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383075+0000", 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026081800000000001 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383324+0000", 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024972200000000002 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.392258+0000", 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089338549999999992 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398665+0000", 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064066649999999998 2024-04-25T15:53:03.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398841+0000", 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017607899999999999 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398993+0000", 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015144799999999999 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:62 3.11 e60/54 3:8969f477:::10000000002.0000003a:head v 60'1, mlcod=0'0)", 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.066620919999998, 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019322214000000001, 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761683+0000", 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761684+0000", 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3399999999999998e-07 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765759+0000", 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040747159999999999 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765801+0000", 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1675999999999999e-05 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765835+0000", 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4518999999999999e-05 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.765978+0000", 2024-04-25T15:53:03.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014263899999999999 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.766025+0000", 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6842999999999999e-05 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780902+0000", 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014877118 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781006+0000", 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010396899999999999 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:73 3.1d e60/54 3:b94578cb:::10000000002.00000045:head v 60'2, mlcod=0'0)", 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.56489552, 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019180661000000002, 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.847 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263409+0000", 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263412+0000", 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9900000000000002e-06 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268799+0000", 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053871980000000002 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268844+0000", 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5062999999999999e-05 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.268870+0000", 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5633e-05 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269028+0000", 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000158135 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.269078+0000", 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0463000000000001e-05 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282434+0000", 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013355156999999999 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282590+0000", 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015602199999999999 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:75 3.35 3:ad096b5b:::10000000002.00000047:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.494795273000001, 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022276672000000001, 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333513+0000", 2024-04-25T15:53:03.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339006+0000", 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054929599999999999 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339071+0000", 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5545999999999999e-05 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339098+0000", 2024-04-25T15:53:03.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6647000000000001e-05 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339299+0000", 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201351 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339703+0000", 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040396399999999998 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:03.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.340160+0000", 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045697899999999999 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.352668+0000", 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012508137000000001 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355605+0000", 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002937013 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355701+0000", 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5681999999999996e-05 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355786+0000", 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4908999999999996e-05 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:79 3.3 3:c0361aff:::10000000002.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:03.857 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.319025178, 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022495096999999999, 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 79 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509282+0000", 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2200000000000001e-06 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517219+0000", 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079362270000000006 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517284+0000", 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5052e-05 2024-04-25T15:53:03.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517310+0000", 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6072999999999999e-05 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517506+0000", 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019606199999999999 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517779+0000", 2024-04-25T15:53:03.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027302799999999998 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.518047+0000", 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000268454 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.528734+0000", 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01068712 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531586+0000", 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028516589999999999 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531689+0000", 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010260900000000001 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531774+0000", 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5593000000000005e-05 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.863 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:144 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.460117731, 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019320924999999999, 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 144 2024-04-25T15:53:03.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368187+0000", 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.368190+0000", 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4170000000000001e-06 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373302+0000", 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005111857 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373366+0000", 2024-04-25T15:53:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3570000000000003e-05 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373413+0000", 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7679000000000002e-05 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373579+0000", 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000165422 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.373836+0000", 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025765800000000002 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.374080+0000", 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024384499999999999 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.386500+0000", 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012419688999999999 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387295+0000", 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00079550599999999997 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387403+0000", 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010791899999999999 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.387508+0000", 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104363 2024-04-25T15:53:03.869 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:165 3.11 e62/54 3:88ab7984:::10000000002.000000a1:head v 62'7, mlcod=61'5)", 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.447230551000001, 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020252645999999999, 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381074+0000", 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.381075+0000", 2024-04-25T15:53:03.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9299999999999997e-07 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386313+0000", 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052379799999999997 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386350+0000", 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7469000000000003e-05 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386384+0000", 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3862999999999997e-05 2024-04-25T15:53:03.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386546+0000", 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016226899999999999 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.386587+0000", 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1026000000000003e-05 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401215+0000", 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01462777 2024-04-25T15:53:03.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401327+0000", 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111576 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:166 3.2c 3:3723e56f:::10000000002.000000a2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.414282169, 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019979538000000002, 2024-04-25T15:53:03.874 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 166 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414022+0000", 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.414025+0000", 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.92e-06 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.419994+0000", 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059684200000000003 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420068+0000", 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4878999999999999e-05 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420093+0000", 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4553e-05 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420239+0000", 2024-04-25T15:53:03.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014577500000000001 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420512+0000", 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027295800000000001 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.420761+0000", 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024945499999999999 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.431910+0000", 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011148739 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.433806+0000", 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018964909999999999 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.433907+0000", 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100179 2024-04-25T15:53:03.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.434002+0000", 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5168999999999996e-05 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:176 3.26 3:66bf65cf:::10000000002.000000ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.001475319000001, 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020587095999999999, 2024-04-25T15:53:03.880 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 176 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826832+0000", 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1200000000000002e-06 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831165+0000", 2024-04-25T15:53:03.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043325289999999999 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831220+0000", 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5686999999999997e-05 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831245+0000", 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4105999999999999e-05 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831400+0000", 2024-04-25T15:53:03.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015513899999999999 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831669+0000", 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026955099999999998 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831947+0000", 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027750399999999998 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.842230+0000", 2024-04-25T15:53:03.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010283537000000001 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847229+0000", 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049990939999999999 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847326+0000", 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7106000000000002e-05 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847416+0000", 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9722999999999995e-05 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:177 3.26 3:66aef6e9:::10000000002.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.951616421000001, 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026823691, 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.886 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876691+0000", 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4939999999999999e-06 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880736+0000", 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040441660000000001 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880793+0000", 2024-04-25T15:53:03.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7581999999999999e-05 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880817+0000", 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881039+0000", 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022171800000000001 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881305+0000", 2024-04-25T15:53:03.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026580799999999997 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881558+0000", 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025290099999999998 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.893182+0000", 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011624242 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903322+0000", 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010139754000000001 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903425+0000", 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102706 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903512+0000", 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7093000000000001e-05 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:182 3.20 3:04677afc:::10000000002.000000b2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.738583265000001, 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018848521, 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 182 2024-04-25T15:53:03.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089721+0000", 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.089724+0000", 2024-04-25T15:53:03.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2270000000000002e-06 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095098+0000", 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053738709999999997 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095165+0000", 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6618999999999994e-05 2024-04-25T15:53:03.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095192+0000", 2024-04-25T15:53:03.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6783000000000001e-05 2024-04-25T15:53:03.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095330+0000", 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013835900000000001 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095586+0000", 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025566400000000001 2024-04-25T15:53:03.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.095830+0000", 2024-04-25T15:53:03.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024478199999999997 2024-04-25T15:53:03.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.104485+0000", 2024-04-25T15:53:03.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0086547110000000007 2024-04-25T15:53:03.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108421+0000", 2024-04-25T15:53:03.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039358010000000001 2024-04-25T15:53:03.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108497+0000", 2024-04-25T15:53:03.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5799999999999999e-05 2024-04-25T15:53:03.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.108570+0000", 2024-04-25T15:53:03.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2904000000000005e-05 2024-04-25T15:53:03.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.901 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:261 3.1d e66/54 3:bbf7764d:::10000000002.00000001:head v 66'8, mlcod=63'6)", 2024-04-25T15:53:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.0684503410000001, 2024-04-25T15:53:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022527077, 2024-04-25T15:53:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:03.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759855+0000", 2024-04-25T15:53:03.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6599999999999996e-07 2024-04-25T15:53:03.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764340+0000", 2024-04-25T15:53:03.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044852190000000004 2024-04-25T15:53:03.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764382+0000", 2024-04-25T15:53:03.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2052999999999999e-05 2024-04-25T15:53:03.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764415+0000", 2024-04-25T15:53:03.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.341e-05 2024-04-25T15:53:03.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764650+0000", 2024-04-25T15:53:03.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023472400000000001 2024-04-25T15:53:03.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764703+0000", 2024-04-25T15:53:03.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3016999999999997e-05 2024-04-25T15:53:03.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782246+0000", 2024-04-25T15:53:03.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017543286000000002 2024-04-25T15:53:03.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782381+0000", 2024-04-25T15:53:03.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134702 2024-04-25T15:53:03.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.912 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.913 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:268 3.14 e67/54 3:29a5f800:::10000000002.00000008:head v 67'6, mlcod=63'4)", 2024-04-25T15:53:03.913 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:03.913 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.7594945179999999, 2024-04-25T15:53:03.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023750529999999999, 2024-04-25T15:53:03.913 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.914 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.914 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.914 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:03.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068811+0000", 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.37e-07 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073130+0000", 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043190069999999997 2024-04-25T15:53:03.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073172+0000", 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2512000000000002e-05 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073198+0000", 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6084e-05 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073426+0000", 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227514 2024-04-25T15:53:03.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073510+0000", 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4469999999999996e-05 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092427+0000", 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018916571 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092560+0000", 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013373500000000001 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:269 3.2c 3:354fa1eb:::10000000002.00000009:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.723015695, 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022386919000000002, 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 269 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105292+0000", 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8839999999999998e-06 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111434+0000", 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061422880000000001 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111491+0000", 2024-04-25T15:53:03.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7306000000000003e-05 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111515+0000", 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3893e-05 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111686+0000", 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017118600000000001 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111859+0000", 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017279099999999999 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.112126+0000", 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000266722 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.123964+0000", 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01183797 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127453+0000", 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034891240000000001 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127555+0000", 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102082 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127676+0000", 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120673 2024-04-25T15:53:03.924 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:273 3.34 3:2fc55728:::10000000002.0000000d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.270390+0000", 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.5579143000000002, 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019567221999999999, 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.925 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 273 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.270390+0000", 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.270390+0000", 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.270393+0000", 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9799999999999998e-06 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.274873+0000", 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044801479999999998 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.274931+0000", 2024-04-25T15:53:03.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8156999999999999e-05 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.274963+0000", 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1965999999999999e-05 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.275139+0000", 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017609799999999999 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.275313+0000", 2024-04-25T15:53:03.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017360300000000001 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.275562+0000", 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024904799999999998 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.288074+0000", 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012512181000000001 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.289762+0000", 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016877120000000001 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.289869+0000", 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010650300000000001 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.289957+0000", 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8826000000000006e-05 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:279 3.10 3:0a7b425d:::10000000002.00000013:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.560792+0000", 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.267512473, 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02017217, 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 279 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.931 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.560792+0000", 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.560792+0000", 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.560795+0000", 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9799999999999998e-06 2024-04-25T15:53:03.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.565701+0000", 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049064190000000004 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.565762+0000", 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0622999999999999e-05 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.565796+0000", 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4153000000000003e-05 2024-04-25T15:53:03.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.566009+0000", 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021265800000000001 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.566184+0000", 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017535200000000001 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.566441+0000", 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025707799999999999 2024-04-25T15:53:03.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.578418+0000", 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011976972000000001 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.580765+0000", 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023471659999999999 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.580869+0000", 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103216 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.580964+0000", 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5552999999999998e-05 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:347 3.11 e68/54 3:8815798b:::10000000002.00000057:head v 68'10, mlcod=63'8)", 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.156855+0000", 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.67144955100000003, 2024-04-25T15:53:03.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020040509000000001, 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156855+0000", 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156855+0000", 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156857+0000", 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4889999999999998e-06 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.159227+0000", 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023694229999999998 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.159255+0000", 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7954e-05 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.159272+0000", 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7362000000000001e-05 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.159442+0000", 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169471 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.159522+0000", 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0096e-05 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.176769+0000", 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017247425 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.176895+0000", 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000126289 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:355 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.33958974800000002, 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025437358, 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 355 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.941 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488717+0000", 2024-04-25T15:53:03.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.874e-06 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494412+0000", 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056951349999999996 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494450+0000", 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7920999999999998e-05 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494464+0000", 2024-04-25T15:53:03.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4728e-05 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494580+0000", 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011599099999999999 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494756+0000", 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176083 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.495038+0000", 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028161800000000003 2024-04-25T15:53:03.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.508282+0000", 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013243981 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514019+0000", 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057372639999999997 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514099+0000", 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9800000000000002e-05 2024-04-25T15:53:03.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514152+0000", 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2963000000000001e-05 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:357 3.34 3:2d119e8b:::10000000002.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.26308162200000001, 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021511328, 2024-04-25T15:53:03.946 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 357 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565225+0000", 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2189999999999998e-06 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569849+0000", 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046243480000000003 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569992+0000", 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014274200000000001 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570006+0000", 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4032999999999999e-05 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570122+0000", 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116358 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:03.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570290+0000", 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016725299999999999 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570528+0000", 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023819800000000001 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.581505+0000", 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01097657 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586619+0000", 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051144119999999996 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586682+0000", 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3309e-05 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586734+0000", 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1885999999999998e-05 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.951 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.952 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.952 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:03.952 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:03.952 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:03.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:03.954 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:03.950+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:04.054 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:04.054+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:04.154 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:04.154+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:04.159 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:04.159 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:04.159 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:04.159 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:04.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:04.255 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:04.254+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:04.328 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:04.328 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:04.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:04.355 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:04.354+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:04.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:53:04.457 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:04.455+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 165.40089157200001, 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:04.503 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:04.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:04.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:04.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 164.80968864799999, 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:04.509 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:53:04.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:53:04.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:53:04.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:53:04.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 164.809677569, 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:53:04.514 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:53:04.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 164.50197453000001, 2024-04-25T15:53:04.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022930816999999999, 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997381+0000", 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997382+0000", 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1699999999999997e-07 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997390+0000", 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2130000000000008e-06 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997566+0000", 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017603200000000001 2024-04-25T15:53:04.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.997577+0000", 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1097000000000001e-05 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008372+0000", 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01079449 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008494+0000", 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012213799999999999 2024-04-25T15:53:04.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.008617+0000", 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012282300000000001 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.009636+0000", 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010194010000000001 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020218+0000", 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581610999999999 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020257+0000", 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9400000000000002e-05 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:20.020312+0000", 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4694999999999997e-05 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:47 3.12 3:489b4446:::10000000002.0000002b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.398831803, 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022914499000000001, 2024-04-25T15:53:04.524 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 47 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100524+0000", 2024-04-25T15:53:04.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.100528+0000", 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3030000000000001e-06 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105104+0000", 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045757710000000002 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105167+0000", 2024-04-25T15:53:04.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2936999999999997e-05 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105194+0000", 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7606e-05 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105424+0000", 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000229998 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105705+0000", 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000280408 2024-04-25T15:53:04.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.105966+0000", 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261258 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.118921+0000", 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012954482999999999 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123247+0000", 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043263199999999998 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123350+0000", 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103423 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.123438+0000", 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7991999999999994e-05 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:62 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.743702677000002, 2024-04-25T15:53:04.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025889728000000001, 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 62 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755657+0000", 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9439999999999998e-06 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760593+0000", 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049360740000000004 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760651+0000", 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8e-05 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760676+0000", 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5219999999999999e-05 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760847+0000", 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017057199999999999 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761123+0000", 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027578100000000001 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761365+0000", 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024248100000000001 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780642+0000", 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019277327 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781358+0000", 2024-04-25T15:53:04.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000715611 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781456+0000", 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7946e-05 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781543+0000", 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6772000000000002e-05 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.534 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:150 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.848183745, 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024331635000000001, 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 150 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:04.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651176+0000", 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8770000000000003e-06 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659329+0000", 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081530219999999994 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659399+0000", 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0376000000000005e-05 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659428+0000", 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8952999999999999e-05 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659580+0000", 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015188199999999999 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659842+0000", 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026179399999999998 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.660120+0000", 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027824799999999998 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.673813+0000", 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013693145 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675307+0000", 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014934600000000001 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675412+0000", 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105233 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675504+0000", 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1644999999999994e-05 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:155 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.626032459999999, 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025243293999999999, 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 155 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873327+0000", 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7699999999999999e-06 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879173+0000", 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005846141 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879239+0000", 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6315999999999993e-05 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879267+0000", 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7137000000000001e-05 2024-04-25T15:53:04.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879415+0000", 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000148715 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879693+0000", 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000278108 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879951+0000", 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025793700000000002 2024-04-25T15:53:04.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.896019+0000", 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016067814999999999 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898153+0000", 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021339280000000002 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898411+0000", 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025760799999999999 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898566+0000", 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015581899999999999 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:158 3.12 3:4b703321:::10000000002.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.489134744999999, 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024089508999999999, 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.545 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 158 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010221+0000", 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.010224+0000", 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3969999999999998e-06 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016098+0000", 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058735000000000002 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016148+0000", 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0640999999999998e-05 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016177+0000", 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.813e-05 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016313+0000", 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136052 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016470+0000", 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000157637 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.016601+0000", 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013075599999999999 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.023425+0000", 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068237799999999998 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034203+0000", 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01077811 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034260+0000", 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6702e-05 2024-04-25T15:53:04.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.034310+0000", 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0803999999999997e-05 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:165 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.130061923, 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032658418000000002, 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.550 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 165 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:04.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369298+0000", 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3429999999999998e-06 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374706+0000", 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054080250000000003 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374776+0000", 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9972e-05 2024-04-25T15:53:04.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374803+0000", 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7384000000000001e-05 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375010+0000", 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206431 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375037+0000", 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7206999999999999e-05 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380232+0000", 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051947859999999998 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380474+0000", 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000241864 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380709+0000", 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023560200000000001 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.392962+0000", 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012252219999999999 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401765+0000", 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088034089999999999 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401866+0000", 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101099 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401952+0000", 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6075999999999994e-05 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:173 3.2b 3:d5480d6b:::10000000002.000000a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.790045442, 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022867683, 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.556 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 173 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709310+0000", 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.709313+0000", 2024-04-25T15:53:04.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8159999999999998e-06 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716389+0000", 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070763739999999999 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716451+0000", 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1958999999999994e-05 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716476+0000", 2024-04-25T15:53:04.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4246000000000001e-05 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716689+0000", 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000212926 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.716951+0000", 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026266700000000001 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.717199+0000", 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024812800000000002 2024-04-25T15:53:04.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.726072+0000", 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088724990000000007 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.731628+0000", 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055558229999999997 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.732003+0000", 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00037564100000000002 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.732178+0000", 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000174604 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.281651091000001, 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.561 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:04.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.187017039000001, 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.566 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:04.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:04.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.571 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:260 3.12 3:4bec3886:::10000000002.00000000:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.8093757979999996, 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028352792000000002, 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 260 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:04.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689983+0000", 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2729999999999998e-06 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696571+0000", 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065880749999999997 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696637+0000", 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5358999999999995e-05 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696662+0000", 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5184000000000001e-05 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697262+0000", 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060013099999999995 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697437+0000", 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017512499999999999 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697672+0000", 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023489800000000001 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.715797+0000", 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018125601000000002 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718141+0000", 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023436300000000002 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718244+0000", 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102379 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718333+0000", 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9136999999999997e-05 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.576 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:268 3.14 3:29a5f800:::10000000002.00000008:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.4384284090000001, 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032265195000000003, 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 268 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060931+0000", 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6629999999999999e-06 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067652+0000", 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067213899999999998 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067713+0000", 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0628999999999998e-05 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067739+0000", 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6492999999999999e-05 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067893+0000", 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015336599999999999 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068083+0000", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019004800000000001 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068328+0000", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024518100000000002 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.087697+0000", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019368720999999998 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092957+0000", 2024-04-25T15:53:04.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052597859999999998 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093058+0000", 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101335 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093192+0000", 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134583 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:271 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:04.581 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3060332599999995, 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026911917, 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 271 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:04.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193326+0000", 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0770000000000001e-06 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198319+0000", 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049928710000000003 2024-04-25T15:53:04.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198382+0000", 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2446000000000005e-05 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198408+0000", 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6676999999999999e-05 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198623+0000", 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021493799999999999 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198914+0000", 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029086100000000001 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.199346+0000", 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000431316 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.219423+0000", 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020076954000000001 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220028+0000", 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060503900000000003 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220153+0000", 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125919 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220234+0000", 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0819e-05 2024-04-25T15:53:04.586 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:347 3.11 3:8815798b:::10000000002.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.3509671169999999, 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029152658000000001, 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.587 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 347 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148391+0000", 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6029999999999999e-06 2024-04-25T15:53:04.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155889+0000", 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074977200000000003 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155942+0000", 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2883999999999997e-05 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155964+0000", 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2762999999999999e-05 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156184+0000", 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156358+0000", 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017470800000000001 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156597+0000", 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238411 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.172619+0000", 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016022649 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177350+0000", 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047301369999999997 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177455+0000", 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010574899999999999 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177541+0000", 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5839000000000003e-05 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:352 3.7 e68/18 3:e0f5f483:::10000000002.0000005c:head v 68'11, mlcod=67'9)", 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.159380597, 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026288315999999999, 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339978+0000", 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8459999999999999e-06 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345285+0000", 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053072630000000004 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345324+0000", 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8535999999999998e-05 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345348+0000", 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4315999999999999e-05 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345503+0000", 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015497300000000001 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345555+0000", 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1814999999999998e-05 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366145+0000", 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020589743000000001 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366263+0000", 2024-04-25T15:53:04.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118824 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:326 3.8 e69/68 3:133e839e:::10000000002.00000042:head v 69'10, mlcod=0'0)", 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.0618589629999999, 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025085139999999999, 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437498+0000", 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.632e-06 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445817+0000", 2024-04-25T15:53:04.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083191740000000004 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445839+0000", 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1314e-05 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445854+0000", 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4986e-05 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445948+0000", 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3735999999999998e-05 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445981+0000", 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3210999999999998e-05 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462484+0000", 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016502871999999998 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462582+0000", 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8214999999999994e-05 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.599 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:367 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.55504261899999996, 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025258800000000001, 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 367 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.600 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944316+0000", 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4880000000000001e-06 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950833+0000", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065179829999999998 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950884+0000", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950902+0000", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8805e-05 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951039+0000", 2024-04-25T15:53:04.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013634300000000001 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951216+0000", 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017729099999999999 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951469+0000", 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025263199999999998 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969276+0000", 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017807630000000001 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969326+0000", 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9639999999999999e-05 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969434+0000", 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108155 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969572+0000", 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013767299999999999 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:04.605 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:04.605 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:04.607 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:04.603+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:04.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c98076640 1 Processor -- start 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c98076640 1 -- start start 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c98076640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c900c92d0 0x7f5c90072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c98076640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9006d8e0 con 0x7f5c900c96e0 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c98076640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c90073c70 con 0x7f5c90160800 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c98076640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5c9006dbc0 con 0x7f5c900c92d0 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c95611640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c900c92d0 0x7f5c90072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c96613640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c900c96e0 0x7f5c90167180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40106/0 (socket says 172.21.15.67:40106) 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c96613640 1 -- 172.21.15.67:0/4276734016 learned_addr learned my addr 172.21.15.67:0/4276734016 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c95611640 1 -- 172.21.15.67:0/4276734016 >> v1:172.21.15.176:6789/0 conn(0x7f5c90160800 legacy=0x7f5c9006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c95611640 1 -- 172.21.15.67:0/4276734016 >> v1:172.21.15.67:6789/0 conn(0x7f5c900c96e0 legacy=0x7f5c90167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.534+0000 7f5c95611640 1 -- 172.21.15.67:0/4276734016 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c90073f50 con 0x7f5c900c92d0 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c95611640 1 --2- 172.21.15.67:0/4276734016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c92d0 0x7f5c90072dc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c800097c0 tx=0x7f5c8001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f8fc2b9b037b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c94e10640 1 -- 172.21.15.67:0/4276734016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c800b0070 con 0x7f5c900c92d0 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c94e10640 1 -- 172.21.15.67:0/4276734016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c800ab600 con 0x7f5c900c92d0 2024-04-25T15:53:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c94e10640 1 -- 172.21.15.67:0/4276734016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c800ab9d0 con 0x7f5c900c92d0 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c98076640 1 -- 172.21.15.67:0/4276734016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c92d0 msgr2=0x7f5c90072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c98076640 1 --2- 172.21.15.67:0/4276734016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c92d0 0x7f5c90072dc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c800097c0 tx=0x7f5c8001c410 comp rx=0 tx=0).stop 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c98076640 1 -- 172.21.15.67:0/4276734016 shutdown_connections 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c98076640 1 --2- 172.21.15.67:0/4276734016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c92d0 0x7f5c90072dc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c98076640 1 -- 172.21.15.67:0/4276734016 >> 172.21.15.67:0/4276734016 conn(0x7f5c90070db0 msgr2=0x7f5c900ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c98076640 1 -- 172.21.15.67:0/4276734016 shutdown_connections 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.538+0000 7f5c98076640 1 -- 172.21.15.67:0/4276734016 wait complete. 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 Processor -- start 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 -- start start 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c900c92d0 0x7f5c900c0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c96e0 0x7f5c900c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c90160800 0x7f5c9015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c90160d90 con 0x7f5c900c96e0 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c90160c10 con 0x7f5c900c92d0 2024-04-25T15:53:04.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c9016b4d0 con 0x7f5c90160800 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c96e0 0x7f5c900c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c96e0 0x7f5c900c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49198/0 (socket says 172.21.15.67:49198) 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 -- 172.21.15.67:0/2270525467 learned_addr learned my addr 172.21.15.67:0/2270525467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c96613640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c90160800 0x7f5c9015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 -- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c90160800 msgr2=0x7f5c9015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c90160800 0x7f5c9015ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 -- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c900c92d0 msgr2=0x7f5c900c0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c900c92d0 0x7f5c900c0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 -- 172.21.15.67:0/2270525467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c900c09a0 con 0x7f5c900c96e0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c95611640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c96e0 0x7f5c900c42f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5c80019020 tx=0x7f5c800191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c867fc640 1 -- 172.21.15.67:0/2270525467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c800b0040 con 0x7f5c900c96e0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c900bf3c0 con 0x7f5c900c96e0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c900bf980 con 0x7f5c900c96e0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c867fc640 1 -- 172.21.15.67:0/2270525467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c800be410 con 0x7f5c900c96e0 2024-04-25T15:53:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.542+0000 7f5c867fc640 1 -- 172.21.15.67:0/2270525467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c800be7c0 con 0x7f5c900c96e0 2024-04-25T15:53:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.546+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c780052a0 con 0x7f5c900c96e0 2024-04-25T15:53:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.546+0000 7f5c867fc640 1 -- 172.21.15.67:0/2270525467 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5c800bc020 con 0x7f5c900c96e0 2024-04-25T15:53:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.546+0000 7f5c867fc640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.546+0000 7f5c867fc640 1 -- 172.21.15.67:0/2270525467 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5886+0+0 (secure 0 0 0) 0x7f5c80128800 con 0x7f5c900c96e0 2024-04-25T15:53:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.546+0000 7f5c95e12640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.546+0000 7f5c95e12640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c000f70 tx=0x7f5c8c0079c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.550+0000 7f5c867fc640 1 -- 172.21.15.67:0/2270525467 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5c80016090 con 0x7f5c900c96e0 2024-04-25T15:53:04.707 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:04.706+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:04.762+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f5c78005860 con 0x7f5c900c96e0 2024-04-25T15:53:04.785 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:04.785 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:04.785 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:04.785 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:04.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:04.807 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:04.806+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:04.907 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:04.906+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:05.009 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:05.006+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:05.022 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:05.022 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:05.022 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:05.023 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:05.023 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:05.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:05.110 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:05.107+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:05.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:05.229 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:05.230+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:05.258 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 149.04563948099999, 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 149.04559209600001, 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:53:05.263 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:05.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:53:05.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 149.04555984199999, 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.268 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:53:05.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.441369779999999, 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:53:05.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.150571636999999, 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:05.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:66 3.b e60/21 3:d381a89f:::10000000002.0000003e:head v 60'1, mlcod=0'0)", 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.311921496, 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015316048000000001, 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945216+0000", 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9e-06 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948749+0000", 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035338119999999999 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948777+0000", 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7915999999999998e-05 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948796+0000", 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.844e-05 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948895+0000", 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9704000000000006e-05 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948916+0000", 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.107e-05 2024-04-25T15:53:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960445+0000", 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011528722999999999 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960530+0000", 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4482999999999997e-05 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:152 3.39 e61/18 3:9e24e8b4:::10000000002.00000094:head v 61'2, mlcod=0'0)", 2024-04-25T15:53:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.520051418000001, 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016125006000000001, 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737086+0000", 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744295+0000", 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072082320000000002 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744341+0000", 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6072000000000002e-05 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744376+0000", 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4903000000000001e-05 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744533+0000", 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015736299999999999 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744590+0000", 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6771000000000002e-05 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753101+0000", 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085115490000000002 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753209+0000", 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107352 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:156 3.3e e61/35 3:7d47d675:::10000000002.00000098:head v 61'1, mlcod=0'0)", 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:05.287 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.322897945000001, 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016362634000000001, 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934240+0000", 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3149999999999999e-06 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939270+0000", 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050303869999999999 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939308+0000", 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8047999999999999e-05 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939335+0000", 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7251999999999999e-05 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939432+0000", 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6637000000000003e-05 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939459+0000", 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7651999999999999e-05 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950534+0000", 2024-04-25T15:53:05.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011074993999999999 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950600+0000", 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5349000000000001e-05 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:158 3.12 e61/40 3:4b703321:::10000000002.0000009a:head v 61'6, mlcod=59'4)", 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.239937818000001, 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017102098, 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017200+0000", 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3199999999999998e-06 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024143+0000", 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069429569999999996 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024169+0000", 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6007e-05 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024190+0000", 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1203999999999999e-05 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024272+0000", 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1833000000000001e-05 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024301+0000", 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9475000000000002e-05 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034243+0000", 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099420589999999996 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034299+0000", 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6242999999999997e-05 2024-04-25T15:53:05.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:162 3.2b e61/40 3:d691b431:::10000000002.0000009e:head v 61'3, mlcod=60'1)", 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.031593807, 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014572351000000001, 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:05.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225544+0000", 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7199999999999998e-06 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230207+0000", 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046626719999999997 2024-04-25T15:53:05.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230239+0000", 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2548000000000001e-05 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230266+0000", 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6599999999999999e-05 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230382+0000", 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115667 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230417+0000", 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5800999999999998e-05 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240029+0000", 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096115349999999992 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240114+0000", 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4808000000000005e-05 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:181 3.1f e62/40 3:f8dc3ee2:::10000000002.000000b1:head v 62'2, mlcod=0'0)", 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.206923212, 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015250434, 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050214+0000", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.013e-06 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054240+0000", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040256609999999998 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054262+0000", 2024-04-25T15:53:05.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2691e-05 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054277+0000", 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4464000000000001e-05 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054365+0000", 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8657000000000002e-05 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054400+0000", 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4505999999999998e-05 2024-04-25T15:53:05.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065391+0000", 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991287000000001 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065462+0000", 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1155000000000006e-05 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:260 3.12 e66/40 3:4bec3886:::10000000002.00000000:head v 66'8, mlcod=61'6)", 2024-04-25T15:53:05.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5586681630000001, 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019802660999999999, 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698470+0000", 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704135+0000", 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056657490000000003 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704159+0000", 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3903000000000001e-05 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704176+0000", 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6461e-05 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704290+0000", 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011446999999999999 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704336+0000", 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5558e-05 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718190+0000", 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013854577999999999 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718270+0000", 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9377999999999998e-05 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:267 3.18 e67/18 3:19a74f15:::10000000002.00000007:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.2411954700000001, 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015975057000000001, 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:05.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019042+0000", 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031019680000000001 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019059+0000", 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6631000000000001e-05 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019075+0000", 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6586e-05 2024-04-25T15:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019143+0000", 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7626999999999995e-05 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019174+0000", 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1078999999999998e-05 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031852+0000", 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012678233000000001 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031915+0000", 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2422999999999995e-05 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:269 3.2c e67/54 3:354fa1eb:::10000000002.00000009:head v 67'3, mlcod=59'1)", 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1441613950000002, 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014629644000000001, 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.310 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4000000000000001e-07 2024-04-25T15:53:05.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119407+0000", 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064328099999999997 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119437+0000", 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9451e-05 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119460+0000", 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640999999999999e-05 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119565+0000", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010474 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119600+0000", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4851999999999998e-05 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127513+0000", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079133090000000003 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127603+0000", 2024-04-25T15:53:05.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0100999999999997e-05 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:275 3.30 e67/18 3:0ca36612:::10000000002.0000000f:head v 67'7, mlcod=62'5)", 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8792228140000002, 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015157541, 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377913+0000", 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2499999999999995e-07 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383613+0000", 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057000189999999997 2024-04-25T15:53:05.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383639+0000", 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6475e-05 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383656+0000", 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.698e-05 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383760+0000", 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103428 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383793+0000", 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3158999999999999e-05 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393003+0000", 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092102839999999991 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393070+0000", 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6670999999999999e-05 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:276 3.1f e67/40 3:f8cab2eb:::10000000002.00000010:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8254897369999998, 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015350925, 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431648+0000", 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3649999999999998e-06 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435504+0000", 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003856221 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435533+0000", 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9461000000000001e-05 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435561+0000", 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7684000000000001e-05 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435659+0000", 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7588000000000001e-05 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435694+0000", 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5185999999999998e-05 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446705+0000", 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011010804000000001 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446996+0000", 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029161600000000002 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.321 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:353 3.1 e69/54 3:838ab1fd:::10000000002.0000005d:head v 69'7, mlcod=62'5)", 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.8599896739999999, 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017336667, 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:05.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397146+0000", 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3700000000000001e-07 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398713+0000", 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001566764 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398740+0000", 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7317e-05 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398777+0000", 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6885999999999999e-05 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398898+0000", 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121723 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398932+0000", 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3491999999999997e-05 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414418+0000", 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015486491 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414482+0000", 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3657000000000004e-05 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:357 3.34 e69/54 3:2d119e8b:::10000000002.00000061:head v 69'10, mlcod=67'8)", 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.6858174849999998, 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015675116999999999, 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571319+0000", 2024-04-25T15:53:05.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.139e-06 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576161+0000", 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048417979999999996 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576182+0000", 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1262000000000001e-05 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576193+0000", 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1197e-05 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576262+0000", 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9165000000000006e-05 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576283+0000", 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0582999999999999e-05 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586920+0000", 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010637275999999999 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586993+0000", 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2696999999999999e-05 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:372 3.10 e69/18 3:0b92729d:::10000000002.00000070:head v 69'7, mlcod=67'5)", 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.1391201149999999, 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015303605, 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118016+0000", 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1930000000000001e-06 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123008+0000", 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049919639999999998 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123026+0000", 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8034999999999999e-05 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123040+0000", 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3895e-05 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123138+0000", 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7929000000000004e-05 2024-04-25T15:53:05.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123177+0000", 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8587999999999997e-05 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133262+0000", 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010085782 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133319+0000", 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6218999999999998e-05 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.332 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:397 3.20 e69/18 3:06fc074c:::10000000002.00000089:head v 69'12, mlcod=68'10)", 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.20863153500000001, 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015916966000000001, 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048506+0000", 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0439999999999998e-06 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054867+0000", 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063613209999999996 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054891+0000", 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4320000000000001e-05 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054921+0000", 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9535000000000001e-05 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054991+0000", 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0314999999999993e-05 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.055025+0000", 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3809000000000001e-05 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064353+0000", 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093282839999999992 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064421+0000", 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7337999999999998e-05 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:05.336 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:05.337 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:05.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:05.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.326+0000 7f2c57484640 1 Processor -- start 2024-04-25T15:53:05.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.326+0000 7f2c57484640 1 -- start start 2024-04-25T15:53:05.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c57484640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c50156f10 0x7f2c500cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c57484640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c5015d210 0x7f2c5015d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c57484640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c50160fa0 con 0x7f2c500cc600 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c57484640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2c50157320 con 0x7f2c50156f10 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c57484640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2c500c6d10 con 0x7f2c5015d210 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55220640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c50156f10 0x7f2c500cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55a21640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2c500cc600 0x7f2c50073c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40118/0 (socket says 172.21.15.67:40118) 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55a21640 1 -- 172.21.15.67:0/3847511498 learned_addr learned my addr 172.21.15.67:0/3847511498 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/3847511498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c5015d210 0x7f2c5015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55220640 1 -- 172.21.15.67:0/3847511498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c5015d210 msgr2=0x7f2c5015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c47fff640 1 -- 172.21.15.67:0/3847511498 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3887981295 0 0) 0x7f2c50160fa0 con 0x7f2c500cc600 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55220640 1 --2- 172.21.15.67:0/3847511498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c5015d210 0x7f2c5015d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55220640 1 -- 172.21.15.67:0/3847511498 >> v1:172.21.15.67:6789/0 conn(0x7f2c500cc600 legacy=0x7f2c50073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55220640 1 -- 172.21.15.67:0/3847511498 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c50160420 con 0x7f2c50156f10 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/3847511498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c5015d210 0x7f2c5015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.330+0000 7f2c55220640 1 --2- 172.21.15.67:0/3847511498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c50156f10 0x7f2c500cbfc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0097c0 tx=0x7f2c4c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65acc7a7ce1e1d74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c47fff640 1 -- 172.21.15.67:0/3847511498 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0b0070 con 0x7f2c50156f10 2024-04-25T15:53:05.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c47fff640 1 -- 172.21.15.67:0/3847511498 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4c0ab600 con 0x7f2c50156f10 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c47fff640 1 -- 172.21.15.67:0/3847511498 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0ab9d0 con 0x7f2c50156f10 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 -- 172.21.15.67:0/3847511498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c50156f10 msgr2=0x7f2c500cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 --2- 172.21.15.67:0/3847511498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c50156f10 0x7f2c500cbfc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0097c0 tx=0x7f2c4c01c4a0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 -- 172.21.15.67:0/3847511498 shutdown_connections 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 --2- 172.21.15.67:0/3847511498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c5015d210 0x7f2c5015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 --2- 172.21.15.67:0/3847511498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c50156f10 0x7f2c500cbfc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 -- 172.21.15.67:0/3847511498 >> 172.21.15.67:0/3847511498 conn(0x7f2c50068c70 msgr2=0x7f2c50071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 -- 172.21.15.67:0/3847511498 shutdown_connections 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 -- 172.21.15.67:0/3847511498 wait complete. 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 Processor -- start 2024-04-25T15:53:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.334+0000 7f2c57484640 1 -- start start 2024-04-25T15:53:05.343 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:05.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c500cc600 0x7f2c501fa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c50156f10 0x7f2c50149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5015d210 0x7f2c500c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c500cce20 con 0x7f2c5015d210 2024-04-25T15:53:05.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5015ccf0 con 0x7f2c500cc600 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2c501600c0 con 0x7f2c50156f10 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5015d210 0x7f2c500c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5015d210 0x7f2c500c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49214/0 (socket says 172.21.15.67:49214) 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 -- 172.21.15.67:0/2256022551 learned_addr learned my addr 172.21.15.67:0/2256022551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c50156f10 0x7f2c50149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c500cc600 0x7f2c501fa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c50156f10 msgr2=0x7f2c50149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c50156f10 0x7f2c50149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c500cc600 msgr2=0x7f2c501fa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c500cc600 0x7f2c501fa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c50148dd0 con 0x7f2c5015d210 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c55a21640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5015d210 0x7f2c500c8410 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800c6d0 tx=0x7f2c4800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:05.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c48018070 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c48004d70 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c501490b0 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800a880 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c50147870 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f2c5006a480 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.338+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2c4801b070 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c45ffb640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2c3406f430 0x7f2c34071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5886+0+0 (secure 0 0 0) 0x7f2c4808ea30 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v69) ==== 54+0+16 (secure 0 0 0) 0x7f2c48093240 con 0x7f2c5015d210 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2c3406f430 0x7f2c34071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f2c5014cda0 0x7f2c50143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c50071ca0 con 0x7f2c5014cda0 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f2c5014cda0 0x7f2c50143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f2c5014cda0 0x7f2c50143300 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2c3406f430 0x7f2c34071940 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0ad000 tx=0x7f2c4c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:05.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.342+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c50071ca0 con 0x7f2c5014cda0 2024-04-25T15:53:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.366+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c5015c9e0 con 0x7f2c5014cda0 2024-04-25T15:53:05.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.370+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c5015c9e0 con 0x7f2c5014cda0 2024-04-25T15:53:05.370 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.370+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2c5015ef10 0x7f2c5015f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.370+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f2c50143d50 con 0x7f2c5015ef10 2024-04-25T15:53:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.370+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2c5015ef10 0x7f2c5015f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.370+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2c5015ef10 0x7f2c5015f320 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.370+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c50143d50 con 0x7f2c5015ef10 2024-04-25T15:53:05.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.374+0000 7f5c867fc640 1 -- 172.21.15.67:0/2270525467 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v70) ==== 87+0+0 (secure 0 0 0) 0x7f5c800f5650 con 0x7f5c900c96e0 2024-04-25T15:53:05.375 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:53:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 msgr2=0x7f5c74071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c000f70 tx=0x7f5c8c0079c0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c96e0 msgr2=0x7f5c900c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c96e0 0x7f5c900c42f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5c80019020 tx=0x7f5c800191b0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 shutdown_connections 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c7406f400 0x7f5c74071910 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c90160800 0x7f5c9015ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c900c96e0 0x7f5c900c42f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 --2- 172.21.15.67:0/2270525467 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c900c92d0 0x7f5c900c0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 >> 172.21.15.67:0/2270525467 conn(0x7f5c90070db0 msgr2=0x7f5c90155f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 shutdown_connections 2024-04-25T15:53:05.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.378+0000 7f5c98076640 1 -- 172.21.15.67:0/2270525467 wait complete. 2024-04-25T15:53:05.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.390+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c500cba60 con 0x7f2c5015ef10 2024-04-25T15:53:05.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c500cba60 con 0x7f2c5015ef10 2024-04-25T15:53:05.394 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 0x7f2c500634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f2c5015fd70 con 0x7f2c50063090 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 0x7f2c500634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c54a1f640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 msgr2=0x7f2c500634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 23 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c54a1f640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 msgr2=0x7f2c500634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 0x7f2c500634a0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:53:05.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.394+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 0x7f2c500634a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:53:05.439 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:05.439+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:05.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:05.561 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:05.562+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:05.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.598+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 0x7f2c500634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.598+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 0x7f2c500634a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:05.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.598+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c5015fd70 con 0x7f2c50063090 2024-04-25T15:53:05.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.618+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c50072360 con 0x7f2c50063090 2024-04-25T15:53:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.618+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c50072360 con 0x7f2c50063090 2024-04-25T15:53:05.621 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.622+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2c5006d750 0x7f2c5006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.622+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f2c5014a640 con 0x7f2c5006d750 2024-04-25T15:53:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.622+0000 7f2c55a21640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2c5006d750 0x7f2c5006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.622+0000 7f2c55a21640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2c5006d750 0x7f2c5006db60 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:05.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.622+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c5014a640 con 0x7f2c5006d750 2024-04-25T15:53:05.641 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:05.639+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:05.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:05.646+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c5006e0a0 con 0x7f2c5006d750 2024-04-25T15:53:05.673 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:05.673 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:05.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:05.781 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:05.782+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:05.842 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:05.839+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:05.893 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:05.893 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:05.913 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:05.957 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:05.958+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:06.044 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:06.042+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:06.123 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:06.124 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:06.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:06.146 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:06.142+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:06.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.158+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c5006e0a0 con 0x7f2c5006d750 2024-04-25T15:53:06.160 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:06.161 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:06.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.158+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f2c5002c9e0 0x7f2c5002cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:06.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.158+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f2c5006e720 con 0x7f2c5002c9e0 2024-04-25T15:53:06.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.158+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f2c5002c9e0 0x7f2c5002cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:06.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.162+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f2c5002c9e0 0x7f2c5002cdf0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:06.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.162+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c5006e720 con 0x7f2c5002c9e0 2024-04-25T15:53:06.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.178+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c5002d330 con 0x7f2c5002c9e0 2024-04-25T15:53:06.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.182+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c5002d330 con 0x7f2c5002c9e0 2024-04-25T15:53:06.183 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:06.183 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:06.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.182+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2c5002d840 0x7f2c5020cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:06.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.182+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f2c5002de90 con 0x7f2c5002d840 2024-04-25T15:53:06.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.182+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2c5002d840 0x7f2c5020cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:06.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.186+0000 7f2c54a1f640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2c5002d840 0x7f2c5020cb00 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:06.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.186+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c5002de90 con 0x7f2c5002d840 2024-04-25T15:53:06.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.214+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c5020d040 con 0x7f2c5002d840 2024-04-25T15:53:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.218+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c5020d040 con 0x7f2c5002d840 2024-04-25T15:53:06.220 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.218+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f2c5020d760 0x7f2c5020fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.218+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f2c50212a80 con 0x7f2c5020d760 2024-04-25T15:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.222+0000 7f2c55a21640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f2c5020d760 0x7f2c5020fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.222+0000 7f2c55a21640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f2c5020d760 0x7f2c5020fb80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.222+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c50212a80 con 0x7f2c5020d760 2024-04-25T15:53:06.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.238+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c50076a30 con 0x7f2c5020d760 2024-04-25T15:53:06.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.242+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c50076a30 con 0x7f2c5020d760 2024-04-25T15:53:06.243 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:06.244 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:06.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.242+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f2c502105d0 0x7f2c502109e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:06.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.242+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f2c50215e50 con 0x7f2c502105d0 2024-04-25T15:53:06.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.242+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f2c502105d0 0x7f2c502109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:06.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.242+0000 7f2c55220640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f2c502105d0 0x7f2c502109e0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:06.245 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:06.243+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:06.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.246+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c50215e50 con 0x7f2c502105d0 2024-04-25T15:53:06.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.262+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f2c50211030 con 0x7f2c502105d0 2024-04-25T15:53:06.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c45ffb640 1 -- 172.21.15.67:0/2256022551 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f2c50211030 con 0x7f2c502105d0 2024-04-25T15:53:06.268 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f2c5014cda0 msgr2=0x7f2c50143300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f2c5014cda0 0x7f2c50143300 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2c5015ef10 msgr2=0x7f2c5015f320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f2c5015ef10 0x7f2c5015f320 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 msgr2=0x7f2c500634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f2c50063090 0x7f2c500634a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2c5006d750 msgr2=0x7f2c5006db60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f2c5006d750 0x7f2c5006db60 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c55a21640 1 -- 172.21.15.67:0/2256022551 reap_dead start 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f2c5002c9e0 msgr2=0x7f2c5002cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f2c5002c9e0 0x7f2c5002cdf0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c55a21640 1 -- 172.21.15.67:0/2256022551 reap_dead start 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2c5002d840 msgr2=0x7f2c5020cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f2c5002d840 0x7f2c5020cb00 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f2c5020d760 msgr2=0x7f2c5020fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f2c5020d760 0x7f2c5020fb80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f2c502105d0 msgr2=0x7f2c502109e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f2c502105d0 0x7f2c502109e0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2c3406f430 msgr2=0x7f2c34071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2c3406f430 0x7f2c34071940 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0ad000 tx=0x7f2c4c008040 comp rx=0 tx=0).stop 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5015d210 msgr2=0x7f2c500c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 --2- 172.21.15.67:0/2256022551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c5015d210 0x7f2c500c8410 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800c6d0 tx=0x7f2c4800cb90 comp rx=0 tx=0).stop 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c55a21640 1 -- 172.21.15.67:0/2256022551 reap_dead start 2024-04-25T15:53:06.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 shutdown_connections 2024-04-25T15:53:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 >> 172.21.15.67:0/2256022551 conn(0x7f2c50068c70 msgr2=0x7f2c50073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 shutdown_connections 2024-04-25T15:53:06.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:06.266+0000 7f2c57484640 1 -- 172.21.15.67:0/2256022551 wait complete. 2024-04-25T15:53:06.344 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:06.344 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:06.346 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:06.346+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:06.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:06.448 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:06.447+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:06.548 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:06.546+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:06.554 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:06.554 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:06.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:06.650 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:06.647+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:06.751 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:06.750+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:06.768 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:06.768 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:06.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:06.852 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:06.851+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:06.952 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:06.950+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:06.979 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:06.979 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:07.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:07.054 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:07.051+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:07.155 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:07.151+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:07.202 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:07.202 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:07.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:07.271 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:07.271+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:07.356 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:07.355+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:07.373 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:07.374 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:07.374 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-25T15:53:07.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-25T15:53:07.393 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:07.393 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:07.403 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:53:07.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:07.459 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:07.459+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:07.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9158a25640 1 Processor -- start 2024-04-25T15:53:07.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9158a25640 1 -- start start 2024-04-25T15:53:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9158a25640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f91540c92d0 0x7f9154072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9158a25640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91540c96e0 0x7f9154073300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9158a25640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915406d8e0 con 0x7f9154160800 2024-04-25T15:53:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9158a25640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f915406dbc0 con 0x7f91540c96e0 2024-04-25T15:53:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9158a25640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9154073c70 con 0x7f91540c92d0 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9152d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91540c96e0 0x7f9154073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9152d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91540c96e0 0x7f9154073300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49224/0 (socket says 172.21.15.67:49224) 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9152d9d640 1 -- 172.21.15.67:0/350617959 learned_addr learned my addr 172.21.15.67:0/350617959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9151d9b640 1 --2- 172.21.15.67:0/350617959 >> v2:172.21.15.176:3300/0 conn(0x7f91540c92d0 0x7f9154072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9152d9d640 1 -- 172.21.15.67:0/350617959 >> v2:172.21.15.176:3300/0 conn(0x7f91540c92d0 msgr2=0x7f9154072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9152d9d640 1 --2- 172.21.15.67:0/350617959 >> v2:172.21.15.176:3300/0 conn(0x7f91540c92d0 0x7f9154072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9152d9d640 1 -- 172.21.15.67:0/350617959 >> v1:172.21.15.176:6789/0 conn(0x7f9154160800 legacy=0x7f915406d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9152d9d640 1 -- 172.21.15.67:0/350617959 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9154073f50 con 0x7f91540c96e0 2024-04-25T15:53:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.534+0000 7f9151d9b640 1 --2- 172.21.15.67:0/350617959 >> v2:172.21.15.176:3300/0 conn(0x7f91540c92d0 0x7f9154072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:07.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9152d9d640 1 --2- 172.21.15.67:0/350617959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 0x7f9154073300 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f915406dd40 tx=0x7f914c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4986d697c0cae04f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f915159a640 1 -- 172.21.15.67:0/350617959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0a5710 con 0x7f91540c96e0 2024-04-25T15:53:07.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f915159a640 1 -- 172.21.15.67:0/350617959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914c0a58b0 con 0x7f91540c96e0 2024-04-25T15:53:07.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f915159a640 1 -- 172.21.15.67:0/350617959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0a5c80 con 0x7f91540c96e0 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 -- 172.21.15.67:0/350617959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 msgr2=0x7f9154073300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 --2- 172.21.15.67:0/350617959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 0x7f9154073300 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f915406dd40 tx=0x7f914c01c960 comp rx=0 tx=0).stop 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 -- 172.21.15.67:0/350617959 shutdown_connections 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 --2- 172.21.15.67:0/350617959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 0x7f9154073300 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 --2- 172.21.15.67:0/350617959 >> v2:172.21.15.176:3300/0 conn(0x7f91540c92d0 0x7f9154072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 -- 172.21.15.67:0/350617959 >> 172.21.15.67:0/350617959 conn(0x7f9154070db0 msgr2=0x7f91540ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 -- 172.21.15.67:0/350617959 shutdown_connections 2024-04-25T15:53:07.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.538+0000 7f9158a25640 1 -- 172.21.15.67:0/350617959 wait complete. 2024-04-25T15:53:07.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 Processor -- start 2024-04-25T15:53:07.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 -- start start 2024-04-25T15:53:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91540c92d0 0x7f91541f9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 0x7f9154157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9154160800 0x7f915414ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9154160d90 con 0x7f91540c96e0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9154160c10 con 0x7f9154160800 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f91541699a0 con 0x7f91540c92d0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 0x7f9154157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 0x7f9154157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49234/0 (socket says 172.21.15.67:49234) 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 -- 172.21.15.67:0/2264931173 learned_addr learned my addr 172.21.15.67:0/2264931173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9152d9d640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9154160800 0x7f915414ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f915259c640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91540c92d0 0x7f91541f9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 -- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91540c92d0 msgr2=0x7f91541f9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91540c92d0 0x7f91541f9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 -- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9154160800 msgr2=0x7f915414ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9154160800 0x7f915414ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 -- 172.21.15.67:0/2264931173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91541f9a60 con 0x7f91540c96e0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9151d9b640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91540c96e0 0x7f9154157c40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f914800d800 tx=0x7f914800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148018020 con 0x7f91540c96e0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91480096c0 con 0x7f91540c96e0 2024-04-25T15:53:07.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148009ab0 con 0x7f91540c96e0 2024-04-25T15:53:07.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 -- 172.21.15.67:0/2264931173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91540c8770 con 0x7f91540c96e0 2024-04-25T15:53:07.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.542+0000 7f9158a25640 1 -- 172.21.15.67:0/2264931173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91540c7d50 con 0x7f91540c96e0 2024-04-25T15:53:07.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9158a25640 1 -- 172.21.15.67:0/2264931173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f915406a480 con 0x7f91540c96e0 2024-04-25T15:53:07.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f914801b020 con 0x7f91540c96e0 2024-04-25T15:53:07.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9142ffd640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f913406f400 0x7f9134071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5886+0+0 (secure 0 0 0) 0x7f914808f6d0 con 0x7f91540c96e0 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9142ffd640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f91340778b0 0x7f9134079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9148010e50 con 0x7f91340778b0 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f9148096050 con 0x7f91540c96e0 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f915259c640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f913406f400 0x7f9134071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9152d9d640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f91340778b0 0x7f9134079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f9152d9d640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f91340778b0 0x7f9134079da0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.546+0000 7f915259c640 1 --2- 172.21.15.67:0/2264931173 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f913406f400 0x7f9134071910 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9144001770 tx=0x7f9144009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.550+0000 7f9142ffd640 1 -- 172.21.15.67:0/2264931173 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9148010e50 con 0x7f91340778b0 2024-04-25T15:53:07.557 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:07.558+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77f8cc640 1 Processor -- start 2024-04-25T15:53:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77f8cc640 1 -- start start 2024-04-25T15:53:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77f8cc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb778073bf0 0x7fb77806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77f8cc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb77806dc90 0x7fb778163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77f8cc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7780c77a0 con 0x7fb7780caf60 2024-04-25T15:53:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77f8cc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb7780cab60 con 0x7fb778073bf0 2024-04-25T15:53:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77f8cc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb778159620 con 0x7fb77806dc90 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77ce67640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb778073bf0 0x7fb77806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77d668640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7780caf60 0x7fb77815d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40120/0 (socket says 172.21.15.67:40120) 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77d668640 1 -- 172.21.15.67:0/1560464080 learned_addr learned my addr 172.21.15.67:0/1560464080 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77de69640 1 --2- 172.21.15.67:0/1560464080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb77806dc90 0x7fb778163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77ce67640 1 -- 172.21.15.67:0/1560464080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb77806dc90 msgr2=0x7fb778163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77ce67640 1 --2- 172.21.15.67:0/1560464080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb77806dc90 0x7fb778163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77ce67640 1 -- 172.21.15.67:0/1560464080 >> v1:172.21.15.67:6789/0 conn(0x7fb7780caf60 legacy=0x7fb77815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb76ffff640 1 -- 172.21.15.67:0/1560464080 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840130655 0 0) 0x7fb7780c77a0 con 0x7fb7780caf60 2024-04-25T15:53:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.578+0000 7fb77ce67640 1 -- 172.21.15.67:0/1560464080 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7780c7a60 con 0x7fb778073bf0 2024-04-25T15:53:07.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77ce67640 1 --2- 172.21.15.67:0/1560464080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778073bf0 0x7fb77806d750 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb7680097c0 tx=0x7fb76801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63dbb4864ae42dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb76ffff640 1 -- 172.21.15.67:0/1560464080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7680b0070 con 0x7fb778073bf0 2024-04-25T15:53:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb76ffff640 1 -- 172.21.15.67:0/1560464080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7680ab600 con 0x7fb778073bf0 2024-04-25T15:53:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb76ffff640 1 -- 172.21.15.67:0/1560464080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7680ab9d0 con 0x7fb778073bf0 2024-04-25T15:53:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-04-25T15:53:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-04-25T15:53:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:bench [] [] [] [] 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs debug_inject_read_zeros 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs files list 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs stats 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:bluestore allocator dump block 2024-04-25T15:53:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 -- 172.21.15.67:0/1560464080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778073bf0 msgr2=0x7fb77806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/1560464080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778073bf0 0x7fb77806d750 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb7680097c0 tx=0x7fb76801c4a0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 -- 172.21.15.67:0/1560464080 shutdown_connections 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/1560464080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb77806dc90 0x7fb778163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/1560464080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb778073bf0 0x7fb77806d750 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 -- 172.21.15.67:0/1560464080 >> 172.21.15.67:0/1560464080 conn(0x7fb778070db0 msgr2=0x7fb7780ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 -- 172.21.15.67:0/1560464080 shutdown_connections 2024-04-25T15:53:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.582+0000 7fb77f8cc640 1 -- 172.21.15.67:0/1560464080 wait complete. 2024-04-25T15:53:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 Processor -- start 2024-04-25T15:53:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 -- start start 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77806dc90 0x7fb7781f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb778073bf0 0x7fb7780ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7780caf60 0x7fb77814d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb77815a570 con 0x7fb77806dc90 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb77815a3f0 con 0x7fb778073bf0 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb778166c20 con 0x7fb7780caf60 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77806dc90 0x7fb7781f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77806dc90 0x7fb7781f4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49252/0 (socket says 172.21.15.67:49252) 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 -- 172.21.15.67:0/2403693307 learned_addr learned my addr 172.21.15.67:0/2403693307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77ce67640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb778073bf0 0x7fb7780ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77de69640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7780caf60 0x7fb77814d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 -- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7780caf60 msgr2=0x7fb77814d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7780caf60 0x7fb77814d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 -- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb778073bf0 msgr2=0x7fb7780ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb778073bf0 0x7fb7780ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 -- 172.21.15.67:0/2403693307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7781f7220 con 0x7fb77806dc90 2024-04-25T15:53:07.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77ce67640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb778073bf0 0x7fb7780ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:07.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77d668640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77806dc90 0x7fb7781f4e50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb774007fd0 tx=0x7fb77400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T15:53:07.591 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-25T15:53:07.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-25T15:53:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb76dffb640 1 -- 172.21.15.67:0/2403693307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774017070 con 0x7fb77806dc90 2024-04-25T15:53:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb76dffb640 1 -- 172.21.15.67:0/2403693307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb774005d10 con 0x7fb77806dc90 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb76dffb640 1 -- 172.21.15.67:0/2403693307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774007310 con 0x7fb77806dc90 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.586+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7781f5390 con 0x7fb77806dc90 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.590+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77815e330 con 0x7fb77806dc90 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.590+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77806a480 con 0x7fb77806dc90 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.590+0000 7fb76dffb640 1 -- 172.21.15.67:0/2403693307 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb77401a070 con 0x7fb77806dc90 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.590+0000 7fb76dffb640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb75c06f400 0x7fb75c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.590+0000 7fb76dffb640 1 -- 172.21.15.67:0/2403693307 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5886+0+0 (secure 0 0 0) 0x7fb77408e620 con 0x7fb77806dc90 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.590+0000 7fb77ce67640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb75c06f400 0x7fb75c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.590+0000 7fb77ce67640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb75c06f400 0x7fb75c071910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb7680ad000 tx=0x7fb7680a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.598+0000 7fb76dffb640 1 -- 172.21.15.67:0/2403693307 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb77405b580 con 0x7fb77806dc90 2024-04-25T15:53:07.623 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:07.623 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:07.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:07.691 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:07.691+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f82414b5640 1 Processor -- start 2024-04-25T15:53:07.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f82414b5640 1 -- start start 2024-04-25T15:53:07.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f82414b5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 0x7f823c0cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f82414b5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f823c15de10 con 0x7f823c0ca540 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f82414b5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f823c072390 con 0x7f823c0cc100 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f82414b5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f823c15ebb0 con 0x7f823c071bc0 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f823b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f823c0ca540 0x7f823c164fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40134/0 (socket says 172.21.15.67:40134) 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f823b7fe640 1 -- 172.21.15.67:0/3545150566 learned_addr learned my addr 172.21.15.67:0/3545150566 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f823affd640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 0x7f823c0cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 449052206 0 0) 0x7f823c15de10 con 0x7f823c0ca540 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8224003530 con 0x7f823c0ca540 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1310341145 0 0) 0x7f8224003530 con 0x7f823c0ca540 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 msgr2=0x7f823c0cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 0x7f823c0cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 >> v1:172.21.15.176:6789/0 conn(0x7f823c0cc100 legacy=0x7f823c15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c15e0d0 con 0x7f823c0ca540 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8234001a20 con 0x7f823c0ca540 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 >> v1:172.21.15.67:6789/0 conn(0x7f823c0ca540 legacy=0x7f823c164fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 0x7f82240039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8224003f50 0x7f8224004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f8224005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f823c15ebb0 con 0x7f8224004d50 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f822400bae0 con 0x7f8224003f50 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.746+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f822400bce0 con 0x7f823c071bc0 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823b7fe640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f8224005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823affd640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 0x7f82240039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823a7fc640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8224003f50 0x7f8224004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823b7fe640 1 -- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 msgr2=0x7f82240039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823b7fe640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c071bc0 0x7f82240039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823b7fe640 1 -- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8224003f50 msgr2=0x7f8224004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823b7fe640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8224003f50 0x7f8224004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823b7fe640 1 -- 172.21.15.67:0/3545150566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8224003530 con 0x7f8224004d50 2024-04-25T15:53:07.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823a7fc640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8224003f50 0x7f8224004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:07.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f823b7fe640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f8224005160 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f823400a870 tx=0x7f823401fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193e7d3c46268489 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823400a4c0 con 0x7f8224004d50 2024-04-25T15:53:07.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82340aa6f0 con 0x7f8224004d50 2024-04-25T15:53:07.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.750+0000 7f8239ffb640 1 -- 172.21.15.67:0/3545150566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82340aaac0 con 0x7f8224004d50 2024-04-25T15:53:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- 172.21.15.67:0/3545150566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 msgr2=0x7f8224005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 --2- 172.21.15.67:0/3545150566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f8224005160 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f823400a870 tx=0x7f823401fac0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f823b7fe640 1 -- 172.21.15.67:0/3545150566 reap_dead start 2024-04-25T15:53:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- 172.21.15.67:0/3545150566 shutdown_connections 2024-04-25T15:53:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- 172.21.15.67:0/3545150566 >> 172.21.15.67:0/3545150566 conn(0x7f823c0700b0 msgr2=0x7f823c0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- 172.21.15.67:0/3545150566 shutdown_connections 2024-04-25T15:53:07.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- 172.21.15.67:0/3545150566 wait complete. 2024-04-25T15:53:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 Processor -- start 2024-04-25T15:53:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- start start 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f823c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c158a90 0x7f823c15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f823c153100 0x7f823c153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f823c0c93b0 con 0x7f8224004d50 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.754+0000 7f82414b5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f823c15e720 con 0x7f823c153100 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82414b5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f823c071fd0 con 0x7f823c158a90 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f823c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f823c0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49282/0 (socket says 172.21.15.67:49282) 2024-04-25T15:53:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 -- 172.21.15.67:0/2306096570 learned_addr learned my addr 172.21.15.67:0/2306096570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823b7fe640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f823c153100 0x7f823c153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 -- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c158a90 msgr2=0x7f823c15d160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f823c158a90 0x7f823c15d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 -- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f823c153100 msgr2=0x7f823c153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f823c153100 0x7f823c153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 -- 172.21.15.67:0/2306096570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823c1f1a30 con 0x7f8224004d50 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f823affd640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8224004d50 0x7f823c0d0300 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f822c009b70 tx=0x7f822c015510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c01f020 con 0x7f8224004d50 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822c015da0 con 0x7f8224004d50 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c01b6c0 con 0x7f8224004d50 2024-04-25T15:53:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82414b5640 1 -- 172.21.15.67:0/2306096570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823c158790 con 0x7f8224004d50 2024-04-25T15:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82414b5640 1 -- 172.21.15.67:0/2306096570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f823c1f1ff0 con 0x7f8224004d50 2024-04-25T15:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82414b5640 1 -- 172.21.15.67:0/2306096570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f823c06a480 con 0x7f8224004d50 2024-04-25T15:53:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f822c022020 con 0x7f8224004d50 2024-04-25T15:53:07.761 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:07.759+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f822806f430 0x7f8228071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5886+0+0 (secure 0 0 0) 0x7f822c093d70 con 0x7f8224004d50 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f82280778e0 0x7f8228079dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f822807a4a0 con 0x7f82280778e0 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.758+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f822c096470 con 0x7f8224004d50 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.762+0000 7f823b7fe640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f82280778e0 0x7f8228079dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.762+0000 7f823a7fc640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f822806f430 0x7f8228071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.762+0000 7f823b7fe640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f82280778e0 0x7f8228079dd0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.762+0000 7f823a7fc640 1 --2- 172.21.15.67:0/2306096570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f822806f430 0x7f8228071940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f823c15a910 tx=0x7f8230009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.762+0000 7f82237fe640 1 -- 172.21.15.67:0/2306096570 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f822807a4a0 con 0x7f82280778e0 2024-04-25T15:53:07.795 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:07.795 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:bench [] [] [] [] 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs debug_inject_read_zeros 2024-04-25T15:53:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs files list 2024-04-25T15:53:07.799 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs stats 2024-04-25T15:53:07.799 INFO:teuthology.orchestra.run.smithi067.stderr:bluestore allocator dump block 2024-04-25T15:53:07.799 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2024-04-25T15:53:07.806 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T15:53:07.806 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-25T15:53:07.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-25T15:53:07.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:07.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.822+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb7780c8a40 con 0x7fb77806dc90 2024-04-25T15:53:07.869 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:07.870+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:07.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.874+0000 7fb76dffb640 1 -- 172.21.15.67:0/2403693307 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v71) ==== 97+0+0 (secure 0 0 0) 0x7fb77405ae80 con 0x7fb77806dc90 2024-04-25T15:53:07.875 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb75c06f400 msgr2=0x7fb75c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb75c06f400 0x7fb75c071910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb7680ad000 tx=0x7fb7680a6000 comp rx=0 tx=0).stop 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77806dc90 msgr2=0x7fb7781f4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77806dc90 0x7fb7781f4e50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb774007fd0 tx=0x7fb77400c8e0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 shutdown_connections 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb75c06f400 0x7fb75c071910 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7780caf60 0x7fb77814d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb778073bf0 0x7fb7780ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 --2- 172.21.15.67:0/2403693307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77806dc90 0x7fb7781f4e50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 >> 172.21.15.67:0/2403693307 conn(0x7fb778070db0 msgr2=0x7fb77815b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 shutdown_connections 2024-04-25T15:53:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.878+0000 7fb77f8cc640 1 -- 172.21.15.67:0/2403693307 wait complete. 2024-04-25T15:53:07.961 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:07.962+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f755049a640 1 Processor -- start 2024-04-25T15:53:07.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f755049a640 1 -- start start 2024-04-25T15:53:07.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f755049a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f754815a720 0x7f754815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f755049a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75480ca540 0x7f75480ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f755049a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f754815de10 con 0x7f7548071bc0 2024-04-25T15:53:07.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f755049a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f754815ebb0 con 0x7f75480ca540 2024-04-25T15:53:07.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f755049a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f754815e720 con 0x7f754815a720 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754ea37640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75480ca540 0x7f75480ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754ea37640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75480ca540 0x7f75480ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49298/0 (socket says 172.21.15.67:49298) 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754ea37640 1 -- 172.21.15.67:0/4287138544 learned_addr learned my addr 172.21.15.67:0/4287138544 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754da35640 1 --2- 172.21.15.67:0/4287138544 >> v2:172.21.15.176:3300/0 conn(0x7f754815a720 0x7f754815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754ea37640 1 -- 172.21.15.67:0/4287138544 >> v2:172.21.15.176:3300/0 conn(0x7f754815a720 msgr2=0x7f754815ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754ea37640 1 --2- 172.21.15.67:0/4287138544 >> v2:172.21.15.176:3300/0 conn(0x7f754815a720 0x7f754815ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754ea37640 1 -- 172.21.15.67:0/4287138544 >> v1:172.21.15.176:6789/0 conn(0x7f7548071bc0 legacy=0x7f75480cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.978+0000 7f754ea37640 1 -- 172.21.15.67:0/4287138544 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7548072390 con 0x7f75480ca540 2024-04-25T15:53:07.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f754ea37640 1 --2- 172.21.15.67:0/4287138544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75480ca540 0x7f75480ca950 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f75480767b0 tx=0x7f754001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791960f9fdf7baba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f754d234640 1 -- 172.21.15.67:0/4287138544 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75400a5760 con 0x7f75480ca540 2024-04-25T15:53:07.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f754d234640 1 -- 172.21.15.67:0/4287138544 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75400a5900 con 0x7f75480ca540 2024-04-25T15:53:07.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f754d234640 1 -- 172.21.15.67:0/4287138544 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75400a5cd0 con 0x7f75480ca540 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 -- 172.21.15.67:0/4287138544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75480ca540 msgr2=0x7f75480ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 --2- 172.21.15.67:0/4287138544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75480ca540 0x7f75480ca950 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f75480767b0 tx=0x7f754001c990 comp rx=0 tx=0).stop 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 -- 172.21.15.67:0/4287138544 shutdown_connections 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 --2- 172.21.15.67:0/4287138544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75480ca540 0x7f75480ca950 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 --2- 172.21.15.67:0/4287138544 >> v2:172.21.15.176:3300/0 conn(0x7f754815a720 0x7f754815ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 -- 172.21.15.67:0/4287138544 >> 172.21.15.67:0/4287138544 conn(0x7f75480700b0 msgr2=0x7f75480704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 -- 172.21.15.67:0/4287138544 shutdown_connections 2024-04-25T15:53:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.982+0000 7f755049a640 1 -- 172.21.15.67:0/4287138544 wait complete. 2024-04-25T15:53:07.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 Processor -- start 2024-04-25T15:53:07.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 -- start start 2024-04-25T15:53:07.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7548071bc0 0x7f754814ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75480ca540 0x7f754806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f754815a720 0x7f754815cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7548071fd0 con 0x7f7548071bc0 2024-04-25T15:53:07.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f75480cc2d0 con 0x7f754815a720 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7548072150 con 0x7f75480ca540 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7548071bc0 0x7f754814ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7548071bc0 0x7f754814ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49312/0 (socket says 172.21.15.67:49312) 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 -- 172.21.15.67:0/1169580542 learned_addr learned my addr 172.21.15.67:0/1169580542 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754da35640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75480ca540 0x7f754806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754ea37640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f754815a720 0x7f754815cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 -- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75480ca540 msgr2=0x7f754806f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75480ca540 0x7f754806f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 -- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f754815a720 msgr2=0x7f754815cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f754815a720 0x7f754815cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 -- 172.21.15.67:0/1169580542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754814f1d0 con 0x7f7548071bc0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f754e236640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7548071bc0 0x7f754814ec90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f754400b440 tx=0x7f754400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544015020 con 0x7f7548071bc0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7544011890 con 0x7f7548071bc0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 -- 172.21.15.67:0/1169580542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7548153370 con 0x7f7548071bc0 2024-04-25T15:53:07.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544011c80 con 0x7f7548071bc0 2024-04-25T15:53:07.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 -- 172.21.15.67:0/1169580542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75481f9970 con 0x7f7548071bc0 2024-04-25T15:53:07.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.986+0000 7f755049a640 1 -- 172.21.15.67:0/1169580542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7548152030 con 0x7f7548071bc0 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7544018020 con 0x7f7548071bc0 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f753effd640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f752c06f400 0x7f752c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7f754408f7b0 con 0x7f7548071bc0 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f753effd640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c0778b0 0x7f752c079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7544020e50 con 0x7f752c0778b0 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f754408fc40 con 0x7f7548071bc0 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f754da35640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f752c06f400 0x7f752c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f754ea37640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c0778b0 0x7f752c079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:07.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f754ea37640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f752c0778b0 0x7f752c079da0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f754da35640 1 --2- 172.21.15.67:0/1169580542 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f752c06f400 0x7f752c071910 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7548154090 tx=0x7f7538008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:07.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:07.990+0000 7f753effd640 1 -- 172.21.15.67:0/1169580542 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7544020e50 con 0x7f752c0778b0 2024-04-25T15:53:08.020 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:08.020 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:bench [] [] [] [] 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs debug_inject_read_zeros 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs files list 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs stats 2024-04-25T15:53:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:bluestore allocator dump block 2024-04-25T15:53:08.025 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2024-04-25T15:53:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T15:53:08.031 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-25T15:53:08.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-04-25T15:53:08.038 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:08.073 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:08.074+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:08.163 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:08.159+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:08.201 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:08.201 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:08.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69aaa40640 1 Processor -- start 2024-04-25T15:53:08.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69aaa40640 1 -- start start 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69aaa40640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f69a415ab30 0x7f69a4165fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69aaa40640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a415de10 con 0x7f69a415a720 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69aaa40640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a4072390 con 0x7f69a4071bc0 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69aaa40640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f69a415ebb0 con 0x7f69a415ab30 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f69a415a720 0x7f69a40ca540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40148/0 (socket says 172.21.15.67:40148) 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a37fe640 1 -- 172.21.15.67:0/844129742 learned_addr learned my addr 172.21.15.67:0/844129742 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a8fdd640 1 --2- 172.21.15.67:0/844129742 >> v2:172.21.15.176:3300/0 conn(0x7f69a415ab30 0x7f69a4165fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3646309021 0 0) 0x7f69a415de10 con 0x7f69a415a720 2024-04-25T15:53:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f698c003530 con 0x7f69a415a720 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1584769983 0 0) 0x7f698c003530 con 0x7f69a415a720 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 >> v2:172.21.15.176:3300/0 conn(0x7f69a415ab30 msgr2=0x7f69a4165fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a2ffd640 1 --2- 172.21.15.67:0/844129742 >> v2:172.21.15.176:3300/0 conn(0x7f69a415ab30 0x7f69a4165fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 >> v1:172.21.15.176:6789/0 conn(0x7f69a4071bc0 legacy=0x7f69a40cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.226+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a415e0d0 con 0x7f69a415a720 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f69980016a0 con 0x7f69a415a720 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 >> v1:172.21.15.67:6789/0 conn(0x7f69a415a720 legacy=0x7f69a40ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f698c0039c0 0x7f698c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f698c004810 0x7f698c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0051d0 0x7f698c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69a415ebb0 con 0x7f698c0039c0 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f698c00c170 con 0x7f698c0051d0 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f698c00c370 con 0x7f698c004810 2024-04-25T15:53:08.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a3fff640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f698c0039c0 0x7f698c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a8fdd640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0051d0 0x7f698c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a37fe640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f698c004810 0x7f698c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a3fff640 1 -- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f698c004810 msgr2=0x7f698c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a3fff640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f698c004810 0x7f698c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a3fff640 1 -- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0051d0 msgr2=0x7f698c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a3fff640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0051d0 0x7f698c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a3fff640 1 -- 172.21.15.67:0/844129742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c003530 con 0x7f698c0039c0 2024-04-25T15:53:08.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a8fdd640 1 -- 172.21.15.67:0/844129742 reap_dead start 2024-04-25T15:53:08.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a3fff640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f698c0039c0 0x7f698c003e30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6994001780 tx=0x7f699401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b93914d7c2b8b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69940013a0 con 0x7f698c0039c0 2024-04-25T15:53:08.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69940050d0 con 0x7f698c0039c0 2024-04-25T15:53:08.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69a2ffd640 1 -- 172.21.15.67:0/844129742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6994003600 con 0x7f698c0039c0 2024-04-25T15:53:08.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69aaa40640 1 -- 172.21.15.67:0/844129742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f698c0039c0 msgr2=0x7f698c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69aaa40640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f698c0039c0 0x7f698c003e30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6994001780 tx=0x7f699401c340 comp rx=0 tx=0).stop 2024-04-25T15:53:08.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69aaa40640 1 -- 172.21.15.67:0/844129742 shutdown_connections 2024-04-25T15:53:08.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69aaa40640 1 --2- 172.21.15.67:0/844129742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f698c0039c0 0x7f698c003e30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.230+0000 7f69aaa40640 1 -- 172.21.15.67:0/844129742 >> 172.21.15.67:0/844129742 conn(0x7f69a40700b0 msgr2=0x7f69a40704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:08.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- 172.21.15.67:0/844129742 shutdown_connections 2024-04-25T15:53:08.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- 172.21.15.67:0/844129742 wait complete. 2024-04-25T15:53:08.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 Processor -- start 2024-04-25T15:53:08.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- start start 2024-04-25T15:53:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0039c0 0x7f69a40d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69a41f7000 0x7f69a41ea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69a4151270 0x7f69a4151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69a4071fd0 con 0x7f69a41f7000 2024-04-25T15:53:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f69a415e720 con 0x7f698c0039c0 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f69a4072150 con 0x7f69a4151270 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69a41f7000 0x7f69a41ea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69a41f7000 0x7f69a41ea4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49330/0 (socket says 172.21.15.67:49330) 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 -- 172.21.15.67:0/4128998920 learned_addr learned my addr 172.21.15.67:0/4128998920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a8fdd640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69a4151270 0x7f69a4151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a3fff640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0039c0 0x7f69a40d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 -- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69a4151270 msgr2=0x7f69a4151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69a4151270 0x7f69a4151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 -- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0039c0 msgr2=0x7f69a40d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0039c0 0x7f69a40d0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 -- 172.21.15.67:0/4128998920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a41f45c0 con 0x7f69a41f7000 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a8fdd640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69a4151270 0x7f69a4151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a3fff640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f698c0039c0 0x7f69a40d0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a37fe640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69a41f7000 0x7f69a41ea4c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6998007290 tx=0x7f6998007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699800f940 con 0x7f69a41f7000 2024-04-25T15:53:08.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6998018070 con 0x7f69a41f7000 2024-04-25T15:53:08.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- 172.21.15.67:0/4128998920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a40b5da0 con 0x7f69a41f7000 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699800fd20 con 0x7f69a41f7000 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.234+0000 7f69aaa40640 1 -- 172.21.15.67:0/4128998920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a40b5ac0 con 0x7f69a41f7000 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69aaa40640 1 -- 172.21.15.67:0/4128998920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f69a406a480 con 0x7f69a41f7000 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f699801d460 con 0x7f69a41f7000 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a0ff9640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f699006f400 0x7f6990071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7f699808fd30 con 0x7f69a41f7000 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a0ff9640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6990077850 0x7f6990079d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6998015e70 con 0x7f6990077850 2024-04-25T15:53:08.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f69980901c0 con 0x7f69a41f7000 2024-04-25T15:53:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a3fff640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f699006f400 0x7f6990071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a8fdd640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6990077850 0x7f6990079d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a3fff640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f699006f400 0x7f6990071910 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6994001750 tx=0x7f69940011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.238+0000 7f69a8fdd640 1 --2- 172.21.15.67:0/4128998920 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6990077850 0x7f6990079d40 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.242+0000 7f69a0ff9640 1 -- 172.21.15.67:0/4128998920 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6998015e70 con 0x7f6990077850 2024-04-25T15:53:08.263 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:08.262+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:bench [] [] [] [] 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs debug_inject_read_zeros 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs files list 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs stats 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:bluestore allocator dump block 2024-04-25T15:53:08.273 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2024-04-25T15:53:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T15:53:08.279 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-25T15:53:08.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-04-25T15:53:08.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:08.365 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:08.366+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:08.422 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:08.422 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:08.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:08.466 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:08.466+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde20d44640 1 Processor -- start 2024-04-25T15:53:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde20d44640 1 -- start start 2024-04-25T15:53:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07aa35f640 1 Processor -- start 2024-04-25T15:53:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07aa35f640 1 -- start start 2024-04-25T15:53:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde20d44640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde1c154ac0 0x7fde1c0cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde20d44640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fde1c15de00 0x7fde1c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde20d44640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde1c0ca8a0 con 0x7fde1c15e210 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde20d44640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fde1c0cab80 con 0x7fde1c154ac0 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde20d44640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fde1c15a920 con 0x7fde1c15de00 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde1ad9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fde1c15e210 0x7fde1c073c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40152/0 (socket says 172.21.15.67:40152) 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde1a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde1c154ac0 0x7fde1c0cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde1ad9d640 1 -- 172.21.15.67:0/4245845706 learned_addr learned my addr 172.21.15.67:0/4245845706 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde19d9b640 1 --2- 172.21.15.67:0/4245845706 >> v2:172.21.15.176:3300/0 conn(0x7fde1c15de00 0x7fde1c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde1a59c640 1 -- 172.21.15.67:0/4245845706 >> v2:172.21.15.176:3300/0 conn(0x7fde1c15de00 msgr2=0x7fde1c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde1a59c640 1 --2- 172.21.15.67:0/4245845706 >> v2:172.21.15.176:3300/0 conn(0x7fde1c15de00 0x7fde1c0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde1a59c640 1 -- 172.21.15.67:0/4245845706 >> v1:172.21.15.67:6789/0 conn(0x7fde1c15e210 legacy=0x7fde1c073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde1a59c640 1 -- 172.21.15.67:0/4245845706 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c15ac00 con 0x7fde1c154ac0 2024-04-25T15:53:08.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7fde19d9b640 1 --2- 172.21.15.67:0/4245845706 >> v2:172.21.15.176:3300/0 conn(0x7fde1c15de00 0x7fde1c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:08.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07aa35f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a40c6c10 0x7f07a4156c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07aa35f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c8200 0x7f07a40c8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07aa35f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a415cc60 con 0x7f07a40c9bc0 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07aa35f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f07a415cf40 con 0x7f07a40c6c10 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07aa35f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07a415d220 con 0x7f07a40c8200 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a40c6c10 0x7f07a4156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07a40c6c10 0x7f07a4156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49350/0 (socket says 172.21.15.67:49350) 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a3fff640 1 -- 172.21.15.67:0/3326467103 learned_addr learned my addr 172.21.15.67:0/3326467103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a88fc640 1 --2- 172.21.15.67:0/3326467103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c8200 0x7f07a40c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a3fff640 1 -- 172.21.15.67:0/3326467103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c8200 msgr2=0x7f07a40c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a3fff640 1 --2- 172.21.15.67:0/3326467103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c8200 0x7f07a40c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a3fff640 1 -- 172.21.15.67:0/3326467103 >> v1:172.21.15.176:6789/0 conn(0x7f07a40c9bc0 legacy=0x7f07a40d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.486+0000 7f07a3fff640 1 -- 172.21.15.67:0/3326467103 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a415d500 con 0x7f07a40c6c10 2024-04-25T15:53:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde1a59c640 1 --2- 172.21.15.67:0/4245845706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c154ac0 0x7fde1c0cbe90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0cd1c0 tx=0x7fde0c01c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa82327789799023 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde1959a640 1 -- 172.21.15.67:0/4245845706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0b0070 con 0x7fde1c154ac0 2024-04-25T15:53:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde1959a640 1 -- 172.21.15.67:0/4245845706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0c0ab600 con 0x7fde1c154ac0 2024-04-25T15:53:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde1959a640 1 -- 172.21.15.67:0/4245845706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c0ab9d0 con 0x7fde1c154ac0 2024-04-25T15:53:08.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07a3fff640 1 --2- 172.21.15.67:0/3326467103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c6c10 0x7f07a4156c60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0794009870 tx=0x7f079401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362f6824ebffc493 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 -- 172.21.15.67:0/4245845706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c154ac0 msgr2=0x7fde1c0cbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 --2- 172.21.15.67:0/4245845706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c154ac0 0x7fde1c0cbe90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0cd1c0 tx=0x7fde0c01c280 comp rx=0 tx=0).stop 2024-04-25T15:53:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07a2ffd640 1 -- 172.21.15.67:0/3326467103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940b5070 con 0x7f07a40c6c10 2024-04-25T15:53:08.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 -- 172.21.15.67:0/4245845706 shutdown_connections 2024-04-25T15:53:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 --2- 172.21.15.67:0/4245845706 >> v2:172.21.15.176:3300/0 conn(0x7fde1c15de00 0x7fde1c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 --2- 172.21.15.67:0/4245845706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c154ac0 0x7fde1c0cbe90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 -- 172.21.15.67:0/4245845706 >> 172.21.15.67:0/4245845706 conn(0x7fde1c068c70 msgr2=0x7fde1c071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07a2ffd640 1 -- 172.21.15.67:0/3326467103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07940a8a50 con 0x7f07a40c6c10 2024-04-25T15:53:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07a2ffd640 1 -- 172.21.15.67:0/3326467103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07940a8e20 con 0x7f07a40c6c10 2024-04-25T15:53:08.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 -- 172.21.15.67:0/4245845706 shutdown_connections 2024-04-25T15:53:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7fde20d44640 1 -- 172.21.15.67:0/4245845706 wait complete. 2024-04-25T15:53:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 -- 172.21.15.67:0/3326467103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c6c10 msgr2=0x7f07a4156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 --2- 172.21.15.67:0/3326467103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c6c10 0x7f07a4156c60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0794009870 tx=0x7f079401ed50 comp rx=0 tx=0).stop 2024-04-25T15:53:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 -- 172.21.15.67:0/3326467103 shutdown_connections 2024-04-25T15:53:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 --2- 172.21.15.67:0/3326467103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c8200 0x7f07a40c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 --2- 172.21.15.67:0/3326467103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c6c10 0x7f07a4156c60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 -- 172.21.15.67:0/3326467103 >> 172.21.15.67:0/3326467103 conn(0x7f07a4068760 msgr2=0x7f07a4068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 -- 172.21.15.67:0/3326467103 shutdown_connections 2024-04-25T15:53:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.490+0000 7f07aa35f640 1 -- 172.21.15.67:0/3326467103 wait complete. 2024-04-25T15:53:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 Processor -- start 2024-04-25T15:53:08.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 -- start start 2024-04-25T15:53:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 Processor -- start 2024-04-25T15:53:08.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 -- start start 2024-04-25T15:53:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde1c15de00 0x7fde1c1f9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde1c15e210 0x7fde1c1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c15f9e0 0x7fde1c14b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c0caec0 con 0x7fde1c15f9e0 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fde1c15b6e0 con 0x7fde1c15e210 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde1c0cb040 con 0x7fde1c15de00 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c15f9e0 0x7fde1c14b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c15f9e0 0x7fde1c14b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49352/0 (socket says 172.21.15.67:49352) 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 -- 172.21.15.67:0/2874296245 learned_addr learned my addr 172.21.15.67:0/2874296245 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde19d9b640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde1c15e210 0x7fde1c1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde1c15de00 0x7fde1c1f9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde1c15de00 msgr2=0x7fde1c1f9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde1c15de00 0x7fde1c1f9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde1c15e210 msgr2=0x7fde1c1516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde1c15e210 0x7fde1c1516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1c1fa310 con 0x7fde1c15f9e0 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde1c15de00 0x7fde1c1f9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:08.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde1ad9d640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c15f9e0 0x7fde1c14b740 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fde1400caf0 tx=0x7fde14002080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde14018070 con 0x7fde1c15f9e0 2024-04-25T15:53:08.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1400a700 con 0x7fde1c15f9e0 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07a40c6c10 0x7f07a4154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c8200 0x7f07a40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c9bc0 0x7f07a4148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07a40c8b50 con 0x7f07a40c8200 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f07a40c7ce0 con 0x7f07a40c6c10 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07a415c8b0 con 0x7f07a40c9bc0 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c8200 0x7f07a40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c8200 0x7f07a40b4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49364/0 (socket says 172.21.15.67:49364) 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 -- 172.21.15.67:0/1716728393 learned_addr learned my addr 172.21.15.67:0/1716728393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a3fff640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07a40c6c10 0x7f07a4154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a88fc640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c9bc0 0x7f07a4148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 -- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c9bc0 msgr2=0x7f07a4148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07a40c9bc0 0x7f07a4148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 -- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07a40c6c10 msgr2=0x7f07a4154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07a40c6c10 0x7f07a4154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 -- 172.21.15.67:0/1716728393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a41f3340 con 0x7f07a40c8200 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a37fe640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a40c8200 0x7f07a40b4d40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f079800c3c0 tx=0x7f079800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1400ab30 con 0x7fde1c15f9e0 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1c15a560 con 0x7fde1c15f9e0 2024-04-25T15:53:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1c15e7c0 con 0x7fde1c15f9e0 2024-04-25T15:53:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798017070 con 0x7f07a40c8200 2024-04-25T15:53:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07980052d0 con 0x7f07a40c8200 2024-04-25T15:53:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.494+0000 7f07aa35f640 1 -- 172.21.15.67:0/1716728393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a41f35e0 con 0x7f07a40c8200 2024-04-25T15:53:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07aa35f640 1 -- 172.21.15.67:0/1716728393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a4155550 con 0x7f07a40c8200 2024-04-25T15:53:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798003820 con 0x7f07a40c8200 2024-04-25T15:53:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fde1c06a480 con 0x7fde1c15f9e0 2024-04-25T15:53:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde1400acd0 con 0x7fde1c15f9e0 2024-04-25T15:53:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07aa35f640 1 -- 172.21.15.67:0/1716728393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0784001040 con 0x7f07a40c8200 2024-04-25T15:53:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f079801a070 con 0x7f07a40c8200 2024-04-25T15:53:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde0affd640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fddfc06f400 0x7fddfc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a0ff9640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f079006f400 0x7f0790071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7fde1408eed0 con 0x7fde1c15f9e0 2024-04-25T15:53:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7f079808e600 con 0x7f07a40c8200 2024-04-25T15:53:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a3fff640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f079006f400 0x7f0790071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a0ff9640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07900778b0 0x7f0790079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fddfc06f400 0x7fddfc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) ==== 54+0+16 (secure 0 0 0) 0x7fde1408f360 con 0x7fde1c15f9e0 2024-04-25T15:53:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f079007a470 con 0x7f07900778b0 2024-04-25T15:53:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f0798095050 con 0x7f07a40c8200 2024-04-25T15:53:08.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a88fc640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07900778b0 0x7f0790079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde1c1f9660 0x7fde1c15eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde1c0cda50 con 0x7fde1c1f9660 2024-04-25T15:53:08.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a3fff640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f079006f400 0x7f0790071910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f07940b2000 tx=0x7f0794009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fddfc06f400 0x7fddfc071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0ad000 tx=0x7fde0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7fde19d9b640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde1c1f9660 0x7fde1c15eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.498+0000 7f07a88fc640 1 --2- 172.21.15.67:0/1716728393 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07900778b0 0x7f0790079da0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.502+0000 7fde19d9b640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde1c1f9660 0x7fde1c15eaa0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.502+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1c0cda50 con 0x7fde1c1f9660 2024-04-25T15:53:08.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.502+0000 7f07a0ff9640 1 -- 172.21.15.67:0/1716728393 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f079007a470 con 0x7f07900778b0 2024-04-25T15:53:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.534+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c06e280 con 0x7fde1c1f9660 2024-04-25T15:53:08.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.534+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c06e280 con 0x7fde1c1f9660 2024-04-25T15:53:08.536 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:08.537 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:08.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.534+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde1c0cf0e0 0x7fde1c1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.534+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fde1c074020 con 0x7fde1c0cf0e0 2024-04-25T15:53:08.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.534+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde1c0cf0e0 0x7fde1c1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.534+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde1c0cf0e0 0x7fde1c1594f0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.538+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1c074020 con 0x7fde1c0cf0e0 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:bench [] [] [] [] 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs debug_inject_read_zeros 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs files list 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs stats 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:bluestore allocator dump block 2024-04-25T15:53:08.548 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2024-04-25T15:53:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T15:53:08.553 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-25T15:53:08.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-04-25T15:53:08.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.554+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c151c10 con 0x7fde1c0cf0e0 2024-04-25T15:53:08.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.554+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c151c10 con 0x7fde1c0cf0e0 2024-04-25T15:53:08.556 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:08.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:08.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.554+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde1c063090 0x7fde1c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.554+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fde1c151e20 con 0x7fde1c063090 2024-04-25T15:53:08.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.554+0000 7fde19d9b640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde1c063090 0x7fde1c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.554+0000 7fde19d9b640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde1c063090 0x7fde1c0634a0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.558+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde10003e90 con 0x7fde1c063090 2024-04-25T15:53:08.566 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:08.563+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:08.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.574+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c0639e0 con 0x7fde1c063090 2024-04-25T15:53:08.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.578+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c0639e0 con 0x7fde1c063090 2024-04-25T15:53:08.579 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.578+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde1c1527a0 0x7fde1c152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.578+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fde1c063fb0 con 0x7fde1c1527a0 2024-04-25T15:53:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.578+0000 7fde1ad9d640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde1c1527a0 0x7fde1c152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.578+0000 7fde1ad9d640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde1c1527a0 0x7fde1c152bb0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.578+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1c063fb0 con 0x7fde1c1527a0 2024-04-25T15:53:08.602 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:08.602 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:08.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.602+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c063fb0 con 0x7fde1c1527a0 2024-04-25T15:53:08.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.606+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c063fb0 con 0x7fde1c1527a0 2024-04-25T15:53:08.606 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:08.607 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:08.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.606+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde1c02d2b0 0x7fde1c02d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.606+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fde1c063fb0 con 0x7fde1c02d2b0 2024-04-25T15:53:08.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.606+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde1c02d2b0 0x7fde1c02d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.606+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde1c02d2b0 0x7fde1c02d6c0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.610+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1c063fb0 con 0x7fde1c02d2b0 2024-04-25T15:53:08.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:08.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.622+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c063fb0 con 0x7fde1c02d2b0 2024-04-25T15:53:08.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.626+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c063fb0 con 0x7fde1c02d2b0 2024-04-25T15:53:08.629 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.630+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde1c02dc00 0x7fde1c20cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.630+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fde1c20fcd0 con 0x7fde1c02dc00 2024-04-25T15:53:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.630+0000 7fde19d9b640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde1c02dc00 0x7fde1c20cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.630+0000 7fde19d9b640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde1c02dc00 0x7fde1c20cf00 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.630+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1c20fcd0 con 0x7fde1c02dc00 2024-04-25T15:53:08.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.650+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c20d4c0 con 0x7fde1c02dc00 2024-04-25T15:53:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.650+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c20d4c0 con 0x7fde1c02dc00 2024-04-25T15:53:08.653 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.654+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde1c20f4a0 0x7fde1c20f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.654+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fde1c2130a0 con 0x7fde1c20f4a0 2024-04-25T15:53:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.654+0000 7fde1ad9d640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde1c20f4a0 0x7fde1c20f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.654+0000 7fde1ad9d640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde1c20f4a0 0x7fde1c20f930 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.654+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1c2130a0 con 0x7fde1c20f4a0 2024-04-25T15:53:08.667 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:08.666+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.674+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c211f20 con 0x7fde1c20f4a0 2024-04-25T15:53:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.682+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c211f20 con 0x7fde1c20f4a0 2024-04-25T15:53:08.682 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:08.682 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:08.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.682+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde1c212640 0x7fde1c212a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.682+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fde1c216470 con 0x7fde1c212640 2024-04-25T15:53:08.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.682+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde1c212640 0x7fde1c212a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.682+0000 7fde1a59c640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde1c212640 0x7fde1c212a50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.682+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde1c216470 con 0x7fde1c212640 2024-04-25T15:53:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.698+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fde1c20d550 con 0x7fde1c212640 2024-04-25T15:53:08.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde0affd640 1 -- 172.21.15.67:0/2874296245 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fde1c20d550 con 0x7fde1c212640 2024-04-25T15:53:08.703 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde1c1f9660 msgr2=0x7fde1c15eaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fde1c1f9660 0x7fde1c15eaa0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde1c0cf0e0 msgr2=0x7fde1c1594f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fde1c0cf0e0 0x7fde1c1594f0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde1c063090 msgr2=0x7fde1c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fde1c063090 0x7fde1c0634a0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde1ad9d640 1 -- 172.21.15.67:0/2874296245 reap_dead start 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde1c1527a0 msgr2=0x7fde1c152bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fde1c1527a0 0x7fde1c152bb0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde1c02d2b0 msgr2=0x7fde1c02d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fde1c02d2b0 0x7fde1c02d6c0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde1c02dc00 msgr2=0x7fde1c20cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fde1c02dc00 0x7fde1c20cf00 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde1c20f4a0 msgr2=0x7fde1c20f930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fde1c20f4a0 0x7fde1c20f930 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde1c212640 msgr2=0x7fde1c212a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fde1c212640 0x7fde1c212a50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde1ad9d640 1 -- 172.21.15.67:0/2874296245 reap_dead start 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fddfc06f400 msgr2=0x7fddfc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fddfc06f400 0x7fddfc071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0ad000 tx=0x7fde0c008040 comp rx=0 tx=0).stop 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c15f9e0 msgr2=0x7fde1c14b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c15f9e0 0x7fde1c14b740 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fde1400caf0 tx=0x7fde14002080 comp rx=0 tx=0).stop 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 shutdown_connections 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fddfc06f400 0x7fddfc071910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 --2- 172.21.15.67:0/2874296245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde1c15f9e0 0x7fde1c14b740 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 >> 172.21.15.67:0/2874296245 conn(0x7fde1c068c70 msgr2=0x7fde1c0c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 shutdown_connections 2024-04-25T15:53:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.702+0000 7fde20d44640 1 -- 172.21.15.67:0/2874296245 wait complete. 2024-04-25T15:53:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.710+0000 7f6d78615640 1 Processor -- start 2024-04-25T15:53:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.710+0000 7f6d78615640 1 -- start start 2024-04-25T15:53:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d78615640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6d70156330 0x7f6d7006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d78615640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d7006cb80 0x7f6d700c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d78615640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d700c4f30 0x7f6d70158d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d78615640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6d700ca6c0 con 0x7f6d7006cb80 2024-04-25T15:53:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d78615640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6d7006cf90 con 0x7f6d70156330 2024-04-25T15:53:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d78615640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6d700c43d0 con 0x7f6d700c4f30 2024-04-25T15:53:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d7006cb80 0x7f6d700c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d7006cb80 0x7f6d700c49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49378/0 (socket says 172.21.15.67:49378) 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 -- 172.21.15.67:0/142969599 learned_addr learned my addr 172.21.15.67:0/142969599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d76bb2640 1 --2- 172.21.15.67:0/142969599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d700c4f30 0x7f6d70158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d763b1640 1 --2- 172.21.15.67:0/142969599 >> v2:172.21.15.176:3300/0 conn(0x7f6d70156330 0x7f6d7006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 -- 172.21.15.67:0/142969599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d700c4f30 msgr2=0x7f6d70158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 --2- 172.21.15.67:0/142969599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d700c4f30 0x7f6d70158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 -- 172.21.15.67:0/142969599 >> v2:172.21.15.176:3300/0 conn(0x7f6d70156330 msgr2=0x7f6d7006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 --2- 172.21.15.67:0/142969599 >> v2:172.21.15.176:3300/0 conn(0x7f6d70156330 0x7f6d7006c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.714+0000 7f6d75bb0640 1 -- 172.21.15.67:0/142969599 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7006fa10 con 0x7f6d7006cb80 2024-04-25T15:53:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d75bb0640 1 --2- 172.21.15.67:0/142969599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7006cb80 0x7f6d700c49f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6d700c8880 tx=0x7f6d6001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad5fa11c0d3e992 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d753af640 1 -- 172.21.15.67:0/142969599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600b5070 con 0x7f6d7006cb80 2024-04-25T15:53:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d753af640 1 -- 172.21.15.67:0/142969599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d600a89b0 con 0x7f6d7006cb80 2024-04-25T15:53:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d753af640 1 -- 172.21.15.67:0/142969599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600a8d80 con 0x7f6d7006cb80 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 -- 172.21.15.67:0/142969599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7006cb80 msgr2=0x7f6d700c49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 --2- 172.21.15.67:0/142969599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7006cb80 0x7f6d700c49f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6d700c8880 tx=0x7f6d6001ecf0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 -- 172.21.15.67:0/142969599 shutdown_connections 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 --2- 172.21.15.67:0/142969599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d700c4f30 0x7f6d70158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 --2- 172.21.15.67:0/142969599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7006cb80 0x7f6d700c49f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 --2- 172.21.15.67:0/142969599 >> v2:172.21.15.176:3300/0 conn(0x7f6d70156330 0x7f6d7006c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 -- 172.21.15.67:0/142969599 >> 172.21.15.67:0/142969599 conn(0x7f6d700ce1c0 msgr2=0x7f6d700ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 -- 172.21.15.67:0/142969599 shutdown_connections 2024-04-25T15:53:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 -- 172.21.15.67:0/142969599 wait complete. 2024-04-25T15:53:08.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 Processor -- start 2024-04-25T15:53:08.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.718+0000 7f6d78615640 1 -- start start 2024-04-25T15:53:08.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d7006cb80 0x7f6d7015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d700c4f30 0x7f6d701f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d70156330 0x7f6d70151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d700c3d80 con 0x7f6d700c4f30 2024-04-25T15:53:08.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6d700c3aa0 con 0x7f6d7006cb80 2024-04-25T15:53:08.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6d700c4060 con 0x7f6d70156330 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d700c4f30 0x7f6d701f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d700c4f30 0x7f6d701f1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49388/0 (socket says 172.21.15.67:49388) 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 -- 172.21.15.67:0/822180805 learned_addr learned my addr 172.21.15.67:0/822180805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d76bb2640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d70156330 0x7f6d70151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d763b1640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d7006cb80 0x7f6d7015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 -- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d70156330 msgr2=0x7f6d70151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6d70156330 0x7f6d70151b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 -- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d7006cb80 msgr2=0x7f6d7015bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6d7006cb80 0x7f6d7015bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 -- 172.21.15.67:0/822180805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7014ff40 con 0x7f6d700c4f30 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d75bb0640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d700c4f30 0x7f6d701f1900 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6d6001f590 tx=0x7f6d6001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600be070 con 0x7f6d700c4f30 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d60003620 con 0x7f6d700c4f30 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600b5070 con 0x7f6d700c4f30 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 -- 172.21.15.67:0/822180805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7015c5c0 con 0x7f6d700c4f30 2024-04-25T15:53:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 -- 172.21.15.67:0/822180805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d701f0e20 con 0x7f6d700c4f30 2024-04-25T15:53:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.722+0000 7f6d78615640 1 -- 172.21.15.67:0/822180805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d7006a480 con 0x7f6d700c4f30 2024-04-25T15:53:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6d600c1070 con 0x7f6d700c4f30 2024-04-25T15:53:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d66ffd640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6d5406f400 0x7f6d54071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7f6d6012be80 con 0x7f6d700c4f30 2024-04-25T15:53:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d66ffd640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6d540778b0 0x7f6d54079da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d600c0e50 con 0x7f6d540778b0 2024-04-25T15:53:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f6d6012c310 con 0x7f6d700c4f30 2024-04-25T15:53:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d763b1640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6d5406f400 0x7f6d54071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d76bb2640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6d540778b0 0x7f6d54079da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.726+0000 7f6d76bb2640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6d540778b0 0x7f6d54079da0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.730+0000 7f6d763b1640 1 --2- 172.21.15.67:0/822180805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6d5406f400 0x7f6d54071910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00b930 tx=0x7f6d6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.730+0000 7f6d66ffd640 1 -- 172.21.15.67:0/822180805 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d600c0e50 con 0x7f6d540778b0 2024-04-25T15:53:08.764 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-04-25T15:53:08.764 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:bench [] [] [] [] 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs debug_inject_read_zeros 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs files list 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs stats 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:bluestore allocator dump block 2024-04-25T15:53:08.765 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2024-04-25T15:53:08.768 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:08.767+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T15:53:08.773 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-25T15:53:08.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-25T15:53:08.825 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:08.825 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:08.825 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:08.825 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:08.826 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:08.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:08.891 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:08.891+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:08.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.966+0000 7f4fcad11640 1 Processor -- start 2024-04-25T15:53:08.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.966+0000 7f4fcad11640 1 -- start start 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fcad11640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fc4073bf0 0x7f4fc406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fcad11640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc4163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fcad11640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fc40c77a0 con 0x7f4fc40caf60 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fcad11640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f4fc40cab60 con 0x7f4fc4073bf0 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fcad11640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc4159620 con 0x7f4fc406dc90 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fc4073bf0 0x7f4fc406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4fc4073bf0 0x7f4fc406d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49396/0 (socket says 172.21.15.67:49396) 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 -- 172.21.15.67:0/2080485657 learned_addr learned my addr 172.21.15.67:0/2080485657 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc92ae640 1 --2- 172.21.15.67:0/2080485657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc4163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 -- 172.21.15.67:0/2080485657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 msgr2=0x7f4fc4163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 --2- 172.21.15.67:0/2080485657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc4163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 -- 172.21.15.67:0/2080485657 >> v1:172.21.15.67:6789/0 conn(0x7f4fc40caf60 legacy=0x7f4fc415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 -- 172.21.15.67:0/2080485657 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc40c7a60 con 0x7f4fc4073bf0 2024-04-25T15:53:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc37fe640 1 -- 172.21.15.67:0/2080485657 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4195550742 0 0) 0x7f4fc40c77a0 con 0x7f4fc40caf60 2024-04-25T15:53:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc92ae640 1 --2- 172.21.15.67:0/2080485657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc4163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:08.972 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:08.967+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:08.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc3fff640 1 --2- 172.21.15.67:0/2080485657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc4073bf0 0x7f4fc406d750 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009870 tx=0x7f4fb001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de894baa956bd530 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.970+0000 7f4fc37fe640 1 -- 172.21.15.67:0/2080485657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00b2070 con 0x7f4fc4073bf0 2024-04-25T15:53:08.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fc37fe640 1 -- 172.21.15.67:0/2080485657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb00a9070 con 0x7f4fc4073bf0 2024-04-25T15:53:08.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fc37fe640 1 -- 172.21.15.67:0/2080485657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00ad880 con 0x7f4fc4073bf0 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- 172.21.15.67:0/2080485657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc4073bf0 msgr2=0x7f4fc406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 --2- 172.21.15.67:0/2080485657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc4073bf0 0x7f4fc406d750 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009870 tx=0x7f4fb001c560 comp rx=0 tx=0).stop 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- 172.21.15.67:0/2080485657 shutdown_connections 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 --2- 172.21.15.67:0/2080485657 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc4163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 --2- 172.21.15.67:0/2080485657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc4073bf0 0x7f4fc406d750 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- 172.21.15.67:0/2080485657 >> 172.21.15.67:0/2080485657 conn(0x7f4fc4070db0 msgr2=0x7f4fc40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- 172.21.15.67:0/2080485657 shutdown_connections 2024-04-25T15:53:08.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- 172.21.15.67:0/2080485657 wait complete. 2024-04-25T15:53:08.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 Processor -- start 2024-04-25T15:53:08.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- start start 2024-04-25T15:53:08.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc41508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fc4073bf0 0x7f4fc414c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc40caf60 0x7f4fc414cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc415a570 con 0x7f4fc40caf60 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc415a3f0 con 0x7f4fc4073bf0 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.974+0000 7f4fcad11640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc406c640 con 0x7f4fc406dc90 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc40caf60 0x7f4fc414cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc40caf60 0x7f4fc414cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49408/0 (socket says 172.21.15.67:49408) 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 -- 172.21.15.67:0/1960254045 learned_addr learned my addr 172.21.15.67:0/1960254045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc8aad640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc41508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc3fff640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fc4073bf0 0x7f4fc414c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 -- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 msgr2=0x7f4fc41508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4fc406dc90 0x7f4fc41508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 -- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fc4073bf0 msgr2=0x7f4fc414c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4fc4073bf0 0x7f4fc414c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:08.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 -- 172.21.15.67:0/1960254045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc4156230 con 0x7f4fc40caf60 2024-04-25T15:53:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc92ae640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fc40caf60 0x7f4fc414cd50 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc007e20 tx=0x7f4fbc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc01a070 con 0x7f4fc40caf60 2024-04-25T15:53:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fbc003ba0 con 0x7f4fc40caf60 2024-04-25T15:53:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fcad11640 1 -- 172.21.15.67:0/1960254045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc4155a40 con 0x7f4fc40caf60 2024-04-25T15:53:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc00de80 con 0x7f4fc40caf60 2024-04-25T15:53:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fcad11640 1 -- 172.21.15.67:0/1960254045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc41564d0 con 0x7f4fc40caf60 2024-04-25T15:53:08.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fcad11640 1 -- 172.21.15.67:0/1960254045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4fc414d290 con 0x7f4fc40caf60 2024-04-25T15:53:08.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4fbc01d070 con 0x7f4fc40caf60 2024-04-25T15:53:08.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4fa806f450 0x7f4fa8071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7f4fbc08fbe0 con 0x7f4fc40caf60 2024-04-25T15:53:08.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4fa8077980 0x7f4fa8079e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:08.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fa807a540 con 0x7f4fa8077980 2024-04-25T15:53:08.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f4fbc0900c0 con 0x7f4fc40caf60 2024-04-25T15:53:08.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc8aad640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4fa806f450 0x7f4fa8071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.978+0000 7f4fc3fff640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4fa8077980 0x7f4fa8079e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:08.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.982+0000 7f4fc3fff640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4fa8077980 0x7f4fa8079e70 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.982+0000 7f4fc8aad640 1 --2- 172.21.15.67:0/1960254045 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4fa806f450 0x7f4fa8071960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8000f70 tx=0x7f4fb8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:08.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:08.982+0000 7f4fc17fa640 1 -- 172.21.15.67:0/1960254045 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fa807a540 con 0x7f4fa8077980 2024-04-25T15:53:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-04-25T15:53:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-04-25T15:53:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-04-25T15:53:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-04-25T15:53:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-04-25T15:53:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-04-25T15:53:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:bench [] [] [] [] 2024-04-25T15:53:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs debug_inject_read_zeros 2024-04-25T15:53:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs files list 2024-04-25T15:53:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:bluefs stats 2024-04-25T15:53:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:bluestore allocator dump block 2024-04-25T15:53:09.013 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2024-04-25T15:53:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-25T15:53:09.018 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-25T15:53:09.054 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:09.054 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:09.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:09.054 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:43 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.148512228000001, 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024229051000000001, 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.055 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902019+0000", 2024-04-25T15:53:09.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9459999999999998e-06 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907768+0000", 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005748525 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907831+0000", 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2842999999999994e-05 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907857+0000", 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6636e-05 2024-04-25T15:53:09.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908056+0000", 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198295 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908464+0000", 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040813600000000002 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908853+0000", 2024-04-25T15:53:09.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038892400000000002 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925332+0000", 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016479074 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925882+0000", 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000550232 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926050+0000", 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016822199999999999 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926245+0000", 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019521799999999999 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:53 3.20 3:068d0381:::10000000002.00000031:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.672330067000001, 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020793931000000002, 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.060 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 53 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378199+0000", 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.378202+0000", 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1669999999999999e-06 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382531+0000", 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043289130000000002 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382593+0000", 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2143000000000005e-05 2024-04-25T15:53:09.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382618+0000", 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.382814+0000", 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195964 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383075+0000", 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026081800000000001 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.383324+0000", 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024972200000000002 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.392258+0000", 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089338549999999992 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398665+0000", 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064066649999999998 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398841+0000", 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017607899999999999 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.398993+0000", 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015144799999999999 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:75 3.35 3:ad096b5b:::10000000002.00000047:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.717019527000001, 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022276672000000001, 2024-04-25T15:53:09.065 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333513+0000", 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339006+0000", 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054929599999999999 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339071+0000", 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5545999999999999e-05 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339098+0000", 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6647000000000001e-05 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339299+0000", 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201351 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339703+0000", 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040396399999999998 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.340160+0000", 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045697899999999999 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.352668+0000", 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012508137000000001 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355605+0000", 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002937013 2024-04-25T15:53:09.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355701+0000", 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5681999999999996e-05 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355786+0000", 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4908999999999996e-05 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.070 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:79 3.3 3:c0361aff:::10000000002.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.541249432000001, 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022495096999999999, 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 79 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:09.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509282+0000", 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2200000000000001e-06 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517219+0000", 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079362270000000006 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517284+0000", 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5052e-05 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517310+0000", 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6072999999999999e-05 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517506+0000", 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019606199999999999 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517779+0000", 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027302799999999998 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.518047+0000", 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000268454 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.528734+0000", 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01068712 2024-04-25T15:53:09.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531586+0000", 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028516589999999999 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531689+0000", 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010260900000000001 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531774+0000", 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5593000000000005e-05 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.075 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:176 3.26 3:66bf65cf:::10000000002.000000ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.223699573000001, 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020587095999999999, 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 176 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.076 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826829+0000", 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.826832+0000", 2024-04-25T15:53:09.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1200000000000002e-06 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831165+0000", 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043325289999999999 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831220+0000", 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5686999999999997e-05 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831245+0000", 2024-04-25T15:53:09.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4105999999999999e-05 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831400+0000", 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015513899999999999 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831669+0000", 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026955099999999998 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:09.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.831947+0000", 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027750399999999998 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.842230+0000", 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010283537000000001 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847229+0000", 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049990939999999999 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847326+0000", 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7106000000000002e-05 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.847416+0000", 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9722999999999995e-05 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:177 3.26 3:66aef6e9:::10000000002.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.173840675000001, 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026823691, 2024-04-25T15:53:09.081 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876691+0000", 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4939999999999999e-06 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880736+0000", 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040441660000000001 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880793+0000", 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7581999999999999e-05 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880817+0000", 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881039+0000", 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022171800000000001 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881305+0000", 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026580799999999997 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881558+0000", 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025290099999999998 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.893182+0000", 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011624242 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903322+0000", 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010139754000000001 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903425+0000", 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102706 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903512+0000", 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7093000000000001e-05 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.086 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:261 3.1d e66/54 3:bbf7764d:::10000000002.00000001:head v 66'8, mlcod=63'6)", 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.2906745950000005, 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022527077, 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759855+0000", 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6599999999999996e-07 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764340+0000", 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044852190000000004 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764382+0000", 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2052999999999999e-05 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764415+0000", 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.341e-05 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764650+0000", 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023472400000000001 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764703+0000", 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3016999999999997e-05 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782246+0000", 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017543286000000002 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782381+0000", 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134702 2024-04-25T15:53:09.090 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:268 3.14 e67/54 3:29a5f800:::10000000002.00000008:head v 67'6, mlcod=63'4)", 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.9817187720000007, 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023750529999999999, 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:09.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068811+0000", 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.37e-07 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073130+0000", 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043190069999999997 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073172+0000", 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2512000000000002e-05 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073198+0000", 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6084e-05 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073426+0000", 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227514 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073510+0000", 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4469999999999996e-05 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092427+0000", 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018916571 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092560+0000", 2024-04-25T15:53:09.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013373500000000001 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:269 3.2c 3:354fa1eb:::10000000002.00000009:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.9452399489999994, 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022386919000000002, 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.095 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 269 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105292+0000", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8839999999999998e-06 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111434+0000", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061422880000000001 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111491+0000", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7306000000000003e-05 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111515+0000", 2024-04-25T15:53:09.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3893e-05 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111686+0000", 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017118600000000001 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111859+0000", 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017279099999999999 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.112126+0000", 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000266722 2024-04-25T15:53:09.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.123964+0000", 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01183797 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127453+0000", 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034891240000000001 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127555+0000", 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102082 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127676+0000", 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120673 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:355 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5618140020000002, 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025437358, 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.100 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 355 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488717+0000", 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.874e-06 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494412+0000", 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056951349999999996 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494450+0000", 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7920999999999998e-05 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494464+0000", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4728e-05 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494580+0000", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011599099999999999 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494756+0000", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176083 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.495038+0000", 2024-04-25T15:53:09.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028161800000000003 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.508282+0000", 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013243981 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514019+0000", 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057372639999999997 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514099+0000", 2024-04-25T15:53:09.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9800000000000002e-05 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514152+0000", 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2963000000000001e-05 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:357 3.34 3:2d119e8b:::10000000002.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.485305876, 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021511328, 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.105 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 357 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565225+0000", 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2189999999999998e-06 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569849+0000", 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046243480000000003 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569992+0000", 2024-04-25T15:53:09.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014274200000000001 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570006+0000", 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4032999999999999e-05 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570122+0000", 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116358 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570290+0000", 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016725299999999999 2024-04-25T15:53:09.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570528+0000", 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023819800000000001 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.581505+0000", 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01097657 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586619+0000", 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051144119999999996 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586682+0000", 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3309e-05 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586734+0000", 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1885999999999998e-05 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:372 3.10 3:0b92729d:::10000000002.00000070:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.110 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.9439906410000001, 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026675501000000001, 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 372 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106541+0000", 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.022e-06 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110718+0000", 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041768669999999999 2024-04-25T15:53:09.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110768+0000", 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110792+0000", 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3577000000000001e-05 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110988+0000", 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019645200000000001 2024-04-25T15:53:09.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111271+0000", 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028314399999999998 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111688+0000", 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041694699999999998 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.123821+0000", 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012133015 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133024+0000", 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092031290000000009 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133125+0000", 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010063199999999999 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133214+0000", 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8555999999999997e-05 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.115 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:388 3.36 3:6dca1976:::10000000002.00000080:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.324781379, 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022516031999999998, 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 388 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.116 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725751+0000", 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.27e-06 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730050+0000", 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004299247 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730109+0000", 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9583000000000002e-05 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730135+0000", 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5316999999999998e-05 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730306+0000", 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000171695 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730503+0000", 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019674499999999999 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730797+0000", 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029401399999999999 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.741175+0000", 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010378 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748075+0000", 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068997829999999996 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748176+0000", 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101545 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748263+0000", 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6833e-05 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:394 3.36 3:6ccb00e9:::10000000002.00000086:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.1189265380000002, 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025589158000000001, 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.121 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 394 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931605+0000", 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0730000000000001e-06 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936840+0000", 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052349129999999999 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936898+0000", 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7793e-05 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936922+0000", 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4513000000000001e-05 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937111+0000", 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018847800000000001 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937292+0000", 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181189 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937542+0000", 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024970800000000002 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956147+0000", 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018605467000000001 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956986+0000", 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083831599999999997 2024-04-25T15:53:09.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957089+0000", 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103379 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957191+0000", 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102329 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.126 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:397 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.0088347600000001, 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022794325000000001, 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 397 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041697+0000", 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8569999999999999e-06 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047000+0000", 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053031220000000004 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047059+0000", 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9499000000000001e-05 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047084+0000", 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4340000000000001e-05 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047231+0000", 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014741600000000001 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047415+0000", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018419500000000001 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047667+0000", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025212800000000001 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.060095+0000", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012427182 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064282+0000", 2024-04-25T15:53:09.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041871410000000001 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064390+0000", 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108142 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064488+0000", 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8302999999999997e-05 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.131 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:400 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.903299429, 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025113648999999998, 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 400 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147232+0000", 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.253e-06 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151612+0000", 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004379226 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151671+0000", 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9036999999999997e-05 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151696+0000", 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4825999999999999e-05 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151891+0000", 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019537499999999999 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.152208+0000", 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000317041 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.152648+0000", 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043998600000000001 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.168097+0000", 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015449137 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172104+0000", 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040067260000000004 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172209+0000", 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105376 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172343+0000", 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133666 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:401 3.34 3:2feeb5a8:::10000000002.0000008d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.872516525, 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022845918, 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 401 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178015+0000", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5730000000000002e-06 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182028+0000", 2024-04-25T15:53:09.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040133800000000004 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182085+0000", 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6656000000000001e-05 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182108+0000", 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.287e-05 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182252+0000", 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144389 2024-04-25T15:53:09.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182427+0000", 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017484799999999999 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182676+0000", 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024908500000000002 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.196912+0000", 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014236367 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200602+0000", 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036898819999999998 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200713+0000", 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011032899999999999 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200858+0000", 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000145539 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.141 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:406 3.11 e69/54 3:8834f03f:::10000000002.00000092:head v 69'13, mlcod=69'11)", 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.6793592569999998, 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023521068999999999, 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371172+0000", 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7800000000000001e-06 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375367+0000", 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004194761 2024-04-25T15:53:09.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375407+0000", 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0222999999999998e-05 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375432+0000", 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5236999999999999e-05 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375634+0000", 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020169400000000001 2024-04-25T15:53:09.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375717+0000", 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2936999999999996e-05 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.394537+0000", 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018820018000000001 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.394690+0000", 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015341899999999999 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.145 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:410 3.34 3:2cf0bbf2:::10000000002.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.5279620619999998, 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026112841000000001, 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 410 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.146 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522570+0000", 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529416+0000", 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006846062 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529476+0000", 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0350000000000003e-05 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529502+0000", 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6259999999999999e-05 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529665+0000", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016305200000000001 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529683+0000", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7935999999999999e-05 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534023+0000", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043396270000000004 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534279+0000", 2024-04-25T15:53:09.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025637099999999998 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534518+0000", 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238762 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.543891+0000", 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093730170000000008 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548495+0000", 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046033610000000003 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548592+0000", 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7793000000000004e-05 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548679+0000", 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7100000000000003e-05 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:412 3.3e 3:7d47d675:::10000000002.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.44690707, 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025248618, 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 412 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603625+0000", 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5329999999999999e-06 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611293+0000", 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076675720000000001 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611358+0000", 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5500000000000006e-05 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611382+0000", 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4300000000000001e-05 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611573+0000", 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019060799999999999 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611863+0000", 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029038099999999998 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.612303+0000", 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043984299999999998 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.627478+0000", 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015174632 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628552+0000", 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001074258 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628728+0000", 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017612500000000001 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628870+0000", 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014186599999999999 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.157 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:09.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:09.159 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:09.158+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:09.259 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:09.255+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:09.360 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:09.358+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:09.368 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:09.368 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:09.368 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:09.368 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:09.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:09.475 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:09.475+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:09.504 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:09.505 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 rewriters = 105637.38 kB/sec 2024-04-25T15:53:09.505 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 rewriters = 105631.11 kB/sec 2024-04-25T15:53:09.505 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 105637.38 kB/sec 2024-04-25T15:53:09.505 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 105637.38 kB/sec 2024-04-25T15:53:09.505 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 105637.38 kB/sec 2024-04-25T15:53:09.505 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:53:09.562 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:09.559+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:09.586 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:09.586 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:09.586 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:09.586 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:09.586 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:09.603 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:09.663 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:09.662+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:09.764 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:09.763+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:09.802 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:09.802 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:09.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:09.802 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:09.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.802 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 170.70033244999999, 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.808 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 170.109129526, 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:53:09.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:53:09.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 170.10911844699999, 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.814 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:62 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.043143555, 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025889728000000001, 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 62 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755657+0000", 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9439999999999998e-06 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760593+0000", 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049360740000000004 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760651+0000", 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8e-05 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760676+0000", 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5219999999999999e-05 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760847+0000", 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017057199999999999 2024-04-25T15:53:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761123+0000", 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027578100000000001 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761365+0000", 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024248100000000001 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780642+0000", 2024-04-25T15:53:09.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019277327 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781358+0000", 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000715611 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781456+0000", 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7946e-05 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781543+0000", 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6772000000000002e-05 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:150 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.147624622999999, 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024331635000000001, 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.825 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 150 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:09.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651176+0000", 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8770000000000003e-06 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659329+0000", 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081530219999999994 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659399+0000", 2024-04-25T15:53:09.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0376000000000005e-05 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659428+0000", 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8952999999999999e-05 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659580+0000", 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015188199999999999 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659842+0000", 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026179399999999998 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.660120+0000", 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027824799999999998 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.673813+0000", 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013693145 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675307+0000", 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014934600000000001 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675412+0000", 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105233 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675504+0000", 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1644999999999994e-05 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:155 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:09.830 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.925473338, 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025243293999999999, 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 155 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873327+0000", 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7699999999999999e-06 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879173+0000", 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005846141 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879239+0000", 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6315999999999993e-05 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879267+0000", 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7137000000000001e-05 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879415+0000", 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000148715 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879693+0000", 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000278108 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879951+0000", 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025793700000000002 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.896019+0000", 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016067814999999999 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898153+0000", 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021339280000000002 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898411+0000", 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025760799999999999 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898566+0000", 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015581899999999999 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.835 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:165 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.429502801000002, 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032658418000000002, 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 165 2024-04-25T15:53:09.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369298+0000", 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3429999999999998e-06 2024-04-25T15:53:09.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374706+0000", 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054080250000000003 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374776+0000", 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9972e-05 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374803+0000", 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7384000000000001e-05 2024-04-25T15:53:09.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375010+0000", 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206431 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375037+0000", 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7206999999999999e-05 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380232+0000", 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051947859999999998 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380474+0000", 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000241864 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380709+0000", 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023560200000000001 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.392962+0000", 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012252219999999999 2024-04-25T15:53:09.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401765+0000", 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088034089999999999 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401866+0000", 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101099 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401952+0000", 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6075999999999994e-05 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.841 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.581091968999999, 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:09.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:09.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:09.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:09.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:09.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.486457916999999, 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.847 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:09.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:09.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:260 3.12 3:4bec3886:::10000000002.00000000:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:09.852 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.108816676, 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028352792000000002, 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 260 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689983+0000", 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2729999999999998e-06 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696571+0000", 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065880749999999997 2024-04-25T15:53:09.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696637+0000", 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5358999999999995e-05 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696662+0000", 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5184000000000001e-05 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697262+0000", 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060013099999999995 2024-04-25T15:53:09.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697437+0000", 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017512499999999999 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697672+0000", 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023489800000000001 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.715797+0000", 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018125601000000002 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718141+0000", 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023436300000000002 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718244+0000", 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102379 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718333+0000", 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9136999999999997e-05 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.857 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:268 3.14 3:29a5f800:::10000000002.00000008:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7378692870000005, 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032265195000000003, 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 268 2024-04-25T15:53:09.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060931+0000", 2024-04-25T15:53:09.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6629999999999999e-06 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067652+0000", 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067213899999999998 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067713+0000", 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0628999999999998e-05 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067739+0000", 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6492999999999999e-05 2024-04-25T15:53:09.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067893+0000", 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015336599999999999 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068083+0000", 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019004800000000001 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068328+0000", 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024518100000000002 2024-04-25T15:53:09.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.087697+0000", 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019368720999999998 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092957+0000", 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052597859999999998 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093058+0000", 2024-04-25T15:53:09.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101335 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093192+0000", 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134583 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:271 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6054741379999999, 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026911917, 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.863 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 271 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:09.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193326+0000", 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0770000000000001e-06 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198319+0000", 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049928710000000003 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198382+0000", 2024-04-25T15:53:09.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2446000000000005e-05 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198408+0000", 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6676999999999999e-05 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198623+0000", 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021493799999999999 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198914+0000", 2024-04-25T15:53:09.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029086100000000001 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.199346+0000", 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000431316 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.219423+0000", 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020076954000000001 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220028+0000", 2024-04-25T15:53:09.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060503900000000003 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220153+0000", 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125919 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220234+0000", 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0819e-05 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:347 3.11 3:8815798b:::10000000002.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.6504079950000001, 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029152658000000001, 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 347 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148391+0000", 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6029999999999999e-06 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155889+0000", 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074977200000000003 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155942+0000", 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2883999999999997e-05 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155964+0000", 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2762999999999999e-05 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156184+0000", 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156358+0000", 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017470800000000001 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156597+0000", 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238411 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.172619+0000", 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016022649 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177350+0000", 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047301369999999997 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177455+0000", 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010574899999999999 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177541+0000", 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5839000000000003e-05 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:352 3.7 e68/18 3:e0f5f483:::10000000002.0000005c:head v 68'11, mlcod=67'9)", 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.4588214749999997, 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026288315999999999, 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.874 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339978+0000", 2024-04-25T15:53:09.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8459999999999999e-06 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345285+0000", 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053072630000000004 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345324+0000", 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8535999999999998e-05 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345348+0000", 2024-04-25T15:53:09.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4315999999999999e-05 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345503+0000", 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015497300000000001 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345555+0000", 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1814999999999998e-05 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366145+0000", 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020589743000000001 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366263+0000", 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118824 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:326 3.8 e69/68 3:133e839e:::10000000002.00000042:head v 69'10, mlcod=0'0)", 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.3612998410000001, 2024-04-25T15:53:09.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025085139999999999, 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437498+0000", 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.632e-06 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445817+0000", 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083191740000000004 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445839+0000", 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1314e-05 2024-04-25T15:53:09.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445854+0000", 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4986e-05 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445948+0000", 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3735999999999998e-05 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445981+0000", 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3210999999999998e-05 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462484+0000", 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016502871999999998 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462582+0000", 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8214999999999994e-05 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:367 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.8544834970000004, 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025258800000000001, 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 367 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944316+0000", 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4880000000000001e-06 2024-04-25T15:53:09.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950833+0000", 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065179829999999998 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950884+0000", 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950902+0000", 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8805e-05 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951039+0000", 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013634300000000001 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951216+0000", 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017729099999999999 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951469+0000", 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025263199999999998 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969276+0000", 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017807630000000001 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969326+0000", 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9639999999999999e-05 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969434+0000", 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108155 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969572+0000", 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013767299999999999 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:406 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.4383767819999997, 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034893977, 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.888 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 406 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360424+0000", 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2300000000000002e-06 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370215+0000", 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0097913340000000005 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370283+0000", 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8028999999999995e-05 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370311+0000", 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8053999999999999e-05 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370464+0000", 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015309899999999999 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370642+0000", 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177648 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370885+0000", 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000242721 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.393703+0000", 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022818037999999999 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395071+0000", 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001368469 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395176+0000", 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104949 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395314+0000", 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013740599999999999 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:411 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.2393091109999999, 2024-04-25T15:53:09.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024254155999999999, 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 411 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559491+0000", 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4699999999999998e-06 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566706+0000", 2024-04-25T15:53:09.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072154760000000002 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566772+0000", 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5646000000000001e-05 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566807+0000", 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4672999999999998e-05 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567067+0000", 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026066400000000002 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567252+0000", 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018467599999999999 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567510+0000", 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025795100000000002 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.582484+0000", 2024-04-25T15:53:09.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014973584 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583549+0000", 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001065365 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583653+0000", 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010430899999999999 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583742+0000", 2024-04-25T15:53:09.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8342000000000002e-05 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:421 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.863323796, 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028370244999999999, 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 421 2024-04-25T15:53:09.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935477+0000", 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0470000000000004e-06 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943048+0000", 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075710159999999999 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943111+0000", 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3026000000000003e-05 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943136+0000", 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.478e-05 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943342+0000", 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020601500000000001 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943637+0000", 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000295254 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.944117+0000", 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00047971599999999999 2024-04-25T15:53:09.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963573+0000", 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019456543 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963639+0000", 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6292999999999997e-05 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963740+0000", 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010014900000000001 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963843+0000", 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010340600000000001 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:428 3.3b 3:ddecd1ef:::10000000002.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.594481547, 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026002537999999999, 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:09.904 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 428 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:09.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204318+0000", 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.027e-06 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212124+0000", 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0078060239999999999 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:09.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212191+0000", 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7162999999999994e-05 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212218+0000", 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7223e-05 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212375+0000", 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015651200000000001 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212559+0000", 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018415799999999999 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212807+0000", 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024789799999999998 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.224975+0000", 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012167661 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:09.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230128+0000", 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051534099999999998 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230234+0000", 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106326 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230318+0000", 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3135999999999998e-05 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:09.910 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:09.910 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:09.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:53:09.911 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:09.911+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:09.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:10.012 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:10.011+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:10.065 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:10.065 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:10.065 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:10.065 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:10.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3722365640 1 Processor -- start 2024-04-25T15:53:10.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3722365640 1 -- start start 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3722365640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 0x7f371c154760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3722365640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f371c15a9e0 0x7f371c15adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3722365640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f371c15a4e0 con 0x7f371c0c8b60 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3722365640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f371c06e360 con 0x7f371c15a9e0 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3722365640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f371c06f5f0 con 0x7f371c0723b0 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f371bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f371c0c8b60 0x7f371c0c7870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40176/0 (socket says 172.21.15.67:40176) 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f371bfff640 1 -- 172.21.15.67:0/1197134142 learned_addr learned my addr 172.21.15.67:0/1197134142 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f371b7fe640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 0x7f371c154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f3720902640 1 --2- 172.21.15.67:0/1197134142 >> v2:172.21.15.176:3300/0 conn(0x7f371c15a9e0 0x7f371c15adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817847274 0 0) 0x7f371c15a4e0 con 0x7f371c0c8b60 2024-04-25T15:53:10.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.070+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3704003530 con 0x7f371c0c8b60 2024-04-25T15:53:10.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3481552776 0 0) 0x7f3704003530 con 0x7f371c0c8b60 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 msgr2=0x7f371c154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 0x7f371c154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 >> v2:172.21.15.176:3300/0 conn(0x7f371c15a9e0 msgr2=0x7f371c15adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 --2- 172.21.15.67:0/1197134142 >> v2:172.21.15.176:3300/0 conn(0x7f371c15a9e0 0x7f371c15adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c06e5b0 con 0x7f371c0c8b60 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f3710001680 con 0x7f371c0c8b60 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 >> v1:172.21.15.67:6789/0 conn(0x7f371c0c8b60 legacy=0x7f371c0c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f3704003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 0x7f3704004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3704004d50 0x7f3704005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f371c06e360 con 0x7f37040039c0 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f371c06f5f0 con 0x7f3704004d50 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f370400b9d0 con 0x7f371c0723b0 2024-04-25T15:53:10.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371bfff640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f3704003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371b7fe640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 0x7f3704004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f3720902640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3704004d50 0x7f3704005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371bfff640 1 -- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 msgr2=0x7f3704004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371bfff640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 0x7f3704004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371bfff640 1 -- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3704004d50 msgr2=0x7f3704005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371bfff640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3704004d50 0x7f3704005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371bfff640 1 -- 172.21.15.67:0/1197134142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3704003530 con 0x7f37040039c0 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f3720902640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3704004d50 0x7f3704005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:10.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371b7fe640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c0723b0 0x7f3704004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:10.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371bfff640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f3704003e30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f371000c210 tx=0x7f371001cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9504c8a0860e844b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3710007d60 con 0x7f37040039c0 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37100a9020 con 0x7f37040039c0 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f371affd640 1 -- 172.21.15.67:0/1197134142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37100a5b50 con 0x7f37040039c0 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f3722365640 1 -- 172.21.15.67:0/1197134142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 msgr2=0x7f3704003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.074+0000 7f3722365640 1 --2- 172.21.15.67:0/1197134142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f3704003e30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f371000c210 tx=0x7f371001cc30 comp rx=0 tx=0).stop 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3720902640 1 -- 172.21.15.67:0/1197134142 reap_dead start 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- 172.21.15.67:0/1197134142 shutdown_connections 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- 172.21.15.67:0/1197134142 >> 172.21.15.67:0/1197134142 conn(0x7f371c0ce330 msgr2=0x7f371c0ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- 172.21.15.67:0/1197134142 shutdown_connections 2024-04-25T15:53:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- 172.21.15.67:0/1197134142 wait complete. 2024-04-25T15:53:10.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 Processor -- start 2024-04-25T15:53:10.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- start start 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f371c1f7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c150880 0x7f371c146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f371c14eaa0 0x7f371c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f371c073190 con 0x7f37040039c0 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f371c154b70 con 0x7f371c14eaa0 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3722365640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f371c0c7f80 con 0x7f371c150880 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f371c1f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f371c1f7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49414/0 (socket says 172.21.15.67:49414) 2024-04-25T15:53:10.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 -- 172.21.15.67:0/1214131462 learned_addr learned my addr 172.21.15.67:0/1214131462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371b7fe640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c150880 0x7f371c146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3720902640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f371c14eaa0 0x7f371c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 -- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c150880 msgr2=0x7f371c146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c150880 0x7f371c146c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 -- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f371c14eaa0 msgr2=0x7f371c156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f371c14eaa0 0x7f371c156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f371bfff640 1 -- 172.21.15.67:0/1214131462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c157930 con 0x7f37040039c0 2024-04-25T15:53:10.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.078+0000 7f3720902640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f371c14eaa0 0x7f371c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:10.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f371bfff640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f371c1f7020 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f371000c210 tx=0x7f3704005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3718ff9640 1 -- 172.21.15.67:0/1214131462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37100a5b30 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3718ff9640 1 -- 172.21.15.67:0/1214131462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37100b3070 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c157bf0 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3718ff9640 1 -- 172.21.15.67:0/1214131462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37100047f0 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c148cf0 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371c06a480 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3718ff9640 1 -- 172.21.15.67:0/1214131462 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3710004990 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3718ff9640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f370806f400 0x7f3708071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f3718ff9640 1 -- 172.21.15.67:0/1214131462 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7f3710128de0 con 0x7f37040039c0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f371b7fe640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f370806f400 0x7f3708071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.082+0000 7f371b7fe640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f370806f400 0x7f3708071910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f371c076140 tx=0x7f370c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.086+0000 7f3718ff9640 1 -- 172.21.15.67:0/1214131462 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3710016970 con 0x7f37040039c0 2024-04-25T15:53:10.133 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:10.134+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:10.214 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:10.211+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:10.220 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:10.221 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:10.221 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:10.221 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:10.221 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:10.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:10.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.258+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f371c1539e0 con 0x7f37040039c0 2024-04-25T15:53:10.315 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:10.311+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:10.406 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:10.406 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:10.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:10.406 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:10.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 154.19334692499999, 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:53:10.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:53:10.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:53:10.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:10.411 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 154.19329954, 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:53:10.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:53:10.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 154.19326728600001, 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:53:10.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:53:10.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:10.419 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.589077224, 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:53:10.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.298279081, 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.423 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:53:10.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.426 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:66 3.b e60/21 3:d381a89f:::10000000002.0000003e:head v 60'1, mlcod=0'0)", 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.459628940000002, 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015316048000000001, 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945216+0000", 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9e-06 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948749+0000", 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035338119999999999 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948777+0000", 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7915999999999998e-05 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948796+0000", 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.844e-05 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948895+0000", 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9704000000000006e-05 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948916+0000", 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.107e-05 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960445+0000", 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011528722999999999 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960530+0000", 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4482999999999997e-05 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:152 3.39 e61/18 3:9e24e8b4:::10000000002.00000094:head v 61'2, mlcod=0'0)", 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.667758861999999, 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016125006000000001, 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737086+0000", 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744295+0000", 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072082320000000002 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744341+0000", 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6072000000000002e-05 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744376+0000", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4903000000000001e-05 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744533+0000", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015736299999999999 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744590+0000", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6771000000000002e-05 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753101+0000", 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085115490000000002 2024-04-25T15:53:10.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753209+0000", 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107352 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:156 3.3e e61/35 3:7d47d675:::10000000002.00000098:head v 61'1, mlcod=0'0)", 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.470605388999999, 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016362634000000001, 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.434 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934240+0000", 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3149999999999999e-06 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939270+0000", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050303869999999999 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939308+0000", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8047999999999999e-05 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939335+0000", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7251999999999999e-05 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939432+0000", 2024-04-25T15:53:10.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6637000000000003e-05 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939459+0000", 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7651999999999999e-05 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950534+0000", 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011074993999999999 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950600+0000", 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5349000000000001e-05 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.437 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:158 3.12 e61/40 3:4b703321:::10000000002.0000009a:head v 61'6, mlcod=59'4)", 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.387645261999999, 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017102098, 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017200+0000", 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3199999999999998e-06 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024143+0000", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069429569999999996 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024169+0000", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6007e-05 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024190+0000", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1203999999999999e-05 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024272+0000", 2024-04-25T15:53:10.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1833000000000001e-05 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024301+0000", 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9475000000000002e-05 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034243+0000", 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099420589999999996 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034299+0000", 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6242999999999997e-05 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.441 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:162 3.2b e61/40 3:d691b431:::10000000002.0000009e:head v 61'3, mlcod=60'1)", 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.179301250999998, 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014572351000000001, 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225544+0000", 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7199999999999998e-06 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230207+0000", 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046626719999999997 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230239+0000", 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2548000000000001e-05 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230266+0000", 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6599999999999999e-05 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230382+0000", 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115667 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230417+0000", 2024-04-25T15:53:10.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5800999999999998e-05 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240029+0000", 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096115349999999992 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240114+0000", 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4808000000000005e-05 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:181 3.1f e62/40 3:f8dc3ee2:::10000000002.000000b1:head v 62'2, mlcod=0'0)", 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.354630656000001, 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015250434, 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050214+0000", 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.013e-06 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054240+0000", 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040256609999999998 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054262+0000", 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2691e-05 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054277+0000", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4464000000000001e-05 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054365+0000", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8657000000000002e-05 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054400+0000", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4505999999999998e-05 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065391+0000", 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991287000000001 2024-04-25T15:53:10.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065462+0000", 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1155000000000006e-05 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:260 3.12 e66/40 3:4bec3886:::10000000002.00000000:head v 66'8, mlcod=61'6)", 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.706375607, 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019802660999999999, 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.449 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698470+0000", 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704135+0000", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056657490000000003 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704159+0000", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3903000000000001e-05 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704176+0000", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6461e-05 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704290+0000", 2024-04-25T15:53:10.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011446999999999999 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704336+0000", 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5558e-05 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718190+0000", 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013854577999999999 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718270+0000", 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9377999999999998e-05 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.452 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:267 3.18 e67/18 3:19a74f15:::10000000002.00000007:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.388902913999999, 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015975057000000001, 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019042+0000", 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031019680000000001 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019059+0000", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6631000000000001e-05 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019075+0000", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6586e-05 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019143+0000", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7626999999999995e-05 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019174+0000", 2024-04-25T15:53:10.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1078999999999998e-05 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031852+0000", 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012678233000000001 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031915+0000", 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2422999999999995e-05 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:269 3.2c e67/54 3:354fa1eb:::10000000002.00000009:head v 67'3, mlcod=59'1)", 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:10.456 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.291868838999999, 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014629644000000001, 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4000000000000001e-07 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119407+0000", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064328099999999997 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119437+0000", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9451e-05 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119460+0000", 2024-04-25T15:53:10.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640999999999999e-05 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119565+0000", 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010474 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119600+0000", 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4851999999999998e-05 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127513+0000", 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079133090000000003 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127603+0000", 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0100999999999997e-05 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:275 3.30 e67/18 3:0ca36612:::10000000002.0000000f:head v 67'7, mlcod=62'5)", 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.026930258, 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015157541, 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377913+0000", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2499999999999995e-07 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383613+0000", 2024-04-25T15:53:10.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057000189999999997 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383639+0000", 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6475e-05 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383656+0000", 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.698e-05 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383760+0000", 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103428 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383793+0000", 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3158999999999999e-05 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393003+0000", 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092102839999999991 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393070+0000", 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6670999999999999e-05 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:276 3.1f e67/40 3:f8cab2eb:::10000000002.00000010:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.9731971809999997, 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015350925, 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:10.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431648+0000", 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3649999999999998e-06 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435504+0000", 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003856221 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435533+0000", 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9461000000000001e-05 2024-04-25T15:53:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435561+0000", 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7684000000000001e-05 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435659+0000", 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7588000000000001e-05 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435694+0000", 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5185999999999998e-05 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446705+0000", 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011010804000000001 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446996+0000", 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029161600000000002 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:353 3.1 e69/54 3:838ab1fd:::10000000002.0000005d:head v 69'7, mlcod=62'5)", 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.0076971180000003, 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017336667, 2024-04-25T15:53:10.467 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397146+0000", 2024-04-25T15:53:10.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3700000000000001e-07 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398713+0000", 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001566764 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398740+0000", 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7317e-05 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398777+0000", 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6885999999999999e-05 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398898+0000", 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121723 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398932+0000", 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3491999999999997e-05 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414418+0000", 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015486491 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414482+0000", 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3657000000000004e-05 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:357 3.34 e69/54 3:2d119e8b:::10000000002.00000061:head v 69'10, mlcod=67'8)", 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.8335249290000002, 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015675116999999999, 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571319+0000", 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.139e-06 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576161+0000", 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048417979999999996 2024-04-25T15:53:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576182+0000", 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1262000000000001e-05 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576193+0000", 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1197e-05 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576262+0000", 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9165000000000006e-05 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576283+0000", 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0582999999999999e-05 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586920+0000", 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010637275999999999 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586993+0000", 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2696999999999999e-05 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:372 3.10 e69/18 3:0b92729d:::10000000002.00000070:head v 69'7, mlcod=67'5)", 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.2868275589999998, 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015303605, 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:10.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118016+0000", 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1930000000000001e-06 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123008+0000", 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049919639999999998 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123026+0000", 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8034999999999999e-05 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123040+0000", 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3895e-05 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123138+0000", 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7929000000000004e-05 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123177+0000", 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8587999999999997e-05 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133262+0000", 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010085782 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133319+0000", 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6218999999999998e-05 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:397 3.20 e69/18 3:06fc074c:::10000000002.00000089:head v 69'12, mlcod=68'10)", 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.3563389790000002, 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015916966000000001, 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:10.478 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048506+0000", 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0439999999999998e-06 2024-04-25T15:53:10.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054867+0000", 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063613209999999996 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054891+0000", 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4320000000000001e-05 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054921+0000", 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9535000000000001e-05 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054991+0000", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0314999999999993e-05 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.055025+0000", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3809000000000001e-05 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064353+0000", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093282839999999992 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064421+0000", 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7337999999999998e-05 2024-04-25T15:53:10.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.482 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.482 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.482 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:10.482 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:10.482 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:10.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:10.483 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:10.483+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:10.584 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:10.583+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:10.640 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:10.640 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:10.640 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:10.640 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:10.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:10.701 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:10.702+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:10.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:10.785 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:10.783+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:10.808 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:10.808 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:10.808 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:10.808 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:10.808 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:10.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:10.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.862+0000 7fe0af1be640 1 Processor -- start 2024-04-25T15:53:10.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.862+0000 7fe0af1be640 1 -- start start 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0af1be640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8162ec0 0x7fe0a81652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0af1be640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a80c77a0 con 0x7fe0a8073bf0 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0af1be640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a80c7a60 con 0x7fe0a80caf60 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0af1be640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a80cab60 con 0x7fe0a8162ec0 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe0a8073bf0 0x7fe0a806d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40178/0 (socket says 172.21.15.67:40178) 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 -- 172.21.15.67:0/1131619050 learned_addr learned my addr 172.21.15.67:0/1131619050 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 235004959 0 0) 0x7fe0a80c77a0 con 0x7fe0a8073bf0 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8162ec0 0x7fe0a81652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe090003530 con 0x7fe0a8073bf0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3155998544 0 0) 0x7fe090003530 con 0x7fe0a8073bf0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8162ec0 msgr2=0x7fe0a81652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8162ec0 0x7fe0a81652e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 >> v1:172.21.15.176:6789/0 conn(0x7fe0a80caf60 legacy=0x7fe0a815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a80c7d40 con 0x7fe0a8073bf0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fe09c0016a0 con 0x7fe0a8073bf0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 >> v1:172.21.15.67:6789/0 conn(0x7fe0a8073bf0 legacy=0x7fe0a806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0900039f0 0x7fe090003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe090004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8162ec0 0x7fe090005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe09000bb60 con 0x7fe090004810 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe09000bce0 con 0x7fe0900039f0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe09000beb0 con 0x7fe0a8162ec0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe090004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0900039f0 0x7fe090003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 -- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8162ec0 msgr2=0x7fe090005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8162ec0 0x7fe090005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 -- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0900039f0 msgr2=0x7fe090003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0900039f0 0x7fe090003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0a7fff640 1 -- 172.21.15.67:0/1131619050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe090003530 con 0x7fe090004810 2024-04-25T15:53:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.866+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0900039f0 0x7fe090003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe090004c20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe0a806e270 tx=0x7fe09c01f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90360d0fd8515618 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c007d40 con 0x7fe090004810 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe09c005ae0 con 0x7fe090004810 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0a77fe640 1 -- 172.21.15.67:0/1131619050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c007070 con 0x7fe090004810 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 -- 172.21.15.67:0/1131619050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 msgr2=0x7fe090004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 --2- 172.21.15.67:0/1131619050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe090004c20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe0a806e270 tx=0x7fe09c01f370 comp rx=0 tx=0).stop 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0ad75b640 1 -- 172.21.15.67:0/1131619050 reap_dead start 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 -- 172.21.15.67:0/1131619050 shutdown_connections 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 -- 172.21.15.67:0/1131619050 >> 172.21.15.67:0/1131619050 conn(0x7fe0a8070db0 msgr2=0x7fe0a80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 -- 172.21.15.67:0/1131619050 shutdown_connections 2024-04-25T15:53:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 -- 172.21.15.67:0/1131619050 wait complete. 2024-04-25T15:53:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 Processor -- start 2024-04-25T15:53:10.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.870+0000 7fe0af1be640 1 -- start start 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe0a80cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8155010 0x7fe0a8154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0a814eb00 0x7fe0a8149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a815d960 con 0x7fe090004810 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a8159620 con 0x7fe0a814eb00 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a815dae0 con 0x7fe0a8155010 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe0a80cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe0a80cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49424/0 (socket says 172.21.15.67:49424) 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 -- 172.21.15.67:0/2933627540 learned_addr learned my addr 172.21.15.67:0/2933627540 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0a814eb00 0x7fe0a8149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8155010 0x7fe0a8154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8155010 msgr2=0x7fe0a8154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0a8155010 0x7fe0a8154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0a814eb00 msgr2=0x7fe0a8149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0a814eb00 0x7fe0a8149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a8154540 con 0x7fe090004810 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe0a80cd270 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000db40 tx=0x7fe0a00020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0003bb0 con 0x7fe090004810 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a0003d50 con 0x7fe090004810 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a80cc7d0 con 0x7fe090004810 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0006d80 con 0x7fe090004810 2024-04-25T15:53:10.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a80cca70 con 0x7fe090004810 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe0a806a480 con 0x7fe090004810 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe0a001a020 con 0x7fe090004810 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a57fa640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe09406f400 0x7fe094071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5886+0+0 (secure 0 0 0) 0x7fe0a008eef0 con 0x7fe090004810 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.874+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v71) ==== 54+0+16 (secure 0 0 0) 0x7fe0a0091570 con 0x7fe090004810 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.878+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe09406f400 0x7fe094071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.878+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe0a80cda70 0x7fe0a8149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.878+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0a80c8a40 con 0x7fe0a80cda70 2024-04-25T15:53:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.878+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe0a80cda70 0x7fe0a8149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.878+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe0a80cda70 0x7fe0a8149970 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.878+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe09406f400 0x7fe094071910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0af000 tx=0x7fe09c0aa000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.878+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a80c8a40 con 0x7fe0a80cda70 2024-04-25T15:53:10.886 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:10.883+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.894+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a80cad40 con 0x7fe0a80cda70 2024-04-25T15:53:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.894+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a80cad40 con 0x7fe0a80cda70 2024-04-25T15:53:10.895 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.894+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe0a81f3db0 0x7fe0a81f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.894+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe0a806c9c0 con 0x7fe0a81f3db0 2024-04-25T15:53:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.894+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe0a81f3db0 0x7fe0a81f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.894+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe0a81f3db0 0x7fe0a81f41c0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.898+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a806c9c0 con 0x7fe0a81f3db0 2024-04-25T15:53:10.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.910+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a806c6e0 con 0x7fe0a81f3db0 2024-04-25T15:53:10.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.914+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a806c6e0 con 0x7fe0a81f3db0 2024-04-25T15:53:10.914 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.914+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe0a8063090 0x7fe0a80634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.914+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe0a81f4c10 con 0x7fe0a8063090 2024-04-25T15:53:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.914+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe0a8063090 0x7fe0a80634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.914+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe0a8063090 0x7fe0a80634a0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.914+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a81f4c10 con 0x7fe0a8063090 2024-04-25T15:53:10.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.930+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a81f4c10 con 0x7fe0a8063090 2024-04-25T15:53:10.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.930+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a81f4c10 con 0x7fe0a8063090 2024-04-25T15:53:10.932 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.930+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe0a8063da0 0x7fe0a814dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.930+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe0a81f4c10 con 0x7fe0a8063da0 2024-04-25T15:53:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.930+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe0a8063da0 0x7fe0a814dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.930+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe0a8063da0 0x7fe0a814dcc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.934+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a81f4c10 con 0x7fe0a8063da0 2024-04-25T15:53:10.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.942+0000 7f3718ff9640 1 -- 172.21.15.67:0/1214131462 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v72) ==== 91+0+0 (secure 0 0 0) 0x7f37100f5cb0 con 0x7f37040039c0 2024-04-25T15:53:10.944 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:53:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f370806f400 msgr2=0x7f3708071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f370806f400 0x7f3708071910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f371c076140 tx=0x7f370c008040 comp rx=0 tx=0).stop 2024-04-25T15:53:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 msgr2=0x7f371c1f7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f371c1f7020 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f371000c210 tx=0x7f3704005840 comp rx=0 tx=0).stop 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 shutdown_connections 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f370806f400 0x7f3708071910 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f371c14eaa0 0x7f371c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f371c150880 0x7f371c146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 --2- 172.21.15.67:0/1214131462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37040039c0 0x7f371c1f7020 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 >> 172.21.15.67:0/1214131462 conn(0x7f371c0ce330 msgr2=0x7f371c1636e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 shutdown_connections 2024-04-25T15:53:10.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.946+0000 7f3722365640 1 -- 172.21.15.67:0/1214131462 wait complete. 2024-04-25T15:53:10.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.950+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a81f4c10 con 0x7fe0a8063da0 2024-04-25T15:53:10.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.950+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a81f4c10 con 0x7fe0a8063da0 2024-04-25T15:53:10.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:10.953 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:10.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.950+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe0a802cef0 0x7fe0a802d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.950+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe0a814e530 con 0x7fe0a802cef0 2024-04-25T15:53:10.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.954+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe0a802cef0 0x7fe0a802d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.954+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe0a802cef0 0x7fe0a802d300 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.954+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a814e530 con 0x7fe0a802cef0 2024-04-25T15:53:10.974 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:10.974 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.982+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a814e530 con 0x7fe0a802cef0 2024-04-25T15:53:10.986 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:10.986+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:10.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.986+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a814e530 con 0x7fe0a802cef0 2024-04-25T15:53:10.989 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.986+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe0a8205150 0x7fe0a8207570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.990+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe0a802de20 con 0x7fe0a8205150 2024-04-25T15:53:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.990+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe0a8205150 0x7fe0a8207570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.990+0000 7fe0a7fff640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe0a8205150 0x7fe0a8207570 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:10.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:10.990+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a802de20 con 0x7fe0a8205150 2024-04-25T15:53:11.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.006+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a80008e0 con 0x7fe0a8205150 2024-04-25T15:53:11.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.010+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a80008e0 con 0x7fe0a8205150 2024-04-25T15:53:11.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:11.011 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:11.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.010+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe0a8207fc0 0x7fe0a820a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:11.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.010+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe0a820d460 con 0x7fe0a8207fc0 2024-04-25T15:53:11.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.010+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe0a8207fc0 0x7fe0a820a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:11.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.010+0000 7fe0ad75b640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe0a8207fc0 0x7fe0a820a3e0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:11.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.014+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a820d460 con 0x7fe0a8207fc0 2024-04-25T15:53:11.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.026+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a802de20 con 0x7fe0a8207fc0 2024-04-25T15:53:11.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.030+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a802de20 con 0x7fe0a8207fc0 2024-04-25T15:53:11.031 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:11.032 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:11.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.030+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe0a820c690 0x7fe0a820cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:11.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.030+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe0a8210830 con 0x7fe0a820c690 2024-04-25T15:53:11.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.030+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe0a820c690 0x7fe0a820cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:11.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.030+0000 7fe0acf5a640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe0a820c690 0x7fe0a820cac0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:11.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.034+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0a8210830 con 0x7fe0a820c690 2024-04-25T15:53:11.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.046+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe0a820aa40 con 0x7fe0a820c690 2024-04-25T15:53:11.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0a57fa640 1 -- 172.21.15.67:0/2933627540 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0a820aa40 con 0x7fe0a820c690 2024-04-25T15:53:11.053 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe0a80cda70 msgr2=0x7fe0a8149970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fe0a80cda70 0x7fe0a8149970 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe0a81f3db0 msgr2=0x7fe0a81f41c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fe0a81f3db0 0x7fe0a81f41c0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe0a8063090 msgr2=0x7fe0a80634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fe0a8063090 0x7fe0a80634a0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe0a8063da0 msgr2=0x7fe0a814dcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fe0a8063da0 0x7fe0a814dcc0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe0a802cef0 msgr2=0x7fe0a802d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fe0a802cef0 0x7fe0a802d300 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe0a8205150 msgr2=0x7fe0a8207570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0ad75b640 1 -- 172.21.15.67:0/2933627540 reap_dead start 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fe0a8205150 0x7fe0a8207570 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0ad75b640 1 -- 172.21.15.67:0/2933627540 reap_dead start 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.050+0000 7fe0ad75b640 1 -- 172.21.15.67:0/2933627540 reap_dead start 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe0a8207fc0 msgr2=0x7fe0a820a3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fe0a8207fc0 0x7fe0a820a3e0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe0a820c690 msgr2=0x7fe0a820cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fe0a820c690 0x7fe0a820cac0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe09406f400 msgr2=0x7fe094071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe09406f400 0x7fe094071910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0af000 tx=0x7fe09c0aa000 comp rx=0 tx=0).stop 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 msgr2=0x7fe0a80cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 --2- 172.21.15.67:0/2933627540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe090004810 0x7fe0a80cd270 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000db40 tx=0x7fe0a00020f0 comp rx=0 tx=0).stop 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0ad75b640 1 -- 172.21.15.67:0/2933627540 reap_dead start 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 shutdown_connections 2024-04-25T15:53:11.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 >> 172.21.15.67:0/2933627540 conn(0x7fe0a8070db0 msgr2=0x7fe0a8157fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:11.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 shutdown_connections 2024-04-25T15:53:11.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:11.054+0000 7fe0af1be640 1 -- 172.21.15.67:0/2933627540 wait complete. 2024-04-25T15:53:11.087 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:11.087+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:11.144 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:11.144 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:11.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:11.205 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:11.206+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:11.290 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:11.287+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:11.307 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:11.307 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:11.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:11.413 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:11.414+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:11.491 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:11.490+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:11.521 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:11.521 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:11.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:11.626 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:11.626+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:11.694 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:11.691+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:11.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:11.797 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:11.798+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:11.895 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:11.895+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:11.952 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:11.953 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:11.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:11.996 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:11.995+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:12.097 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:12.098+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:12.126 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:12.126 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:12.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:12.197 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:12.195+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:12.298 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:12.298+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:12.301 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:12.301 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:12.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:12.407 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:12.407+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:12.500 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:12.499+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:12.510 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:12.510 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:12.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:12.601 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:12.602+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:12.702 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:12.702+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:12.714 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:12.714 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:12.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:12.803 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:12.802+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:12.883 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:12.883 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:12.900 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:12.903 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:12.902+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:12.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:53:13.004 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:13.003+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:13.051 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:13.051 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:13.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:13.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:13.111 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:13.111+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.134+0000 7fdf1d2a8640 1 Processor -- start 2024-04-25T15:53:13.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.134+0000 7fdf1d2a8640 1 -- start start 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf1d2a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806d460 0x7fdf180736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf1d2a8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdf181547e0 0x7fdf1806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf1d2a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf1806cb80 0x7fdf1815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf1d2a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdf18068590 con 0x7fdf1806cb80 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf1d2a8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdf180caa60 con 0x7fdf181547e0 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf1d2a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdf1815a5f0 con 0x7fdf1806d460 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf1806cb80 0x7fdf1815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf1806cb80 0x7fdf1815aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42378/0 (socket says 172.21.15.67:42378) 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 -- 172.21.15.67:0/2695266022 learned_addr learned my addr 172.21.15.67:0/2695266022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf167fc640 1 --2- 172.21.15.67:0/2695266022 >> v2:172.21.15.176:3300/0 conn(0x7fdf181547e0 0x7fdf1806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf16ffd640 1 --2- 172.21.15.67:0/2695266022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806d460 0x7fdf180736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 -- 172.21.15.67:0/2695266022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806d460 msgr2=0x7fdf180736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 --2- 172.21.15.67:0/2695266022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806d460 0x7fdf180736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 -- 172.21.15.67:0/2695266022 >> v2:172.21.15.176:3300/0 conn(0x7fdf181547e0 msgr2=0x7fdf1806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 --2- 172.21.15.67:0/2695266022 >> v2:172.21.15.176:3300/0 conn(0x7fdf181547e0 0x7fdf1806c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 -- 172.21.15.67:0/2695266022 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf180ca100 con 0x7fdf1806cb80 2024-04-25T15:53:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf177fe640 1 --2- 172.21.15.67:0/2695266022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806cb80 0x7fdf1815aaf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdf18075390 tx=0x7fdf1001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d925235dc5d8c266 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf15ffb640 1 -- 172.21.15.67:0/2695266022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf100a5760 con 0x7fdf1806cb80 2024-04-25T15:53:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf15ffb640 1 -- 172.21.15.67:0/2695266022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf100a5900 con 0x7fdf1806cb80 2024-04-25T15:53:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.138+0000 7fdf15ffb640 1 -- 172.21.15.67:0/2695266022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf100a5cd0 con 0x7fdf1806cb80 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/2695266022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806cb80 msgr2=0x7fdf1815aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/2695266022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806cb80 0x7fdf1815aaf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdf18075390 tx=0x7fdf1001c9f0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/2695266022 shutdown_connections 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/2695266022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806cb80 0x7fdf1815aaf0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/2695266022 >> v2:172.21.15.176:3300/0 conn(0x7fdf181547e0 0x7fdf1806c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/2695266022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806d460 0x7fdf180736d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/2695266022 >> 172.21.15.67:0/2695266022 conn(0x7fdf180ce1c0 msgr2=0x7fdf180ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/2695266022 shutdown_connections 2024-04-25T15:53:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/2695266022 wait complete. 2024-04-25T15:53:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 Processor -- start 2024-04-25T15:53:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- start start 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806cb80 0x7fdf180b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806d460 0x7fdf1814f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf181547e0 0x7fdf1814f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf18154bf0 con 0x7fdf1806d460 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1815b440 con 0x7fdf181547e0 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.142+0000 7fdf1d2a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdf18154d70 con 0x7fdf1806cb80 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806d460 0x7fdf1814f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806d460 0x7fdf1814f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42384/0 (socket says 172.21.15.67:42384) 2024-04-25T15:53:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 -- 172.21.15.67:0/3576415530 learned_addr learned my addr 172.21.15.67:0/3576415530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf16ffd640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806cb80 0x7fdf180b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf177fe640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf181547e0 0x7fdf1814f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 -- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806cb80 msgr2=0x7fdf180b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806cb80 0x7fdf180b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 -- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf181547e0 msgr2=0x7fdf1814f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf181547e0 0x7fdf1814f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 -- 172.21.15.67:0/3576415530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf180b2ca0 con 0x7fdf1806d460 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf167fc640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806d460 0x7fdf1814f0f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00e870 tx=0x7fdf0c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdeff7fe640 1 -- 172.21.15.67:0/3576415530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c017070 con 0x7fdf1806d460 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdeff7fe640 1 -- 172.21.15.67:0/3576415530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf0c00c940 con 0x7fdf1806d460 2024-04-25T15:53:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf180b1fa0 con 0x7fdf1806d460 2024-04-25T15:53:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdeff7fe640 1 -- 172.21.15.67:0/3576415530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c00cd30 con 0x7fdf1806d460 2024-04-25T15:53:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf180b32b0 con 0x7fdf1806d460 2024-04-25T15:53:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf1806a480 con 0x7fdf1806d460 2024-04-25T15:53:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.146+0000 7fdeff7fe640 1 -- 172.21.15.67:0/3576415530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdf0c01a070 con 0x7fdf1806d460 2024-04-25T15:53:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.150+0000 7fdeff7fe640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdf0406f400 0x7fdf04071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.150+0000 7fdeff7fe640 1 -- 172.21.15.67:0/3576415530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5944+0+0 (secure 0 0 0) 0x7fdf0c08f980 con 0x7fdf1806d460 2024-04-25T15:53:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.150+0000 7fdf16ffd640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdf0406f400 0x7fdf04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.150+0000 7fdf16ffd640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdf0406f400 0x7fdf04071910 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800b990 tx=0x7fdf08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.154+0000 7fdeff7fe640 1 -- 172.21.15.67:0/3576415530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdf0c05c310 con 0x7fdf1806d460 2024-04-25T15:53:13.206 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:13.206+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:13.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ecab640 1 Processor -- start 2024-04-25T15:53:13.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ecab640 1 -- start start 2024-04-25T15:53:13.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ecab640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e180caf60 0x7f7e1815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ecab640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e18073bf0 0x7f7e18163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ecab640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e180c77a0 con 0x7f7e1815d790 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ecab640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f7e180cab60 con 0x7f7e18073bf0 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ecab640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7e18159620 con 0x7f7e180caf60 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1d248640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e18073bf0 0x7f7e18163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1d248640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e18073bf0 0x7f7e18163ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42388/0 (socket says 172.21.15.67:42388) 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1d248640 1 -- 172.21.15.67:0/928781684 learned_addr learned my addr 172.21.15.67:0/928781684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/928781684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e180caf60 0x7f7e1815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1d248640 1 -- 172.21.15.67:0/928781684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e180caf60 msgr2=0x7f7e1815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1d248640 1 --2- 172.21.15.67:0/928781684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e180caf60 0x7f7e1815d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1d248640 1 -- 172.21.15.67:0/928781684 >> v1:172.21.15.176:6789/0 conn(0x7f7e1815d790 legacy=0x7f7e1806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.234+0000 7f7e1d248640 1 -- 172.21.15.67:0/928781684 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e180c7a60 con 0x7f7e18073bf0 2024-04-25T15:53:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1d248640 1 --2- 172.21.15.67:0/928781684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e18073bf0 0x7f7e18163ec0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009870 tx=0x7f7e0801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb30927717009c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e177fe640 1 -- 172.21.15.67:0/928781684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080b2070 con 0x7f7e18073bf0 2024-04-25T15:53:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e177fe640 1 -- 172.21.15.67:0/928781684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e080a9070 con 0x7f7e18073bf0 2024-04-25T15:53:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e177fe640 1 -- 172.21.15.67:0/928781684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080ad880 con 0x7f7e18073bf0 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 -- 172.21.15.67:0/928781684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e18073bf0 msgr2=0x7f7e18163ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/928781684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e18073bf0 0x7f7e18163ec0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009870 tx=0x7f7e0801c560 comp rx=0 tx=0).stop 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 -- 172.21.15.67:0/928781684 shutdown_connections 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/928781684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e18073bf0 0x7f7e18163ec0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/928781684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e180caf60 0x7f7e1815d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 -- 172.21.15.67:0/928781684 >> 172.21.15.67:0/928781684 conn(0x7f7e18070db0 msgr2=0x7f7e180ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 -- 172.21.15.67:0/928781684 shutdown_connections 2024-04-25T15:53:13.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.238+0000 7f7e1ecab640 1 -- 172.21.15.67:0/928781684 wait complete. 2024-04-25T15:53:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 Processor -- start 2024-04-25T15:53:13.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 -- start start 2024-04-25T15:53:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e18073bf0 0x7f7e18154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e180caf60 0x7f7e181555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e1815d790 0x7f7e180cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1815a570 con 0x7f7e180caf60 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1815a3f0 con 0x7f7e18073bf0 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ecab640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7e18166c20 con 0x7f7e1815d790 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e180caf60 0x7f7e181555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e180caf60 0x7f7e181555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42400/0 (socket says 172.21.15.67:42400) 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 -- 172.21.15.67:0/164185638 learned_addr learned my addr 172.21.15.67:0/164185638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1d248640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e1815d790 0x7f7e180cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e18073bf0 0x7f7e18154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e1815d790 msgr2=0x7f7e180cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e1815d790 0x7f7e180cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e18073bf0 msgr2=0x7f7e18154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e18073bf0 0x7f7e18154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e18155ae0 con 0x7f7e180caf60 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e18073bf0 0x7f7e18154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:13.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.242+0000 7f7e17fff640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e180caf60 0x7f7e181555a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00c3f0 tx=0x7f7e0c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c0052d0 con 0x7f7e180caf60 2024-04-25T15:53:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0c005470 con 0x7f7e180caf60 2024-04-25T15:53:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e180cd170 con 0x7f7e180caf60 2024-04-25T15:53:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c007260 con 0x7f7e180caf60 2024-04-25T15:53:13.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e181f4b40 con 0x7f7e180caf60 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7e1806a480 con 0x7f7e180caf60 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7e0c007400 con 0x7f7e180caf60 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e157fa640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7e0406f400 0x7f7e04071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5944+0+0 (secure 0 0 0) 0x7f7e0c08e590 con 0x7f7e180caf60 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v74) ==== 54+0+16 (secure 0 0 0) 0x7f7e0c094170 con 0x7f7e180caf60 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7e0406f400 0x7f7e04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7e181505c0 0x7f7e1814a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e18157da0 con 0x7f7e181505c0 2024-04-25T15:53:13.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1d248640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7e181505c0 0x7f7e1814a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1d248640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7e181505c0 0x7f7e1814a180 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.246+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7e0406f400 0x7f7e04071910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7e10003fd0 tx=0x7f7e100039e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.250+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e18157da0 con 0x7f7e181505c0 2024-04-25T15:53:13.261 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:13.261 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.262+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e18151a30 con 0x7f7e181505c0 2024-04-25T15:53:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e18151a30 con 0x7f7e181505c0 2024-04-25T15:53:13.268 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 0x7f7e181f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7e180c8d80 con 0x7f7e181f7210 2024-04-25T15:53:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 0x7f7e181f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e1ca47640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 msgr2=0x7f7e181f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 22 2024-04-25T15:53:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e1ca47640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 msgr2=0x7f7e181f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:53:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 0x7f7e181f7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:53:13.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.266+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 0x7f7e181f7620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:53:13.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:13.306 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:13.306+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:13.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.330+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fdf180cdc90 con 0x7fdf1806d460 2024-04-25T15:53:13.406 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:13.403+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:13.427 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:13.427 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 readers = 6992983.00 kB/sec 2024-04-25T15:53:13.428 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 readers = 6961503.00 kB/sec 2024-04-25T15:53:13.428 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 6992983.00 kB/sec 2024-04-25T15:53:13.428 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 6992983.00 kB/sec 2024-04-25T15:53:13.428 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 6992983.00 kB/sec 2024-04-25T15:53:13.428 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:53:13.428 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:13.428 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:13.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.470+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 0x7f7e181f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.470+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 0x7f7e181f7620 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.470+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e180c8d80 con 0x7f7e181f7210 2024-04-25T15:53:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.498+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e181641e0 con 0x7f7e181f7210 2024-04-25T15:53:13.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.502+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e181641e0 con 0x7f7e181f7210 2024-04-25T15:53:13.502 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:13.502 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:13.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.502+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7e18063090 0x7f7e180634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.502+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7e181f8070 con 0x7f7e18063090 2024-04-25T15:53:13.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.502+0000 7f7e17fff640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7e18063090 0x7f7e180634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.502+0000 7f7e17fff640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7e18063090 0x7f7e180634a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.502+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e181f8070 con 0x7f7e18063090 2024-04-25T15:53:13.507 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:13.506+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:13.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.534+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e180c8a40 con 0x7f7e18063090 2024-04-25T15:53:13.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.542+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e180c8a40 con 0x7f7e18063090 2024-04-25T15:53:13.542 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:13.542 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:13.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.542+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7e18151fe0 0x7f7e181523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.542+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7e180c90d0 con 0x7f7e18151fe0 2024-04-25T15:53:13.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.542+0000 7f7e1d248640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7e18151fe0 0x7f7e181523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.542+0000 7f7e1d248640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7e18151fe0 0x7f7e181523f0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.542+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e180c90d0 con 0x7f7e18151fe0 2024-04-25T15:53:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.566+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e18152930 con 0x7f7e18151fe0 2024-04-25T15:53:13.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.566+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e18152930 con 0x7f7e18151fe0 2024-04-25T15:53:13.568 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.566+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 0x7f7e1802cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.566+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7e1802d330 con 0x7f7e1802c9e0 2024-04-25T15:53:13.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.570+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 0x7f7e1802cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.570+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 0x7f7e1802cdf0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.570+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e1802d330 con 0x7f7e1802c9e0 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e1802d330 con 0x7f7e1802c9e0 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e1ca47640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 msgr2=0x7f7e1802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 25 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e1ca47640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 msgr2=0x7f7e1802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 0x7f7e1802cdf0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 0x7f7e1802cdf0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 msgr2=0x7f7e1802cdf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e157fa640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e1802c9e0 0x7f7e1802cdf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e157fa640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e04079840 0x7f7e0407bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e180c90d0 con 0x7f7e04079840 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=75}) -- 0x7f7e0407c1c0 con 0x7f7e180caf60 2024-04-25T15:53:13.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e04079840 0x7f7e0407bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.590+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e04079840 0x7f7e0407bc60 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.594+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e180c90d0 con 0x7f7e04079840 2024-04-25T15:53:13.597 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:13.598 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:13.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.598+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7e1802da80 0x7f7e18208590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.598+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7e1802de90 con 0x7f7e1802da80 2024-04-25T15:53:13.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.598+0000 7f7e17fff640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7e1802da80 0x7f7e18208590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.598+0000 7f7e17fff640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7e1802da80 0x7f7e18208590 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.598+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e1802de90 con 0x7f7e1802da80 2024-04-25T15:53:13.608 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:13.607+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:13.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.622+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e1802de90 con 0x7f7e1802da80 2024-04-25T15:53:13.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.626+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e1802de90 con 0x7f7e1802da80 2024-04-25T15:53:13.626 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.626+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7e18208fe0 0x7f7e1820b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.626+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7e1802d130 con 0x7f7e18208fe0 2024-04-25T15:53:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.626+0000 7f7e1d248640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7e18208fe0 0x7f7e1820b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.626+0000 7f7e1d248640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7e18208fe0 0x7f7e1820b400 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.630+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e1802d130 con 0x7f7e18208fe0 2024-04-25T15:53:13.639 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:13.639 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:13.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.650+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e1820b940 con 0x7f7e18208fe0 2024-04-25T15:53:13.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.654+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e1820b940 con 0x7f7e18208fe0 2024-04-25T15:53:13.654 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:13.655 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:13.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.654+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7e1820c060 0x7f7e1820c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:13.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.654+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7e182113c0 con 0x7f7e1820c060 2024-04-25T15:53:13.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.654+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7e1820c060 0x7f7e1820c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:13.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.654+0000 7f7e1ca47640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7e1820c060 0x7f7e1820c470 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:13.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:13.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.658+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e182113c0 con 0x7f7e1820c060 2024-04-25T15:53:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.678+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7e18210240 con 0x7f7e1820c060 2024-04-25T15:53:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e157fa640 1 -- 172.21.15.67:0/164185638 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e18210240 con 0x7f7e1820c060 2024-04-25T15:53:13.683 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7e181505c0 msgr2=0x7f7e1814a180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f7e181505c0 0x7f7e1814a180 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 msgr2=0x7f7e181f7620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f7e181f7210 0x7f7e181f7620 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7e18063090 msgr2=0x7f7e180634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f7e18063090 0x7f7e180634a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7e18151fe0 msgr2=0x7f7e181523f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f7e18151fe0 0x7f7e181523f0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e04079840 msgr2=0x7f7e0407bc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f7e04079840 0x7f7e0407bc60 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7e1802da80 msgr2=0x7f7e18208590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f7e1802da80 0x7f7e18208590 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7e18208fe0 msgr2=0x7f7e1820b400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f7e18208fe0 0x7f7e1820b400 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7e1820c060 msgr2=0x7f7e1820c470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f7e1820c060 0x7f7e1820c470 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1d248640 1 -- 172.21.15.67:0/164185638 reap_dead start 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1d248640 1 -- 172.21.15.67:0/164185638 reap_dead start 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7e0406f400 msgr2=0x7f7e04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f7e0406f400 0x7f7e04071910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7e10003fd0 tx=0x7f7e100039e0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1d248640 1 -- 172.21.15.67:0/164185638 reap_dead start 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e180caf60 msgr2=0x7f7e181555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 --2- 172.21.15.67:0/164185638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e180caf60 0x7f7e181555a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00c3f0 tx=0x7f7e0c00c8b0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1d248640 1 -- 172.21.15.67:0/164185638 reap_dead start 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1d248640 1 -- 172.21.15.67:0/164185638 reap_dead start 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1d248640 1 -- 172.21.15.67:0/164185638 reap_dead start 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 shutdown_connections 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 >> 172.21.15.67:0/164185638 conn(0x7f7e18070db0 msgr2=0x7f7e181586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 shutdown_connections 2024-04-25T15:53:13.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.682+0000 7f7e1ecab640 1 -- 172.21.15.67:0/164185638 wait complete. 2024-04-25T15:53:13.750 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:13.750+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:13.809 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:13.807+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:43 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.947414594000001, 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024229051000000001, 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.853 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902019+0000", 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9459999999999998e-06 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907768+0000", 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005748525 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907831+0000", 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2842999999999994e-05 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907857+0000", 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6636e-05 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908056+0000", 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198295 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908464+0000", 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040813600000000002 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908853+0000", 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038892400000000002 2024-04-25T15:53:13.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925332+0000", 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016479074 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925882+0000", 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000550232 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926050+0000", 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016822199999999999 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926245+0000", 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019521799999999999 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:75 3.35 3:ad096b5b:::10000000002.00000047:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 28.515921893000002, 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022276672000000001, 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.858 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333513+0000", 2024-04-25T15:53:13.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339006+0000", 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054929599999999999 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339071+0000", 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5545999999999999e-05 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339098+0000", 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6647000000000001e-05 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339299+0000", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201351 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339703+0000", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040396399999999998 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.340160+0000", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045697899999999999 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.352668+0000", 2024-04-25T15:53:13.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012508137000000001 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355605+0000", 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002937013 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355701+0000", 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5681999999999996e-05 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355786+0000", 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4908999999999996e-05 2024-04-25T15:53:13.862 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:79 3.3 3:c0361aff:::10000000002.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 28.340151798000001, 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022495096999999999, 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 79 2024-04-25T15:53:13.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509282+0000", 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2200000000000001e-06 2024-04-25T15:53:13.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517219+0000", 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079362270000000006 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517284+0000", 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5052e-05 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517310+0000", 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6072999999999999e-05 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517506+0000", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019606199999999999 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517779+0000", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027302799999999998 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.518047+0000", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000268454 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.528734+0000", 2024-04-25T15:53:13.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01068712 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531586+0000", 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028516589999999999 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531689+0000", 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010260900000000001 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531774+0000", 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5593000000000005e-05 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.867 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:177 3.26 3:66aef6e9:::10000000002.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.972743041000001, 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026823691, 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876691+0000", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4939999999999999e-06 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880736+0000", 2024-04-25T15:53:13.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040441660000000001 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880793+0000", 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7581999999999999e-05 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880817+0000", 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881039+0000", 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022171800000000001 2024-04-25T15:53:13.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881305+0000", 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026580799999999997 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881558+0000", 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025290099999999998 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.893182+0000", 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011624242 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903322+0000", 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010139754000000001 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903425+0000", 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102706 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903512+0000", 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7093000000000001e-05 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.872 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:261 3.1d e66/54 3:bbf7764d:::10000000002.00000001:head v 66'8, mlcod=63'6)", 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.089576961000001, 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022527077, 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759855+0000", 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6599999999999996e-07 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764340+0000", 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044852190000000004 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764382+0000", 2024-04-25T15:53:13.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2052999999999999e-05 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764415+0000", 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.341e-05 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764650+0000", 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023472400000000001 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764703+0000", 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3016999999999997e-05 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782246+0000", 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017543286000000002 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782381+0000", 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134702 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.876 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:268 3.14 e67/54 3:29a5f800:::10000000002.00000008:head v 67'6, mlcod=63'4)", 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.780621138000001, 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023750529999999999, 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:13.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068811+0000", 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.37e-07 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073130+0000", 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043190069999999997 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073172+0000", 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2512000000000002e-05 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073198+0000", 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6084e-05 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073426+0000", 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227514 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073510+0000", 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4469999999999996e-05 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092427+0000", 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018916571 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092560+0000", 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013373500000000001 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:269 3.2c 3:354fa1eb:::10000000002.00000009:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.744142314999999, 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022386919000000002, 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.880 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 269 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105292+0000", 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8839999999999998e-06 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111434+0000", 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061422880000000001 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111491+0000", 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7306000000000003e-05 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111515+0000", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3893e-05 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111686+0000", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017118600000000001 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111859+0000", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017279099999999999 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.112126+0000", 2024-04-25T15:53:13.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000266722 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.123964+0000", 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01183797 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127453+0000", 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034891240000000001 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127555+0000", 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102082 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127676+0000", 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120673 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:355 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.360716368, 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025437358, 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.885 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 355 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488717+0000", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.874e-06 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494412+0000", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056951349999999996 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494450+0000", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7920999999999998e-05 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494464+0000", 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4728e-05 2024-04-25T15:53:13.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494580+0000", 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011599099999999999 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494756+0000", 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176083 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.495038+0000", 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028161800000000003 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.508282+0000", 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013243981 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514019+0000", 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057372639999999997 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514099+0000", 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9800000000000002e-05 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514152+0000", 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2963000000000001e-05 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:357 3.34 3:2d119e8b:::10000000002.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.284208242, 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021511328, 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.890 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 357 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565225+0000", 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2189999999999998e-06 2024-04-25T15:53:13.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569849+0000", 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046243480000000003 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569992+0000", 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014274200000000001 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570006+0000", 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4032999999999999e-05 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570122+0000", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116358 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570290+0000", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016725299999999999 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570528+0000", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023819800000000001 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.581505+0000", 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01097657 2024-04-25T15:53:13.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586619+0000", 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051144119999999996 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586682+0000", 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3309e-05 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586734+0000", 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1885999999999998e-05 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.894 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:372 3.10 3:0b92729d:::10000000002.00000070:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7428930069999993, 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026675501000000001, 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 372 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106541+0000", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.022e-06 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110718+0000", 2024-04-25T15:53:13.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041768669999999999 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110768+0000", 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110792+0000", 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3577000000000001e-05 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110988+0000", 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019645200000000001 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111271+0000", 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028314399999999998 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111688+0000", 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041694699999999998 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.123821+0000", 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012133015 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133024+0000", 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092031290000000009 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133125+0000", 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010063199999999999 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133214+0000", 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8555999999999997e-05 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:388 3.36 3:6dca1976:::10000000002.00000080:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.1236837449999992, 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022516031999999998, 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 388 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:13.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725751+0000", 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.27e-06 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730050+0000", 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004299247 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730109+0000", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9583000000000002e-05 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730135+0000", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5316999999999998e-05 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730306+0000", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000171695 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730503+0000", 2024-04-25T15:53:13.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019674499999999999 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730797+0000", 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029401399999999999 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.741175+0000", 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010378 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748075+0000", 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068997829999999996 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748176+0000", 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101545 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748263+0000", 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6833e-05 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:394 3.36 3:6ccb00e9:::10000000002.00000086:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.9178289040000003, 2024-04-25T15:53:13.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025589158000000001, 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 394 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931605+0000", 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0730000000000001e-06 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936840+0000", 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052349129999999999 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936898+0000", 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7793e-05 2024-04-25T15:53:13.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936922+0000", 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4513000000000001e-05 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937111+0000", 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018847800000000001 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937292+0000", 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181189 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937542+0000", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024970800000000002 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956147+0000", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018605467000000001 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956986+0000", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083831599999999997 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957089+0000", 2024-04-25T15:53:13.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103379 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957191+0000", 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102329 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:397 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.8077371259999993, 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022794325000000001, 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.909 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 397 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041697+0000", 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8569999999999999e-06 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047000+0000", 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053031220000000004 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047059+0000", 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9499000000000001e-05 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047084+0000", 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4340000000000001e-05 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047231+0000", 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014741600000000001 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047415+0000", 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018419500000000001 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047667+0000", 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025212800000000001 2024-04-25T15:53:13.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.060095+0000", 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012427182 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064282+0000", 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041871410000000001 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064390+0000", 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108142 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064488+0000", 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8302999999999997e-05 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:400 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.7022017950000006, 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025113648999999998, 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.914 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 400 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147232+0000", 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.253e-06 2024-04-25T15:53:13.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151612+0000", 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004379226 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151671+0000", 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9036999999999997e-05 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151696+0000", 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4825999999999999e-05 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151891+0000", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019537499999999999 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.152208+0000", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000317041 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.152648+0000", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043998600000000001 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.168097+0000", 2024-04-25T15:53:13.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015449137 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172104+0000", 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040067260000000004 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172209+0000", 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105376 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172343+0000", 2024-04-25T15:53:13.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133666 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:401 3.34 3:2feeb5a8:::10000000002.0000008d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.6714188910000001, 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022845918, 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 401 2024-04-25T15:53:13.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178015+0000", 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5730000000000002e-06 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182028+0000", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040133800000000004 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182085+0000", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6656000000000001e-05 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182108+0000", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.287e-05 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182252+0000", 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144389 2024-04-25T15:53:13.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182427+0000", 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017484799999999999 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182676+0000", 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024908500000000002 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.196912+0000", 2024-04-25T15:53:13.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014236367 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200602+0000", 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036898819999999998 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200713+0000", 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011032899999999999 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200858+0000", 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000145539 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.923 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:406 3.11 e69/54 3:8834f03f:::10000000002.00000092:head v 69'13, mlcod=69'11)", 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.4782616229999999, 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023521068999999999, 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371172+0000", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7800000000000001e-06 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375367+0000", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004194761 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375407+0000", 2024-04-25T15:53:13.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0222999999999998e-05 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375432+0000", 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5236999999999999e-05 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375634+0000", 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020169400000000001 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375717+0000", 2024-04-25T15:53:13.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2936999999999996e-05 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.394537+0000", 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018820018000000001 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.394690+0000", 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015341899999999999 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:410 3.34 3:2cf0bbf2:::10000000002.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:13.927 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.3268644280000004, 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026112841000000001, 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 410 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522570+0000", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529416+0000", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006846062 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529476+0000", 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0350000000000003e-05 2024-04-25T15:53:13.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529502+0000", 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6259999999999999e-05 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529665+0000", 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016305200000000001 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529683+0000", 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7935999999999999e-05 2024-04-25T15:53:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534023+0000", 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043396270000000004 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534279+0000", 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025637099999999998 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534518+0000", 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238762 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.543891+0000", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093730170000000008 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548495+0000", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046033610000000003 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548592+0000", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7793000000000004e-05 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548679+0000", 2024-04-25T15:53:13.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7100000000000003e-05 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:412 3.3e 3:7d47d675:::10000000002.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.245809436, 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025248618, 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 412 2024-04-25T15:53:13.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603625+0000", 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5329999999999999e-06 2024-04-25T15:53:13.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611293+0000", 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076675720000000001 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611358+0000", 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5500000000000006e-05 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611382+0000", 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4300000000000001e-05 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611573+0000", 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019060799999999999 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611863+0000", 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029038099999999998 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.612303+0000", 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043984299999999998 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.627478+0000", 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015174632 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628552+0000", 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001074258 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628728+0000", 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017612500000000001 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628870+0000", 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014186599999999999 2024-04-25T15:53:13.937 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:505 3.3e 3:7d99c36c:::10000000002.000000f5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:09.131489+0000", 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.717941648, 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025085643000000001, 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 505 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.938 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.131489+0000", 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.131489+0000", 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.131493+0000", 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8060000000000001e-06 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.138731+0000", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072381370000000004 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.138792+0000", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0402000000000002e-05 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.138817+0000", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5117e-05 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.139023+0000", 2024-04-25T15:53:13.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000205861 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.139314+0000", 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029163799999999999 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.139731+0000", 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041711599999999998 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.154735+0000", 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015003448000000001 2024-04-25T15:53:13.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.156386+0000", 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001650966 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.156482+0000", 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6491999999999997e-05 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.156575+0000", 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2659999999999997e-05 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.942 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:514 3.19 e71/68 3:982db27a:::10000000002.000000ff:head v 71'6, mlcod=67'4)", 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:09.482959+0000", 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3664715540000003, 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021499580000000001, 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.482959+0000", 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.482959+0000", 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.482963+0000", 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.569e-06 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.490855+0000", 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0078918070000000007 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:13.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.490891+0000", 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6117e-05 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.490913+0000", 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2011999999999999e-05 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.491117+0000", 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020398700000000001 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.491187+0000", 2024-04-25T15:53:13.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9632e-05 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.504371+0000", 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013184226 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.504459+0000", 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8230000000000001e-05 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:13.946 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:13.947 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:13.948 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:13.947+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.990+0000 7fdeff7fe640 1 -- 172.21.15.67:0/3576415530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v75) ==== 101+0+0 (secure 0 0 0) 0x7fdf0c060610 con 0x7fdf1806d460 2024-04-25T15:53:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:53:13.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdf0406f400 msgr2=0x7fdf04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdf0406f400 0x7fdf04071910 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800b990 tx=0x7fdf08008040 comp rx=0 tx=0).stop 2024-04-25T15:53:13.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806d460 msgr2=0x7fdf1814f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:13.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806d460 0x7fdf1814f0f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00e870 tx=0x7fdf0c00ed30 comp rx=0 tx=0).stop 2024-04-25T15:53:13.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 shutdown_connections 2024-04-25T15:53:13.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdf0406f400 0x7fdf04071910 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf181547e0 0x7fdf1814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1806d460 0x7fdf1814f0f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 --2- 172.21.15.67:0/3576415530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf1806cb80 0x7fdf180b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:13.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 >> 172.21.15.67:0/3576415530 conn(0x7fdf180ce1c0 msgr2=0x7fdf180cac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:13.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 shutdown_connections 2024-04-25T15:53:13.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:13.994+0000 7fdf1d2a8640 1 -- 172.21.15.67:0/3576415530 wait complete. 2024-04-25T15:53:14.019 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:14.019 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:14.019 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-04-25T15:53:14.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-04-25T15:53:14.049 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:14.047+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:14.147 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:14.147 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:14.147 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:14.147 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:14.150 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:14.147+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:14.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.170+0000 7f81b4ef9640 1 Processor -- start 2024-04-25T15:53:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.170+0000 7f81b4ef9640 1 -- start start 2024-04-25T15:53:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81b4ef9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81b015ab30 0x7f81b0165fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81b4ef9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b015de10 con 0x7f81b0071bc0 2024-04-25T15:53:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81b4ef9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b0072390 con 0x7f81b015a720 2024-04-25T15:53:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81b4ef9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f81b015ebb0 con 0x7f81b015ab30 2024-04-25T15:53:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81aed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81b015ab30 0x7f81b0165fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81ae59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81b0071bc0 0x7f81b00cbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55304/0 (socket says 172.21.15.67:55304) 2024-04-25T15:53:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81ae59c640 1 -- 172.21.15.67:0/3038040468 learned_addr learned my addr 172.21.15.67:0/3038040468 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81aed9d640 1 -- 172.21.15.67:0/3038040468 >> v1:172.21.15.176:6789/0 conn(0x7f81b015a720 legacy=0x7f81b00ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81aed9d640 1 -- 172.21.15.67:0/3038040468 >> v1:172.21.15.67:6789/0 conn(0x7f81b0071bc0 legacy=0x7f81b00cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81add9b640 1 -- 172.21.15.67:0/3038040468 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2058822976 0 0) 0x7f81b015de10 con 0x7f81b0071bc0 2024-04-25T15:53:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.174+0000 7f81aed9d640 1 -- 172.21.15.67:0/3038040468 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b015e0d0 con 0x7f81b015ab30 2024-04-25T15:53:14.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81aed9d640 1 --2- 172.21.15.67:0/3038040468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b0165fd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009d90 tx=0x7f81a801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c295c94ee552b446 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:14.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81add9b640 1 -- 172.21.15.67:0/3038040468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a80b2070 con 0x7f81b015ab30 2024-04-25T15:53:14.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81add9b640 1 -- 172.21.15.67:0/3038040468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a80a5940 con 0x7f81b015ab30 2024-04-25T15:53:14.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81add9b640 1 -- 172.21.15.67:0/3038040468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a80a5d10 con 0x7f81b015ab30 2024-04-25T15:53:14.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3038040468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 msgr2=0x7f81b0165fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:14.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3038040468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b0165fd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009d90 tx=0x7f81a801c970 comp rx=0 tx=0).stop 2024-04-25T15:53:14.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3038040468 shutdown_connections 2024-04-25T15:53:14.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3038040468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b0165fd0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:14.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3038040468 >> 172.21.15.67:0/3038040468 conn(0x7f81b00700b0 msgr2=0x7f81b00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:14.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3038040468 shutdown_connections 2024-04-25T15:53:14.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.178+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3038040468 wait complete. 2024-04-25T15:53:14.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 Processor -- start 2024-04-25T15:53:14.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 -- start start 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81b0071bc0 0x7f81b01f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81b015a720 0x7f81b00cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b01465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f81b00cc2d0 con 0x7f81b015ab30 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f81b015e720 con 0x7f81b0071bc0 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f81b0071fd0 con 0x7f81b015a720 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b01465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b01465a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42424/0 (socket says 172.21.15.67:42424) 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 -- 172.21.15.67:0/3289570228 learned_addr learned my addr 172.21.15.67:0/3289570228 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81ae59c640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81b0071bc0 0x7f81b01f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81a7fff640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81b015a720 0x7f81b00cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:14.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 -- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81b015a720 msgr2=0x7f81b00cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81b015a720 0x7f81b00cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 -- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81b0071bc0 msgr2=0x7f81b01f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81b0071bc0 0x7f81b01f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 -- 172.21.15.67:0/3289570228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b01f76c0 con 0x7f81b015ab30 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81aed9d640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b01465a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f81a801a5a0 tx=0x7f81a8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81a6ffd640 1 -- 172.21.15.67:0/3289570228 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a80b2040 con 0x7f81b015ab30 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81a6ffd640 1 -- 172.21.15.67:0/3289570228 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a80bf070 con 0x7f81b015ab30 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b014f8a0 con 0x7f81b015ab30 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81a6ffd640 1 -- 172.21.15.67:0/3289570228 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a80ad8c0 con 0x7f81b015ab30 2024-04-25T15:53:14.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b014fe60 con 0x7f81b015ab30 2024-04-25T15:53:14.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.182+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b006a480 con 0x7f81b015ab30 2024-04-25T15:53:14.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.186+0000 7f81a6ffd640 1 -- 172.21.15.67:0/3289570228 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f81a80ada60 con 0x7f81b015ab30 2024-04-25T15:53:14.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.186+0000 7f81a6ffd640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f819006f400 0x7f8190071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:14.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.186+0000 7f81a6ffd640 1 -- 172.21.15.67:0/3289570228 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5944+0+0 (secure 0 0 0) 0x7f81a812a360 con 0x7f81b015ab30 2024-04-25T15:53:14.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.186+0000 7f81ae59c640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f819006f400 0x7f8190071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:14.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.186+0000 7f81ae59c640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f819006f400 0x7f8190071910 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8198007c40 tx=0x7f8198007b90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:14.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.190+0000 7f81a6ffd640 1 -- 172.21.15.67:0/3289570228 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f81a8016560 con 0x7f81b015ab30 2024-04-25T15:53:14.251 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:14.251+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:14.352 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:14.351+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:14.360 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:14.360 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:14.360 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:14.360 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:14.361 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:14.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.366+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd in", "ids": ["4"]} v 0) -- 0x7f81b014c2b0 con 0x7f81b015ab30 2024-04-25T15:53:14.369 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-25T15:53:14.370+0000 7fa4fdaf0640 -1 mon.a@0(leader).osd e75 definitely_dead 0 2024-04-25T15:53:14.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:14.467 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:14.467+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:14.554 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:14.551+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:14.568 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:14.568 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:14.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:14.568 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:14.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.568 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:14.568 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 175.466595638, 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:14.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:14.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 174.87539271399999, 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:53:14.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:53:14.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:14.579 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 174.875381635, 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.584 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:62 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.809406743, 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025889728000000001, 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 62 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755657+0000", 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9439999999999998e-06 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760593+0000", 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049360740000000004 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760651+0000", 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8e-05 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760676+0000", 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5219999999999999e-05 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760847+0000", 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017057199999999999 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761123+0000", 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027578100000000001 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761365+0000", 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024248100000000001 2024-04-25T15:53:14.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780642+0000", 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019277327 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781358+0000", 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000715611 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781456+0000", 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7946e-05 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781543+0000", 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6772000000000002e-05 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:150 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.913887810999999, 2024-04-25T15:53:14.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024331635000000001, 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 150 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651176+0000", 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8770000000000003e-06 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659329+0000", 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081530219999999994 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659399+0000", 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0376000000000005e-05 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659428+0000", 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8952999999999999e-05 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659580+0000", 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015188199999999999 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659842+0000", 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026179399999999998 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.660120+0000", 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027824799999999998 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.673813+0000", 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013693145 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675307+0000", 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014934600000000001 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675412+0000", 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105233 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675504+0000", 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1644999999999994e-05 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:155 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.691736526, 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025243293999999999, 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 155 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873327+0000", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7699999999999999e-06 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879173+0000", 2024-04-25T15:53:14.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005846141 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879239+0000", 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6315999999999993e-05 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879267+0000", 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7137000000000001e-05 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879415+0000", 2024-04-25T15:53:14.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000148715 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879693+0000", 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000278108 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879951+0000", 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025793700000000002 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.896019+0000", 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016067814999999999 2024-04-25T15:53:14.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898153+0000", 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021339280000000002 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898411+0000", 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025760799999999999 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898566+0000", 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015581899999999999 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.600 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:165 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.195765989000002, 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032658418000000002, 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 165 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.601 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369298+0000", 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3429999999999998e-06 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374706+0000", 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054080250000000003 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374776+0000", 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9972e-05 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374803+0000", 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7384000000000001e-05 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375010+0000", 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206431 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375037+0000", 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7206999999999999e-05 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380232+0000", 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051947859999999998 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380474+0000", 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000241864 2024-04-25T15:53:14.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380709+0000", 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023560200000000001 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.392962+0000", 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012252219999999999 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401765+0000", 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088034089999999999 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401866+0000", 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101099 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401952+0000", 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6075999999999994e-05 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:14.606 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.347355156999999, 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:14.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.252721104999999, 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:14.611 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:14.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:260 3.12 3:4bec3886:::10000000002.00000000:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.875079864, 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028352792000000002, 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.616 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 260 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689983+0000", 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2729999999999998e-06 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696571+0000", 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065880749999999997 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696637+0000", 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5358999999999995e-05 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696662+0000", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5184000000000001e-05 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697262+0000", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060013099999999995 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697437+0000", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017512499999999999 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697672+0000", 2024-04-25T15:53:14.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023489800000000001 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.715797+0000", 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018125601000000002 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718141+0000", 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023436300000000002 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718244+0000", 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102379 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718333+0000", 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9136999999999997e-05 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:268 3.14 3:29a5f800:::10000000002.00000008:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.504132475, 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032265195000000003, 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 268 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060931+0000", 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6629999999999999e-06 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067652+0000", 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067213899999999998 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067713+0000", 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0628999999999998e-05 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067739+0000", 2024-04-25T15:53:14.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6492999999999999e-05 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067893+0000", 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015336599999999999 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068083+0000", 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019004800000000001 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068328+0000", 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024518100000000002 2024-04-25T15:53:14.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.087697+0000", 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019368720999999998 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092957+0000", 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052597859999999998 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093058+0000", 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101335 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093192+0000", 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134583 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:271 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.371737326, 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026911917, 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 271 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193326+0000", 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0770000000000001e-06 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198319+0000", 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049928710000000003 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198382+0000", 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2446000000000005e-05 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198408+0000", 2024-04-25T15:53:14.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6676999999999999e-05 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198623+0000", 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021493799999999999 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198914+0000", 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029086100000000001 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.199346+0000", 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000431316 2024-04-25T15:53:14.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.219423+0000", 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020076954000000001 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220028+0000", 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060503900000000003 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220153+0000", 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125919 2024-04-25T15:53:14.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220234+0000", 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0819e-05 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:347 3.11 3:8815798b:::10000000002.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.416671183, 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029152658000000001, 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.631 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 347 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148391+0000", 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6029999999999999e-06 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155889+0000", 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074977200000000003 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155942+0000", 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2883999999999997e-05 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155964+0000", 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2762999999999999e-05 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156184+0000", 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156358+0000", 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017470800000000001 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156597+0000", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238411 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.172619+0000", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016022649 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177350+0000", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047301369999999997 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177455+0000", 2024-04-25T15:53:14.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010574899999999999 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177541+0000", 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5839000000000003e-05 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:352 3.7 e68/18 3:e0f5f483:::10000000002.0000005c:head v 68'11, mlcod=67'9)", 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.225084663000001, 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026288315999999999, 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.636 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339978+0000", 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8459999999999999e-06 2024-04-25T15:53:14.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345285+0000", 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053072630000000004 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345324+0000", 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8535999999999998e-05 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345348+0000", 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4315999999999999e-05 2024-04-25T15:53:14.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345503+0000", 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015497300000000001 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345555+0000", 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1814999999999998e-05 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366145+0000", 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020589743000000001 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366263+0000", 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118824 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:326 3.8 e69/68 3:133e839e:::10000000002.00000042:head v 69'10, mlcod=0'0)", 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.127563028999999, 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025085139999999999, 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437498+0000", 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.632e-06 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445817+0000", 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083191740000000004 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445839+0000", 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1314e-05 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445854+0000", 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4986e-05 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445948+0000", 2024-04-25T15:53:14.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3735999999999998e-05 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445981+0000", 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3210999999999998e-05 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462484+0000", 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016502871999999998 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462582+0000", 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8214999999999994e-05 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:367 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.620746685, 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025258800000000001, 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 367 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944316+0000", 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4880000000000001e-06 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950833+0000", 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065179829999999998 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950884+0000", 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950902+0000", 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8805e-05 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951039+0000", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013634300000000001 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951216+0000", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017729099999999999 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951469+0000", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025263199999999998 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969276+0000", 2024-04-25T15:53:14.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017807630000000001 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969326+0000", 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9639999999999999e-05 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969434+0000", 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108155 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969572+0000", 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013767299999999999 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.648 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:406 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.2046399700000006, 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034893977, 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 406 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.649 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360424+0000", 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2300000000000002e-06 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370215+0000", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0097913340000000005 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370283+0000", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8028999999999995e-05 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370311+0000", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8053999999999999e-05 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370464+0000", 2024-04-25T15:53:14.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015309899999999999 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370642+0000", 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177648 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370885+0000", 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000242721 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.393703+0000", 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022818037999999999 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395071+0000", 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001368469 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395176+0000", 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104949 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395314+0000", 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013740599999999999 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:411 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.0055722990000007, 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024254155999999999, 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 411 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559491+0000", 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4699999999999998e-06 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566706+0000", 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072154760000000002 2024-04-25T15:53:14.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566772+0000", 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5646000000000001e-05 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566807+0000", 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4672999999999998e-05 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567067+0000", 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026066400000000002 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567252+0000", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018467599999999999 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567510+0000", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025795100000000002 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.582484+0000", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014973584 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583549+0000", 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001065365 2024-04-25T15:53:14.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583653+0000", 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010430899999999999 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583742+0000", 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8342000000000002e-05 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:421 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:14.658 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.6295869839999995, 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028370244999999999, 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 421 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935477+0000", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0470000000000004e-06 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943048+0000", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075710159999999999 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943111+0000", 2024-04-25T15:53:14.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3026000000000003e-05 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943136+0000", 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.478e-05 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943342+0000", 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020601500000000001 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943637+0000", 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000295254 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.944117+0000", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00047971599999999999 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963573+0000", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019456543 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963639+0000", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6292999999999997e-05 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963740+0000", 2024-04-25T15:53:14.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010014900000000001 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963843+0000", 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010340600000000001 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:428 3.3b 3:ddecd1ef:::10000000002.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.3607447350000008, 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026002537999999999, 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:14.663 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 428 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204318+0000", 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.027e-06 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212124+0000", 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0078060239999999999 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212191+0000", 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7162999999999994e-05 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:14.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212218+0000", 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7223e-05 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212375+0000", 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015651200000000001 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212559+0000", 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018415799999999999 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212807+0000", 2024-04-25T15:53:14.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024789799999999998 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.224975+0000", 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012167661 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230128+0000", 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051534099999999998 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230234+0000", 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106326 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:14.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230318+0000", 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3135999999999998e-05 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:14.668 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:14.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:14.670 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:14.667+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:14.770 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:14.770+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:14.831 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:14.831 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:14.831 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:14.831 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:14.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:14.871 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:14.871+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:14.972 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:14.970+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:14.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.990+0000 7f81a6ffd640 1 -- 172.21.15.67:0/3289570228 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["4"]}]=0 marked in osd.4. v76) ==== 85+0+0 (secure 0 0 0) 0x7f81a80f6ba0 con 0x7f81b015ab30 2024-04-25T15:53:14.992 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-04-25T15:53:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f819006f400 msgr2=0x7f8190071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f819006f400 0x7f8190071910 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8198007c40 tx=0x7f8198007b90 comp rx=0 tx=0).stop 2024-04-25T15:53:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 msgr2=0x7f81b01465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b01465a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f81a801a5a0 tx=0x7f81a8019710 comp rx=0 tx=0).stop 2024-04-25T15:53:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 shutdown_connections 2024-04-25T15:53:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f819006f400 0x7f8190071910 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81b015ab30 0x7f81b01465a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f81b015a720 0x7f81b00cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 --2- 172.21.15.67:0/3289570228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81b0071bc0 0x7f81b01f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 >> 172.21.15.67:0/3289570228 conn(0x7f81b00700b0 msgr2=0x7f81b00ca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 shutdown_connections 2024-04-25T15:53:14.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:14.994+0000 7f81b4ef9640 1 -- 172.21.15.67:0/3289570228 wait complete. 2024-04-25T15:53:15.026 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-04-25T15:53:15.050 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:15.050 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:15.051 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:15.051 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:15.051 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:15.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:15.109 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:15.110+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:15.174 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:15.171+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 159.04953584, 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.263 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:53:15.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.267 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 159.04948845499999, 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:53:15.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 159.049456201, 2024-04-25T15:53:15.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:53:15.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:53:15.275 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.445266139000001, 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:53:15.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:53:15.279 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.154467996000001, 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:53:15.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:66 3.b e60/21 3:d381a89f:::10000000002.0000003e:head v 60'1, mlcod=0'0)", 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.315817854999999, 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015316048000000001, 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945216+0000", 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9e-06 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948749+0000", 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035338119999999999 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948777+0000", 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7915999999999998e-05 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948796+0000", 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.844e-05 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948895+0000", 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9704000000000006e-05 2024-04-25T15:53:15.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948916+0000", 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.107e-05 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960445+0000", 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011528722999999999 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960530+0000", 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4482999999999997e-05 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.286 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:152 3.39 e61/18 3:9e24e8b4:::10000000002.00000094:head v 61'2, mlcod=0'0)", 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.523947777, 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016125006000000001, 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737086+0000", 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744295+0000", 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072082320000000002 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744341+0000", 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6072000000000002e-05 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744376+0000", 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4903000000000001e-05 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744533+0000", 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015736299999999999 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744590+0000", 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6771000000000002e-05 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753101+0000", 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085115490000000002 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753209+0000", 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107352 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:156 3.3e e61/35 3:7d47d675:::10000000002.00000098:head v 61'1, mlcod=0'0)", 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.326794304, 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016362634000000001, 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934240+0000", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3149999999999999e-06 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939270+0000", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050303869999999999 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939308+0000", 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8047999999999999e-05 2024-04-25T15:53:15.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939335+0000", 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7251999999999999e-05 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939432+0000", 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6637000000000003e-05 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939459+0000", 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7651999999999999e-05 2024-04-25T15:53:15.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950534+0000", 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011074993999999999 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950600+0000", 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5349000000000001e-05 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:158 3.12 e61/40 3:4b703321:::10000000002.0000009a:head v 61'6, mlcod=59'4)", 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:15.294 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.243834177, 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017102098, 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017200+0000", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3199999999999998e-06 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024143+0000", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069429569999999996 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024169+0000", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6007e-05 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024190+0000", 2024-04-25T15:53:15.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1203999999999999e-05 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024272+0000", 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1833000000000001e-05 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024301+0000", 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9475000000000002e-05 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034243+0000", 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099420589999999996 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034299+0000", 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6242999999999997e-05 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:162 3.2b e61/40 3:d691b431:::10000000002.0000009e:head v 61'3, mlcod=60'1)", 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.035490165999999, 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014572351000000001, 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.298 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225544+0000", 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7199999999999998e-06 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230207+0000", 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046626719999999997 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230239+0000", 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2548000000000001e-05 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230266+0000", 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6599999999999999e-05 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230382+0000", 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115667 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230417+0000", 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5800999999999998e-05 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240029+0000", 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096115349999999992 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240114+0000", 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4808000000000005e-05 2024-04-25T15:53:15.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:181 3.1f e62/40 3:f8dc3ee2:::10000000002.000000b1:head v 62'2, mlcod=0'0)", 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.210819570999998, 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015250434, 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050214+0000", 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.013e-06 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054240+0000", 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040256609999999998 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054262+0000", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2691e-05 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054277+0000", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4464000000000001e-05 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054365+0000", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8657000000000002e-05 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054400+0000", 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4505999999999998e-05 2024-04-25T15:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065391+0000", 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991287000000001 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065462+0000", 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1155000000000006e-05 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:260 3.12 e66/40 3:4bec3886:::10000000002.00000000:head v 66'8, mlcod=61'6)", 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.562564522000001, 2024-04-25T15:53:15.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019802660999999999, 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698470+0000", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704135+0000", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056657490000000003 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704159+0000", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3903000000000001e-05 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704176+0000", 2024-04-25T15:53:15.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6461e-05 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704290+0000", 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011446999999999999 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704336+0000", 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5558e-05 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718190+0000", 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013854577999999999 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718270+0000", 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9377999999999998e-05 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:267 3.18 e67/18 3:19a74f15:::10000000002.00000007:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.245091829, 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015975057000000001, 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019042+0000", 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031019680000000001 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019059+0000", 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6631000000000001e-05 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019075+0000", 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6586e-05 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019143+0000", 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7626999999999995e-05 2024-04-25T15:53:15.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019174+0000", 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1078999999999998e-05 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031852+0000", 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012678233000000001 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031915+0000", 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2422999999999995e-05 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:269 3.2c e67/54 3:354fa1eb:::10000000002.00000009:head v 67'3, mlcod=59'1)", 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.148057754, 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014629644000000001, 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4000000000000001e-07 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119407+0000", 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064328099999999997 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119437+0000", 2024-04-25T15:53:15.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9451e-05 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119460+0000", 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640999999999999e-05 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119565+0000", 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010474 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119600+0000", 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4851999999999998e-05 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127513+0000", 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079133090000000003 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127603+0000", 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0100999999999997e-05 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:275 3.30 e67/18 3:0ca36612:::10000000002.0000000f:head v 67'7, mlcod=62'5)", 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.883119173000001, 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015157541, 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.316 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377913+0000", 2024-04-25T15:53:15.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2499999999999995e-07 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383613+0000", 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057000189999999997 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383639+0000", 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6475e-05 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383656+0000", 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.698e-05 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383760+0000", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103428 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383793+0000", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3158999999999999e-05 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393003+0000", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092102839999999991 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393070+0000", 2024-04-25T15:53:15.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6670999999999999e-05 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:276 3.1f e67/40 3:f8cab2eb:::10000000002.00000010:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.829386096, 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015350925, 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:15.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431648+0000", 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3649999999999998e-06 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435504+0000", 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003856221 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435533+0000", 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9461000000000001e-05 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435561+0000", 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7684000000000001e-05 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435659+0000", 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7588000000000001e-05 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435694+0000", 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5185999999999998e-05 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446705+0000", 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011010804000000001 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446996+0000", 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029161600000000002 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.323 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:353 3.1 e69/54 3:838ab1fd:::10000000002.0000005d:head v 69'7, mlcod=62'5)", 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.863886033, 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017336667, 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397146+0000", 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3700000000000001e-07 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398713+0000", 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001566764 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398740+0000", 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7317e-05 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398777+0000", 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6885999999999999e-05 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398898+0000", 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121723 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398932+0000", 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3491999999999997e-05 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414418+0000", 2024-04-25T15:53:15.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015486491 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414482+0000", 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3657000000000004e-05 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:357 3.34 e69/54 3:2d119e8b:::10000000002.00000061:head v 69'10, mlcod=67'8)", 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.689713844, 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015675116999999999, 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.327 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571319+0000", 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.139e-06 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576161+0000", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048417979999999996 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576182+0000", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1262000000000001e-05 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576193+0000", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1197e-05 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576262+0000", 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9165000000000006e-05 2024-04-25T15:53:15.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576283+0000", 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0582999999999999e-05 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586920+0000", 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010637275999999999 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586993+0000", 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2696999999999999e-05 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.330 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:372 3.10 e69/18 3:0b92729d:::10000000002.00000070:head v 69'7, mlcod=67'5)", 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.143016474, 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015303605, 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:15.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118016+0000", 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1930000000000001e-06 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123008+0000", 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049919639999999998 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123026+0000", 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8034999999999999e-05 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123040+0000", 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3895e-05 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123138+0000", 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7929000000000004e-05 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123177+0000", 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8587999999999997e-05 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133262+0000", 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010085782 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133319+0000", 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6218999999999998e-05 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.334 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:397 3.20 e69/18 3:06fc074c:::10000000002.00000089:head v 69'12, mlcod=68'10)", 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.212527894000001, 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015916966000000001, 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048506+0000", 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0439999999999998e-06 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054867+0000", 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063613209999999996 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054891+0000", 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4320000000000001e-05 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054921+0000", 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9535000000000001e-05 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054991+0000", 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0314999999999993e-05 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.055025+0000", 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3809000000000001e-05 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:15.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064353+0000", 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093282839999999992 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064421+0000", 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7337999999999998e-05 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:15.338 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:15.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:15.340 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:15.338+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:15.440 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:15.438+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:15.537 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:15.537 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:15.540 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:15.539+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:15.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:15.641 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:15.639+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:15.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:15.749 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:15.749 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:15.749 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:15.749 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:15.749 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:15.749 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:15.750+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:15.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:15.837 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:15.837 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 re-readers = 5399047.00 kB/sec 2024-04-25T15:53:15.837 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 re-readers = 5380486.78 kB/sec 2024-04-25T15:53:15.838 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 5399047.00 kB/sec 2024-04-25T15:53:15.838 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 5399047.00 kB/sec 2024-04-25T15:53:15.838 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 5399047.00 kB/sec 2024-04-25T15:53:15.838 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:53:15.843 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:15.843+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:15.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a265e640 1 Processor -- start 2024-04-25T15:53:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a265e640 1 -- start start 2024-04-25T15:53:15.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a265e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f279c156c70 0x7f279c0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a265e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f279c0cd9b0 0x7f279c15cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a265e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279c0b7c20 con 0x7f279c0ce260 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a265e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f279c0ce840 con 0x7f279c156c70 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a265e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f279c0ceb20 con 0x7f279c0cd9b0 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f279b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f279c156c70 0x7f279c0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f279b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f279c156c70 0x7f279c0cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42428/0 (socket says 172.21.15.67:42428) 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f279b7fe640 1 -- 172.21.15.67:0/1037342883 learned_addr learned my addr 172.21.15.67:0/1037342883 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1037342883 >> v2:172.21.15.176:3300/0 conn(0x7f279c0cd9b0 0x7f279c15cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f279b7fe640 1 -- 172.21.15.67:0/1037342883 >> v2:172.21.15.176:3300/0 conn(0x7f279c0cd9b0 msgr2=0x7f279c15cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1037342883 >> v2:172.21.15.176:3300/0 conn(0x7f279c0cd9b0 0x7f279c15cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f279b7fe640 1 -- 172.21.15.67:0/1037342883 >> v1:172.21.15.176:6789/0 conn(0x7f279c0ce260 legacy=0x7f279c0b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.862+0000 7f279b7fe640 1 -- 172.21.15.67:0/1037342883 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c0cee00 con 0x7f279c156c70 2024-04-25T15:53:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1037342883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c156c70 0x7f279c0cd470 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f279c0ce670 tx=0x7f278401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bdebf764a631f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f279affd640 1 -- 172.21.15.67:0/1037342883 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840b5070 con 0x7f279c156c70 2024-04-25T15:53:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f279affd640 1 -- 172.21.15.67:0/1037342883 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27840a89d0 con 0x7f279c156c70 2024-04-25T15:53:15.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f279affd640 1 -- 172.21.15.67:0/1037342883 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840a8da0 con 0x7f279c156c70 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 -- 172.21.15.67:0/1037342883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c156c70 msgr2=0x7f279c0cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 --2- 172.21.15.67:0/1037342883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c156c70 0x7f279c0cd470 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f279c0ce670 tx=0x7f278401ecd0 comp rx=0 tx=0).stop 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 -- 172.21.15.67:0/1037342883 shutdown_connections 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 --2- 172.21.15.67:0/1037342883 >> v2:172.21.15.176:3300/0 conn(0x7f279c0cd9b0 0x7f279c15cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 --2- 172.21.15.67:0/1037342883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c156c70 0x7f279c0cd470 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 -- 172.21.15.67:0/1037342883 >> 172.21.15.67:0/1037342883 conn(0x7f279c06e700 msgr2=0x7f279c06eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 -- 172.21.15.67:0/1037342883 shutdown_connections 2024-04-25T15:53:15.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.866+0000 7f27a265e640 1 -- 172.21.15.67:0/1037342883 wait complete. 2024-04-25T15:53:15.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 Processor -- start 2024-04-25T15:53:15.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 -- start start 2024-04-25T15:53:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f279c0cd9b0 0x7f279c1f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c0ce260 0x7f279c15aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f279c156c70 0x7f279c14d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f279c157080 con 0x7f279c0ce260 2024-04-25T15:53:15.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f279c0cf3c0 con 0x7f279c0cd9b0 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f279c157200 con 0x7f279c156c70 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c0ce260 0x7f279c15aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c0ce260 0x7f279c15aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42442/0 (socket says 172.21.15.67:42442) 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 -- 172.21.15.67:0/1771092073 learned_addr learned my addr 172.21.15.67:0/1771092073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f279c0cd9b0 0x7f279c1f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f279c156c70 0x7f279c14d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f279c156c70 msgr2=0x7f279c14d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f279c156c70 0x7f279c14d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f279c0cd9b0 msgr2=0x7f279c1f7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f279c0cd9b0 0x7f279c1f7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c1585e0 con 0x7f279c0ce260 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f279c0cd9b0 0x7f279c1f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c0ce260 0x7f279c15aee0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f27840b3040 tx=0x7f278401ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840c0070 con 0x7f279c0ce260 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27840b0a50 con 0x7f279c0ce260 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27840b5070 con 0x7f279c0ce260 2024-04-25T15:53:15.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279c15b420 con 0x7f279c0ce260 2024-04-25T15:53:15.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.870+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f279c157d40 con 0x7f279c0ce260 2024-04-25T15:53:15.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f279c06a480 con 0x7f279c0ce260 2024-04-25T15:53:15.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f27840ba3e0 con 0x7f279c0ce260 2024-04-25T15:53:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f2798ff9640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f277c06f430 0x7f277c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5915+0+0 (secure 0 0 0) 0x7f278412c650 con 0x7f279c0ce260 2024-04-25T15:53:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v76) ==== 54+0+16 (secure 0 0 0) 0x7f2784018830 con 0x7f279c0ce260 2024-04-25T15:53:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f277c06f430 0x7f277c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f279c14dbd0 0x7f279c068590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f279c06eb10 con 0x7f279c14dbd0 2024-04-25T15:53:15.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f279c14dbd0 0x7f279c068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f279c14dbd0 0x7f279c068590 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f277c06f430 0x7f277c071940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f278c009fc0 tx=0x7f278c009550 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.874+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c06eb10 con 0x7f279c14dbd0 2024-04-25T15:53:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.902+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c15a470 con 0x7f279c14dbd0 2024-04-25T15:53:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.906+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c15a470 con 0x7f279c14dbd0 2024-04-25T15:53:15.906 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.906+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f279c14aa30 0x7f279c1588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.906+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f279c0ce040 con 0x7f279c14aa30 2024-04-25T15:53:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.906+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f279c14aa30 0x7f279c1588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.906+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f279c14aa30 0x7f279c1588f0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.906+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c0ce040 con 0x7f279c14aa30 2024-04-25T15:53:15.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.922+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c0c7620 con 0x7f279c14aa30 2024-04-25T15:53:15.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.922+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c0c7620 con 0x7f279c14aa30 2024-04-25T15:53:15.924 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:15.925 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:15.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.922+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f279c159340 0x7f279c063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.922+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f279c159750 con 0x7f279c159340 2024-04-25T15:53:15.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.922+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f279c159340 0x7f279c063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.922+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f279c159340 0x7f279c063090 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.926+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c159750 con 0x7f279c159340 2024-04-25T15:53:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.938+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c0cca30 con 0x7f279c159340 2024-04-25T15:53:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.942+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c0cca30 con 0x7f279c159340 2024-04-25T15:53:15.942 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.942+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f279c063ae0 0x7f279c0cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.942+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f279c063ef0 con 0x7f279c063ae0 2024-04-25T15:53:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.942+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f279c063ae0 0x7f279c0cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.942+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f279c063ae0 0x7f279c0cfb70 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.942+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c063ef0 con 0x7f279c063ae0 2024-04-25T15:53:15.944 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:15.943+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:15.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.958+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c063ef0 con 0x7f279c063ae0 2024-04-25T15:53:15.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.958+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c063ef0 con 0x7f279c063ae0 2024-04-25T15:53:15.960 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.958+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f279c0d05c0 0x7f279c02c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.958+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f279c0d09d0 con 0x7f279c0d05c0 2024-04-25T15:53:15.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.958+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f279c0d05c0 0x7f279c02c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.962+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f279c0d05c0 0x7f279c02c9e0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.962+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c0d09d0 con 0x7f279c0d05c0 2024-04-25T15:53:15.977 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:15.977 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:15.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.978+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c0d09d0 con 0x7f279c0d05c0 2024-04-25T15:53:15.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.982+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c0d09d0 con 0x7f279c0d05c0 2024-04-25T15:53:15.982 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.982+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f279c02d430 0x7f279c02d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.982+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f279c02dd80 con 0x7f279c02d430 2024-04-25T15:53:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.982+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f279c02d430 0x7f279c02d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:15.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.982+0000 7f279b7fe640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f279c02d430 0x7f279c02d840 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:15.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:15.986+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c02dd80 con 0x7f279c02d430 2024-04-25T15:53:15.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.002+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c02dd80 con 0x7f279c02d430 2024-04-25T15:53:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.014+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c02dd80 con 0x7f279c02d430 2024-04-25T15:53:16.016 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:16.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.014+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f279c20a320 0x7f279c20c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.014+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f279c20f590 con 0x7f279c20a320 2024-04-25T15:53:16.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.018+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f279c20a320 0x7f279c20c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.018+0000 7f27a0bfb640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f279c20a320 0x7f279c20c740 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:16.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:53:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.026+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c20f590 con 0x7f279c20a320 2024-04-25T15:53:16.044 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:16.042+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.046+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c20cf50 con 0x7f279c20a320 2024-04-25T15:53:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.050+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c20cf50 con 0x7f279c20a320 2024-04-25T15:53:16.052 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.050+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f279c20d670 0x7f279c20da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.050+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f279c212960 con 0x7f279c20d670 2024-04-25T15:53:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.050+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f279c20d670 0x7f279c20da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.054+0000 7f279bfff640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f279c20d670 0x7f279c20da80 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.058+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f279c212960 con 0x7f279c20d670 2024-04-25T15:53:16.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.070+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f279c211860 con 0x7f279c20d670 2024-04-25T15:53:16.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f2798ff9640 1 -- 172.21.15.67:0/1771092073 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f279c211860 con 0x7f279c20d670 2024-04-25T15:53:16.076 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f279c14dbd0 msgr2=0x7f279c068590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f279c14dbd0 0x7f279c068590 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f279c14aa30 msgr2=0x7f279c1588f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f279c14aa30 0x7f279c1588f0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f279c159340 msgr2=0x7f279c063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f279c159340 0x7f279c063090 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f279c063ae0 msgr2=0x7f279c0cfb70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f279c063ae0 0x7f279c0cfb70 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f279c0d05c0 msgr2=0x7f279c02c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f279c0d05c0 0x7f279c02c9e0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f279c02d430 msgr2=0x7f279c02d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f279c02d430 0x7f279c02d840 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a0bfb640 1 -- 172.21.15.67:0/1771092073 reap_dead start 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f279c20a320 msgr2=0x7f279c20c740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f279c20a320 0x7f279c20c740 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a0bfb640 1 -- 172.21.15.67:0/1771092073 reap_dead start 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f279c20d670 msgr2=0x7f279c20da80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f279c20d670 0x7f279c20da80 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a0bfb640 1 -- 172.21.15.67:0/1771092073 reap_dead start 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a0bfb640 1 -- 172.21.15.67:0/1771092073 reap_dead start 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f277c06f430 msgr2=0x7f277c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f277c06f430 0x7f277c071940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f278c009fc0 tx=0x7f278c009550 comp rx=0 tx=0).stop 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c0ce260 msgr2=0x7f279c15aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.074+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c0ce260 0x7f279c15aee0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f27840b3040 tx=0x7f278401ba70 comp rx=0 tx=0).stop 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.078+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 shutdown_connections 2024-04-25T15:53:16.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.078+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f277c06f430 0x7f277c071940 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.078+0000 7f27a265e640 1 --2- 172.21.15.67:0/1771092073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f279c0ce260 0x7f279c15aee0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.078+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 >> 172.21.15.67:0/1771092073 conn(0x7f279c06e700 msgr2=0x7f279c0c7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.078+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 shutdown_connections 2024-04-25T15:53:16.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.078+0000 7f27a265e640 1 -- 172.21.15.67:0/1771092073 wait complete. 2024-04-25T15:53:16.146 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:16.146+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:16.180 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:16.181 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0861767640 1 Processor -- start 2024-04-25T15:53:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0861767640 1 -- start start 2024-04-25T15:53:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0861767640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f085c0c92d0 0x7f085c072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0861767640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f085c06d8e0 con 0x7f085c160800 2024-04-25T15:53:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0861767640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f085c073c70 con 0x7f085c0c96e0 2024-04-25T15:53:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0861767640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f085c06dbc0 con 0x7f085c0c92d0 2024-04-25T15:53:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f085c160800 0x7f085c06d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55316/0 (socket says 172.21.15.67:55316) 2024-04-25T15:53:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085affd640 1 -- 172.21.15.67:0/2757478442 learned_addr learned my addr 172.21.15.67:0/2757478442 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085a7fc640 1 --2- 172.21.15.67:0/2757478442 >> v2:172.21.15.176:3300/0 conn(0x7f085c0c92d0 0x7f085c072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3215194429 0 0) 0x7f085c06d8e0 con 0x7f085c160800 2024-04-25T15:53:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0844003530 con 0x7f085c160800 2024-04-25T15:53:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2674260456 0 0) 0x7f0844003530 con 0x7f085c160800 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 >> v2:172.21.15.176:3300/0 conn(0x7f085c0c92d0 msgr2=0x7f085c072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 --2- 172.21.15.67:0/2757478442 >> v2:172.21.15.176:3300/0 conn(0x7f085c0c92d0 0x7f085c072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 >> v1:172.21.15.176:6789/0 conn(0x7f085c0c96e0 legacy=0x7f085c167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c073f50 con 0x7f085c160800 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f084c001680 con 0x7f085c160800 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 >> v1:172.21.15.67:6789/0 conn(0x7f085c160800 legacy=0x7f085c06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0844003a10 0x7f0844003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0844004810 0x7f0844004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08440052b0 0x7f084400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f085c06dbc0 con 0x7f08440052b0 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f084400c240 con 0x7f0844004810 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f084400c440 con 0x7f0844003a10 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085b7fe640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08440052b0 0x7f084400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085a7fc640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0844004810 0x7f0844004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085affd640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0844003a10 0x7f0844003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085b7fe640 1 -- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0844003a10 msgr2=0x7f0844003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085b7fe640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0844003a10 0x7f0844003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085b7fe640 1 -- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0844004810 msgr2=0x7f0844004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085b7fe640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0844004810 0x7f0844004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.186+0000 7f085b7fe640 1 -- 172.21.15.67:0/2757478442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0844003530 con 0x7f08440052b0 2024-04-25T15:53:16.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f085b7fe640 1 -- 172.21.15.67:0/2757478442 reap_dead start 2024-04-25T15:53:16.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f085b7fe640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08440052b0 0x7f084400b9d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0854001e70 tx=0x7f085401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2a856424901fe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:16.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854001a90 con 0x7f08440052b0 2024-04-25T15:53:16.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0854003bc0 con 0x7f08440052b0 2024-04-25T15:53:16.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0859ffb640 1 -- 172.21.15.67:0/2757478442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854005020 con 0x7f08440052b0 2024-04-25T15:53:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0861767640 1 -- 172.21.15.67:0/2757478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08440052b0 msgr2=0x7f084400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0861767640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08440052b0 0x7f084400b9d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0854001e70 tx=0x7f085401c780 comp rx=0 tx=0).stop 2024-04-25T15:53:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0861767640 1 -- 172.21.15.67:0/2757478442 shutdown_connections 2024-04-25T15:53:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0861767640 1 --2- 172.21.15.67:0/2757478442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08440052b0 0x7f084400b9d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0861767640 1 -- 172.21.15.67:0/2757478442 >> 172.21.15.67:0/2757478442 conn(0x7f085c070db0 msgr2=0x7f085c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0861767640 1 -- 172.21.15.67:0/2757478442 shutdown_connections 2024-04-25T15:53:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.190+0000 7f0861767640 1 -- 172.21.15.67:0/2757478442 wait complete. 2024-04-25T15:53:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 Processor -- start 2024-04-25T15:53:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 -- start start 2024-04-25T15:53:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08440052b0 0x7f085c15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c15f650 0x7f085c14e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c1533b0 0x7f085c148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f085c074a30 con 0x7f085c15f650 2024-04-25T15:53:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f085c074510 con 0x7f085c1533b0 2024-04-25T15:53:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f0861767640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f085c160c10 con 0x7f08440052b0 2024-04-25T15:53:16.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c15f650 0x7f085c14e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c15f650 0x7f085c14e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42462/0 (socket says 172.21.15.67:42462) 2024-04-25T15:53:16.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 -- 172.21.15.67:0/744000612 learned_addr learned my addr 172.21.15.67:0/744000612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:16.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085b7fe640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c1533b0 0x7f085c148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085affd640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08440052b0 0x7f085c15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 -- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08440052b0 msgr2=0x7f085c15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08440052b0 0x7f085c15dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 -- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c1533b0 msgr2=0x7f085c148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c1533b0 0x7f085c148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 -- 172.21.15.67:0/744000612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c1f5d20 con 0x7f085c15f650 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085affd640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08440052b0 0x7f085c15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085b7fe640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c1533b0 0x7f085c148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f085a7fc640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c15f650 0x7f085c14e900 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0850011990 tx=0x7f0850011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.194+0000 7f08437fe640 1 -- 172.21.15.67:0/744000612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085000fb30 con 0x7f085c15f650 2024-04-25T15:53:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f08437fe640 1 -- 172.21.15.67:0/744000612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085000fcd0 con 0x7f085c15f650 2024-04-25T15:53:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085c14ee40 con 0x7f085c15f650 2024-04-25T15:53:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f08437fe640 1 -- 172.21.15.67:0/744000612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0850014660 con 0x7f085c15f650 2024-04-25T15:53:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085c15ee10 con 0x7f085c15f650 2024-04-25T15:53:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f085c06a480 con 0x7f085c15f650 2024-04-25T15:53:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f08437fe640 1 -- 172.21.15.67:0/744000612 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0850012040 con 0x7f085c15f650 2024-04-25T15:53:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f08437fe640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f084806f400 0x7f0848071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f08437fe640 1 -- 172.21.15.67:0/744000612 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 5915+0+0 (secure 0 0 0) 0x7f085008fa10 con 0x7f085c15f650 2024-04-25T15:53:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.198+0000 7f085affd640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f084806f400 0x7f0848071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:16.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.202+0000 7f085affd640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f084806f400 0x7f0848071910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f084c000c10 tx=0x7f084c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:16.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.206+0000 7f08437fe640 1 -- 172.21.15.67:0/744000612 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0850019030 con 0x7f085c15f650 2024-04-25T15:53:16.249 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:16.250+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:16.347 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:16.346+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:16.353 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:16.353 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:16.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:16.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:16.390+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f085c0c8c50 con 0x7f085c15f650 2024-04-25T15:53:16.448 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:16.446+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:16.542 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:16.543 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:16.549 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:16.550+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:16.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:16.650 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:16.650+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:16.751 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:16.751+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:16.774 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:16.774 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:16.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:16.851 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:16.851+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:16.952 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:16.950+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:16.987 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:16.987 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:17.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:17.053 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:17.051+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:17.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.054+0000 7f08437fe640 1 -- 172.21.15.67:0/744000612 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v78) ==== 87+0+0 (secure 0 0 0) 0x7f085005c7b0 con 0x7f085c15f650 2024-04-25T15:53:17.057 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:53:17.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f084806f400 msgr2=0x7f0848071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:17.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f084806f400 0x7f0848071910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f084c000c10 tx=0x7f084c009040 comp rx=0 tx=0).stop 2024-04-25T15:53:17.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c15f650 msgr2=0x7f085c14e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:17.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c15f650 0x7f085c14e900 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0850011990 tx=0x7f0850011e50 comp rx=0 tx=0).stop 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 shutdown_connections 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f084806f400 0x7f0848071910 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f085c1533b0 0x7f085c148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f085c15f650 0x7f085c14e900 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 --2- 172.21.15.67:0/744000612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08440052b0 0x7f085c15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 >> 172.21.15.67:0/744000612 conn(0x7f085c070db0 msgr2=0x7f085c155460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 shutdown_connections 2024-04-25T15:53:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:17.058+0000 7f0861767640 1 -- 172.21.15.67:0/744000612 wait complete. 2024-04-25T15:53:17.153 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:17.154+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:17.217 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:17.217 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:17.235 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:17.283 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:17.283+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:17.355 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:17.354+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:17.408 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:17.408 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:17.430 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:17.471 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:17.471+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:17.557 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:17.558+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:17.617 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:17.617 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:17.634 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:17.658 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:17.658+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:17.759 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:17.758+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:17.782 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:17.782 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:17.799 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:17.859 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:17.859+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:17.943 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:17.943 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:17.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:17.961 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:17.958+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:18.062 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:18.062+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:18.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:18.155 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:18.155 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:18.162 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:18.159+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:18.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:18.263 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:18.263+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:18.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a72ac640 1 Processor -- start 2024-04-25T15:53:18.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a72ac640 1 -- start start 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a72ac640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04a0153a00 0x7f04a00c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a72ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a00725c0 0x7f04a015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a72ac640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04a00c3bc0 con 0x7f04a0073f90 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a72ac640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f04a0153e10 con 0x7f04a0153a00 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a72ac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f04a015a5f0 con 0x7f04a00725c0 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a5048640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04a0153a00 0x7f04a00c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a5048640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04a0153a00 0x7f04a00c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42470/0 (socket says 172.21.15.67:42470) 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a5048640 1 -- 172.21.15.67:0/3841304341 learned_addr learned my addr 172.21.15.67:0/3841304341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a4847640 1 --2- 172.21.15.67:0/3841304341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a00725c0 0x7f04a015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a5048640 1 -- 172.21.15.67:0/3841304341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a00725c0 msgr2=0x7f04a015a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:18.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a5048640 1 --2- 172.21.15.67:0/3841304341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a00725c0 0x7f04a015a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a5048640 1 -- 172.21.15.67:0/3841304341 >> v1:172.21.15.176:6789/0 conn(0x7f04a0073f90 legacy=0x7f04a00743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a5048640 1 -- 172.21.15.67:0/3841304341 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a00c53a0 con 0x7f04a0153a00 2024-04-25T15:53:18.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.266+0000 7f04a4847640 1 --2- 172.21.15.67:0/3841304341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a00725c0 0x7f04a015a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a5048640 1 --2- 172.21.15.67:0/3841304341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a0153a00 0x7f04a00c4010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f049c009870 tx=0x7f049c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfe33b17ff195e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f0497fff640 1 -- 172.21.15.67:0/3841304341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0b0070 con 0x7f04a0153a00 2024-04-25T15:53:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f0497fff640 1 -- 172.21.15.67:0/3841304341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049c0ab600 con 0x7f04a0153a00 2024-04-25T15:53:18.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f0497fff640 1 -- 172.21.15.67:0/3841304341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0ab9d0 con 0x7f04a0153a00 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 -- 172.21.15.67:0/3841304341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a0153a00 msgr2=0x7f04a00c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 --2- 172.21.15.67:0/3841304341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a0153a00 0x7f04a00c4010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f049c009870 tx=0x7f049c01c4d0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 -- 172.21.15.67:0/3841304341 shutdown_connections 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 --2- 172.21.15.67:0/3841304341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a00725c0 0x7f04a015a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 --2- 172.21.15.67:0/3841304341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a0153a00 0x7f04a00c4010 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 -- 172.21.15.67:0/3841304341 >> 172.21.15.67:0/3841304341 conn(0x7f04a00ce1c0 msgr2=0x7f04a00ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 -- 172.21.15.67:0/3841304341 shutdown_connections 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.270+0000 7f04a72ac640 1 -- 172.21.15.67:0/3841304341 wait complete. 2024-04-25T15:53:18.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 Processor -- start 2024-04-25T15:53:18.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 -- start start 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a00725c0 0x7f04a015a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04a0073f90 0x7f04a01525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a0153a00 0x7f04a014cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f04a00729d0 con 0x7f04a00725c0 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04a00686c0 con 0x7f04a0073f90 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f04a0074ab0 con 0x7f04a0153a00 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a00725c0 0x7f04a015a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a00725c0 0x7f04a015a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42478/0 (socket says 172.21.15.67:42478) 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 -- 172.21.15.67:0/302364908 learned_addr learned my addr 172.21.15.67:0/302364908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04a0073f90 0x7f04a01525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a0153a00 0x7f04a014cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a0153a00 msgr2=0x7f04a014cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a0153a00 0x7f04a014cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04a0073f90 msgr2=0x7f04a01525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04a0073f90 0x7f04a01525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a015acd0 con 0x7f04a00725c0 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04a0153a00 0x7f04a014cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04a0073f90 0x7f04a01525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a00725c0 0x7f04a015a790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f049c0ae040 tx=0x7f049c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0b0040 con 0x7f04a00725c0 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049c0bc070 con 0x7f04a00725c0 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a0158140 con 0x7f04a00725c0 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0b6640 con 0x7f04a00725c0 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a0158a00 con 0x7f04a00725c0 2024-04-25T15:53:18.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.274+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f04a006a480 con 0x7f04a00725c0 2024-04-25T15:53:18.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f049c0b67e0 con 0x7f04a00725c0 2024-04-25T15:53:18.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f0495ffb640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f048406f400 0x7f0484071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5915+0+0 (secure 0 0 0) 0x7f049c12a080 con 0x7f04a00725c0 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v78) ==== 54+0+16 (secure 0 0 0) 0x7f04a00c53a0 con 0x7f04a00725c0 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f048406f400 0x7f0484071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f04a014d020 0x7f04a014f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04a00c95a0 con 0x7f04a014d020 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f04a014d020 0x7f04a014f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f04a014d020 0x7f04a014f8f0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f048406f400 0x7f0484071910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0490004660 tx=0x7f0490008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.278+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a00c95a0 con 0x7f04a014d020 2024-04-25T15:53:18.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.294+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a00b0780 con 0x7f04a014d020 2024-04-25T15:53:18.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.294+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a00b0780 con 0x7f04a014d020 2024-04-25T15:53:18.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:18.297 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:18.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.294+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f04a01ec960 0x7f04a01ecd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.294+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f04a01ed2b0 con 0x7f04a01ec960 2024-04-25T15:53:18.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.294+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f04a01ec960 0x7f04a01ecd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.298+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f04a01ec960 0x7f04a01ecd70 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.298+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a01ed2b0 con 0x7f04a01ec960 2024-04-25T15:53:18.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.310+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a00c32b0 con 0x7f04a01ec960 2024-04-25T15:53:18.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.314+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a00c32b0 con 0x7f04a01ec960 2024-04-25T15:53:18.314 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:18.315 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:18.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.314+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f04a00635a0 0x7f04a00639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.314+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f04a0153340 con 0x7f04a00635a0 2024-04-25T15:53:18.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.314+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f04a00635a0 0x7f04a00639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.314+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f04a00635a0 0x7f04a00639b0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.314+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a0153340 con 0x7f04a00635a0 2024-04-25T15:53:18.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.330+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a015a040 con 0x7f04a00635a0 2024-04-25T15:53:18.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.330+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a015a040 con 0x7f04a00635a0 2024-04-25T15:53:18.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:18.331 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:18.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.330+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f04a01517b0 0x7f04a0151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.330+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f04a0152180 con 0x7f04a01517b0 2024-04-25T15:53:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.330+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f04a01517b0 0x7f04a0151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.330+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f04a01517b0 0x7f04a0151bc0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.330+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a0152180 con 0x7f04a01517b0 2024-04-25T15:53:18.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.346+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a0152bb0 con 0x7f04a01517b0 2024-04-25T15:53:18.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.346+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a0152bb0 con 0x7f04a01517b0 2024-04-25T15:53:18.348 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:18.348 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:18.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.346+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f04a002cef0 0x7f04a002d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.346+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f04a01592d0 con 0x7f04a002cef0 2024-04-25T15:53:18.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.346+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f04a002cef0 0x7f04a002d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.346+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f04a002cef0 0x7f04a002d300 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.350+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a01592d0 con 0x7f04a002cef0 2024-04-25T15:53:18.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.362+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a0159250 con 0x7f04a002cef0 2024-04-25T15:53:18.364 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:18.363+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:18.366 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:18.366 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:18.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.366+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a0159250 con 0x7f04a002cef0 2024-04-25T15:53:18.369 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:18.369 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:18.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.366+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f04a020a650 0x7f04a020ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.366+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f04a002ddd0 con 0x7f04a020a650 2024-04-25T15:53:18.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.370+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f04a020a650 0x7f04a020ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.370+0000 7f04a4847640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f04a020a650 0x7f04a020ca70 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.370+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a002ddd0 con 0x7f04a020a650 2024-04-25T15:53:18.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.386+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a002ddd0 con 0x7f04a020a650 2024-04-25T15:53:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.390+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a002ddd0 con 0x7f04a020a650 2024-04-25T15:53:18.390 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.390+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f04a020d4c0 0x7f04a020f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.390+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f04a0158c40 con 0x7f04a020d4c0 2024-04-25T15:53:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.390+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f04a020d4c0 0x7f04a020f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.390+0000 7f04a5849640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f04a020d4c0 0x7f04a020f8e0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.390+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a0158c40 con 0x7f04a020d4c0 2024-04-25T15:53:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.406+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a0158c40 con 0x7f04a020d4c0 2024-04-25T15:53:18.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.410+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a0158c40 con 0x7f04a020d4c0 2024-04-25T15:53:18.409 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.410+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f04a0210330 0x7f04a0212750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.410+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f04a0215680 con 0x7f04a0210330 2024-04-25T15:53:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.410+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f04a0210330 0x7f04a0212750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.410+0000 7f04a5048640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f04a0210330 0x7f04a0212750 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.410+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a0215680 con 0x7f04a0210330 2024-04-25T15:53:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.422+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f04a02155d0 con 0x7f04a0210330 2024-04-25T15:53:18.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f0495ffb640 1 -- 172.21.15.67:0/302364908 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f04a02155d0 con 0x7f04a0210330 2024-04-25T15:53:18.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f04a014d020 msgr2=0x7f04a014f8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f04a014d020 0x7f04a014f8f0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f04a01ec960 msgr2=0x7f04a01ecd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f04a01ec960 0x7f04a01ecd70 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f04a00635a0 msgr2=0x7f04a00639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f04a00635a0 0x7f04a00639b0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f04a01517b0 msgr2=0x7f04a0151bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a5849640 1 -- 172.21.15.67:0/302364908 reap_dead start 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f04a01517b0 0x7f04a0151bc0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f04a002cef0 msgr2=0x7f04a002d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f04a002cef0 0x7f04a002d300 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f04a020a650 msgr2=0x7f04a020ca70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f04a020a650 0x7f04a020ca70 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f04a020d4c0 msgr2=0x7f04a020f8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f04a020d4c0 0x7f04a020f8e0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f04a0210330 msgr2=0x7f04a0212750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f04a0210330 0x7f04a0212750 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a5849640 1 -- 172.21.15.67:0/302364908 reap_dead start 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f048406f400 msgr2=0x7f0484071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f048406f400 0x7f0484071910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0490004660 tx=0x7f0490008040 comp rx=0 tx=0).stop 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a00725c0 msgr2=0x7f04a015a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a00725c0 0x7f04a015a790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f049c0ae040 tx=0x7f049c01ad10 comp rx=0 tx=0).stop 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 shutdown_connections 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f048406f400 0x7f0484071910 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 --2- 172.21.15.67:0/302364908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04a00725c0 0x7f04a015a790 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 >> 172.21.15.67:0/302364908 conn(0x7f04a00ce1c0 msgr2=0x7f04a0163390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 shutdown_connections 2024-04-25T15:53:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:18.426+0000 7f04a72ac640 1 -- 172.21.15.67:0/302364908 wait complete. 2024-04-25T15:53:18.475 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:18.475+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:18.565 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:18.563+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:18.574 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:18.575 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:18.575 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:18.575 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:18.592 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:18.665 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:18.666+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:18.766 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:18.763+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:18.781 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:18.781 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:18.782 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:18.782 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:18.782 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:18.799 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:18.869 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:18.866+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:18.968 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:18.967+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:43 3.26 3:64018a55:::10000000002.00000027:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.091194137999999, 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024229051000000001, 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:18.997 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 43 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902016+0000", 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:18.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.902019+0000", 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9459999999999998e-06 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907768+0000", 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005748525 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907831+0000", 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2842999999999994e-05 2024-04-25T15:53:18.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.907857+0000", 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6636e-05 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908056+0000", 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198295 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908464+0000", 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040813600000000002 2024-04-25T15:53:19.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.908853+0000", 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038892400000000002 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925332+0000", 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016479074 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.925882+0000", 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000550232 2024-04-25T15:53:19.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926050+0000", 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016822199999999999 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:43.926245+0000", 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019521799999999999 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:75 3.35 3:ad096b5b:::10000000002.00000047:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.659701437000002, 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022276672000000001, 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 75 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333509+0000", 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.333513+0000", 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339006+0000", 2024-04-25T15:53:19.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054929599999999999 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339071+0000", 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5545999999999999e-05 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339098+0000", 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6647000000000001e-05 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339299+0000", 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201351 2024-04-25T15:53:19.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.339703+0000", 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00040396399999999998 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.340160+0000", 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045697899999999999 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.352668+0000", 2024-04-25T15:53:19.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012508137000000001 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355605+0000", 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002937013 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355701+0000", 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5681999999999996e-05 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.355786+0000", 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4908999999999996e-05 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:79 3.3 3:c0361aff:::10000000002.0000004b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.483931341999998, 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022495096999999999, 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.008 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 79 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509279+0000", 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.509282+0000", 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2200000000000001e-06 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517219+0000", 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079362270000000006 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517284+0000", 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5052e-05 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517310+0000", 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6072999999999999e-05 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517506+0000", 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019606199999999999 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.517779+0000", 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027302799999999998 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.518047+0000", 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000268454 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.528734+0000", 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01068712 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531586+0000", 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028516589999999999 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531689+0000", 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010260900000000001 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.531774+0000", 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5593000000000005e-05 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:177 3.26 3:66aef6e9:::10000000002.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.116522584999998, 2024-04-25T15:53:19.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026823691, 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876691+0000", 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4939999999999999e-06 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880736+0000", 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040441660000000001 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880793+0000", 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7581999999999999e-05 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880817+0000", 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881039+0000", 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022171800000000001 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881305+0000", 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026580799999999997 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881558+0000", 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025290099999999998 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.893182+0000", 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011624242 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903322+0000", 2024-04-25T15:53:19.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010139754000000001 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903425+0000", 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102706 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903512+0000", 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7093000000000001e-05 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.018 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:261 3.1d e66/54 3:bbf7764d:::10000000002.00000001:head v 66'8, mlcod=63'6)", 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.233356505, 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022527077, 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759854+0000", 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759855+0000", 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6599999999999996e-07 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764340+0000", 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044852190000000004 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764382+0000", 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2052999999999999e-05 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764415+0000", 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.341e-05 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764650+0000", 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023472400000000001 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.764703+0000", 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3016999999999997e-05 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782246+0000", 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017543286000000002 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782381+0000", 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134702 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:268 3.14 e67/54 3:29a5f800:::10000000002.00000008:head v 67'6, mlcod=63'4)", 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.924400682000002, 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023750529999999999, 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.023 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068810+0000", 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068811+0000", 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.37e-07 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073130+0000", 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043190069999999997 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073172+0000", 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2512000000000002e-05 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073198+0000", 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6084e-05 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073426+0000", 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227514 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.073510+0000", 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4469999999999996e-05 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092427+0000", 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018916571 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092560+0000", 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013373500000000001 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:269 3.2c 3:354fa1eb:::10000000002.00000009:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:19.027 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.887921858999999, 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022386919000000002, 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 269 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105289+0000", 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.105292+0000", 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8839999999999998e-06 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111434+0000", 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061422880000000001 2024-04-25T15:53:19.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111491+0000", 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7306000000000003e-05 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111515+0000", 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3893e-05 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111686+0000", 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017118600000000001 2024-04-25T15:53:19.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.111859+0000", 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017279099999999999 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.112126+0000", 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000266722 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.123964+0000", 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01183797 2024-04-25T15:53:19.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127453+0000", 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034891240000000001 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127555+0000", 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102082 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.127676+0000", 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120673 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.032 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:355 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.504495911999999, 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025437358, 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.033 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 355 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488717+0000", 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.874e-06 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494412+0000", 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056951349999999996 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494450+0000", 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7920999999999998e-05 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494464+0000", 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4728e-05 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494580+0000", 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011599099999999999 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494756+0000", 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176083 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.495038+0000", 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028161800000000003 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.508282+0000", 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013243981 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514019+0000", 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057372639999999997 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514099+0000", 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9800000000000002e-05 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514152+0000", 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2963000000000001e-05 2024-04-25T15:53:19.038 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:357 3.34 3:2d119e8b:::10000000002.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.427987785999999, 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021511328, 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 357 2024-04-25T15:53:19.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565223+0000", 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.565225+0000", 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2189999999999998e-06 2024-04-25T15:53:19.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569849+0000", 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046243480000000003 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.569992+0000", 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014274200000000001 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570006+0000", 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4032999999999999e-05 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570122+0000", 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116358 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570290+0000", 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016725299999999999 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.570528+0000", 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023819800000000001 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.581505+0000", 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01097657 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586619+0000", 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051144119999999996 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586682+0000", 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3309e-05 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.586734+0000", 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1885999999999998e-05 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:372 3.10 3:0b92729d:::10000000002.00000070:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.886672551, 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026675501000000001, 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.044 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 372 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106541+0000", 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.022e-06 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110718+0000", 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041768669999999999 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110768+0000", 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110792+0000", 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3577000000000001e-05 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110988+0000", 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019645200000000001 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111271+0000", 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028314399999999998 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111688+0000", 2024-04-25T15:53:19.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041694699999999998 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.123821+0000", 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012133015 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133024+0000", 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092031290000000009 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133125+0000", 2024-04-25T15:53:19.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010063199999999999 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133214+0000", 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8555999999999997e-05 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:388 3.36 3:6dca1976:::10000000002.00000080:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.267463289, 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022516031999999998, 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.049 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 388 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725747+0000", 2024-04-25T15:53:19.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.725751+0000", 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.27e-06 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730050+0000", 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004299247 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730109+0000", 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9583000000000002e-05 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730135+0000", 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5316999999999998e-05 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730306+0000", 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000171695 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730503+0000", 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019674499999999999 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:19.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.730797+0000", 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029401399999999999 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.741175+0000", 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010378 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748075+0000", 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068997829999999996 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748176+0000", 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101545 2024-04-25T15:53:19.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.748263+0000", 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6833e-05 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:394 3.36 3:6ccb00e9:::10000000002.00000086:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.061608447999999, 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025589158000000001, 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.054 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 394 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931605+0000", 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0730000000000001e-06 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936840+0000", 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052349129999999999 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936898+0000", 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7793e-05 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936922+0000", 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4513000000000001e-05 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937111+0000", 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018847800000000001 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937292+0000", 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181189 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937542+0000", 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024970800000000002 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956147+0000", 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018605467000000001 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956986+0000", 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083831599999999997 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957089+0000", 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103379 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957191+0000", 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102329 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:397 3.20 3:06fc074c:::10000000002.00000089:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.95151667, 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022794325000000001, 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.059 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 397 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041694+0000", 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.041697+0000", 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8569999999999999e-06 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047000+0000", 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053031220000000004 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047059+0000", 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9499000000000001e-05 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047084+0000", 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4340000000000001e-05 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047231+0000", 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014741600000000001 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047415+0000", 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018419500000000001 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.047667+0000", 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025212800000000001 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.060095+0000", 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012427182 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064282+0000", 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041871410000000001 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064390+0000", 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108142 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.064488+0000", 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8302999999999997e-05 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.064 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:400 3.26 3:6771a79d:::10000000002.0000008c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.845981339, 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025113648999999998, 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 400 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147229+0000", 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.147232+0000", 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.253e-06 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151612+0000", 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004379226 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151671+0000", 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9036999999999997e-05 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151696+0000", 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4825999999999999e-05 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.151891+0000", 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019537499999999999 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.152208+0000", 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000317041 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.152648+0000", 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043998600000000001 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.168097+0000", 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015449137 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172104+0000", 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040067260000000004 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172209+0000", 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105376 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.172343+0000", 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133666 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:401 3.34 3:2feeb5a8:::10000000002.0000008d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.815198434999999, 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022845918, 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 401 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178012+0000", 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.178015+0000", 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5730000000000002e-06 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182028+0000", 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040133800000000004 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182085+0000", 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6656000000000001e-05 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182108+0000", 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.287e-05 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182252+0000", 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144389 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182427+0000", 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017484799999999999 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.182676+0000", 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024908500000000002 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.196912+0000", 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014236367 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200602+0000", 2024-04-25T15:53:19.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036898819999999998 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200713+0000", 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011032899999999999 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.200858+0000", 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000145539 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:406 3.11 e69/54 3:8834f03f:::10000000002.00000092:head v 69'13, mlcod=69'11)", 2024-04-25T15:53:19.074 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.622041167000001, 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023521068999999999, 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371169+0000", 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.371172+0000", 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7800000000000001e-06 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375367+0000", 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004194761 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375407+0000", 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0222999999999998e-05 2024-04-25T15:53:19.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375432+0000", 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5236999999999999e-05 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375634+0000", 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020169400000000001 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.375717+0000", 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2936999999999996e-05 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.394537+0000", 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018820018000000001 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.394690+0000", 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015341899999999999 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:410 3.34 3:2cf0bbf2:::10000000002.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.470643972, 2024-04-25T15:53:19.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026112841000000001, 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 410 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522570+0000", 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529416+0000", 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006846062 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529476+0000", 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0350000000000003e-05 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529502+0000", 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6259999999999999e-05 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529665+0000", 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016305200000000001 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529683+0000", 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7935999999999999e-05 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534023+0000", 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043396270000000004 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534279+0000", 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025637099999999998 2024-04-25T15:53:19.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534518+0000", 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238762 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.543891+0000", 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093730170000000008 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548495+0000", 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046033610000000003 2024-04-25T15:53:19.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548592+0000", 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7793000000000004e-05 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548679+0000", 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7100000000000003e-05 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:412 3.3e 3:7d47d675:::10000000002.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:19.084 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.389588979999999, 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025248618, 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 412 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:19.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603622+0000", 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.603625+0000", 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5329999999999999e-06 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611293+0000", 2024-04-25T15:53:19.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076675720000000001 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611358+0000", 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5500000000000006e-05 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611382+0000", 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4300000000000001e-05 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611573+0000", 2024-04-25T15:53:19.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019060799999999999 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.611863+0000", 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029038099999999998 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.612303+0000", 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043984299999999998 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.627478+0000", 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015174632 2024-04-25T15:53:19.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628552+0000", 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001074258 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628728+0000", 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017612500000000001 2024-04-25T15:53:19.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.628870+0000", 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014186599999999999 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:505 3.3e 3:7d99c36c:::10000000002.000000f5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:09.131489+0000", 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.8617211919999992, 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025085643000000001, 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.090 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 505 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.131489+0000", 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.131489+0000", 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.131493+0000", 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8060000000000001e-06 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.138731+0000", 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072381370000000004 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.138792+0000", 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0402000000000002e-05 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.138817+0000", 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5117e-05 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.139023+0000", 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000205861 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.139314+0000", 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029163799999999999 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.139731+0000", 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041711599999999998 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.154735+0000", 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015003448000000001 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.156386+0000", 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001650966 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.156482+0000", 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6491999999999997e-05 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.156575+0000", 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2659999999999997e-05 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:514 3.19 e71/68 3:982db27a:::10000000002.000000ff:head v 71'6, mlcod=67'4)", 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:09.482959+0000", 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.5102510979999995, 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021499580000000001, 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.482959+0000", 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.482959+0000", 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.482963+0000", 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.569e-06 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.490855+0000", 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0078918070000000007 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.490891+0000", 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6117e-05 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.490913+0000", 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2011999999999999e-05 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.491117+0000", 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020398700000000001 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.491187+0000", 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9632e-05 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.504371+0000", 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013184226 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.504459+0000", 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8230000000000001e-05 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.099 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.100 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.100 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:19.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:53:19.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:19.102 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:19.102+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:19.202 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:19.199+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f872bcc6640 1 Processor -- start 2024-04-25T15:53:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f872bcc6640 1 -- start start 2024-04-25T15:53:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f872bcc6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f87240c99e0 0x7f87240ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f872bcc6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8724074720 con 0x7f87240733f0 2024-04-25T15:53:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f872bcc6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8724074990 con 0x7f8724164300 2024-04-25T15:53:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f872bcc6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f872406d050 con 0x7f87240c99e0 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8729a62640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f87240733f0 0x7f87240c92d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55330/0 (socket says 172.21.15.67:55330) 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8729a62640 1 -- 172.21.15.67:0/3464718766 learned_addr learned my addr 172.21.15.67:0/3464718766 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8729261640 1 --2- 172.21.15.67:0/3464718766 >> v2:172.21.15.176:3300/0 conn(0x7f87240c99e0 0x7f87240ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8728a60640 1 -- 172.21.15.67:0/3464718766 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4229436212 0 0) 0x7f8724074720 con 0x7f87240733f0 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8728a60640 1 -- 172.21.15.67:0/3464718766 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8710003530 con 0x7f87240733f0 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8728a60640 1 -- 172.21.15.67:0/3464718766 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1405553714 0 0) 0x7f8710003530 con 0x7f87240733f0 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8729261640 1 -- 172.21.15.67:0/3464718766 >> v1:172.21.15.176:6789/0 conn(0x7f8724164300 legacy=0x7f8724164710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8729261640 1 -- 172.21.15.67:0/3464718766 >> v1:172.21.15.67:6789/0 conn(0x7f87240733f0 legacy=0x7f87240c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.270+0000 7f8729261640 1 -- 172.21.15.67:0/3464718766 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872406d2a0 con 0x7f87240c99e0 2024-04-25T15:53:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f8729261640 1 --2- 172.21.15.67:0/3464718766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87240c99e0 0x7f87240ca780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f87140097c0 tx=0x7f871401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45cc8a58d7ec9546 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f8728a60640 1 -- 172.21.15.67:0/3464718766 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140b0070 con 0x7f87240c99e0 2024-04-25T15:53:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f8728a60640 1 -- 172.21.15.67:0/3464718766 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87140ab400 con 0x7f87240c99e0 2024-04-25T15:53:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f8728a60640 1 -- 172.21.15.67:0/3464718766 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c001a00 con 0x7f87240c99e0 2024-04-25T15:53:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f872bcc6640 1 -- 172.21.15.67:0/3464718766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87240c99e0 msgr2=0x7f87240ca780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3464718766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87240c99e0 0x7f87240ca780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f87140097c0 tx=0x7f871401c3c0 comp rx=0 tx=0).stop 2024-04-25T15:53:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f872bcc6640 1 -- 172.21.15.67:0/3464718766 shutdown_connections 2024-04-25T15:53:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3464718766 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87240c99e0 0x7f87240ca780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f872bcc6640 1 -- 172.21.15.67:0/3464718766 >> 172.21.15.67:0/3464718766 conn(0x7f8724070db0 msgr2=0x7f87240ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f872bcc6640 1 -- 172.21.15.67:0/3464718766 shutdown_connections 2024-04-25T15:53:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.274+0000 7f872bcc6640 1 -- 172.21.15.67:0/3464718766 wait complete. 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 Processor -- start 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 -- start start 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87240733f0 0x7f87241fe7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87240c99e0 0x7f872414d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8724164300 0x7f87241588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f872416f1f0 con 0x7f87240c99e0 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f872416f070 con 0x7f8724164300 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f872416f370 con 0x7f87240733f0 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87240c99e0 0x7f872414d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87240c99e0 0x7f872414d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42486/0 (socket says 172.21.15.67:42486) 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 -- 172.21.15.67:0/3025638639 learned_addr learned my addr 172.21.15.67:0/3025638639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729a62640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87240733f0 0x7f87241fe7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872a263640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8724164300 0x7f87241588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:19.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 -- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87240733f0 msgr2=0x7f87241fe7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87240733f0 0x7f87241fe7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 -- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8724164300 msgr2=0x7f87241588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8724164300 0x7f87241588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 -- 172.21.15.67:0/3025638639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87240cd820 con 0x7f87240c99e0 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872a263640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8724164300 0x7f87241588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f8729261640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87240c99e0 0x7f872414d6e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f87140ae040 tx=0x7f8714019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f871a7fc640 1 -- 172.21.15.67:0/3025638639 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140b0040 con 0x7f87240c99e0 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f871a7fc640 1 -- 172.21.15.67:0/3025638639 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8714003620 con 0x7f87240c99e0 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f871a7fc640 1 -- 172.21.15.67:0/3025638639 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140039d0 con 0x7f87240c99e0 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87240cdb00 con 0x7f87240c99e0 2024-04-25T15:53:19.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.278+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87241fe290 con 0x7f87240c99e0 2024-04-25T15:53:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.282+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872406a480 con 0x7f87240c99e0 2024-04-25T15:53:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.282+0000 7f871a7fc640 1 -- 172.21.15.67:0/3025638639 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f87140bd020 con 0x7f87240c99e0 2024-04-25T15:53:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.282+0000 7f871a7fc640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f870806f430 0x7f8708071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.282+0000 7f871a7fc640 1 -- 172.21.15.67:0/3025638639 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5915+0+0 (secure 0 0 0) 0x7f8714129690 con 0x7f87240c99e0 2024-04-25T15:53:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.282+0000 7f8729a62640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f870806f430 0x7f8708071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:19.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.282+0000 7f8729a62640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f870806f430 0x7f8708071940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f871c005c10 tx=0x7f871c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:19.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.286+0000 7f871a7fc640 1 -- 172.21.15.67:0/3025638639 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8714015fb0 con 0x7f87240c99e0 2024-04-25T15:53:19.299 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:19.299 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:19.302 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:19.299+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:19.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:19.403 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:19.403+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:19.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:19.458+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f8724150420 con 0x7f87240c99e0 2024-04-25T15:53:19.503 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:19.502+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:19.509 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:19.509 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:19.509 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:19.509 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:19.509 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:19.526 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:19.611 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:19.611+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:19.705 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:19.703+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:19.720 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:19.720 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:19.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:19.720 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:19.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.720 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:19.720 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 180.61854162099999, 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:19.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:19.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:19.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:19.726 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 180.027338697, 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024492090000000001, 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 51 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.727 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689667+0000", 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689668+0000", 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5000000000000001e-07 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689675+0000", 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8159999999999996e-06 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.728 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690012+0000", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033731700000000001 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690020+0000", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3000000000000002e-06 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709229+0000", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019208668000000002 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709500+0000", 2024-04-25T15:53:19.729 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000270881 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.709927+0000", 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042707599999999997 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.712761+0000", 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002834455 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713978+0000", 2024-04-25T15:53:19.730 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012166220000000001 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714076+0000", 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8163000000000002e-05 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714159+0000", 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2942000000000006e-05 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.731 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 180.02732761799999, 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025018706000000002, 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 52 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:19.732 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689678+0000", 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689679+0000", 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1399999999999997e-07 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.689695+0000", 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6342999999999999e-05 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690038+0000", 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034292699999999998 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.690051+0000", 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-05 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711285+0000", 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021233213000000001 2024-04-25T15:53:19.734 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711546+0000", 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000261568 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.711972+0000", 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042627000000000002 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.713408+0000", 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014350999999999999 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714555+0000", 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001146989 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714633+0000", 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8926999999999998e-05 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.714697+0000", 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3335000000000003e-05 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.736 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:62 3.11 3:8969f477:::10000000002.0000003a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.961352726000001, 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025889728000000001, 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 62 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.737 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755653+0000", 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.755657+0000", 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9439999999999998e-06 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.738 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760593+0000", 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049360740000000004 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760651+0000", 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8e-05 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760676+0000", 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5219999999999999e-05 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.739 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.760847+0000", 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017057199999999999 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761123+0000", 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027578100000000001 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.761365+0000", 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024248100000000001 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.780642+0000", 2024-04-25T15:53:19.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019277327 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781358+0000", 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000715611 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781456+0000", 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7946e-05 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.781543+0000", 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6772000000000002e-05 2024-04-25T15:53:19.741 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:150 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.065833794, 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024331635000000001, 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.742 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 150 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651172+0000", 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.651176+0000", 2024-04-25T15:53:19.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8770000000000003e-06 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659329+0000", 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081530219999999994 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659399+0000", 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0376000000000005e-05 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659428+0000", 2024-04-25T15:53:19.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8952999999999999e-05 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659580+0000", 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015188199999999999 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.659842+0000", 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026179399999999998 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.660120+0000", 2024-04-25T15:53:19.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027824799999999998 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.673813+0000", 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013693145 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675307+0000", 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014934600000000001 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675412+0000", 2024-04-25T15:53:19.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105233 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.675504+0000", 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1644999999999994e-05 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:155 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 30.843682509000001, 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025243293999999999, 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.747 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 155 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873323+0000", 2024-04-25T15:53:19.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.873327+0000", 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7699999999999999e-06 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879173+0000", 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005846141 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879239+0000", 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6315999999999993e-05 2024-04-25T15:53:19.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879267+0000", 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7137000000000001e-05 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879415+0000", 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000148715 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879693+0000", 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000278108 2024-04-25T15:53:19.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.879951+0000", 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025793700000000002 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.896019+0000", 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016067814999999999 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898153+0000", 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021339280000000002 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898411+0000", 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025760799999999999 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:48.898566+0000", 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015581899999999999 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:165 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:19.752 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 30.347711971999999, 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032658418000000002, 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 165 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:19.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369294+0000", 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.369298+0000", 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3429999999999998e-06 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374706+0000", 2024-04-25T15:53:19.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054080250000000003 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374776+0000", 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9972e-05 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.374803+0000", 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7384000000000001e-05 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375010+0000", 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206431 2024-04-25T15:53:19.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.375037+0000", 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7206999999999999e-05 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380232+0000", 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051947859999999998 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380474+0000", 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000241864 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.380709+0000", 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023560200000000001 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.392962+0000", 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012252219999999999 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401765+0000", 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088034089999999999 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401866+0000", 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101099 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.401952+0000", 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6075999999999994e-05 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:19.758 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.49930114, 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:19.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:19.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:19.763 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.404667088, 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.768 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:260 3.12 3:4bec3886:::10000000002.00000000:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.027025847000001, 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028352792000000002, 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 260 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.769 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689980+0000", 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.689983+0000", 2024-04-25T15:53:19.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2729999999999998e-06 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696571+0000", 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065880749999999997 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696637+0000", 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5358999999999995e-05 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.696662+0000", 2024-04-25T15:53:19.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5184000000000001e-05 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697262+0000", 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060013099999999995 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697437+0000", 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017512499999999999 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.697672+0000", 2024-04-25T15:53:19.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023489800000000001 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.715797+0000", 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018125601000000002 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718141+0000", 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023436300000000002 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718244+0000", 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102379 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.718333+0000", 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9136999999999997e-05 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:268 3.14 3:29a5f800:::10000000002.00000008:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.656078458, 2024-04-25T15:53:19.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032265195000000003, 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 268 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060927+0000", 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.060931+0000", 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6629999999999999e-06 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067652+0000", 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067213899999999998 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067713+0000", 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0628999999999998e-05 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067739+0000", 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6492999999999999e-05 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.067893+0000", 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015336599999999999 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068083+0000", 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019004800000000001 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.068328+0000", 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024518100000000002 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.087697+0000", 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019368720999999998 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.092957+0000", 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052597859999999998 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093058+0000", 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101335 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.093192+0000", 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134583 2024-04-25T15:53:19.779 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:271 3.d 3:b0681b21:::10000000002.0000000b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.523683308999999, 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026911917, 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 271 2024-04-25T15:53:19.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193322+0000", 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.193326+0000", 2024-04-25T15:53:19.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0770000000000001e-06 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198319+0000", 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049928710000000003 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198382+0000", 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2446000000000005e-05 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198408+0000", 2024-04-25T15:53:19.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6676999999999999e-05 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198623+0000", 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021493799999999999 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.198914+0000", 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029086100000000001 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.199346+0000", 2024-04-25T15:53:19.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000431316 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.219423+0000", 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020076954000000001 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220028+0000", 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060503900000000003 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220153+0000", 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125919 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:00.220234+0000", 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0819e-05 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:347 3.11 3:8815798b:::10000000002.00000057:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.568617165999999, 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029152658000000001, 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.785 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 347 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148389+0000", 2024-04-25T15:53:19.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.148391+0000", 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6029999999999999e-06 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155889+0000", 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074977200000000003 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155942+0000", 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2883999999999997e-05 2024-04-25T15:53:19.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.155964+0000", 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2762999999999999e-05 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156184+0000", 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021919499999999999 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156358+0000", 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017470800000000001 2024-04-25T15:53:19.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.156597+0000", 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238411 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.172619+0000", 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016022649 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177350+0000", 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047301369999999997 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177455+0000", 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010574899999999999 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.177541+0000", 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5839000000000003e-05 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:352 3.7 e68/18 3:e0f5f483:::10000000002.0000005c:head v 68'11, mlcod=67'9)", 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.377030646000001, 2024-04-25T15:53:19.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026288315999999999, 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339975+0000", 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.339978+0000", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8459999999999999e-06 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345285+0000", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053072630000000004 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345324+0000", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8535999999999998e-05 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345348+0000", 2024-04-25T15:53:19.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4315999999999999e-05 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345503+0000", 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015497300000000001 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.345555+0000", 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1814999999999998e-05 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366145+0000", 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020589743000000001 2024-04-25T15:53:19.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.366263+0000", 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118824 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:326 3.8 e69/68 3:133e839e:::10000000002.00000042:head v 69'10, mlcod=0'0)", 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.279509011999998, 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025085139999999999, 2024-04-25T15:53:19.794 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437497+0000", 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.437498+0000", 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.632e-06 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445817+0000", 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083191740000000004 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445839+0000", 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1314e-05 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445854+0000", 2024-04-25T15:53:19.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4986e-05 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445948+0000", 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3735999999999998e-05 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.445981+0000", 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3210999999999998e-05 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462484+0000", 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016502871999999998 2024-04-25T15:53:19.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.462582+0000", 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8214999999999994e-05 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:367 3.11 3:8b4c71c3:::10000000002.0000006b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.772692667999999, 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025258800000000001, 2024-04-25T15:53:19.798 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 367 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944313+0000", 2024-04-25T15:53:19.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.944316+0000", 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4880000000000001e-06 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950833+0000", 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065179829999999998 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950884+0000", 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:19.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.950902+0000", 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8805e-05 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951039+0000", 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013634300000000001 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951216+0000", 2024-04-25T15:53:19.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017729099999999999 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.951469+0000", 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025263199999999998 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969276+0000", 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017807630000000001 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969326+0000", 2024-04-25T15:53:19.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9639999999999999e-05 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969434+0000", 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108155 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.969572+0000", 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013767299999999999 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.803 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:406 3.11 3:8834f03f:::10000000002.00000092:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.356585953, 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034893977, 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 406 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360420+0000", 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.360424+0000", 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2300000000000002e-06 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370215+0000", 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0097913340000000005 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370283+0000", 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8028999999999995e-05 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370311+0000", 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8053999999999999e-05 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370464+0000", 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015309899999999999 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370642+0000", 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177648 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.370885+0000", 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000242721 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.393703+0000", 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022818037999999999 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395071+0000", 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001368469 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395176+0000", 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104949 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.395314+0000", 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013740599999999999 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:411 3.11 3:8988b152:::10000000002.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.157518282, 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024254155999999999, 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 411 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559487+0000", 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.559491+0000", 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4699999999999998e-06 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566706+0000", 2024-04-25T15:53:19.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072154760000000002 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566772+0000", 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5646000000000001e-05 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.566807+0000", 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4672999999999998e-05 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567067+0000", 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026066400000000002 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567252+0000", 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018467599999999999 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.567510+0000", 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025795100000000002 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.582484+0000", 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014973584 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583549+0000", 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001065365 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583653+0000", 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010430899999999999 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.583742+0000", 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8342000000000002e-05 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:421 3.11 3:88ab7984:::10000000002.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.781532967, 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028370244999999999, 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 421 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935473+0000", 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.935477+0000", 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0470000000000004e-06 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943048+0000", 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075710159999999999 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943111+0000", 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3026000000000003e-05 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943136+0000", 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.478e-05 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943342+0000", 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020601500000000001 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.943637+0000", 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000295254 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.944117+0000", 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00047971599999999999 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963573+0000", 2024-04-25T15:53:19.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019456543 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963639+0000", 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6292999999999997e-05 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963740+0000", 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010014900000000001 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.963843+0000", 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010340600000000001 2024-04-25T15:53:19.818 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:428 3.3b 3:ddecd1ef:::10000000002.000000a8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.512690718, 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026002537999999999, 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 428 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.819 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204315+0000", 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.204318+0000", 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.027e-06 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212124+0000", 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0078060239999999999 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212191+0000", 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7162999999999994e-05 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212218+0000", 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7223e-05 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212375+0000", 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015651200000000001 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212559+0000", 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018415799999999999 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.212807+0000", 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024789799999999998 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.224975+0000", 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012167661 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230128+0000", 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051534099999999998 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230234+0000", 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106326 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:19.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.230318+0000", 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3135999999999998e-05 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:19.824 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:19.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:19.826 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:19.823+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:19.926 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:19.923+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:19.985 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:19.985 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:20.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:20.027 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:20.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:20.027 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T15:53:20.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:53:20.028 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:20.027+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:20.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f871a7fc640 1 -- 172.21.15.67:0/3025638639 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v79) ==== 97+0+0 (secure 0 0 0) 0x7f87140f6620 con 0x7f87240c99e0 2024-04-25T15:53:20.069 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f870806f430 msgr2=0x7f8708071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f870806f430 0x7f8708071940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f871c005c10 tx=0x7f871c009040 comp rx=0 tx=0).stop 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87240c99e0 msgr2=0x7f872414d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87240c99e0 0x7f872414d6e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f87140ae040 tx=0x7f8714019160 comp rx=0 tx=0).stop 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 shutdown_connections 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f870806f430 0x7f8708071940 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8724164300 0x7f87241588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87240c99e0 0x7f872414d6e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 --2- 172.21.15.67:0/3025638639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87240733f0 0x7f87241fe7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 >> 172.21.15.67:0/3025638639 conn(0x7f8724070db0 msgr2=0x7f87240c82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:20.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 shutdown_connections 2024-04-25T15:53:20.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.070+0000 7f872bcc6640 1 -- 172.21.15.67:0/3025638639 wait complete. 2024-04-25T15:53:20.141 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:20.142+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:20.156 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:20.156 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:20.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:20.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 Processor -- start 2024-04-25T15:53:20.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 -- start start 2024-04-25T15:53:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92780caf60 0x7f927815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f927815d790 0x7f927806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9278073bf0 0x7f927806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f92780cab60 con 0x7f92780caf60 2024-04-25T15:53:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9278159620 con 0x7f927815d790 2024-04-25T15:53:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927e50a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f92780c77a0 con 0x7f9278073bf0 2024-04-25T15:53:20.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92780caf60 0x7f927815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92780caf60 0x7f927815d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42502/0 (socket says 172.21.15.67:42502) 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 -- 172.21.15.67:0/520324378 learned_addr learned my addr 172.21.15.67:0/520324378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f92777fe640 1 --2- 172.21.15.67:0/520324378 >> v2:172.21.15.176:3300/0 conn(0x7f927815d790 0x7f927806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f927caa7640 1 --2- 172.21.15.67:0/520324378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9278073bf0 0x7f927806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 -- 172.21.15.67:0/520324378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9278073bf0 msgr2=0x7f927806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 --2- 172.21.15.67:0/520324378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9278073bf0 0x7f927806dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 -- 172.21.15.67:0/520324378 >> v2:172.21.15.176:3300/0 conn(0x7f927815d790 msgr2=0x7f927806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 --2- 172.21.15.67:0/520324378 >> v2:172.21.15.176:3300/0 conn(0x7f927815d790 0x7f927806d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.190+0000 7f9277fff640 1 -- 172.21.15.67:0/520324378 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92780c7a60 con 0x7f92780caf60 2024-04-25T15:53:20.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f9277fff640 1 --2- 172.21.15.67:0/520324378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92780caf60 0x7f927815d250 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f92780cf8d0 tx=0x7f927001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134f8282c4c51854 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f9276ffd640 1 -- 172.21.15.67:0/520324378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92700b0070 con 0x7f92780caf60 2024-04-25T15:53:20.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f9276ffd640 1 -- 172.21.15.67:0/520324378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92700ab600 con 0x7f92780caf60 2024-04-25T15:53:20.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f9276ffd640 1 -- 172.21.15.67:0/520324378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92700ab9d0 con 0x7f92780caf60 2024-04-25T15:53:20.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 -- 172.21.15.67:0/520324378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92780caf60 msgr2=0x7f927815d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 --2- 172.21.15.67:0/520324378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92780caf60 0x7f927815d250 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f92780cf8d0 tx=0x7f927001c520 comp rx=0 tx=0).stop 2024-04-25T15:53:20.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 -- 172.21.15.67:0/520324378 shutdown_connections 2024-04-25T15:53:20.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 --2- 172.21.15.67:0/520324378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9278073bf0 0x7f927806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 --2- 172.21.15.67:0/520324378 >> v2:172.21.15.176:3300/0 conn(0x7f927815d790 0x7f927806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 --2- 172.21.15.67:0/520324378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92780caf60 0x7f927815d250 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 -- 172.21.15.67:0/520324378 >> 172.21.15.67:0/520324378 conn(0x7f9278070db0 msgr2=0x7f92780ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 -- 172.21.15.67:0/520324378 shutdown_connections 2024-04-25T15:53:20.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 -- 172.21.15.67:0/520324378 wait complete. 2024-04-25T15:53:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 Processor -- start 2024-04-25T15:53:20.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.194+0000 7f927e50a640 1 -- start start 2024-04-25T15:53:20.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9278073bf0 0x7f9278164da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92780caf60 0x7f9278156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f927815d790 0x7f9278163f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f92780733d0 con 0x7f9278073bf0 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f927815a600 con 0x7f927815d790 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9278073550 con 0x7f92780caf60 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9278073bf0 0x7f9278164da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9278073bf0 0x7f9278164da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42514/0 (socket says 172.21.15.67:42514) 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 -- 172.21.15.67:0/1660884505 learned_addr learned my addr 172.21.15.67:0/1660884505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927caa7640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f927815d790 0x7f9278163f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f92777fe640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92780caf60 0x7f9278156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 -- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92780caf60 msgr2=0x7f9278156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92780caf60 0x7f9278156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 -- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f927815d790 msgr2=0x7f9278163f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f927815d790 0x7f9278163f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 -- 172.21.15.67:0/1660884505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92781652e0 con 0x7f9278073bf0 2024-04-25T15:53:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9277fff640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9278073bf0 0x7f9278164da0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f92700ae040 tx=0x7f92700192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9274ff9640 1 -- 172.21.15.67:0/1660884505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92700ba070 con 0x7f9278073bf0 2024-04-25T15:53:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9274ff9640 1 -- 172.21.15.67:0/1660884505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92700b4640 con 0x7f9278073bf0 2024-04-25T15:53:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f9274ff9640 1 -- 172.21.15.67:0/1660884505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92700abe60 con 0x7f9278073bf0 2024-04-25T15:53:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92781f5e80 con 0x7f9278073bf0 2024-04-25T15:53:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.198+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92781f6440 con 0x7f9278073bf0 2024-04-25T15:53:20.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.202+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f927806a480 con 0x7f9278073bf0 2024-04-25T15:53:20.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.202+0000 7f9274ff9640 1 -- 172.21.15.67:0/1660884505 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f92700ab4c0 con 0x7f9278073bf0 2024-04-25T15:53:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.202+0000 7f9274ff9640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f925806f400 0x7f9258071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.202+0000 7f9274ff9640 1 -- 172.21.15.67:0/1660884505 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5915+0+0 (secure 0 0 0) 0x7f9270128630 con 0x7f9278073bf0 2024-04-25T15:53:20.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.202+0000 7f92777fe640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f925806f400 0x7f9258071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.202+0000 7f92777fe640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f925806f400 0x7f9258071910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f92781f3650 tx=0x7f9260008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.206+0000 7f9274ff9640 1 -- 172.21.15.67:0/1660884505 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f92700c3080 con 0x7f9278073bf0 2024-04-25T15:53:20.229 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:20.226+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:20.329 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:20.330+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:5 2.1f 2:ff5b34d6:::1.00000000:head [create,setxattr parent (30) in=6b,setxattr layout (30) in=6b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 164.12068333299999, 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015036353000000001, 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211496+0000", 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211497+0000", 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1170000000000001e-06 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211505+0000", 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.259e-06 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211748+0000", 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000243012 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211759+0000", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1232e-05 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214713+0000", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0029532569999999999 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.214967+0000", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025407400000000002 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.215371+0000", 2024-04-25T15:53:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00040476999999999999 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.225904+0000", 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010532992999999999 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226253+0000", 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00034850199999999999 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226407+0000", 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015422999999999999 2024-04-25T15:53:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226532+0000", 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000124907 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:26 2.1f 2:fe07bba1:::100.00000000.inode:head [writefull 0~582] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 164.120635948, 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015301949, 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211543+0000", 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211544+0000", 2024-04-25T15:53:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.176e-06 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211551+0000", 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2690000000000002e-06 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211971+0000", 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000419613 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212015+0000", 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4280999999999998e-05 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216052+0000", 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040368160000000004 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216389+0000", 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033694399999999999 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.216696+0000", 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030701099999999999 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226033+0000", 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093367989999999998 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226565+0000", 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053194699999999995 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226693+0000", 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.226845+0000", 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015183300000000001 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.0.4:35 2.1f 2:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e19)", 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 164.12060369400001, 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015493484, 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211575+0000", 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211576+0000", 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0839999999999999e-06 2024-04-25T15:53:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.211595+0000", 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8298999999999999e-05 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212128+0000", 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00053311000000000005 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.212145+0000", 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6973000000000002e-05 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219256+0000", 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007110819 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.219591+0000", 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00033512100000000002 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:36.227069+0000", 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007478078 2024-04-25T15:53:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:41 3.f e59/54 3:f045ad74:::10000000002.00000025:head v 59'1, mlcod=0'0)", 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.516413632000003, 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015885976, 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815765+0000", 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.815769+0000", 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.704e-06 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820849+0000", 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050802190000000004 2024-04-25T15:53:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820897+0000", 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7833000000000001e-05 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.820928+0000", 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1006000000000001e-05 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821049+0000", 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121056 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.821089+0000", 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891999999999997e-05 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831496+0000", 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010407342999999999 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:43.831651+0000", 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015492300000000001 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:47 3.12 e59/40 3:489b4446:::10000000002.0000002b:head v 59'4, mlcod=58'2)", 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.225615488999999, 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016539835999999999, 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106564+0000", 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.106566+0000", 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640000000000002e-06 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112447+0000", 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058808569999999998 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112466+0000", 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9457e-05 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112479+0000", 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2992e-05 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112562+0000", 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2582000000000003e-05 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.112584+0000", 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2353000000000001e-05 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123028+0000", 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010443885999999999 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.123103+0000", 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5345000000000005e-05 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:66 3.b e60/21 3:d381a89f:::10000000002.0000003e:head v 60'1, mlcod=0'0)", 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.386965347999997, 2024-04-25T15:53:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015316048000000001, 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945214+0000", 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.945216+0000", 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9e-06 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948749+0000", 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035338119999999999 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948777+0000", 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7915999999999998e-05 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948796+0000", 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.844e-05 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948895+0000", 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9704000000000006e-05 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.948916+0000", 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.107e-05 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960445+0000", 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011528722999999999 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:44.960530+0000", 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4482999999999997e-05 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:152 3.39 e61/18 3:9e24e8b4:::10000000002.00000094:head v 61'2, mlcod=0'0)", 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.595095270000002, 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016125006000000001, 2024-04-25T15:53:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737084+0000", 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.737086+0000", 2024-04-25T15:53:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7640000000000001e-06 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744295+0000", 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072082320000000002 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744341+0000", 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6072000000000002e-05 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744376+0000", 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4903000000000001e-05 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744533+0000", 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015736299999999999 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.744590+0000", 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6771000000000002e-05 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753101+0000", 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085115490000000002 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.753209+0000", 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107352 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:156 3.3e e61/35 3:7d47d675:::10000000002.00000098:head v 61'1, mlcod=0'0)", 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.397941797000001, 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016362634000000001, 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934237+0000", 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934240+0000", 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3149999999999999e-06 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939270+0000", 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050303869999999999 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939308+0000", 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8047999999999999e-05 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939335+0000", 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7251999999999999e-05 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939432+0000", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6637000000000003e-05 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.939459+0000", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7651999999999999e-05 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950534+0000", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011074993999999999 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950600+0000", 2024-04-25T15:53:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5349000000000001e-05 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:158 3.12 e61/40 3:4b703321:::10000000002.0000009a:head v 61'6, mlcod=59'4)", 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.314981670000002, 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017102098, 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017197+0000", 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.017200+0000", 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3199999999999998e-06 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024143+0000", 2024-04-25T15:53:20.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069429569999999996 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024169+0000", 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6007e-05 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024190+0000", 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1203999999999999e-05 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024272+0000", 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1833000000000001e-05 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.024301+0000", 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9475000000000002e-05 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034243+0000", 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0099420589999999996 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034299+0000", 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6242999999999997e-05 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:162 3.2b e61/40 3:d691b431:::10000000002.0000009e:head v 61'3, mlcod=60'1)", 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.106637659, 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014572351000000001, 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225541+0000", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.225544+0000", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7199999999999998e-06 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230207+0000", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046626719999999997 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230239+0000", 2024-04-25T15:53:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2548000000000001e-05 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230266+0000", 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6599999999999999e-05 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230382+0000", 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115667 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.230417+0000", 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5800999999999998e-05 2024-04-25T15:53:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240029+0000", 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096115349999999992 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.240114+0000", 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4808000000000005e-05 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:181 3.1f e62/40 3:f8dc3ee2:::10000000002.000000b1:head v 62'2, mlcod=0'0)", 2024-04-25T15:53:20.373 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.281967064, 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015250434, 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050212+0000", 2024-04-25T15:53:20.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.050214+0000", 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.013e-06 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054240+0000", 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040256609999999998 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054262+0000", 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2691e-05 2024-04-25T15:53:20.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054277+0000", 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4464000000000001e-05 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054365+0000", 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8657000000000002e-05 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.054400+0000", 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4505999999999998e-05 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065391+0000", 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010991287000000001 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.065462+0000", 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1155000000000006e-05 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:260 3.12 e66/40 3:4bec3886:::10000000002.00000000:head v 66'8, mlcod=61'6)", 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.633712015, 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019802660999999999, 2024-04-25T15:53:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698467+0000", 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.698470+0000", 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704135+0000", 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056657490000000003 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704159+0000", 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3903000000000001e-05 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704176+0000", 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6461e-05 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704290+0000", 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011446999999999999 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.704336+0000", 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5558e-05 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718190+0000", 2024-04-25T15:53:20.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013854577999999999 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:59.718270+0000", 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9377999999999998e-05 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:267 3.18 e67/18 3:19a74f15:::10000000002.00000007:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.316239322000001, 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015975057000000001, 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.381 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.015940+0000", 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0999999999999999e-07 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019042+0000", 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031019680000000001 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019059+0000", 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6631000000000001e-05 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019075+0000", 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6586e-05 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019143+0000", 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7626999999999995e-05 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.019174+0000", 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1078999999999998e-05 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031852+0000", 2024-04-25T15:53:20.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012678233000000001 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.031915+0000", 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2422999999999995e-05 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:269 3.2c e67/54 3:354fa1eb:::10000000002.00000009:head v 67'3, mlcod=59'1)", 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.219205247000001, 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014629644000000001, 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.112974+0000", 2024-04-25T15:53:20.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4000000000000001e-07 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119407+0000", 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064328099999999997 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119437+0000", 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9451e-05 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119460+0000", 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3640999999999999e-05 2024-04-25T15:53:20.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119565+0000", 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010474 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.119600+0000", 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4851999999999998e-05 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127513+0000", 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079133090000000003 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.127603+0000", 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0100999999999997e-05 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:275 3.30 e67/18 3:0ca36612:::10000000002.0000000f:head v 67'7, mlcod=62'5)", 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.954266665999999, 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015157541, 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377912+0000", 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.377913+0000", 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2499999999999995e-07 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383613+0000", 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0057000189999999997 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383639+0000", 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6475e-05 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383656+0000", 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.698e-05 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383760+0000", 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103428 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.383793+0000", 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3158999999999999e-05 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393003+0000", 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092102839999999991 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.393070+0000", 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6670999999999999e-05 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:276 3.1f e67/40 3:f8cab2eb:::10000000002.00000010:head v 67'4, mlcod=62'2)", 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.900533588999998, 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015350925, 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431645+0000", 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.431648+0000", 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3649999999999998e-06 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435504+0000", 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003856221 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435533+0000", 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9461000000000001e-05 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435561+0000", 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7684000000000001e-05 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435659+0000", 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7588000000000001e-05 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.435694+0000", 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5185999999999998e-05 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446705+0000", 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011010804000000001 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:00.446996+0000", 2024-04-25T15:53:20.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029161600000000002 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:353 3.1 e69/54 3:838ab1fd:::10000000002.0000005d:head v 69'7, mlcod=62'5)", 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.935033526000002, 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017336667, 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:20.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397145+0000", 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.397146+0000", 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3700000000000001e-07 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398713+0000", 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001566764 2024-04-25T15:53:20.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398740+0000", 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7317e-05 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398777+0000", 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6885999999999999e-05 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398898+0000", 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121723 2024-04-25T15:53:20.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.398932+0000", 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3491999999999997e-05 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414418+0000", 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015486491 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.414482+0000", 2024-04-25T15:53:20.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3657000000000004e-05 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:357 3.34 e69/54 3:2d119e8b:::10000000002.00000061:head v 69'10, mlcod=67'8)", 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.760861337000001, 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015675116999999999, 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571318+0000", 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.571319+0000", 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.139e-06 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576161+0000", 2024-04-25T15:53:20.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048417979999999996 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576182+0000", 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1262000000000001e-05 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576193+0000", 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1197e-05 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576262+0000", 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9165000000000006e-05 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.576283+0000", 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0582999999999999e-05 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586920+0000", 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010637275999999999 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.586993+0000", 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2696999999999999e-05 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:372 3.10 e69/18 3:0b92729d:::10000000002.00000070:head v 69'7, mlcod=67'5)", 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.214163967000001, 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015303605, 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118015+0000", 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.118016+0000", 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1930000000000001e-06 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123008+0000", 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0049919639999999998 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123026+0000", 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8034999999999999e-05 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123040+0000", 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3895e-05 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123138+0000", 2024-04-25T15:53:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7929000000000004e-05 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.123177+0000", 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8587999999999997e-05 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133262+0000", 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010085782 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:04.133319+0000", 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6218999999999998e-05 2024-04-25T15:53:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:397 3.20 e69/18 3:06fc074c:::10000000002.00000089:head v 69'12, mlcod=68'10)", 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.283675387000001, 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015916966000000001, 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:20.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048504+0000", 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.048506+0000", 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0439999999999998e-06 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054867+0000", 2024-04-25T15:53:20.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063613209999999996 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054891+0000", 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4320000000000001e-05 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054921+0000", 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9535000000000001e-05 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.054991+0000", 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0314999999999993e-05 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.055025+0000", 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3809000000000001e-05 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064353+0000", 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093282839999999992 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:05.064421+0000", 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7337999999999998e-05 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.413 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.413 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:20.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.370+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f92780c8a40 con 0x7f9278073bf0 2024-04-25T15:53:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f9274ff9640 1 -- 172.21.15.67:0/1660884505 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v79) ==== 81+0+35368 (secure 0 0 0) 0x7f9270016110 con 0x7f9278073bf0 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 79, 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:53:20.062481+0000", 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:53:14.990625+0000", 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5806080, 2024-04-25T15:53:20.415 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:53:20.416 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:20.417 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:20.418 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:20.419 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:20.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.422 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:20.423 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:20.424 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:20.426 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:20.427 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.75, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.75, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:20.428 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:20.429 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:20.430 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:20.431 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:20.432 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.3200000524520874, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.3200000524520874, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:20.433 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.434 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.435 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.436 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.437 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:53:20.438 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0.2953948974609375, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:20.439 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.443 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:20.444 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 73, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.446 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:53:20.447 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.448 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.449 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.451 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.454 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.455 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.456 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.458 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:20.459 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:53:20.460 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.462 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:20.464 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:53:20.465 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.466 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:53:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:20.469 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.472 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.473 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.474 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.476 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:53:20.477 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.478 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.18", 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: 4 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.480 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:20.481 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.3e", 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:20.482 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 0, 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:53:20.483 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:20.484 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f925806f400 msgr2=0x7f9258071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f925806f400 0x7f9258071910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f92781f3650 tx=0x7f9260008040 comp rx=0 tx=0).stop 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9278073bf0 msgr2=0x7f9278164da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9278073bf0 0x7f9278164da0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f92700ae040 tx=0x7f92700192c0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 shutdown_connections 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f925806f400 0x7f9258071910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f927815d790 0x7f9278163f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92780caf60 0x7f9278156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 --2- 172.21.15.67:0/1660884505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9278073bf0 0x7f9278164da0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 >> 172.21.15.67:0/1660884505 conn(0x7f9278070db0 msgr2=0x7f92781597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 shutdown_connections 2024-04-25T15:53:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.374+0000 7f927e50a640 1 -- 172.21.15.67:0/1660884505 wait complete. 2024-04-25T15:53:20.487 INFO:tasks.thrashosds.thrasher:j is {'epoch': 79, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:53:20.062481+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:53:14.990625+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 0.2953948974609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [1, 4]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '3.3e', 'mappings': [{'from': 0, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:53:20.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:53:20.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:20.531 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:20.530+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:20.590 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:20.590+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.666+0000 7f9d5da5b640 1 Processor -- start 2024-04-25T15:53:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.666+0000 7f9d5da5b640 1 -- start start 2024-04-25T15:53:20.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d5da5b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d580c92d0 0x7f9d58072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d5da5b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f9d580c96e0 0x7f9d58073300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.670 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:20.670 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:20.670 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:20.670 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d5da5b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5806d8e0 con 0x7f9d58160800 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d5da5b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9d5806dbc0 con 0x7f9d580c92d0 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d5da5b640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f9d58073c70 con 0x7f9d580c96e0 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d580c92d0 0x7f9d58072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d580c92d0 0x7f9d58072dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42522/0 (socket says 172.21.15.67:42522) 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 -- 172.21.15.67:0/7499648 learned_addr learned my addr 172.21.15.67:0/7499648 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d577fe640 1 --2- 172.21.15.67:0/7499648 >> v2:172.21.15.176:3300/0 conn(0x7f9d580c96e0 0x7f9d58073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 -- 172.21.15.67:0/7499648 >> v2:172.21.15.176:3300/0 conn(0x7f9d580c96e0 msgr2=0x7f9d58073300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 --2- 172.21.15.67:0/7499648 >> v2:172.21.15.176:3300/0 conn(0x7f9d580c96e0 0x7f9d58073300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 -- 172.21.15.67:0/7499648 >> v1:172.21.15.176:6789/0 conn(0x7f9d58160800 legacy=0x7f9d5806d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 -- 172.21.15.67:0/7499648 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d58073f50 con 0x7f9d580c92d0 2024-04-25T15:53:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d567fc640 1 --2- 172.21.15.67:0/7499648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c92d0 0x7f9d58072dc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9d5806dd40 tx=0x7f9d4c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ce09a5f0806eb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d55ffb640 1 -- 172.21.15.67:0/7499648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0b0070 con 0x7f9d580c92d0 2024-04-25T15:53:20.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d55ffb640 1 -- 172.21.15.67:0/7499648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4c0ab600 con 0x7f9d580c92d0 2024-04-25T15:53:20.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.670+0000 7f9d55ffb640 1 -- 172.21.15.67:0/7499648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0ab9d0 con 0x7f9d580c92d0 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- 172.21.15.67:0/7499648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c92d0 msgr2=0x7f9d58072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 --2- 172.21.15.67:0/7499648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c92d0 0x7f9d58072dc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9d5806dd40 tx=0x7f9d4c01c450 comp rx=0 tx=0).stop 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- 172.21.15.67:0/7499648 shutdown_connections 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 --2- 172.21.15.67:0/7499648 >> v2:172.21.15.176:3300/0 conn(0x7f9d580c96e0 0x7f9d58073300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 --2- 172.21.15.67:0/7499648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c92d0 0x7f9d58072dc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- 172.21.15.67:0/7499648 >> 172.21.15.67:0/7499648 conn(0x7f9d58070db0 msgr2=0x7f9d580ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- 172.21.15.67:0/7499648 shutdown_connections 2024-04-25T15:53:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- 172.21.15.67:0/7499648 wait complete. 2024-04-25T15:53:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 Processor -- start 2024-04-25T15:53:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- start start 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d580c92d0 0x7f9d581f9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c96e0 0x7f9d580cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58160800 0x7f9d5814ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58160d90 con 0x7f9d580c96e0 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58160c10 con 0x7f9d58160800 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d5da5b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9d581699a0 con 0x7f9d580c92d0 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d567fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c96e0 0x7f9d580cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d567fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c96e0 0x7f9d580cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42526/0 (socket says 172.21.15.67:42526) 2024-04-25T15:53:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.674+0000 7f9d567fc640 1 -- 172.21.15.67:0/831797978 learned_addr learned my addr 172.21.15.67:0/831797978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d580c92d0 0x7f9d581f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d577fe640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58160800 0x7f9d5814ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d567fc640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d580c92d0 msgr2=0x7f9d581f9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d567fc640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d580c92d0 0x7f9d581f9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d567fc640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58160800 msgr2=0x7f9d5814ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d567fc640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d58160800 0x7f9d5814ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d567fc640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d581f9ca0 con 0x7f9d580c96e0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d567fc640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c96e0 0x7f9d580cd770 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c01a080 tx=0x7f9d4c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0b9070 con 0x7f9d580c96e0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4c0b4420 con 0x7f9d580c96e0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d5da5b640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d580c89b0 con 0x7f9d580c96e0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c0b0040 con 0x7f9d580c96e0 2024-04-25T15:53:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d5da5b640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d58159c70 con 0x7f9d580c96e0 2024-04-25T15:53:20.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9d30005280 con 0x7f9d580c96e0 2024-04-25T15:53:20.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f07d6809640 1 Processor -- start 2024-04-25T15:53:20.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f07d6809640 1 -- start start 2024-04-25T15:53:20.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9d4c0bc070 con 0x7f9d580c96e0 2024-04-25T15:53:20.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3ffff640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9d4406f400 0x7f9d44071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5915+0+0 (secure 0 0 0) 0x7f9d4c1283e0 con 0x7f9d580c96e0 2024-04-25T15:53:20.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v79) ==== 54+0+16 (secure 0 0 0) 0x7f9d4c0160d0 con 0x7f9d580c96e0 2024-04-25T15:53:20.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9d4406f400 0x7f9d44071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9d300045f0 0x7f9d30004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f07d6809640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d006d460 0x7f07d00736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d30005560 con 0x7f9d300045f0 2024-04-25T15:53:20.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f07d6809640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07d01547e0 0x7f07d006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f07d6809640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d00caa60 con 0x7f07d006cb80 2024-04-25T15:53:20.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f07d6809640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f07d0068590 con 0x7f07d01547e0 2024-04-25T15:53:20.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.678+0000 7f07d6809640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07d015a5f0 con 0x7f07d006d460 2024-04-25T15:53:20.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f9d577fe640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9d300045f0 0x7f9d30004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07d01547e0 0x7f07d006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07d01547e0 0x7f07d006c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42542/0 (socket says 172.21.15.67:42542) 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 -- 172.21.15.67:0/2320911582 learned_addr learned my addr 172.21.15.67:0/2320911582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07cffff640 1 --2- 172.21.15.67:0/2320911582 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d006d460 0x7f07d00736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 -- 172.21.15.67:0/2320911582 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d006d460 msgr2=0x7f07d00736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 --2- 172.21.15.67:0/2320911582 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d006d460 0x7f07d00736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 -- 172.21.15.67:0/2320911582 >> v1:172.21.15.176:6789/0 conn(0x7f07d006cb80 legacy=0x7f07d015aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 -- 172.21.15.67:0/2320911582 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d00ca100 con 0x7f07d01547e0 2024-04-25T15:53:20.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f9d577fe640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9d300045f0 0x7f9d30004a00 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9d4406f400 0x7f9d44071910 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800b810 tx=0x7f9d48008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d30005560 con 0x7f9d300045f0 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07c7fff640 1 --2- 172.21.15.67:0/2320911582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d01547e0 0x7f07d006c640 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f07b8006fd0 tx=0x7f07b801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c489aba342b3c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07cf7fe640 1 -- 172.21.15.67:0/2320911582 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b80ab430 con 0x7f07d01547e0 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07cf7fe640 1 -- 172.21.15.67:0/2320911582 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b80b0070 con 0x7f07d01547e0 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07cf7fe640 1 -- 172.21.15.67:0/2320911582 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b80ab7f0 con 0x7f07d01547e0 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 -- 172.21.15.67:0/2320911582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d01547e0 msgr2=0x7f07d006c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 --2- 172.21.15.67:0/2320911582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d01547e0 0x7f07d006c640 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f07b8006fd0 tx=0x7f07b801c540 comp rx=0 tx=0).stop 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 -- 172.21.15.67:0/2320911582 shutdown_connections 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 --2- 172.21.15.67:0/2320911582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d01547e0 0x7f07d006c640 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 --2- 172.21.15.67:0/2320911582 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d006d460 0x7f07d00736d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 -- 172.21.15.67:0/2320911582 >> 172.21.15.67:0/2320911582 conn(0x7f07d00ce1c0 msgr2=0x7f07d00ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 -- 172.21.15.67:0/2320911582 shutdown_connections 2024-04-25T15:53:20.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.682+0000 7f07d6809640 1 -- 172.21.15.67:0/2320911582 wait complete. 2024-04-25T15:53:20.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 Processor -- start 2024-04-25T15:53:20.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 -- start start 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d006cb80 0x7f07d01f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07d006d460 0x7f07d00b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d01547e0 0x7f07d014b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0154d70 con 0x7f07d006cb80 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0154bf0 con 0x7f07d006d460 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d6809640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07d0167790 con 0x7f07d01547e0 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d006cb80 0x7f07d01f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d006cb80 0x7f07d01f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42546/0 (socket says 172.21.15.67:42546) 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 -- 172.21.15.67:0/2940365644 learned_addr learned my addr 172.21.15.67:0/2940365644 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07c7fff640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07d006d460 0x7f07d00b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07d4da6640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d01547e0 0x7f07d014b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 -- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d01547e0 msgr2=0x7f07d014b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d01547e0 0x7f07d014b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 -- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07d006d460 msgr2=0x7f07d00b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07d006d460 0x7f07d00b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 -- 172.21.15.67:0/2940365644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d00b2600 con 0x7f07d006cb80 2024-04-25T15:53:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07c7fff640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07d006d460 0x7f07d00b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:20.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.686+0000 7f07cffff640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d006cb80 0x7f07d01f38a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e870 tx=0x7f07c000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07cd7fa640 1 -- 172.21.15.67:0/2940365644 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c0017070 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07cd7fa640 1 -- 172.21.15.67:0/2940365644 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c000c940 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d01f3e90 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07cd7fa640 1 -- 172.21.15.67:0/2940365644 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c000ccf0 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d01f4710 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d006a480 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07cd7fa640 1 -- 172.21.15.67:0/2940365644 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f07c001a070 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07cd7fa640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07b006f430 0x7f07b0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07cd7fa640 1 -- 172.21.15.67:0/2940365644 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5915+0+0 (secure 0 0 0) 0x7f07c008f600 con 0x7f07d006cb80 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07c7fff640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07b006f430 0x7f07b0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.690+0000 7f07c7fff640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07b006f430 0x7f07b0071940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f07b80af000 tx=0x7f07b80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.695 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:20.691+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.694+0000 7f07cd7fa640 1 -- 172.21.15.67:0/2940365644 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f07c005c420 con 0x7f07d006cb80 2024-04-25T15:53:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.698+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d30001280 con 0x7f9d300045f0 2024-04-25T15:53:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.698+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d30001280 con 0x7f9d300045f0 2024-04-25T15:53:20.700 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.698+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9d30001d40 0x7f9d300029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.698+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9d30002370 con 0x7f9d30001d40 2024-04-25T15:53:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.698+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9d30001d40 0x7f9d300029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.698+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9d30001d40 0x7f9d300029a0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.702+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d30002370 con 0x7f9d30001d40 2024-04-25T15:53:20.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.714+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d30003210 con 0x7f9d30001d40 2024-04-25T15:53:20.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.714+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d30003210 con 0x7f9d30001d40 2024-04-25T15:53:20.717 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:20.717 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:20.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.714+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9d30009c90 0x7f9d3000a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.714+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9d3000a660 con 0x7f9d30009c90 2024-04-25T15:53:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.718+0000 7f9d567fc640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9d30009c90 0x7f9d3000a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.718+0000 7f9d567fc640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9d30009c90 0x7f9d3000a0a0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.718+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d3000a660 con 0x7f9d30009c90 2024-04-25T15:53:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.730+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d3000ab70 con 0x7f9d30009c90 2024-04-25T15:53:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.734+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d3000ab70 con 0x7f9d30009c90 2024-04-25T15:53:20.734 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.734+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9d3000d8f0 0x7f9d3000fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.734+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9d30012b60 con 0x7f9d3000d8f0 2024-04-25T15:53:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.734+0000 7f9d577fe640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9d3000d8f0 0x7f9d3000fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.734+0000 7f9d577fe640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9d3000d8f0 0x7f9d3000fd10 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.734+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d30012b60 con 0x7f9d3000d8f0 2024-04-25T15:53:20.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.746+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d30010640 con 0x7f9d3000d8f0 2024-04-25T15:53:20.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.750+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d30010640 con 0x7f9d3000d8f0 2024-04-25T15:53:20.752 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:20.752 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:20.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.750+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9d300124c0 0x7f9d30010850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.750+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9d30012af0 con 0x7f9d300124c0 2024-04-25T15:53:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.750+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9d300124c0 0x7f9d30010850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.754+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9d300124c0 0x7f9d30010850 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.754+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d30012af0 con 0x7f9d300124c0 2024-04-25T15:53:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.766+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d30012af0 con 0x7f9d300124c0 2024-04-25T15:53:20.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.774+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d30012af0 con 0x7f9d300124c0 2024-04-25T15:53:20.773 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.774+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9d30011300 0x7f9d30011710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.774+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9d30019620 con 0x7f9d30011300 2024-04-25T15:53:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.774+0000 7f9d567fc640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9d30011300 0x7f9d30011710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.774+0000 7f9d567fc640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9d30011300 0x7f9d30011710 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.774+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d30019620 con 0x7f9d30011300 2024-04-25T15:53:20.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.790+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d30016d90 con 0x7f9d30011300 2024-04-25T15:53:20.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.794+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d30016d90 con 0x7f9d30011300 2024-04-25T15:53:20.795 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:20.795 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:20.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.794+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9d30017410 0x7f9d30019830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.794+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9d3001c8f0 con 0x7f9d30017410 2024-04-25T15:53:20.796 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:20.794+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.794+0000 7f9d577fe640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9d30017410 0x7f9d30019830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.794+0000 7f9d577fe640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9d30017410 0x7f9d30019830 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.798+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d3001c8f0 con 0x7f9d30017410 2024-04-25T15:53:20.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.810+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d3001c880 con 0x7f9d30017410 2024-04-25T15:53:20.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.814+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d3001c880 con 0x7f9d30017410 2024-04-25T15:53:20.816 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:20.817 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:20.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.814+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9d3001bae0 0x7f9d3001bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:20.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.814+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9d3001fcc0 con 0x7f9d3001bae0 2024-04-25T15:53:20.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.818+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9d3001bae0 0x7f9d3001bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:20.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.818+0000 7f9d56ffd640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9d3001bae0 0x7f9d3001bf10 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:20.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.818+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d3001fcc0 con 0x7f9d3001bae0 2024-04-25T15:53:20.830 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:20.830 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:20.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.834+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f9d30019e90 con 0x7f9d3001bae0 2024-04-25T15:53:20.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3ffff640 1 -- 172.21.15.67:0/831797978 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9d30019e90 con 0x7f9d3001bae0 2024-04-25T15:53:20.838 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9d300045f0 msgr2=0x7f9d30004a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f9d300045f0 0x7f9d30004a00 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9d30001d40 msgr2=0x7f9d300029a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f9d30001d40 0x7f9d300029a0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9d30009c90 msgr2=0x7f9d3000a0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f9d30009c90 0x7f9d3000a0a0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9d3000d8f0 msgr2=0x7f9d3000fd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9d3000d8f0 0x7f9d3000fd10 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9d300124c0 msgr2=0x7f9d30010850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f9d300124c0 0x7f9d30010850 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9d30011300 msgr2=0x7f9d30011710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f9d30011300 0x7f9d30011710 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9d30017410 msgr2=0x7f9d30019830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f9d30017410 0x7f9d30019830 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d577fe640 1 -- 172.21.15.67:0/831797978 reap_dead start 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9d3001bae0 msgr2=0x7f9d3001bf10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9d3001bae0 0x7f9d3001bf10 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d577fe640 1 -- 172.21.15.67:0/831797978 reap_dead start 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d577fe640 1 -- 172.21.15.67:0/831797978 reap_dead start 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d577fe640 1 -- 172.21.15.67:0/831797978 reap_dead start 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9d4406f400 msgr2=0x7f9d44071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9d4406f400 0x7f9d44071910 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800b810 tx=0x7f9d48008040 comp rx=0 tx=0).stop 2024-04-25T15:53:20.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c96e0 msgr2=0x7f9d580cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c96e0 0x7f9d580cd770 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c01a080 tx=0x7f9d4c0191f0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 shutdown_connections 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f9d3001bae0 0x7f9d3001bf10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9d4406f400 0x7f9d44071910 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 --2- 172.21.15.67:0/831797978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d580c96e0 0x7f9d580cd770 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 >> 172.21.15.67:0/831797978 conn(0x7f9d58070db0 msgr2=0x7f9d58155be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 shutdown_connections 2024-04-25T15:53:20.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.838+0000 7f9d3dffb640 1 -- 172.21.15.67:0/831797978 wait complete. 2024-04-25T15:53:20.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:20.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.870+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f07d00cbaa0 con 0x7f07b006f430 2024-04-25T15:53:20.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.886+0000 7f07cd7fa640 1 -- 172.21.15.67:0/2940365644 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253465 (secure 0 0 0) 0x7f07d00cbaa0 con 0x7f07b006f430 2024-04-25T15:53:20.890 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:53:20.892 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07b006f430 msgr2=0x7f07b0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07b006f430 0x7f07b0071940 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f07b80af000 tx=0x7f07b80a7000 comp rx=0 tx=0).stop 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d006cb80 msgr2=0x7f07d01f38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d006cb80 0x7f07d01f38a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f07c000e870 tx=0x7f07c000ec70 comp rx=0 tx=0).stop 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 shutdown_connections 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07b006f430 0x7f07b0071940 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07d01547e0 0x7f07d014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07d006d460 0x7f07d00b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 --2- 172.21.15.67:0/2940365644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d006cb80 0x7f07d01f38a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:20.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 >> 172.21.15.67:0/2940365644 conn(0x7f07d00ce1c0 msgr2=0x7f07d00cac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:20.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 shutdown_connections 2024-04-25T15:53:20.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:20.890+0000 7f07d6809640 1 -- 172.21.15.67:0/2940365644 wait complete. 2024-04-25T15:53:20.896 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:20.895+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:20.911 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-25T15:53:20.072007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1074404384,"num_objects":356,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":859,"num_write_kb":2098923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":541072643,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":800,"ondisk_log_size":800,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":221,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2488476,"kb_used_data":2238984,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":752486244,"statfs":{"total":773094113280,"available":770545913856,"internally_reserved":0,"allocated":2292719616,"data_stored":2288531084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":394266187,"num_keys_recovered":4,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.077555"},"pg_stats":[{"pgid":"3.3f","version":"71'6","reported_seq":133,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.070342+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:53:17.070342+0000","last_peered":"2024-04-25T15:53:17.070342+0000","last_clean":"2024-04-25T15:53:17.070342+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:53:17.070342+0000","last_undegraded":"2024-04-25T15:53:17.070342+0000","last_fullsized":"2024-04-25T15:53:17.070342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"71'6","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083520+0000","last_change":"2024-04-25T15:53:13.069367+0000","last_active":"2024-04-25T15:53:17.083520+0000","last_peered":"2024-04-25T15:53:17.083520+0000","last_clean":"2024-04-25T15:53:17.083520+0000","last_became_active":"2024-04-25T15:53:12.990829+0000","last_became_peered":"2024-04-25T15:53:12.990829+0000","last_unstale":"2024-04-25T15:53:17.083520+0000","last_undegraded":"2024-04-25T15:53:17.083520+0000","last_fullsized":"2024-04-25T15:53:17.083520+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:46.276338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"70'6","reported_seq":90,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083937+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:53:17.083937+0000","last_peered":"2024-04-25T15:53:17.083937+0000","last_clean":"2024-04-25T15:53:17.083937+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:53:17.083937+0000","last_undegraded":"2024-04-25T15:53:17.083937+0000","last_fullsized":"2024-04-25T15:53:17.083937+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"70'6","reported_seq":135,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083035+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:53:17.083035+0000","last_peered":"2024-04-25T15:53:17.083035+0000","last_clean":"2024-04-25T15:53:17.083035+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:53:17.083035+0000","last_undegraded":"2024-04-25T15:53:17.083035+0000","last_fullsized":"2024-04-25T15:53:17.083035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"71'10","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066209+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:53:17.066209+0000","last_peered":"2024-04-25T15:53:17.066209+0000","last_clean":"2024-04-25T15:53:17.066209+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:53:17.066209+0000","last_undegraded":"2024-04-25T15:53:17.066209+0000","last_fullsized":"2024-04-25T15:53:17.066209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"70'4","reported_seq":133,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082252+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:53:17.082252+0000","last_peered":"2024-04-25T15:53:17.082252+0000","last_clean":"2024-04-25T15:53:17.082252+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:53:17.082252+0000","last_undegraded":"2024-04-25T15:53:17.082252+0000","last_fullsized":"2024-04-25T15:53:17.082252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"71'14","reported_seq":141,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.067233+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:53:17.067233+0000","last_peered":"2024-04-25T15:53:17.067233+0000","last_clean":"2024-04-25T15:53:17.067233+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:53:17.067233+0000","last_undegraded":"2024-04-25T15:53:17.067233+0000","last_fullsized":"2024-04-25T15:53:17.067233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"70'2","reported_seq":131,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083932+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:53:17.083932+0000","last_peered":"2024-04-25T15:53:17.083932+0000","last_clean":"2024-04-25T15:53:17.083932+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:53:17.083932+0000","last_undegraded":"2024-04-25T15:53:17.083932+0000","last_fullsized":"2024-04-25T15:53:17.083932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"70'8","reported_seq":129,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083666+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:53:17.083666+0000","last_peered":"2024-04-25T15:53:17.083666+0000","last_clean":"2024-04-25T15:53:17.083666+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:53:17.083666+0000","last_undegraded":"2024-04-25T15:53:17.083666+0000","last_fullsized":"2024-04-25T15:53:17.083666+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"69'10","reported_seq":139,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066341+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:53:17.066341+0000","last_peered":"2024-04-25T15:53:17.066341+0000","last_clean":"2024-04-25T15:53:17.066341+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:53:17.066341+0000","last_undegraded":"2024-04-25T15:53:17.066341+0000","last_fullsized":"2024-04-25T15:53:17.066341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"70'4","reported_seq":133,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.063501+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:53:17.063501+0000","last_peered":"2024-04-25T15:53:17.063501+0000","last_clean":"2024-04-25T15:53:17.063501+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:53:17.063501+0000","last_undegraded":"2024-04-25T15:53:17.063501+0000","last_fullsized":"2024-04-25T15:53:17.063501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"70'14","reported_seq":28,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085125+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:53:17.085125+0000","last_peered":"2024-04-25T15:53:17.085125+0000","last_clean":"2024-04-25T15:53:17.085125+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:53:17.085125+0000","last_undegraded":"2024-04-25T15:53:17.085125+0000","last_fullsized":"2024-04-25T15:53:17.085125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"71'4","reported_seq":133,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083144+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:53:17.083144+0000","last_peered":"2024-04-25T15:53:17.083144+0000","last_clean":"2024-04-25T15:53:17.083144+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:53:17.083144+0000","last_undegraded":"2024-04-25T15:53:17.083144+0000","last_fullsized":"2024-04-25T15:53:17.083144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"71'12","reported_seq":142,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081998+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:53:17.081998+0000","last_peered":"2024-04-25T15:53:17.081998+0000","last_clean":"2024-04-25T15:53:17.081998+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:53:17.081998+0000","last_undegraded":"2024-04-25T15:53:17.081998+0000","last_fullsized":"2024-04-25T15:53:17.081998+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"70'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.150784+0000","last_change":"2024-04-25T15:53:17.148839+0000","last_active":"2024-04-25T15:53:17.150784+0000","last_peered":"2024-04-25T15:53:17.150784+0000","last_clean":"2024-04-25T15:53:17.150784+0000","last_became_active":"2024-04-25T15:53:16.064416+0000","last_became_peered":"2024-04-25T15:53:16.064416+0000","last_unstale":"2024-04-25T15:53:17.150784+0000","last_undegraded":"2024-04-25T15:53:17.150784+0000","last_fullsized":"2024-04-25T15:53:17.150784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:58.798689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"71'12","reported_seq":141,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082017+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:53:17.082017+0000","last_peered":"2024-04-25T15:53:17.082017+0000","last_clean":"2024-04-25T15:53:17.082017+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:53:17.082017+0000","last_undegraded":"2024-04-25T15:53:17.082017+0000","last_fullsized":"2024-04-25T15:53:17.082017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"71'8","reported_seq":70,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085370+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:53:17.085370+0000","last_peered":"2024-04-25T15:53:17.085370+0000","last_clean":"2024-04-25T15:53:17.085370+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:53:17.085370+0000","last_undegraded":"2024-04-25T15:53:17.085370+0000","last_fullsized":"2024-04-25T15:53:17.085370+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"69'4","reported_seq":132,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083654+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:53:17.083654+0000","last_peered":"2024-04-25T15:53:17.083654+0000","last_clean":"2024-04-25T15:53:17.083654+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:53:17.083654+0000","last_undegraded":"2024-04-25T15:53:17.083654+0000","last_fullsized":"2024-04-25T15:53:17.083654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"71'4","reported_seq":131,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.068621+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:53:17.068621+0000","last_peered":"2024-04-25T15:53:17.068621+0000","last_clean":"2024-04-25T15:53:17.068621+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:53:17.068621+0000","last_undegraded":"2024-04-25T15:53:17.068621+0000","last_fullsized":"2024-04-25T15:53:17.068621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"70'4","reported_seq":149,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.234347+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:53:17.234347+0000","last_peered":"2024-04-25T15:53:17.234347+0000","last_clean":"2024-04-25T15:53:17.234347+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:53:17.234347+0000","last_undegraded":"2024-04-25T15:53:17.234347+0000","last_fullsized":"2024-04-25T15:53:17.234347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"71'12","reported_seq":95,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.065885+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:53:17.065885+0000","last_peered":"2024-04-25T15:53:17.065885+0000","last_clean":"2024-04-25T15:53:17.065885+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:53:17.065885+0000","last_undegraded":"2024-04-25T15:53:17.065885+0000","last_fullsized":"2024-04-25T15:53:17.065885+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"70'10","reported_seq":131,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081152+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:53:17.081152+0000","last_peered":"2024-04-25T15:53:17.081152+0000","last_clean":"2024-04-25T15:53:17.081152+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:53:17.081152+0000","last_undegraded":"2024-04-25T15:53:17.081152+0000","last_fullsized":"2024-04-25T15:53:17.081152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"71'4","reported_seq":149,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.287218+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:53:17.287218+0000","last_peered":"2024-04-25T15:53:17.287218+0000","last_clean":"2024-04-25T15:53:17.287218+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:53:17.287218+0000","last_undegraded":"2024-04-25T15:53:17.287218+0000","last_fullsized":"2024-04-25T15:53:17.287218+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"71'8","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082252+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:53:17.082252+0000","last_peered":"2024-04-25T15:53:17.082252+0000","last_clean":"2024-04-25T15:53:17.082252+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:53:17.082252+0000","last_undegraded":"2024-04-25T15:53:17.082252+0000","last_fullsized":"2024-04-25T15:53:17.082252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"71'4","reported_seq":149,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.119870+0000","last_change":"2024-04-25T15:53:17.119870+0000","last_active":"2024-04-25T15:53:17.119870+0000","last_peered":"2024-04-25T15:53:17.119870+0000","last_clean":"2024-04-25T15:53:17.119870+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:53:17.119870+0000","last_undegraded":"2024-04-25T15:53:17.119870+0000","last_fullsized":"2024-04-25T15:53:17.119870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:05:09.853949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"70'18","reported_seq":147,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.065972+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:53:17.065972+0000","last_peered":"2024-04-25T15:53:17.065972+0000","last_clean":"2024-04-25T15:53:17.065972+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:53:17.065972+0000","last_undegraded":"2024-04-25T15:53:17.065972+0000","last_fullsized":"2024-04-25T15:53:17.065972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"70'6","reported_seq":115,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083571+0000","last_change":"2024-04-25T15:53:03.808515+0000","last_active":"2024-04-25T15:53:17.083571+0000","last_peered":"2024-04-25T15:53:17.083571+0000","last_clean":"2024-04-25T15:53:17.083571+0000","last_became_active":"2024-04-25T15:53:03.367253+0000","last_became_peered":"2024-04-25T15:53:03.367253+0000","last_unstale":"2024-04-25T15:53:17.083571+0000","last_undegraded":"2024-04-25T15:53:17.083571+0000","last_fullsized":"2024-04-25T15:53:17.083571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:05.644251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"71'8","reported_seq":172,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.549084+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:53:17.549084+0000","last_peered":"2024-04-25T15:53:17.549084+0000","last_clean":"2024-04-25T15:53:17.549084+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:53:17.549084+0000","last_undegraded":"2024-04-25T15:53:17.549084+0000","last_fullsized":"2024-04-25T15:53:17.549084+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"70'6","reported_seq":91,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083088+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:53:17.083088+0000","last_peered":"2024-04-25T15:53:17.083088+0000","last_clean":"2024-04-25T15:53:17.083088+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:53:17.083088+0000","last_undegraded":"2024-04-25T15:53:17.083088+0000","last_fullsized":"2024-04-25T15:53:17.083088+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081659+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:53:17.081659+0000","last_peered":"2024-04-25T15:53:17.081659+0000","last_clean":"2024-04-25T15:53:17.081659+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:53:17.081659+0000","last_undegraded":"2024-04-25T15:53:17.081659+0000","last_fullsized":"2024-04-25T15:53:17.081659+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"71'10","reported_seq":148,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.080976+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:53:17.080976+0000","last_peered":"2024-04-25T15:53:17.080976+0000","last_clean":"2024-04-25T15:53:17.080976+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:53:17.080976+0000","last_undegraded":"2024-04-25T15:53:17.080976+0000","last_fullsized":"2024-04-25T15:53:17.080976+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"71'14","reported_seq":143,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.062197+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:53:17.062197+0000","last_peered":"2024-04-25T15:53:17.062197+0000","last_clean":"2024-04-25T15:53:17.062197+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:53:17.062197+0000","last_undegraded":"2024-04-25T15:53:17.062197+0000","last_fullsized":"2024-04-25T15:53:17.062197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"70'8","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083630+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:53:17.083630+0000","last_peered":"2024-04-25T15:53:17.083630+0000","last_clean":"2024-04-25T15:53:17.083630+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:53:17.083630+0000","last_undegraded":"2024-04-25T15:53:17.083630+0000","last_fullsized":"2024-04-25T15:53:17.083630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083227+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:53:17.083227+0000","last_peered":"2024-04-25T15:53:17.083227+0000","last_clean":"2024-04-25T15:53:17.083227+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:53:17.083227+0000","last_undegraded":"2024-04-25T15:53:17.083227+0000","last_fullsized":"2024-04-25T15:53:17.083227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"70'6","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.084158+0000","last_change":"2024-04-25T15:53:16.510837+0000","last_active":"2024-04-25T15:53:17.084158+0000","last_peered":"2024-04-25T15:53:17.084158+0000","last_clean":"2024-04-25T15:53:17.084158+0000","last_became_active":"2024-04-25T15:53:16.041557+0000","last_became_peered":"2024-04-25T15:53:16.041557+0000","last_unstale":"2024-04-25T15:53:17.084158+0000","last_undegraded":"2024-04-25T15:53:17.084158+0000","last_fullsized":"2024-04-25T15:53:17.084158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:06.600570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":164,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.062935+0000","last_change":"2024-04-25T15:53:16.102712+0000","last_active":"2024-04-25T15:53:17.062935+0000","last_peered":"2024-04-25T15:53:17.062935+0000","last_clean":"2024-04-25T15:53:17.062935+0000","last_became_active":"2024-04-25T15:53:16.052729+0000","last_became_peered":"2024-04-25T15:53:16.052729+0000","last_unstale":"2024-04-25T15:53:17.062935+0000","last_undegraded":"2024-04-25T15:53:17.062935+0000","last_fullsized":"2024-04-25T15:53:17.062935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:53.161289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":134,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082944+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:53:17.082944+0000","last_peered":"2024-04-25T15:53:17.082944+0000","last_clean":"2024-04-25T15:53:17.082944+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:53:17.082944+0000","last_undegraded":"2024-04-25T15:53:17.082944+0000","last_fullsized":"2024-04-25T15:53:17.082944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"71'6","reported_seq":30,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081280+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:53:17.081280+0000","last_peered":"2024-04-25T15:53:17.081280+0000","last_clean":"2024-04-25T15:53:17.081280+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:53:17.081280+0000","last_undegraded":"2024-04-25T15:53:17.081280+0000","last_fullsized":"2024-04-25T15:53:17.081280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":152,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082331+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:53:17.082331+0000","last_peered":"2024-04-25T15:53:17.082331+0000","last_clean":"2024-04-25T15:53:17.082331+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:53:17.082331+0000","last_undegraded":"2024-04-25T15:53:17.082331+0000","last_fullsized":"2024-04-25T15:53:17.082331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"71'6","reported_seq":125,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082555+0000","last_change":"2024-04-25T15:53:16.206656+0000","last_active":"2024-04-25T15:53:17.082555+0000","last_peered":"2024-04-25T15:53:17.082555+0000","last_clean":"2024-04-25T15:53:17.082555+0000","last_became_active":"2024-04-25T15:53:16.049908+0000","last_became_peered":"2024-04-25T15:53:16.049908+0000","last_unstale":"2024-04-25T15:53:17.082555+0000","last_undegraded":"2024-04-25T15:53:17.082555+0000","last_fullsized":"2024-04-25T15:53:17.082555+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.217418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"22'5","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.065940+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:53:17.065940+0000","last_peered":"2024-04-25T15:53:17.065940+0000","last_clean":"2024-04-25T15:53:17.065940+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:53:17.065940+0000","last_undegraded":"2024-04-25T15:53:17.065940+0000","last_fullsized":"2024-04-25T15:53:17.065940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"71'6","reported_seq":135,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085209+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:53:17.085209+0000","last_peered":"2024-04-25T15:53:17.085209+0000","last_clean":"2024-04-25T15:53:17.085209+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:53:17.085209+0000","last_undegraded":"2024-04-25T15:53:17.085209+0000","last_fullsized":"2024-04-25T15:53:17.085209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":135,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085607+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:53:17.085607+0000","last_peered":"2024-04-25T15:53:17.085607+0000","last_clean":"2024-04-25T15:53:17.085607+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:53:17.085607+0000","last_undegraded":"2024-04-25T15:53:17.085607+0000","last_fullsized":"2024-04-25T15:53:17.085607+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"69'10","reported_seq":95,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085765+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:53:17.085765+0000","last_peered":"2024-04-25T15:53:17.085765+0000","last_clean":"2024-04-25T15:53:17.085765+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:53:17.085765+0000","last_undegraded":"2024-04-25T15:53:17.085765+0000","last_fullsized":"2024-04-25T15:53:17.085765+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":87,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.084701+0000","last_change":"2024-04-25T15:53:03.435546+0000","last_active":"2024-04-25T15:53:17.084701+0000","last_peered":"2024-04-25T15:53:17.084701+0000","last_clean":"2024-04-25T15:53:17.084701+0000","last_became_active":"2024-04-25T15:53:03.387235+0000","last_became_peered":"2024-04-25T15:53:03.387235+0000","last_unstale":"2024-04-25T15:53:17.084701+0000","last_undegraded":"2024-04-25T15:53:17.084701+0000","last_fullsized":"2024-04-25T15:53:17.084701+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:30.444213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1344,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"71'10","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085452+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:53:17.085452+0000","last_peered":"2024-04-25T15:53:17.085452+0000","last_clean":"2024-04-25T15:53:17.085452+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:53:17.085452+0000","last_undegraded":"2024-04-25T15:53:17.085452+0000","last_fullsized":"2024-04-25T15:53:17.085452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"71'36","reported_seq":167,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.063416+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:53:17.063416+0000","last_peered":"2024-04-25T15:53:17.063416+0000","last_clean":"2024-04-25T15:53:17.063416+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:53:17.063416+0000","last_undegraded":"2024-04-25T15:53:17.063416+0000","last_fullsized":"2024-04-25T15:53:17.063416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64447,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"69'6","reported_seq":133,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.065523+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:53:17.065523+0000","last_peered":"2024-04-25T15:53:17.065523+0000","last_clean":"2024-04-25T15:53:17.065523+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:53:17.065523+0000","last_undegraded":"2024-04-25T15:53:17.065523+0000","last_fullsized":"2024-04-25T15:53:17.065523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.084552+0000","last_change":"2024-04-25T15:53:16.094342+0000","last_active":"2024-04-25T15:53:17.084552+0000","last_peered":"2024-04-25T15:53:17.084552+0000","last_clean":"2024-04-25T15:53:17.084552+0000","last_became_active":"2024-04-25T15:53:16.041484+0000","last_became_peered":"2024-04-25T15:53:16.041484+0000","last_unstale":"2024-04-25T15:53:17.084552+0000","last_undegraded":"2024-04-25T15:53:17.084552+0000","last_fullsized":"2024-04-25T15:53:17.084552+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:42.531801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"70'13","reported_seq":96,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.068473+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:53:17.068473+0000","last_peered":"2024-04-25T15:53:17.068473+0000","last_clean":"2024-04-25T15:53:17.068473+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:53:17.068473+0000","last_undegraded":"2024-04-25T15:53:17.068473+0000","last_fullsized":"2024-04-25T15:53:17.068473+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082332+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:53:17.082332+0000","last_peered":"2024-04-25T15:53:17.082332+0000","last_clean":"2024-04-25T15:53:17.082332+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:53:17.082332+0000","last_undegraded":"2024-04-25T15:53:17.082332+0000","last_fullsized":"2024-04-25T15:53:17.082332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'8","reported_seq":155,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066580+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:53:17.066580+0000","last_peered":"2024-04-25T15:53:17.066580+0000","last_clean":"2024-04-25T15:53:17.066580+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:53:17.066580+0000","last_undegraded":"2024-04-25T15:53:17.066580+0000","last_fullsized":"2024-04-25T15:53:17.066580+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":176,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.063197+0000","last_change":"2024-04-25T15:53:16.115440+0000","last_active":"2024-04-25T15:53:17.063197+0000","last_peered":"2024-04-25T15:53:17.063197+0000","last_clean":"2024-04-25T15:53:17.063197+0000","last_became_active":"2024-04-25T15:53:16.053229+0000","last_became_peered":"2024-04-25T15:53:16.053229+0000","last_unstale":"2024-04-25T15:53:17.063197+0000","last_undegraded":"2024-04-25T15:53:17.063197+0000","last_fullsized":"2024-04-25T15:53:17.063197+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:13.860370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3087,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.062378+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:53:17.062378+0000","last_peered":"2024-04-25T15:53:17.062378+0000","last_clean":"2024-04-25T15:53:17.062378+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:53:17.062378+0000","last_undegraded":"2024-04-25T15:53:17.062378+0000","last_fullsized":"2024-04-25T15:53:17.062378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":216,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.080898+0000","last_change":"2024-04-25T15:53:03.421455+0000","last_active":"2024-04-25T15:53:17.080898+0000","last_peered":"2024-04-25T15:53:17.080898+0000","last_clean":"2024-04-25T15:53:17.080898+0000","last_became_active":"2024-04-25T15:53:03.385909+0000","last_became_peered":"2024-04-25T15:53:03.385909+0000","last_unstale":"2024-04-25T15:53:17.080898+0000","last_undegraded":"2024-04-25T15:53:17.080898+0000","last_fullsized":"2024-04-25T15:53:17.080898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:54.677309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"70'16","reported_seq":171,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.460515+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:53:17.460515+0000","last_peered":"2024-04-25T15:53:17.460515+0000","last_clean":"2024-04-25T15:53:17.460515+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:53:17.460515+0000","last_undegraded":"2024-04-25T15:53:17.460515+0000","last_fullsized":"2024-04-25T15:53:17.460515+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085989+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:53:17.085989+0000","last_peered":"2024-04-25T15:53:17.085989+0000","last_clean":"2024-04-25T15:53:17.085989+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:53:17.085989+0000","last_undegraded":"2024-04-25T15:53:17.085989+0000","last_fullsized":"2024-04-25T15:53:17.085989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'4","reported_seq":147,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.178176+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:53:17.178176+0000","last_peered":"2024-04-25T15:53:17.178176+0000","last_clean":"2024-04-25T15:53:17.178176+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:53:17.178176+0000","last_undegraded":"2024-04-25T15:53:17.178176+0000","last_fullsized":"2024-04-25T15:53:17.178176+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":135,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083783+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:53:17.083783+0000","last_peered":"2024-04-25T15:53:17.083783+0000","last_clean":"2024-04-25T15:53:17.083783+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:53:17.083783+0000","last_undegraded":"2024-04-25T15:53:17.083783+0000","last_fullsized":"2024-04-25T15:53:17.083783+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"70'8","reported_seq":155,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.070438+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:53:17.070438+0000","last_peered":"2024-04-25T15:53:17.070438+0000","last_clean":"2024-04-25T15:53:17.070438+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:53:17.070438+0000","last_undegraded":"2024-04-25T15:53:17.070438+0000","last_fullsized":"2024-04-25T15:53:17.070438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085290+0000","last_change":"2024-04-25T15:53:16.098410+0000","last_active":"2024-04-25T15:53:17.085290+0000","last_peered":"2024-04-25T15:53:17.085290+0000","last_clean":"2024-04-25T15:53:17.085290+0000","last_became_active":"2024-04-25T15:53:16.053979+0000","last_became_peered":"2024-04-25T15:53:16.053979+0000","last_unstale":"2024-04-25T15:53:17.085290+0000","last_undegraded":"2024-04-25T15:53:17.085290+0000","last_fullsized":"2024-04-25T15:53:17.085290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:24.859180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"70'8","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083401+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:53:17.083401+0000","last_peered":"2024-04-25T15:53:17.083401+0000","last_clean":"2024-04-25T15:53:17.083401+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:53:17.083401+0000","last_undegraded":"2024-04-25T15:53:17.083401+0000","last_fullsized":"2024-04-25T15:53:17.083401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081545+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:53:17.081545+0000","last_peered":"2024-04-25T15:53:17.081545+0000","last_clean":"2024-04-25T15:53:17.081545+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:53:17.081545+0000","last_undegraded":"2024-04-25T15:53:17.081545+0000","last_fullsized":"2024-04-25T15:53:17.081545+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"69'4","reported_seq":131,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.068282+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:53:17.068282+0000","last_peered":"2024-04-25T15:53:17.068282+0000","last_clean":"2024-04-25T15:53:17.068282+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:53:17.068282+0000","last_undegraded":"2024-04-25T15:53:17.068282+0000","last_fullsized":"2024-04-25T15:53:17.068282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":225,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066534+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:53:17.066534+0000","last_peered":"2024-04-25T15:53:17.066534+0000","last_clean":"2024-04-25T15:53:17.066534+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:53:17.066534+0000","last_undegraded":"2024-04-25T15:53:17.066534+0000","last_fullsized":"2024-04-25T15:53:17.066534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082398+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:53:17.082398+0000","last_peered":"2024-04-25T15:53:17.082398+0000","last_clean":"2024-04-25T15:53:17.082398+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:53:17.082398+0000","last_undegraded":"2024-04-25T15:53:17.082398+0000","last_fullsized":"2024-04-25T15:53:17.082398+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'10","reported_seq":139,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083160+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:53:17.083160+0000","last_peered":"2024-04-25T15:53:17.083160+0000","last_clean":"2024-04-25T15:53:17.083160+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:53:17.083160+0000","last_undegraded":"2024-04-25T15:53:17.083160+0000","last_fullsized":"2024-04-25T15:53:17.083160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":160,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066209+0000","last_change":"2024-04-25T15:53:03.424512+0000","last_active":"2024-04-25T15:53:17.066209+0000","last_peered":"2024-04-25T15:53:17.066209+0000","last_clean":"2024-04-25T15:53:17.066209+0000","last_became_active":"2024-04-25T15:53:03.385988+0000","last_became_peered":"2024-04-25T15:53:03.385988+0000","last_unstale":"2024-04-25T15:53:17.066209+0000","last_undegraded":"2024-04-25T15:53:17.066209+0000","last_fullsized":"2024-04-25T15:53:17.066209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:55.224996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"70'10","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085072+0000","last_change":"2024-04-25T15:53:16.680938+0000","last_active":"2024-04-25T15:53:17.085072+0000","last_peered":"2024-04-25T15:53:17.085072+0000","last_clean":"2024-04-25T15:53:17.085072+0000","last_became_active":"2024-04-25T15:53:16.041368+0000","last_became_peered":"2024-04-25T15:53:16.041368+0000","last_unstale":"2024-04-25T15:53:17.085072+0000","last_undegraded":"2024-04-25T15:53:17.085072+0000","last_fullsized":"2024-04-25T15:53:17.085072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:49.681769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081025+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:53:17.081025+0000","last_peered":"2024-04-25T15:53:17.081025+0000","last_clean":"2024-04-25T15:53:17.081025+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:53:17.081025+0000","last_undegraded":"2024-04-25T15:53:17.081025+0000","last_fullsized":"2024-04-25T15:53:17.081025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"70'8","reported_seq":116,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081098+0000","last_change":"2024-04-25T15:53:03.530328+0000","last_active":"2024-04-25T15:53:17.081098+0000","last_peered":"2024-04-25T15:53:17.081098+0000","last_clean":"2024-04-25T15:53:17.081098+0000","last_became_active":"2024-04-25T15:53:03.382259+0000","last_became_peered":"2024-04-25T15:53:03.382259+0000","last_unstale":"2024-04-25T15:53:17.081098+0000","last_undegraded":"2024-04-25T15:53:17.081098+0000","last_fullsized":"2024-04-25T15:53:17.081098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:43.788748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":158,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082593+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:53:17.082593+0000","last_peered":"2024-04-25T15:53:17.082593+0000","last_clean":"2024-04-25T15:53:17.082593+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:53:17.082593+0000","last_undegraded":"2024-04-25T15:53:17.082593+0000","last_fullsized":"2024-04-25T15:53:17.082593+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"69'6","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.084844+0000","last_change":"2024-04-25T15:53:16.788986+0000","last_active":"2024-04-25T15:53:17.084844+0000","last_peered":"2024-04-25T15:53:17.084844+0000","last_clean":"2024-04-25T15:53:17.084844+0000","last_became_active":"2024-04-25T15:53:16.046426+0000","last_became_peered":"2024-04-25T15:53:16.046426+0000","last_unstale":"2024-04-25T15:53:17.084844+0000","last_undegraded":"2024-04-25T15:53:17.084844+0000","last_fullsized":"2024-04-25T15:53:17.084844+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:33.326656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":27,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066336+0000","last_change":"2024-04-25T15:53:03.381919+0000","last_active":"2024-04-25T15:53:17.066336+0000","last_peered":"2024-04-25T15:53:17.066336+0000","last_clean":"2024-04-25T15:53:17.066336+0000","last_became_active":"2024-04-25T15:53:03.381463+0000","last_became_peered":"2024-04-25T15:53:03.381463+0000","last_unstale":"2024-04-25T15:53:17.066336+0000","last_undegraded":"2024-04-25T15:53:17.066336+0000","last_fullsized":"2024-04-25T15:53:17.066336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:09.993891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"70'8","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.084360+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:53:17.084360+0000","last_peered":"2024-04-25T15:53:17.084360+0000","last_clean":"2024-04-25T15:53:17.084360+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:53:17.084360+0000","last_undegraded":"2024-04-25T15:53:17.084360+0000","last_fullsized":"2024-04-25T15:53:17.084360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":133,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.068834+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:53:17.068834+0000","last_peered":"2024-04-25T15:53:17.068834+0000","last_clean":"2024-04-25T15:53:17.068834+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:53:17.068834+0000","last_undegraded":"2024-04-25T15:53:17.068834+0000","last_fullsized":"2024-04-25T15:53:17.068834+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"70'4","reported_seq":98,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081387+0000","last_change":"2024-04-25T15:53:03.480191+0000","last_active":"2024-04-25T15:53:17.081387+0000","last_peered":"2024-04-25T15:53:17.081387+0000","last_clean":"2024-04-25T15:53:17.081387+0000","last_became_active":"2024-04-25T15:53:03.380081+0000","last_became_peered":"2024-04-25T15:53:03.380081+0000","last_unstale":"2024-04-25T15:53:17.081387+0000","last_undegraded":"2024-04-25T15:53:17.081387+0000","last_fullsized":"2024-04-25T15:53:17.081387+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:50.114177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"67'12","reported_seq":136,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066328+0000","last_change":"2024-04-25T15:53:16.083270+0000","last_active":"2024-04-25T15:53:17.066328+0000","last_peered":"2024-04-25T15:53:17.066328+0000","last_clean":"2024-04-25T15:53:17.066328+0000","last_became_active":"2024-04-25T15:53:16.045200+0000","last_became_peered":"2024-04-25T15:53:16.045200+0000","last_unstale":"2024-04-25T15:53:17.066328+0000","last_undegraded":"2024-04-25T15:53:17.066328+0000","last_fullsized":"2024-04-25T15:53:17.066328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:38.382894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"71'14","reported_seq":143,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083362+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:53:17.083362+0000","last_peered":"2024-04-25T15:53:17.083362+0000","last_clean":"2024-04-25T15:53:17.083362+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:53:17.083362+0000","last_undegraded":"2024-04-25T15:53:17.083362+0000","last_fullsized":"2024-04-25T15:53:17.083362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":138,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066136+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:53:17.066136+0000","last_peered":"2024-04-25T15:53:17.066136+0000","last_clean":"2024-04-25T15:53:17.066136+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:53:17.066136+0000","last_undegraded":"2024-04-25T15:53:17.066136+0000","last_fullsized":"2024-04-25T15:53:17.066136+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'4","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.086397+0000","last_change":"2024-04-25T15:53:16.863394+0000","last_active":"2024-04-25T15:53:17.086397+0000","last_peered":"2024-04-25T15:53:17.086397+0000","last_clean":"2024-04-25T15:53:17.086397+0000","last_became_active":"2024-04-25T15:53:16.048669+0000","last_became_peered":"2024-04-25T15:53:16.048669+0000","last_unstale":"2024-04-25T15:53:17.086397+0000","last_undegraded":"2024-04-25T15:53:17.086397+0000","last_fullsized":"2024-04-25T15:53:17.086397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:53.132724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":131,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.065915+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:53:17.065915+0000","last_peered":"2024-04-25T15:53:17.065915+0000","last_clean":"2024-04-25T15:53:17.065915+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:53:17.065915+0000","last_undegraded":"2024-04-25T15:53:17.065915+0000","last_fullsized":"2024-04-25T15:53:17.065915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"71'12","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083947+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:53:17.083947+0000","last_peered":"2024-04-25T15:53:17.083947+0000","last_clean":"2024-04-25T15:53:17.083947+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:53:17.083947+0000","last_undegraded":"2024-04-25T15:53:17.083947+0000","last_fullsized":"2024-04-25T15:53:17.083947+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":103,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082290+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:53:17.082290+0000","last_peered":"2024-04-25T15:53:17.082290+0000","last_clean":"2024-04-25T15:53:17.082290+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:53:17.082290+0000","last_undegraded":"2024-04-25T15:53:17.082290+0000","last_fullsized":"2024-04-25T15:53:17.082290+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"71'14","reported_seq":174,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.084189+0000","last_change":"2024-04-25T15:53:03.761415+0000","last_active":"2024-04-25T15:53:17.084189+0000","last_peered":"2024-04-25T15:53:17.084189+0000","last_clean":"2024-04-25T15:53:17.084189+0000","last_became_active":"2024-04-25T15:53:03.384666+0000","last_became_peered":"2024-04-25T15:53:03.384666+0000","last_unstale":"2024-04-25T15:53:17.084189+0000","last_undegraded":"2024-04-25T15:53:17.084189+0000","last_fullsized":"2024-04-25T15:53:17.084189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:14.524576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":77,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.082858+0000","last_change":"2024-04-25T15:53:03.409633+0000","last_active":"2024-04-25T15:53:17.082858+0000","last_peered":"2024-04-25T15:53:17.082858+0000","last_clean":"2024-04-25T15:53:17.082858+0000","last_became_active":"2024-04-25T15:53:03.384646+0000","last_became_peered":"2024-04-25T15:53:03.384646+0000","last_unstale":"2024-04-25T15:53:17.082858+0000","last_undegraded":"2024-04-25T15:53:17.082858+0000","last_fullsized":"2024-04-25T15:53:17.082858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:43.442571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"69'8","reported_seq":129,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.081310+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:53:17.081310+0000","last_peered":"2024-04-25T15:53:17.081310+0000","last_clean":"2024-04-25T15:53:17.081310+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:53:17.081310+0000","last_undegraded":"2024-04-25T15:53:17.081310+0000","last_fullsized":"2024-04-25T15:53:17.081310+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":155,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066985+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:53:17.066985+0000","last_peered":"2024-04-25T15:53:17.066985+0000","last_clean":"2024-04-25T15:53:17.066985+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:53:17.066985+0000","last_undegraded":"2024-04-25T15:53:17.066985+0000","last_fullsized":"2024-04-25T15:53:17.066985+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"69'2","reported_seq":157,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.062581+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:53:17.062581+0000","last_peered":"2024-04-25T15:53:17.062581+0000","last_clean":"2024-04-25T15:53:17.062581+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:53:17.062581+0000","last_undegraded":"2024-04-25T15:53:17.062581+0000","last_fullsized":"2024-04-25T15:53:17.062581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"71'14","reported_seq":169,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083994+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:53:17.083994+0000","last_peered":"2024-04-25T15:53:17.083994+0000","last_clean":"2024-04-25T15:53:17.083994+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:53:17.083994+0000","last_undegraded":"2024-04-25T15:53:17.083994+0000","last_fullsized":"2024-04-25T15:53:17.083994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":134,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.066852+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:53:17.066852+0000","last_peered":"2024-04-25T15:53:17.066852+0000","last_clean":"2024-04-25T15:53:17.066852+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:53:17.066852+0000","last_undegraded":"2024-04-25T15:53:17.066852+0000","last_fullsized":"2024-04-25T15:53:17.066852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"70'12","reported_seq":184,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083112+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:53:17.083112+0000","last_peered":"2024-04-25T15:53:17.083112+0000","last_clean":"2024-04-25T15:53:17.083112+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:53:17.083112+0000","last_undegraded":"2024-04-25T15:53:17.083112+0000","last_fullsized":"2024-04-25T15:53:17.083112+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":141,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.068483+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:53:17.068483+0000","last_peered":"2024-04-25T15:53:17.068483+0000","last_clean":"2024-04-25T15:53:17.068483+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:53:17.068483+0000","last_undegraded":"2024-04-25T15:53:17.068483+0000","last_fullsized":"2024-04-25T15:53:17.068483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"71'6","reported_seq":91,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.085699+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:53:17.085699+0000","last_peered":"2024-04-25T15:53:17.085699+0000","last_clean":"2024-04-25T15:53:17.085699+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:53:17.085699+0000","last_undegraded":"2024-04-25T15:53:17.085699+0000","last_fullsized":"2024-04-25T15:53:17.085699+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":24576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":133,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083568+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:53:17.083568+0000","last_peered":"2024-04-25T15:53:17.083568+0000","last_clean":"2024-04-25T15:53:17.083568+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:53:17.083568+0000","last_undegraded":"2024-04-25T15:53:17.083568+0000","last_fullsized":"2024-04-25T15:53:17.083568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'8","reported_seq":137,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.083549+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:53:17.083549+0000","last_peered":"2024-04-25T15:53:17.083549+0000","last_clean":"2024-04-25T15:53:17.083549+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:53:17.083549+0000","last_undegraded":"2024-04-25T15:53:17.083549+0000","last_fullsized":"2024-04-25T15:53:17.083549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":107,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-25T15:53:17.070600+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:53:17.070600+0000","last_peered":"2024-04-25T15:53:17.070600+0000","last_clean":"2024-04-25T15:53:17.070600+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:53:17.070600+0000","last_undegraded":"2024-04-25T15:53:17.070600+0000","last_fullsized":"2024-04-25T15:53:17.070600+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1073741824,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":2097152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":541065216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2147483648,"data_stored":2147483648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":513,"ondisk_log_size":513,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72192,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":246,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":7427,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":144384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454596,"kb_used_data":423408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93917244,"statfs":{"total":96636764160,"available":96171257856,"internally_reserved":0,"allocated":433569792,"data_stored":433032679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388520,"kb_used_data":357336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93983320,"statfs":{"total":96636764160,"available":96238919680,"internally_reserved":0,"allocated":365912064,"data_stored":365395980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363904,"kb_used_data":332720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94007936,"statfs":{"total":96636764160,"available":96264126464,"internally_reserved":0,"allocated":340705280,"data_stored":340168851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320876,"kb_used_data":289688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94050964,"statfs":{"total":96636764160,"available":96308187136,"internally_reserved":0,"allocated":296640512,"data_stored":296126057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":282032,"kb_used_data":250848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94089808,"statfs":{"total":96636764160,"available":96347963392,"internally_reserved":0,"allocated":256868352,"data_stored":256344180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228724,"kb_used_data":197536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94143116,"statfs":{"total":96636764160,"available":96402550784,"internally_reserved":0,"allocated":202276864,"data_stored":201754349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85340,"kb_used_data":54152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286500,"statfs":{"total":96636764160,"available":96549376000,"internally_reserved":0,"allocated":55451648,"data_stored":54952484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364484,"kb_used_data":333296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94007356,"statfs":{"total":96636764160,"available":96263532544,"internally_reserved":0,"allocated":341295104,"data_stored":340756504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":65782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":314572800,"data_stored":314572800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54525952,"data_stored":54525952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":201326592,"data_stored":201326592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":255852544,"data_stored":255852544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":364904448,"data_stored":364904448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":339738624,"data_stored":339738624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":230686720,"data_stored":230686720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":385875968,"data_stored":385875968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:53:20.914 INFO:tasks.thrashosds.thrasher:Setting 2.10 to [7, 3, 4, 6] 2024-04-25T15:53:20.914 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.10', '7', '3', '4', '6'] 2024-04-25T15:53:20.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.10 7 3 4 6 2024-04-25T15:53:20.996 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:20.995+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:21.007 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:21.007 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:21.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:21.096 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:21.094+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:21.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.102+0000 7f686c89e640 1 Processor -- start 2024-04-25T15:53:21.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.102+0000 7f686c89e640 1 -- start start 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686c89e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f68680d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686c89e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6868162e00 0x7f6868165220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686c89e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68680c73c0 con 0x7f68680ca370 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686c89e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f68680cb6e0 con 0x7f6868162e00 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686c89e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f68680c9680 con 0x7f68680cff30 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f68680ca370 0x7f686815aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55346/0 (socket says 172.21.15.67:55346) 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686659c640 1 -- 172.21.15.67:0/2368236698 learned_addr learned my addr 172.21.15.67:0/2368236698 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f6866d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6868162e00 0x7f6868165220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686559a640 1 -- 172.21.15.67:0/2368236698 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682432312 0 0) 0x7f68680c73c0 con 0x7f68680ca370 2024-04-25T15:53:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f6865d9b640 1 --2- 172.21.15.67:0/2368236698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f68680d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:21.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f686559a640 1 -- 172.21.15.67:0/2368236698 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6854003530 con 0x7f68680ca370 2024-04-25T15:53:21.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f6866d9d640 1 -- 172.21.15.67:0/2368236698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 msgr2=0x7f68680d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:21.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f6866d9d640 1 --2- 172.21.15.67:0/2368236698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f68680d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:21.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f6866d9d640 1 -- 172.21.15.67:0/2368236698 >> v1:172.21.15.67:6789/0 conn(0x7f68680ca370 legacy=0x7f686815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:21.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.106+0000 7f6866d9d640 1 -- 172.21.15.67:0/2368236698 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68680c9960 con 0x7f6868162e00 2024-04-25T15:53:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f6866d9d640 1 --2- 172.21.15.67:0/2368236698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868165220 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6858009d90 tx=0x7f685801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5d99e276cdc4cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686559a640 1 -- 172.21.15.67:0/2368236698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580b4070 con 0x7f6868162e00 2024-04-25T15:53:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686559a640 1 -- 172.21.15.67:0/2368236698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68580af600 con 0x7f6868162e00 2024-04-25T15:53:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686559a640 1 -- 172.21.15.67:0/2368236698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580af9d0 con 0x7f6868162e00 2024-04-25T15:53:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 -- 172.21.15.67:0/2368236698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 msgr2=0x7f6868165220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 --2- 172.21.15.67:0/2368236698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868165220 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6858009d90 tx=0x7f685801f150 comp rx=0 tx=0).stop 2024-04-25T15:53:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 -- 172.21.15.67:0/2368236698 shutdown_connections 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 --2- 172.21.15.67:0/2368236698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868165220 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 --2- 172.21.15.67:0/2368236698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f68680d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 -- 172.21.15.67:0/2368236698 >> 172.21.15.67:0/2368236698 conn(0x7f6868068990 msgr2=0x7f6868068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 -- 172.21.15.67:0/2368236698 shutdown_connections 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.110+0000 7f686c89e640 1 -- 172.21.15.67:0/2368236698 wait complete. 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 Processor -- start 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 -- start start 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68680ca370 0x7f686814e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f6868148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6868072770 con 0x7f6868162e00 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f686815a680 con 0x7f68680ca370 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f68680ca910 con 0x7f68680cff30 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868148650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42572/0 (socket says 172.21.15.67:42572) 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 -- 172.21.15.67:0/3538875015 learned_addr learned my addr 172.21.15.67:0/3538875015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6865d9b640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f6868148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686659c640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68680ca370 0x7f686814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 -- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 msgr2=0x7f6868148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f6868148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 -- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68680ca370 msgr2=0x7f686814e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68680ca370 0x7f686814e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 -- 172.21.15.67:0/3538875015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68681f17c0 con 0x7f6868162e00 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f6866d9d640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868148650 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f68580b2040 tx=0x7f685801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f685effd640 1 -- 172.21.15.67:0/3538875015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580b4040 con 0x7f6868162e00 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f685effd640 1 -- 172.21.15.67:0/3538875015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68580c1070 con 0x7f6868162e00 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f685effd640 1 -- 172.21.15.67:0/3538875015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858009450 con 0x7f6868162e00 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68681f0a00 con 0x7f6868162e00 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.114+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68681f0fc0 con 0x7f6868162e00 2024-04-25T15:53:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.118+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68681557e0 con 0x7f6868162e00 2024-04-25T15:53:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.118+0000 7f685effd640 1 -- 172.21.15.67:0/3538875015 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f68580ba6b0 con 0x7f6868162e00 2024-04-25T15:53:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.118+0000 7f685effd640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f684806f400 0x7f6848071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.118+0000 7f685effd640 1 -- 172.21.15.67:0/3538875015 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5915+0+0 (secure 0 0 0) 0x7f685812dc70 con 0x7f6868162e00 2024-04-25T15:53:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.118+0000 7f686659c640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f684806f400 0x7f6848071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.118+0000 7f686659c640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f684806f400 0x7f6848071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6860005890 tx=0x7f6860009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:21.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.122+0000 7f685effd640 1 -- 172.21.15.67:0/3538875015 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6858018cb0 con 0x7f6868162e00 2024-04-25T15:53:21.171 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:21.172 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:21.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:21.197 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:21.195+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:21.298 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:21.295+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:21.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:21.302+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [7, 3, 4, 6]} v 0) -- 0x7f686806a480 con 0x7f6868162e00 2024-04-25T15:53:21.378 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:21.378 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:21.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:21.398 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:21.395+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:21.499 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:21.498+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:21.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:21.609 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:21.610+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:21.701 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:21.698+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:21.765 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:21.765 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:21.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:21.829 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:21.830+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:21.902 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:21.899+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:21.985 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:21.985 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:22.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:22.004 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:22.003+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:22.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.078+0000 7f685effd640 1 -- 172.21.15.67:0/3538875015 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "2.10", "id": [7, 3, 4, 6]}]=0 set 2.10 pg_upmap_items mapping to [7->3,4->6] v80) ==== 148+0+0 (secure 0 0 0) 0x7f68580fab20 con 0x7f6868162e00 2024-04-25T15:53:22.080 INFO:teuthology.orchestra.run.smithi067.stderr:set 2.10 pg_upmap_items mapping to [7->3,4->6] 2024-04-25T15:53:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f684806f400 msgr2=0x7f6848071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f684806f400 0x7f6848071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6860005890 tx=0x7f6860009040 comp rx=0 tx=0).stop 2024-04-25T15:53:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 msgr2=0x7f6868148650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868148650 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f68580b2040 tx=0x7f685801bef0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 shutdown_connections 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f684806f400 0x7f6848071910 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6868162e00 0x7f6868148650 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68680cff30 0x7f6868148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 --2- 172.21.15.67:0/3538875015 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68680ca370 0x7f686814e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 >> 172.21.15.67:0/3538875015 conn(0x7f6868068990 msgr2=0x7f68680cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 shutdown_connections 2024-04-25T15:53:22.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.082+0000 7f686c89e640 1 -- 172.21.15.67:0/3538875015 wait complete. 2024-04-25T15:53:22.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:53:22.104 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:22.103+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:22.158 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:22.158 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:22.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:22.227 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:22.227+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.242+0000 7fee3cab7640 1 Processor -- start 2024-04-25T15:53:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.242+0000 7fee3cab7640 1 -- start start 2024-04-25T15:53:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3cab7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee380cc6b0 0x7fee380cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3cab7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fee38160430 0x7fee38160840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3cab7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee380ccc80 con 0x7fee3806d650 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3cab7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fee380c9c20 con 0x7fee380cc6b0 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3cab7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fee380c9e70 con 0x7fee38160430 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee380cc6b0 0x7fee380cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee36d9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee3806d650 0x7fee38068590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55354/0 (socket says 172.21.15.67:55354) 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3659c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee380cc6b0 0x7fee380cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42576/0 (socket says 172.21.15.67:42576) 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee36d9d640 1 -- 172.21.15.67:0/617735279 learned_addr learned my addr 172.21.15.67:0/617735279 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee35d9b640 1 --2- 172.21.15.67:0/617735279 >> v2:172.21.15.176:3300/0 conn(0x7fee38160430 0x7fee38160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3659c640 1 -- 172.21.15.67:0/617735279 >> v2:172.21.15.176:3300/0 conn(0x7fee38160430 msgr2=0x7fee38160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3659c640 1 --2- 172.21.15.67:0/617735279 >> v2:172.21.15.176:3300/0 conn(0x7fee38160430 0x7fee38160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3659c640 1 -- 172.21.15.67:0/617735279 >> v1:172.21.15.67:6789/0 conn(0x7fee3806d650 legacy=0x7fee38068590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3659c640 1 -- 172.21.15.67:0/617735279 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee380ca150 con 0x7fee380cc6b0 2024-04-25T15:53:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.246+0000 7fee3559a640 1 -- 172.21.15.67:0/617735279 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069829900 0 0) 0x7fee380ccc80 con 0x7fee3806d650 2024-04-25T15:53:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3659c640 1 --2- 172.21.15.67:0/617735279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee380cc6b0 0x7fee380cdc70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fee38072150 tx=0x7fee2801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f58504080d3b1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3559a640 1 -- 172.21.15.67:0/617735279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee280b0070 con 0x7fee380cc6b0 2024-04-25T15:53:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3559a640 1 -- 172.21.15.67:0/617735279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee280ab600 con 0x7fee380cc6b0 2024-04-25T15:53:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3559a640 1 -- 172.21.15.67:0/617735279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee280ab9d0 con 0x7fee380cc6b0 2024-04-25T15:53:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 -- 172.21.15.67:0/617735279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee380cc6b0 msgr2=0x7fee380cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 --2- 172.21.15.67:0/617735279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee380cc6b0 0x7fee380cdc70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fee38072150 tx=0x7fee2801c3d0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 -- 172.21.15.67:0/617735279 shutdown_connections 2024-04-25T15:53:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 --2- 172.21.15.67:0/617735279 >> v2:172.21.15.176:3300/0 conn(0x7fee38160430 0x7fee38160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 --2- 172.21.15.67:0/617735279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee380cc6b0 0x7fee380cdc70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 -- 172.21.15.67:0/617735279 >> 172.21.15.67:0/617735279 conn(0x7fee380705c0 msgr2=0x7fee380709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 -- 172.21.15.67:0/617735279 shutdown_connections 2024-04-25T15:53:22.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.250+0000 7fee3cab7640 1 -- 172.21.15.67:0/617735279 wait complete. 2024-04-25T15:53:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 Processor -- start 2024-04-25T15:53:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 -- start start 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3806d650 0x7fee380c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee380cc6b0 0x7fee3815f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee38160430 0x7fee38148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fee3816a2b0 con 0x7fee3806d650 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fee3816a130 con 0x7fee380cc6b0 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee3816a430 con 0x7fee38160430 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3806d650 0x7fee380c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3806d650 0x7fee380c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42590/0 (socket says 172.21.15.67:42590) 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 -- 172.21.15.67:0/1497149485 learned_addr learned my addr 172.21.15.67:0/1497149485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee36d9d640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee38160430 0x7fee38148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee35d9b640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee380cc6b0 0x7fee3815f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 -- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee38160430 msgr2=0x7fee38148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee38160430 0x7fee38148050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 -- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee380cc6b0 msgr2=0x7fee3815f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee380cc6b0 0x7fee3815f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 -- 172.21.15.67:0/1497149485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee380c70f0 con 0x7fee3806d650 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3659c640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3806d650 0x7fee380c6bb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fee28018fe0 tx=0x7fee2801ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee26ffd640 1 -- 172.21.15.67:0/1497149485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee280b0050 con 0x7fee3806d650 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee26ffd640 1 -- 172.21.15.67:0/1497149485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee280bc070 con 0x7fee3806d650 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3814e010 con 0x7fee3806d650 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee26ffd640 1 -- 172.21.15.67:0/1497149485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28006dc0 con 0x7fee3806d650 2024-04-25T15:53:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3814e5d0 con 0x7fee3806d650 2024-04-25T15:53:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.254+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee38050750 con 0x7fee3806d650 2024-04-25T15:53:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.258+0000 7fee26ffd640 1 -- 172.21.15.67:0/1497149485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fee280b5830 con 0x7fee3806d650 2024-04-25T15:53:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.258+0000 7fee26ffd640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1806f430 0x7fee18071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.258+0000 7fee26ffd640 1 -- 172.21.15.67:0/1497149485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5915+0+0 (secure 0 0 0) 0x7fee28129940 con 0x7fee3806d650 2024-04-25T15:53:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.258+0000 7fee35d9b640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1806f430 0x7fee18071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.258+0000 7fee35d9b640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1806f430 0x7fee18071940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fee38073f40 tx=0x7fee2c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.262+0000 7fee26ffd640 1 -- 172.21.15.67:0/1497149485 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fee28016050 con 0x7fee3806d650 2024-04-25T15:53:22.306 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:22.303+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:22.339 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:22.339 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:22.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:22.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:22.430+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fee380cd040 con 0x7fee3806d650 2024-04-25T15:53:22.443 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:22.443+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:22.506 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:22.503+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:22.546 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:22.546 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:22.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:22.568 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:22.568 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:22.568 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:22.568 INFO:tasks.workunit.client.0.smithi067.stdout:iozone test complete. 2024-04-25T15:53:22.570 INFO:tasks.workunit.client.0.smithi067.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-25T15:53:22.607 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:22.606+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:22.639 INFO:tasks.workunit.client.0.smithi067.stdout: Iozone: Performance Test of File I/O 2024-04-25T15:53:22.639 INFO:tasks.workunit.client.0.smithi067.stdout: Version $Revision: 3.489 $ 2024-04-25T15:53:22.639 INFO:tasks.workunit.client.0.smithi067.stdout: Compiled for 64 bit mode. 2024-04-25T15:53:22.639 INFO:tasks.workunit.client.0.smithi067.stdout: Build: linux-AMD64 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Run began: Thu Apr 25 15:53:22 2024 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Include close in write timing 2024-04-25T15:53:22.640 INFO:tasks.workunit.client.0.smithi067.stdout: Include fsync in write timing 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: File size set to 1048576 kB 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Record Size 1024 kB 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Output is in kBytes/sec 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Time Resolution = 0.000001 seconds. 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache size set to 1024 kBytes. 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache line size set to 32 bytes. 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: File stride size set to 17 * record size. 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Throughput test with 1 process 2024-04-25T15:53:22.641 INFO:tasks.workunit.client.0.smithi067.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-04-25T15:53:22.708 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:22.707+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:22.780 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:22.781 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:22.809 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:22.810+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:22.812 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:22.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:22.909 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:22.907+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:23.010 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:23.007+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:23.038 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:23.038 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:23.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.070+0000 7fb0611e9640 1 Processor -- start 2024-04-25T15:53:23.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.070+0000 7fb0611e9640 1 -- start start 2024-04-25T15:53:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb0611e9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb05c156330 0x7fb05c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb0611e9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 0x7fb05c0c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb0611e9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb05c0c4f30 0x7fb05c158d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb0611e9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb05c0ca6c0 con 0x7fb05c156330 2024-04-25T15:53:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb0611e9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb05c06cf90 con 0x7fb05c0c4f30 2024-04-25T15:53:23.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb0611e9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb05c0c43d0 con 0x7fb05c06cb80 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb05c156330 0x7fb05c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb05c156330 0x7fb05c06c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40550/0 (socket says 172.21.15.67:40550) 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 -- 172.21.15.67:0/2302842002 learned_addr learned my addr 172.21.15.67:0/2302842002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb053fff640 1 --2- 172.21.15.67:0/2302842002 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 0x7fb05c0c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05b59e640 1 --2- 172.21.15.67:0/2302842002 >> v2:172.21.15.176:3300/0 conn(0x7fb05c0c4f30 0x7fb05c158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 -- 172.21.15.67:0/2302842002 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 msgr2=0x7fb05c0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/2302842002 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 0x7fb05c0c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 -- 172.21.15.67:0/2302842002 >> v2:172.21.15.176:3300/0 conn(0x7fb05c0c4f30 msgr2=0x7fb05c158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/2302842002 >> v2:172.21.15.176:3300/0 conn(0x7fb05c0c4f30 0x7fb05c158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.074+0000 7fb05ad9d640 1 -- 172.21.15.67:0/2302842002 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c06fa10 con 0x7fb05c156330 2024-04-25T15:53:23.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/2302842002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c156330 0x7fb05c06c640 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0c8830 tx=0x7fb04c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c8266c6d84f7d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb05a59c640 1 -- 172.21.15.67:0/2302842002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c0b5070 con 0x7fb05c156330 2024-04-25T15:53:23.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb05a59c640 1 -- 172.21.15.67:0/2302842002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04c0a89b0 con 0x7fb05c156330 2024-04-25T15:53:23.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb05a59c640 1 -- 172.21.15.67:0/2302842002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c0a8d80 con 0x7fb05c156330 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 -- 172.21.15.67:0/2302842002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c156330 msgr2=0x7fb05c06c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 --2- 172.21.15.67:0/2302842002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c156330 0x7fb05c06c640 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0c8830 tx=0x7fb04c01ecf0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 -- 172.21.15.67:0/2302842002 shutdown_connections 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 --2- 172.21.15.67:0/2302842002 >> v2:172.21.15.176:3300/0 conn(0x7fb05c0c4f30 0x7fb05c158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 --2- 172.21.15.67:0/2302842002 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 0x7fb05c0c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 --2- 172.21.15.67:0/2302842002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c156330 0x7fb05c06c640 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 -- 172.21.15.67:0/2302842002 >> 172.21.15.67:0/2302842002 conn(0x7fb05c0ce1c0 msgr2=0x7fb05c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:23.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 -- 172.21.15.67:0/2302842002 shutdown_connections 2024-04-25T15:53:23.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.078+0000 7fb0611e9640 1 -- 172.21.15.67:0/2302842002 wait complete. 2024-04-25T15:53:23.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 Processor -- start 2024-04-25T15:53:23.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 -- start start 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 0x7fb05c0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c0c4f30 0x7fb05c1f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb05c156330 0x7fb05c0cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c0c3d80 con 0x7fb05c0c4f30 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c0c3aa0 con 0x7fb05c156330 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb05c0c4060 con 0x7fb05c06cb80 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c0c4f30 0x7fb05c1f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c0c4f30 0x7fb05c1f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40552/0 (socket says 172.21.15.67:40552) 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 -- 172.21.15.67:0/1134009379 learned_addr learned my addr 172.21.15.67:0/1134009379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 0x7fb05c0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb05c156330 0x7fb05c0cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 msgr2=0x7fb05c0c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb05c06cb80 0x7fb05c0c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb05c156330 msgr2=0x7fb05c0cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb05c156330 0x7fb05c0cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c1f77b0 con 0x7fb05c0c4f30 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb05c156330 0x7fb05c0cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb053fff640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c0c4f30 0x7fb05c1f7270 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb04400e730 tx=0x7fb04400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04400b040 con 0x7fb05c0c4f30 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04400ca40 con 0x7fb05c0c4f30 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c0c80a0 con 0x7fb05c0c4f30 2024-04-25T15:53:23.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04400ce10 con 0x7fb05c0c4f30 2024-04-25T15:53:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.082+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c15a1f0 con 0x7fb05c0c4f30 2024-04-25T15:53:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.086+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb05c1f7c80 con 0x7fb05c0c4f30 2024-04-25T15:53:23.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.086+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb0440059a0 con 0x7fb05c0c4f30 2024-04-25T15:53:23.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.086+0000 7fb0537fe640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb03c06f400 0x7fb03c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.086+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5915+0+0 (secure 0 0 0) 0x7fb04408f270 con 0x7fb05c0c4f30 2024-04-25T15:53:23.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.090+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb03c06f400 0x7fb03c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.090+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb03c06f400 0x7fb03c071910 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0b2000 tx=0x7fb04c0aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.098+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v80) ==== 54+0+16 (secure 0 0 0) 0x7fb044016020 con 0x7fb05c0c4f30 2024-04-25T15:53:23.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.098+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb05c154390 0x7fb05c15a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.098+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb05c06a480 con 0x7fb05c154390 2024-04-25T15:53:23.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.098+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb05c154390 0x7fb05c15a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.098+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb05c154390 0x7fb05c15a430 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.102+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c06a480 con 0x7fb05c154390 2024-04-25T15:53:23.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.102+0000 7fee26ffd640 1 -- 172.21.15.67:0/1497149485 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v81) ==== 91+0+0 (secure 0 0 0) 0x7fee280f6760 con 0x7fee3806d650 2024-04-25T15:53:23.103 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:53:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1806f430 msgr2=0x7fee18071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1806f430 0x7fee18071940 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fee38073f40 tx=0x7fee2c008040 comp rx=0 tx=0).stop 2024-04-25T15:53:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3806d650 msgr2=0x7fee380c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3806d650 0x7fee380c6bb0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fee28018fe0 tx=0x7fee2801ac10 comp rx=0 tx=0).stop 2024-04-25T15:53:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 shutdown_connections 2024-04-25T15:53:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee1806f430 0x7fee18071940 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee38160430 0x7fee38148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee380cc6b0 0x7fee3815f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 --2- 172.21.15.67:0/1497149485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3806d650 0x7fee380c6bb0 secure :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fee28018fe0 tx=0x7fee2801ac10 comp rx=0 tx=0).stop 2024-04-25T15:53:23.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 >> 172.21.15.67:0/1497149485 conn(0x7fee380705c0 msgr2=0x7fee38155e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:23.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 shutdown_connections 2024-04-25T15:53:23.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.106+0000 7fee3cab7640 1 -- 172.21.15.67:0/1497149485 wait complete. 2024-04-25T15:53:23.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.130+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c152bc0 con 0x7fb05c154390 2024-04-25T15:53:23.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.134+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c152bc0 con 0x7fb05c154390 2024-04-25T15:53:23.133 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:23.134 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:23.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.134+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb05c0b0660 0x7fb05c14de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.134+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb05c14e3b0 con 0x7fb05c0b0660 2024-04-25T15:53:23.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.134+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb05c0b0660 0x7fb05c14de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.134+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb05c0b0660 0x7fb05c14de70 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.134+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c14e3b0 con 0x7fb05c0b0660 2024-04-25T15:53:23.147 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:23.147+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:23.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.154+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c0c89e0 con 0x7fb05c0b0660 2024-04-25T15:53:23.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.154+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c0c89e0 con 0x7fb05c0b0660 2024-04-25T15:53:23.159 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:23.160 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:23.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.158+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb05c15c7d0 0x7fb05c15cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.158+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb05c0cbfc0 con 0x7fb05c15c7d0 2024-04-25T15:53:23.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.158+0000 7fb053fff640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb05c15c7d0 0x7fb05c15cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.158+0000 7fb053fff640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb05c15c7d0 0x7fb05c15cbe0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.158+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c0cbfc0 con 0x7fb05c15c7d0 2024-04-25T15:53:23.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.178+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c0cc4f0 con 0x7fb05c15c7d0 2024-04-25T15:53:23.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.182+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c0cc4f0 con 0x7fb05c15c7d0 2024-04-25T15:53:23.182 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:23.183 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:23.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.182+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb05c063090 0x7fb05c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.182+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb05c0cbfc0 con 0x7fb05c063090 2024-04-25T15:53:23.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.182+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb05c063090 0x7fb05c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.182+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb05c063090 0x7fb05c0634a0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.186+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c0cbfc0 con 0x7fb05c063090 2024-04-25T15:53:23.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.206+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c0cbfc0 con 0x7fb05c063090 2024-04-25T15:53:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.206+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c0cbfc0 con 0x7fb05c063090 2024-04-25T15:53:23.209 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.210+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb05c02c9e0 0x7fb05c02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.210+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb05c0cbfc0 con 0x7fb05c02c9e0 2024-04-25T15:53:23.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.210+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb05c02c9e0 0x7fb05c02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.210+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb05c02c9e0 0x7fb05c02cdf0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.211 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:23.207+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:23.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.210+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c0cbfc0 con 0x7fb05c02c9e0 2024-04-25T15:53:23.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.234+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c0cbfc0 con 0x7fb05c02c9e0 2024-04-25T15:53:23.290 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:23.290 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:23.290 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:23.290 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:23.290 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:23.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.234+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c0cbfc0 con 0x7fb05c02c9e0 2024-04-25T15:53:23.291 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.238+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb05c02d840 0x7fb05c20d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.238+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb05c02dc50 con 0x7fb05c02d840 2024-04-25T15:53:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.238+0000 7fb053fff640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb05c02d840 0x7fb05c20d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.238+0000 7fb053fff640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb05c02d840 0x7fb05c20d900 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.238+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c02dc50 con 0x7fb05c02d840 2024-04-25T15:53:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.258+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c02dc50 con 0x7fb05c02d840 2024-04-25T15:53:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.262+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c02dc50 con 0x7fb05c02d840 2024-04-25T15:53:23.292 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.262+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb05c20e350 0x7fb05c210770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.262+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb05c213630 con 0x7fb05c20e350 2024-04-25T15:53:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.262+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb05c20e350 0x7fb05c210770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.262+0000 7fb05b59e640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb05c20e350 0x7fb05c210770 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.266+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c213630 con 0x7fb05c20e350 2024-04-25T15:53:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.290+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c210f80 con 0x7fb05c20e350 2024-04-25T15:53:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.294+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c210f80 con 0x7fb05c20e350 2024-04-25T15:53:23.296 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.294+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb05c2116a0 0x7fb05c211ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:23.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.294+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb05c216a00 con 0x7fb05c2116a0 2024-04-25T15:53:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.294+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb05c2116a0 0x7fb05c211ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.298+0000 7fb05ad9d640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb05c2116a0 0x7fb05c211ab0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:23.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.298+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb05c216a00 con 0x7fb05c2116a0 2024-04-25T15:53:23.304 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:23.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.318+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb05c211ff0 con 0x7fb05c2116a0 2024-04-25T15:53:23.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0537fe640 1 -- 172.21.15.67:0/1134009379 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb05c211ff0 con 0x7fb05c2116a0 2024-04-25T15:53:23.323 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb05c154390 msgr2=0x7fb05c15a430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb05c154390 0x7fb05c15a430 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb05c0b0660 msgr2=0x7fb05c14de70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb05c0b0660 0x7fb05c14de70 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb05c15c7d0 msgr2=0x7fb05c15cbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb05c15c7d0 0x7fb05c15cbe0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb05c063090 msgr2=0x7fb05c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb05c063090 0x7fb05c0634a0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb05c02c9e0 msgr2=0x7fb05c02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb05c02c9e0 0x7fb05c02cdf0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb05c02d840 msgr2=0x7fb05c20d900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb05c02d840 0x7fb05c20d900 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb05c20e350 msgr2=0x7fb05c210770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb05c20e350 0x7fb05c210770 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb05b59e640 1 -- 172.21.15.67:0/1134009379 reap_dead start 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb05b59e640 1 -- 172.21.15.67:0/1134009379 reap_dead start 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb05b59e640 1 -- 172.21.15.67:0/1134009379 reap_dead start 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb05c2116a0 msgr2=0x7fb05c211ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb05c2116a0 0x7fb05c211ab0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb03c06f400 msgr2=0x7fb03c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb03c06f400 0x7fb03c071910 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0b2000 tx=0x7fb04c0aa070 comp rx=0 tx=0).stop 2024-04-25T15:53:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c0c4f30 msgr2=0x7fb05c1f7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c0c4f30 0x7fb05c1f7270 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb04400e730 tx=0x7fb04400ebf0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 shutdown_connections 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb05c20e350 0x7fb05c210770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb05c2116a0 0x7fb05c211ab0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb03c06f400 0x7fb03c071910 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 --2- 172.21.15.67:0/1134009379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb05c0c4f30 0x7fb05c1f7270 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 >> 172.21.15.67:0/1134009379 conn(0x7fb05c0ce1c0 msgr2=0x7fb05c151dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 shutdown_connections 2024-04-25T15:53:23.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:23.322+0000 7fb0611e9640 1 -- 172.21.15.67:0/1134009379 wait complete. 2024-04-25T15:53:23.347 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:23.347+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:23.414 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:23.414+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:23.513 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:23.514+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:67 3.28 3:1772fc85:::10000000002.0000003f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:44.996236+0000", 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.547641818999999, 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025389129, 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.547 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 67 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.996236+0000", 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.996236+0000", 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:44.996240+0000", 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1829999999999998e-06 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005298+0000", 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.009058254 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005367+0000", 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9162e-05 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005392+0000", 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5510000000000001e-05 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005587+0000", 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000194792 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.005874+0000", 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000286564 2024-04-25T15:53:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.006129+0000", 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025482400000000001 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.015438+0000", 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093097019999999996 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.021435+0000", 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059962790000000002 2024-04-25T15:53:23.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.021539+0000", 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010387000000000001 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.021626+0000", 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6989000000000003e-05 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.552 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:73 3.1d 3:b94578cb:::10000000002.00000045:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:45.257557+0000", 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 38.286321383999997, 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025594236999999999, 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 73 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.257557+0000", 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.257557+0000", 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.257560+0000", 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.213e-06 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262293+0000", 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047326570000000004 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262353+0000", 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0009000000000001e-05 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262377+0000", 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4593e-05 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262599+0000", 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022189200000000001 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.262870+0000", 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027080099999999998 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.263123+0000", 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025334799999999997 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.272154+0000", 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090310040000000005 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.282961+0000", 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010806827999999999 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.283065+0000", 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010398 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:45.283151+0000", 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5912e-05 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:168 3.3a 3:5e20dd20:::10000000002.000000a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.499594+0000", 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.044284058000002, 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025637967000000001, 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.558 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 168 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.499594+0000", 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.499594+0000", 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.499598+0000", 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506582+0000", 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069845569999999997 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506649+0000", 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6500000000000004e-05 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506684+0000", 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4999999999999997e-05 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.506896+0000", 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021213499999999999 2024-04-25T15:53:23.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.507193+0000", 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029704699999999999 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.507463+0000", 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027019400000000002 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.520156+0000", 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012692914 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.524914+0000", 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047575960000000002 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.525096+0000", 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018226599999999999 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.525232+0000", 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013627800000000001 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:180 3.2 e62/18 3:42ce5726:::10000000002.000000b0:head v 62'3, mlcod=59'1)", 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.003402+0000", 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.540476435999999, 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029043465000000001, 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.003402+0000", 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.003402+0000", 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.003405+0000", 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7000000000000002e-06 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.013858+0000", 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010452258000000001 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.013898+0000", 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0033000000000001e-05 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.013922+0000", 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4686e-05 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.014112+0000", 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018924499999999999 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.014192+0000", 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0742999999999996e-05 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.032267+0000", 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018074921000000001 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.032445+0000", 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017787900000000001 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:181 3.1f 3:f8dc3ee2:::10000000002.000000b1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:50.040594+0000", 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.503283916999997, 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02520416, 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.568 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 181 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.040594+0000", 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.040594+0000", 2024-04-25T15:53:23.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.040597+0000", 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1489999999999998e-06 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048339+0000", 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077415790000000002 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048410+0000", 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0545999999999998e-05 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048436+0000", 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6426000000000001e-05 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.048640+0000", 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020419200000000001 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.049063+0000", 2024-04-25T15:53:23.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000422626 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.049532+0000", 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00046956999999999999 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.064111+0000", 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014578420999999999 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.065438+0000", 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001327665 2024-04-25T15:53:23.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.065621+0000", 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018292800000000001 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:50.065798+0000", 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017705800000000001 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:261 3.1d 3:bbf7764d:::10000000002.00000001:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:59.752398+0000", 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.791480019000002, 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030613782999999999, 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.574 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 261 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.752398+0000", 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.752398+0000", 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.752402+0000", 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.783e-06 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.758763+0000", 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063605900000000002 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.758830+0000", 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7768999999999994e-05 2024-04-25T15:53:23.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.758860+0000", 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0034000000000001e-05 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759094+0000", 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000233351 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759271+0000", 2024-04-25T15:53:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017722899999999999 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.759518+0000", 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024753399999999999 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.776213+0000", 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016694102999999998 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782820+0000", 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066076980000000004 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.782930+0000", 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109633 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:59.783012+0000", 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2058999999999995e-05 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.324851589000001, 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.580 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:53:23.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:53:23.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:53:23.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:53:23.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:327 3.4 e69/68 3:20ca167d:::10000000002.00000043:head v 69'3, mlcod=0'0)", 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.440429+0000", 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.103449424000001, 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025176512000000002, 2024-04-25T15:53:23.587 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.440429+0000", 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.440429+0000", 2024-04-25T15:53:23.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.440430+0000", 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4759999999999999e-06 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.448079+0000", 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076487559999999996 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.448098+0000", 2024-04-25T15:53:23.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9437999999999999e-05 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.448118+0000", 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9352e-05 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.448205+0000", 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6975e-05 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.448235+0000", 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0297000000000001e-05 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.465420+0000", 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017184699000000001 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.465605+0000", 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018551900000000001 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:396 3.1c 3:384ad1d9:::10000000002.00000088:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.007054+0000", 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.536824034999999, 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029039829, 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 396 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.592 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.007054+0000", 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.007054+0000", 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.007058+0000", 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.49e-06 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.014395+0000", 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073375089999999999 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.014455+0000", 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0071999999999997e-05 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.014489+0000", 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4069999999999997e-05 2024-04-25T15:53:23.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.014681+0000", 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019176499999999999 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.014863+0000", 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018195900000000001 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.015116+0000", 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025321700000000003 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.034821+0000", 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019705001999999999 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.035869+0000", 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010480610000000001 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.035974+0000", 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104689 2024-04-25T15:53:23.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.036094+0000", 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011999500000000001 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:402 3.16 3:6bf10b49:::10000000002.0000008e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.216579+0000", 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.327299135000001, 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025809295999999999, 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.597 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 402 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.216579+0000", 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.216579+0000", 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.216582+0000", 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.993e-06 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.221852+0000", 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005270406 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.221916+0000", 2024-04-25T15:53:23.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3303e-05 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.221949+0000", 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3073e-05 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.222156+0000", 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020745799999999999 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.222334+0000", 2024-04-25T15:53:23.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177589 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.222582+0000", 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024859099999999999 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.234097+0000", 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01151487 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.242191+0000", 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080934549999999994 2024-04-25T15:53:23.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.242296+0000", 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104929 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.242388+0000", 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2628999999999997e-05 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:405 3.1c 3:3afba7d1:::10000000002.00000091:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.320906+0000", 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.222972461000001, 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025336379999999999, 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 405 2024-04-25T15:53:23.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.320906+0000", 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.320906+0000", 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.320909+0000", 2024-04-25T15:53:23.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.253e-06 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.326090+0000", 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051808840000000002 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.326155+0000", 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5508999999999999e-05 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.326190+0000", 2024-04-25T15:53:23.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4567000000000003e-05 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.326403+0000", 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000213544 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.326588+0000", 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018490499999999999 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.326901+0000", 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031261499999999997 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.339937+0000", 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013036275999999999 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.346039+0000", 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006101403 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.346145+0000", 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010664500000000001 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.346242+0000", 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6779000000000003e-05 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:427 3.3c 3:3ecd43ae:::10000000002.000000a7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:23.608 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:06.164512+0000", 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 17.379366477000001, 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028826747999999999, 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 427 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.164512+0000", 2024-04-25T15:53:23.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.164512+0000", 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.164515+0000", 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5599999999999998e-06 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.179624+0000", 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015108266 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.179682+0000", 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8665999999999997e-05 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.179718+0000", 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5389999999999998e-05 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.179966+0000", 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024861799999999998 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.180175+0000", 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020925100000000001 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.180431+0000", 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025580500000000001 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.189565+0000", 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0091339799999999999 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.193154+0000", 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0035889609999999999 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.193252+0000", 2024-04-25T15:53:23.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7822e-05 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:06.193338+0000", 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6428999999999995e-05 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:506 3.16 3:689177d4:::10000000002.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:09.167440+0000", 2024-04-25T15:53:23.614 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.376437920000001, 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025730487999999999, 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 506 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.167440+0000", 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.167440+0000", 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.167444+0000", 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6799999999999999e-06 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.175178+0000", 2024-04-25T15:53:23.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077338379999999998 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.175244+0000", 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6156999999999997e-05 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.175273+0000", 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8679000000000001e-05 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.175477+0000", 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020393200000000001 2024-04-25T15:53:23.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.175659+0000", 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018203100000000001 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.175930+0000", 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027181200000000002 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.192224+0000", 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016293269999999999 2024-04-25T15:53:23.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.192980+0000", 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00075641399999999996 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.193082+0000", 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101436 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:09.193171+0000", 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9239000000000004e-05 2024-04-25T15:53:23.619 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:589 3.16 3:6ac5d82f:::100000001f7.00000049:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.315253+0000", 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.22862486200000001, 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025102218999999999, 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.620 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 589 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.315253+0000", 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.315253+0000", 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.315256+0000", 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7879999999999998e-06 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.320721+0000", 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054647749999999998 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.320769+0000", 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8028999999999997e-05 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.320790+0000", 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0613000000000001e-05 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.320893+0000", 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010379599999999999 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.321063+0000", 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169765 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.321181+0000", 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011817299999999999 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.339785+0000", 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018603742999999999 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.340201+0000", 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041605199999999998 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.340278+0000", 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6669999999999996e-05 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.340356+0000", 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7815e-05 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:594 3.16 3:6a58e86c:::100000001f7.0000004e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.196674142, 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034501508, 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 594 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347207+0000", 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8660000000000002e-06 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353453+0000", 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062464160000000003 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353489+0000", 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5283000000000001e-05 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353501+0000", 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2860999999999999e-05 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353629+0000", 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000127632 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353808+0000", 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017904800000000001 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353996+0000", 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018766099999999999 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.380705+0000", 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026708789 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381525+0000", 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00082074100000000005 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381619+0000", 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3153999999999996e-05 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381706+0000", 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7057000000000004e-05 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:596 3.16 3:6941c83d:::100000001f7.00000050:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:23.631 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.18195966, 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049183123000000002, 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 596 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361922+0000", 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367484+0000", 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055618409999999997 2024-04-25T15:53:23.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367522+0000", 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8260000000000003e-05 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367543+0000", 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0795999999999999e-05 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367646+0000", 2024-04-25T15:53:23.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102913 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367857+0000", 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021190400000000001 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368091+0000", 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023323699999999999 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.405516+0000", 2024-04-25T15:53:23.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037425424999999998 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410955+0000", 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054389859999999998 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411034+0000", 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8421e-05 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411102+0000", 2024-04-25T15:53:23.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8137000000000002e-05 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:600 3.5 3:a2751c53:::100000001f7.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.15847615400000001, 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035430837, 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.637 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 600 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385405+0000", 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8959999999999999e-06 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390062+0000", 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046572949999999997 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390104+0000", 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1943000000000001e-05 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390123+0000", 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9147000000000002e-05 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390225+0000", 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101983 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390407+0000", 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181819 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390593+0000", 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018578900000000001 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409139+0000", 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018546360000000001 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420728+0000", 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011589083 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420783+0000", 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5025000000000003e-05 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420833+0000", 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9496999999999997e-05 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:601 3.38 e81/18 3:1c31e34b:::100000001f7.00000055:head v 81'5, mlcod=80'3)", 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.395838+0000", 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.148040062, 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025463844999999999, 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.395838+0000", 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.395838+0000", 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.395839+0000", 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.049e-06 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400491+0000", 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046516680000000003 2024-04-25T15:53:23.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400515+0000", 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4396000000000002e-05 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400529+0000", 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3658e-05 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400606+0000", 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7233999999999999e-05 2024-04-25T15:53:23.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400634+0000", 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7367000000000001e-05 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.421206+0000", 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020572292999999998 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.421302+0000", 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6180000000000004e-05 2024-04-25T15:53:23.646 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:606 3.31 e81/76 3:8d161915:::100000001f7.0000005a:head v 81'11, mlcod=80'9)", 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.436189+0000", 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.107689146, 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024934036, 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.436189+0000", 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.436189+0000", 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.436191+0000", 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.883e-06 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.439909+0000", 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003718191 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.439942+0000", 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3052999999999997e-05 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.439963+0000", 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0653999999999999e-05 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.440072+0000", 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109177 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.440107+0000", 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4555999999999999e-05 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461062+0000", 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020955438 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461123+0000", 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1084e-05 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:613 3.3a 3:5ecf435d:::100000001f7.00000061:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.478047+0000", 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.065831524000000002, 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027997125000000001, 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 613 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.478047+0000", 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.478047+0000", 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.478050+0000", 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.703e-06 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.482552+0000", 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045018439999999996 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.482584+0000", 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1526999999999998e-05 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.482599+0000", 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4925e-05 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.482695+0000", 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6589999999999995e-05 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.482842+0000", 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014636200000000001 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.482990+0000", 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014839599999999999 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.493230+0000", 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010240404 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.505847+0000", 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012616502999999999 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.505955+0000", 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107774 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.506044+0000", 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9097000000000004e-05 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.656 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.657 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.657 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:23.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:23.659 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:23.655+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:23.759 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:23.759+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:23.862 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:23.859+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.25849008800000001, 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.25855047599999997, 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.918 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:23.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:53:23.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.922 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:88 2.14 2:2e2fa760:::200.00000001:head [write 97242~3394 [fadvise_dontneed] in=3394b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.026426928999999998, 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026596748, 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:23.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891429+0000", 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8190000000000001e-06 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891440+0000", 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1029999999999999e-05 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891499+0000", 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8961e-05 2024-04-25T15:53:23.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891530+0000", 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1049e-05 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891630+0000", 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010053899999999999 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891787+0000", 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015704200000000001 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891996+0000", 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020865400000000001 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:640 3.10 3:092a6a2c:::100000001f7.0000007c:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.039458806999999999, 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039693608999999998, 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.926 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 640 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:23.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878397+0000", 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.198e-06 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894140+0000", 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015742477000000001 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894240+0000", 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010061299999999999 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894258+0000", 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7779999999999999e-05 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894341+0000", 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3168000000000001e-05 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894526+0000", 2024-04-25T15:53:23.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018456399999999999 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894632+0000", 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010651500000000001 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:647 3.a 3:524e89bd:::100000001f7.00000083:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.023704131, 2024-04-25T15:53:23.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024018576, 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 647 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894152+0000", 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.968e-06 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902694+0000", 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085420719999999995 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903482+0000", 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078797999999999997 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903499+0000", 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7082e-05 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903593+0000", 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4452999999999998e-05 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903774+0000", 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180803 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903953+0000", 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179106 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:652 3.36 3:6c98fff9:::100000001f7.00000088:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:23.934 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.015148826000000001, 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015546525, 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 652 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902707+0000", 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5209999999999999e-06 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910035+0000", 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073280000000000003 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910344+0000", 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030890699999999998 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910417+0000", 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3900999999999996e-05 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910547+0000", 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012940400000000001 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910876+0000", 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000329181 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911062+0000", 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018587000000000001 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 5 2024-04-25T15:53:23.938 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:23.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:23.961 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:23.962+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:24.061 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:24.059+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:24.162 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:24.159+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:24.169 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:24.169 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:24.169 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:24.170 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:24.170 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:24.194 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:24.263 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:24.262+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:24.364 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:24.363+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:24.455 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:177 3.26 3:66aef6e9:::10000000002.000000ad:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.576282966999997, 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026823691, 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.456 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876688+0000", 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.876691+0000", 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4939999999999999e-06 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880736+0000", 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040441660000000001 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880793+0000", 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7581999999999999e-05 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.880817+0000", 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881039+0000", 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022171800000000001 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881305+0000", 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026580799999999997 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.881558+0000", 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025290099999999998 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.893182+0000", 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011624242 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903322+0000", 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010139754000000001 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903425+0000", 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102706 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:49.903512+0000", 2024-04-25T15:53:24.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7093000000000001e-05 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:355 3.26 3:6477918e:::10000000002.0000005f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.964256293999998, 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025437358, 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.462 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 355 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488715+0000", 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.488717+0000", 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.874e-06 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494412+0000", 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056951349999999996 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494450+0000", 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7920999999999998e-05 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494464+0000", 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4728e-05 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494580+0000", 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011599099999999999 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.494756+0000", 2024-04-25T15:53:24.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000176083 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.495038+0000", 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028161800000000003 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.508282+0000", 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013243981 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514019+0000", 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057372639999999997 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514099+0000", 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9800000000000002e-05 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.514152+0000", 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2963000000000001e-05 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:372 3.10 3:0b92729d:::10000000002.00000070:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.346432932999999, 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026675501000000001, 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 372 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106538+0000", 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.106541+0000", 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.022e-06 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110718+0000", 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041768669999999999 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110768+0000", 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0160000000000001e-05 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110792+0000", 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3577000000000001e-05 2024-04-25T15:53:24.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.110988+0000", 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019645200000000001 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111271+0000", 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028314399999999998 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.111688+0000", 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041694699999999998 2024-04-25T15:53:24.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.123821+0000", 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012133015 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133024+0000", 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092031290000000009 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133125+0000", 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010063199999999999 2024-04-25T15:53:24.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.133214+0000", 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8555999999999997e-05 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:394 3.36 3:6ccb00e9:::10000000002.00000086:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.52136883, 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025589158000000001, 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.473 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 394 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931602+0000", 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.931605+0000", 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0730000000000001e-06 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936840+0000", 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052349129999999999 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936898+0000", 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7793e-05 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.936922+0000", 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4513000000000001e-05 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937111+0000", 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018847800000000001 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937292+0000", 2024-04-25T15:53:24.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181189 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.937542+0000", 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024970800000000002 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956147+0000", 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018605467000000001 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.956986+0000", 2024-04-25T15:53:24.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083831599999999997 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957089+0000", 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103379 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:04.957191+0000", 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102329 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:410 3.34 3:2cf0bbf2:::10000000002.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.930404354, 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026112841000000001, 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 410 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522567+0000", 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.522570+0000", 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529416+0000", 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006846062 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529476+0000", 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0350000000000003e-05 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529502+0000", 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6259999999999999e-05 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529665+0000", 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016305200000000001 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.529683+0000", 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7935999999999999e-05 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534023+0000", 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043396270000000004 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534279+0000", 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025637099999999998 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.534518+0000", 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000238762 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.543891+0000", 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093730170000000008 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548495+0000", 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046033610000000003 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548592+0000", 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7793000000000004e-05 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:05.548679+0000", 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7100000000000003e-05 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.484 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:602 3.20 3:073599bb:::100000001f7.00000056:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.400749+0000", 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.052221646, 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026254452000000001, 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 602 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.485 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400749+0000", 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400749+0000", 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.400751+0000", 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.035e-06 2024-04-25T15:53:24.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.406373+0000", 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056220480000000001 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.406406+0000", 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2155e-05 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.406421+0000", 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5075999999999999e-05 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.406519+0000", 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8145000000000003e-05 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.406717+0000", 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000198252 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.406917+0000", 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199948 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.419267+0000", 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012349597 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.426894+0000", 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076277610000000003 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.426942+0000", 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.427004+0000", 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1265000000000004e-05 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:608 3.20 3:04d3e86f:::100000001f7.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.441288+0000", 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.0116827209999999, 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027991305000000001, 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.490 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 608 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.441288+0000", 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.441288+0000", 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.441291+0000", 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5490000000000001e-06 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.446245+0000", 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049537970000000002 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.446285+0000", 2024-04-25T15:53:24.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0763000000000003e-05 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.446306+0000", 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0438999999999999e-05 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.446424+0000", 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118322 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.446785+0000", 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00036123200000000001 2024-04-25T15:53:24.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.447096+0000", 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031040899999999999 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.459666+0000", 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012570288000000001 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.469177+0000", 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095107390000000007 2024-04-25T15:53:24.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.469230+0000", 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2734e-05 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.469280+0000", 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0033e-05 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.495 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:625 3.10 3:08d8b931:::100000001f7.0000006d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.565297+0000", 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.88767417400000004, 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026065893999999999, 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 625 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.565297+0000", 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.565297+0000", 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.565299+0000", 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0600000000000002e-06 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.569829+0000", 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045304689999999996 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.569862+0000", 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2529e-05 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.569876+0000", 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4105e-05 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.569950+0000", 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4486000000000005e-05 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.570122+0000", 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000172041 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.570300+0000", 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017734099999999999 2024-04-25T15:53:24.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.578470+0000", 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081703309999999994 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.591273+0000", 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012802391 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.591319+0000", 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6650999999999998e-05 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.591363+0000", 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3489999999999999e-05 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.79360706400000003, 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.26243838600000002, 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.501 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:53:24.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920373+0000", 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039448372000000002 2024-04-25T15:53:24.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921649+0000", 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012757210000000001 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921724+0000", 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5549000000000005e-05 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921802+0000", 2024-04-25T15:53:24.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8150000000000002e-05 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:640 3.10 3:092a6a2c:::100000001f7.0000007c:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.57457578300000001, 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065130602999999995, 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.507 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 640 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:24.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878397+0000", 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.198e-06 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894140+0000", 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015742477000000001 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894240+0000", 2024-04-25T15:53:24.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010061299999999999 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894258+0000", 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7779999999999999e-05 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894341+0000", 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3168000000000001e-05 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894526+0000", 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018456399999999999 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894632+0000", 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010651500000000001 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.923409+0000", 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028776380000000001 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943396+0000", 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019987393999999999 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943459+0000", 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2389000000000002e-05 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943526+0000", 2024-04-25T15:53:24.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7125000000000005e-05 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:88 2.14 2:2e2fa760:::200.00000001:head [write 97242~3394 [fadvise_dontneed] in=3394b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.56154390499999995, 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052104437000000003, 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891429+0000", 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8190000000000001e-06 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891440+0000", 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1029999999999999e-05 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891499+0000", 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8961e-05 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891530+0000", 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1049e-05 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891630+0000", 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010053899999999999 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891787+0000", 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015704200000000001 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891996+0000", 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020865400000000001 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920805+0000", 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028808585000000001 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943394+0000", 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022589115 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943465+0000", 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1254000000000006e-05 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943531+0000", 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:647 3.a 3:524e89bd:::100000001f7.00000083:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.55882110699999998, 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083393058000000006, 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.518 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 647 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:24.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894152+0000", 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.968e-06 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902694+0000", 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085420719999999995 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903482+0000", 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078797999999999997 2024-04-25T15:53:24.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903499+0000", 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7082e-05 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903593+0000", 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4452999999999998e-05 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903774+0000", 2024-04-25T15:53:24.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180803 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903953+0000", 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179106 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.965763+0000", 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061810057000000002 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977390+0000", 2024-04-25T15:53:24.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01162697 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977474+0000", 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3812000000000003e-05 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977543+0000", 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8755000000000002e-05 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:652 3.36 3:6c98fff9:::100000001f7.00000088:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.55026580199999997, 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065992254, 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 652 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902707+0000", 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5209999999999999e-06 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910035+0000", 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073280000000000003 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910344+0000", 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030890699999999998 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910417+0000", 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3900999999999996e-05 2024-04-25T15:53:24.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910547+0000", 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012940400000000001 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910876+0000", 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000329181 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911062+0000", 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018587000000000001 2024-04-25T15:53:24.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966069+0000", 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.055007038000000001 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968579+0000", 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002509598 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968649+0000", 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0189000000000002e-05 2024-04-25T15:53:24.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968697+0000", 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8644999999999999e-05 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:660 3.26 3:65a7570a:::100000001f7.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.54292242300000004, 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058617787999999997, 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.529 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 660 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910051+0000", 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7599999999999998e-06 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925001+0000", 2024-04-25T15:53:24.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014949405000000001 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925210+0000", 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020949799999999999 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925223+0000", 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3086e-05 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925337+0000", 2024-04-25T15:53:24.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011333899999999999 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925525+0000", 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018789300000000001 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925638+0000", 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011354 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967535+0000", 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041897074999999999 2024-04-25T15:53:24.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968553+0000", 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010179480000000001 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968609+0000", 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5653999999999999e-05 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968666+0000", 2024-04-25T15:53:24.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7590000000000003e-05 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:675 3.a 3:5122fcae:::100000001f7.0000009f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.52796057600000001, 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052715010999999999, 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 675 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925013+0000", 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3089999999999998e-06 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929333+0000", 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043198170000000001 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929365+0000", 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2323000000000002e-05 2024-04-25T15:53:24.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929380+0000", 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5421999999999999e-05 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929460+0000", 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0148000000000006e-05 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929572+0000", 2024-04-25T15:53:24.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011171 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929688+0000", 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011547 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966801+0000", 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037113896 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977633+0000", 2024-04-25T15:53:24.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010831294999999999 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977681+0000", 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7926000000000002e-05 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977725+0000", 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4694999999999998e-05 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:697 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.075373+0000", 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.37759759199999998, 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028709926, 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 697 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.075373+0000", 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.075373+0000", 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.075376+0000", 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.112e-06 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.080690+0000", 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053133859999999998 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.080738+0000", 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8201999999999997e-05 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.080757+0000", 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8944e-05 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.080856+0000", 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9047000000000002e-05 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.080995+0000", 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000138777 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.081116+0000", 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121242 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.086601+0000", 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005484895 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.103941+0000", 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017340518999999999 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.104007+0000", 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5753999999999994e-05 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.104083+0000", 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6048000000000001e-05 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:702 3.20 3:062e7a7c:::100000001f7.000000ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.106239+0000", 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.34673158399999998, 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029329181999999999, 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.546 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 702 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.106239+0000", 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.106239+0000", 2024-04-25T15:53:24.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.106243+0000", 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0409999999999999e-06 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.111436+0000", 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051922899999999996 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.111472+0000", 2024-04-25T15:53:24.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6285000000000002e-05 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.111485+0000", 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2604e-05 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.111572+0000", 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7700000000000004e-05 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.111756+0000", 2024-04-25T15:53:24.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018371199999999999 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.111940+0000", 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018357399999999999 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.122211+0000", 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010271568999999999 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.135477+0000", 2024-04-25T15:53:24.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013265992000000001 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.135526+0000", 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8665999999999998e-05 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.135569+0000", 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2749e-05 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.551 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:721 3.36 3:6f382920:::100000001f7.000000cd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.230574+0000", 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.22239729899999999, 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026554219, 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 721 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230574+0000", 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230574+0000", 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230578+0000", 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0119999999999999e-06 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.236919+0000", 2024-04-25T15:53:24.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063410050000000003 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.236998+0000", 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8943999999999995e-05 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.237036+0000", 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7963000000000002e-05 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.237124+0000", 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8110999999999997e-05 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.237238+0000", 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114695 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.237342+0000", 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103853 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.253152+0000", 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015810076999999999 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.256942+0000", 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037900630000000002 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.257049+0000", 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106613 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.257128+0000", 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8882999999999997e-05 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:731 3.35 3:ad5d525e:::100000001f7.000000d7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.292933+0000", 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.16003809599999999, 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029493505, 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.557 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 731 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292933+0000", 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292933+0000", 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292936+0000", 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7599999999999998e-06 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298756+0000", 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058200919999999998 2024-04-25T15:53:24.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298797+0000", 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1221999999999999e-05 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298815+0000", 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8338999999999998e-05 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298928+0000", 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112321 2024-04-25T15:53:24.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.299152+0000", 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022447400000000001 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.299340+0000", 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000188087 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.318980+0000", 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019639997999999999 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.322285+0000", 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033045779999999999 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.322361+0000", 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6544000000000004e-05 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.322426+0000", 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5090000000000002e-05 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:742 3.a 3:528c53b9:::100000001f7.000000e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.362254+0000", 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.090717409999999998, 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025499542, 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:24.563 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 742 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.362254+0000", 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.362254+0000", 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:24.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.362260+0000", 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2349999999999996e-06 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.367220+0000", 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049599309999999999 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.367253+0000", 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3309999999999998e-05 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.367268+0000", 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4734e-05 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.367337+0000", 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9597999999999993e-05 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.367471+0000", 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000133986 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.367600+0000", 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012841099999999999 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.386574+0000", 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018974184000000002 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.387601+0000", 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010275359999999999 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.387685+0000", 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3697999999999997e-05 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.387753+0000", 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7918999999999998e-05 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.568 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.569 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.569 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:24.569 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:24.569 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:24.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:24.570 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:24.570+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:24.602 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:24.602 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 initial writers = 546704.25 kB/sec 2024-04-25T15:53:24.602 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 initial writers = 546544.67 kB/sec 2024-04-25T15:53:24.602 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 546704.25 kB/sec 2024-04-25T15:53:24.602 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 546704.25 kB/sec 2024-04-25T15:53:24.602 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 546704.25 kB/sec 2024-04-25T15:53:24.602 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:53:24.671 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:24.667+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:24.750 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:24.750 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:24.751 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:24.751 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:24.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:24.771 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:24.770+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:24.872 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:24.870+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:24.954 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:24.954 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:24.954 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:24.954 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:24.955 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:24.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:24.972 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:24.971+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:25.074 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:25.074+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:25.120 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.120 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 186.01770264300001, 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:25.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:25.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:25.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:25.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.898462162000001, 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:25.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.131 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.803828110000001, 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.132 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:25.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:25.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:595 3.e 3:7000a0d4:::100000001f7.0000004f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.355581+0000", 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.760586078, 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036677335999999998, 2024-04-25T15:53:25.137 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 595 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.355581+0000", 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.355581+0000", 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.355584+0000", 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7979999999999999e-06 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.359983+0000", 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043987560000000002 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.360061+0000", 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7455999999999998e-05 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.360088+0000", 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7053e-05 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.360209+0000", 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012146000000000001 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.360340+0000", 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130708 2024-04-25T15:53:25.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.360457+0000", 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116903 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373892+0000", 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013435701 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.392084+0000", 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018191217999999999 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.392174+0000", 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0303999999999995e-05 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.392258+0000", 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3979000000000004e-05 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:597 3.e 3:72b88a04:::100000001f7.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.7480501880000001, 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045014089, 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 597 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:25.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368119+0000", 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.311e-06 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372689+0000", 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004569666 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372724+0000", 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5407000000000002e-05 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372735+0000", 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1528e-05 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373941+0000", 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001205788 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374137+0000", 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195725 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374337+0000", 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020006099999999999 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412835+0000", 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038497525999999997 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413004+0000", 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016924900000000001 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413078+0000", 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4159000000000006e-05 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413131+0000", 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2669e-05 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:596 3.16 e81/18 3:6941c83d:::100000001f7.00000050:head v 81'12, mlcod=81'9)", 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.7474637469999998, 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042032427999999997, 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368705+0000", 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8419999999999999e-06 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.371969+0000", 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032644509999999998 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372007+0000", 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7419000000000002e-05 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372032+0000", 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4905999999999998e-05 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372147+0000", 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115716 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372181+0000", 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3578999999999998e-05 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410659+0000", 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038477931 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410735+0000", 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6583999999999997e-05 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:598 3.11 3:89a80ce8:::100000001f7.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.742836276, 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045571093999999999, 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 598 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373332+0000", 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.035e-06 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379271+0000", 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059388929999999998 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379309+0000", 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7287999999999999e-05 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379327+0000", 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8508e-05 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379451+0000", 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012354999999999999 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379662+0000", 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000211277 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379850+0000", 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018827000000000001 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413014+0000", 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033163661999999997 2024-04-25T15:53:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418816+0000", 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058018840000000002 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418863+0000", 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6910000000000003e-05 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418902+0000", 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8816999999999998e-05 2024-04-25T15:53:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.533150158, 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:53:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:53:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:53:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:53:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:635 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.3023636380000001, 2024-04-25T15:53:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044093436999999999, 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 635 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813805+0000", 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.66e-06 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832121+0000", 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018315765000000001 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832272+0000", 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015122099999999999 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832299+0000", 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7013e-05 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832408+0000", 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109121 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832613+0000", 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020534899999999999 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832814+0000", 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020120399999999999 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.848197+0000", 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015382290999999999 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857706+0000", 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095096690000000001 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857804+0000", 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8047999999999994e-05 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857897+0000", 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2095999999999994e-05 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:654 3.11 3:892e96a3:::100000001f7.0000008a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2755895609999999, 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061060665, 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 654 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840579+0000", 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.894e-06 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849327+0000", 2024-04-25T15:53:25.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087475109999999995 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851420+0000", 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020932490000000002 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851444+0000", 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4652999999999999e-05 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851555+0000", 2024-04-25T15:53:25.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110671 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851751+0000", 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195921 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851947+0000", 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019585699999999999 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877957+0000", 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02600968 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901479+0000", 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023522661 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901570+0000", 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1168000000000005e-05 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901638+0000", 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7399999999999998e-05 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:663 3.d 3:b1139e77:::100000001f7.00000093:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2668249039999999, 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042145086999999998, 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.173 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 663 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849344+0000", 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854735+0000", 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053907210000000002 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857691+0000", 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002955877 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857704+0000", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2668e-05 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857835+0000", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131786 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858065+0000", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022928099999999999 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858248+0000", 2024-04-25T15:53:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183124 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891161+0000", 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032912917 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891323+0000", 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016235699999999999 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891410+0000", 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7183999999999997e-05 2024-04-25T15:53:25.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891487+0000", 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6607999999999996e-05 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:664 3.d 3:b353c7f2:::100000001f7.00000094:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2614172670000001, 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079861626000000005, 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.178 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 664 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854752+0000", 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.762e-06 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865333+0000", 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581248999999999 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865758+0000", 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042434800000000001 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865779+0000", 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1209999999999999e-05 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868028+0000", 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022485339999999999 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868222+0000", 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000193986 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868340+0000", 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118163 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920976+0000", 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052636792000000002 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934415+0000", 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013438662000000001 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934516+0000", 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010058199999999999 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934611+0000", 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5338000000000001e-05 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:666 3.15 3:a92ac213:::100000001f7.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.183 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.250818792, 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10217435, 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 666 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:25.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865350+0000", 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9360000000000002e-06 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871962+0000", 2024-04-25T15:53:25.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006612349 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872932+0000", 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097010200000000003 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872950+0000", 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7193999999999998e-05 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873133+0000", 2024-04-25T15:53:25.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183265 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873494+0000", 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00036074300000000001 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873747+0000", 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025347100000000001 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920781+0000", 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047033951999999997 2024-04-25T15:53:25.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967388+0000", 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046606748000000003 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967458+0000", 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999994e-05 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967522+0000", 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4250000000000003e-05 2024-04-25T15:53:25.188 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:667 3.2d 3:b6615a40:::100000001f7.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2441888800000001, 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095661866999999998, 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.189 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 667 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871981+0000", 2024-04-25T15:53:25.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0759999999999999e-06 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877165+0000", 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051845709999999998 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878247+0000", 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001081448 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878262+0000", 2024-04-25T15:53:25.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5196e-05 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885690+0000", 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074279159999999997 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886266+0000", 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057628900000000001 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886495+0000", 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022871600000000001 2024-04-25T15:53:25.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910525+0000", 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024029452 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967541+0000", 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.057016940000000002 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967590+0000", 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8585999999999999e-05 2024-04-25T15:53:25.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967640+0000", 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9676999999999999e-05 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:670 3.8 e81/68 3:1323134e:::100000001f7.0000009a:head v 81'17, mlcod=80'15)", 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2390466390000001, 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090558897999999999, 2024-04-25T15:53:25.194 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877122+0000", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.883982+0000", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068602339999999998 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884003+0000", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0418999999999999e-05 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884024+0000", 2024-04-25T15:53:25.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1225000000000001e-05 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884154+0000", 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977099999999999 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884218+0000", 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3830000000000004e-05 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967623+0000", 2024-04-25T15:53:25.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083405807999999998 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967679+0000", 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5559000000000001e-05 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:669 3.f 3:f0675d7f:::100000001f7.00000099:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.238991033, 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090729281999999994, 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.198 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 669 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877178+0000", 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.88e-06 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884938+0000", 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077602569999999996 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884975+0000", 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7157999999999999e-05 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884991+0000", 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5637999999999998e-05 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885075+0000", 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4696000000000004e-05 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885226+0000", 2024-04-25T15:53:25.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015028799999999999 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885331+0000", 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010580400000000001 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967662+0000", 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.082330667999999996 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967792+0000", 2024-04-25T15:53:25.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130259 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967861+0000", 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9002000000000001e-05 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967905+0000", 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3631999999999998e-05 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:660 3.26 e81/18 3:65a7570a:::100000001f7.00000090:head v 81'20, mlcod=70'18)", 2024-04-25T15:53:25.203 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.1903739339999999, 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042555325999999997, 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925794+0000", 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4950000000000001e-06 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.927977+0000", 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002183192 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928011+0000", 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3037000000000002e-05 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928031+0000", 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0863999999999999e-05 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928165+0000", 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013332700000000001 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928204+0000", 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.888e-05 2024-04-25T15:53:25.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968300+0000", 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040096738999999999 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968348+0000", 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7791999999999999e-05 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:732 3.15 3:aadcba4a:::100000001f7.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.81723226500000001, 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.064849247999999998, 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 732 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298937+0000", 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8930000000000001e-06 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306019+0000", 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070820759999999996 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306065+0000", 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5235999999999999e-05 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306086+0000", 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0883e-05 2024-04-25T15:53:25.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306275+0000", 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018911500000000001 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306470+0000", 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019556900000000001 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306646+0000", 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017527999999999999 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319323+0000", 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012677034 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363674+0000", 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044351773999999997 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363724+0000", 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0016999999999999e-05 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363784+0000", 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9370999999999998e-05 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:745 3.d 3:b225ffa8:::100000001f7.000000e5:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:25.213 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.72698670499999996, 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047290008000000001, 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 745 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:25.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389181+0000", 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1500000000000003e-07 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401220+0000", 2024-04-25T15:53:25.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012039525000000001 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401248+0000", 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7922000000000001e-05 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401261+0000", 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2224e-05 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401480+0000", 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021909300000000001 2024-04-25T15:53:25.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401643+0000", 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016358 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401799+0000", 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015555699999999999 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.412210+0000", 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010410777 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436343+0000", 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024133884000000001 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436416+0000", 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2268000000000007e-05 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436470+0000", 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4262999999999999e-05 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:761 3.3b 3:df080f3c:::100000001f7.000000f5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.60846256600000004, 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045918457000000003, 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 761 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507707+0000", 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.717e-06 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:25.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.524752+0000", 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017044996999999999 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525517+0000", 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076479300000000005 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525526+0000", 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3829999999999998e-06 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525610+0000", 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3758999999999995e-05 2024-04-25T15:53:25.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525797+0000", 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187302 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525954+0000", 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015707000000000001 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.536701+0000", 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01074656 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553463+0000", 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016761804000000002 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553540+0000", 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7026000000000004e-05 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553623+0000", 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3046000000000004e-05 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.223 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.224 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:25.224 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:25.224 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:25.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:53:25.225 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:25.223+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:25.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:25.327 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:25.326+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:25.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:25.378 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:25.378 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:25.378 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:25.378 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:25.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.382+0000 7f481c901640 1 Processor -- start 2024-04-25T15:53:25.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f481c901640 1 -- start start 2024-04-25T15:53:25.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f481c901640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f481c901640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4818071b10 0x7f4818071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f481c901640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481815d110 con 0x7f481815c880 2024-04-25T15:53:25.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f481c901640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f48180cac50 con 0x7f4818071b10 2024-04-25T15:53:25.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f481c901640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4818157010 con 0x7f4818156c00 2024-04-25T15:53:25.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f4816d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4818071b10 0x7f4818071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f480ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f481815c880 0x7f48180ca540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39454/0 (socket says 172.21.15.67:39454) 2024-04-25T15:53:25.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f480ffff640 1 -- 172.21.15.67:0/647079020 learned_addr learned my addr 172.21.15.67:0/647079020 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:25.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f481659c640 1 --2- 172.21.15.67:0/647079020 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f4816d9d640 1 -- 172.21.15.67:0/647079020 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 msgr2=0x7f48181608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:25.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f4816d9d640 1 --2- 172.21.15.67:0/647079020 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f4816d9d640 1 -- 172.21.15.67:0/647079020 >> v1:172.21.15.67:6789/0 conn(0x7f481815c880 legacy=0x7f48180ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.386+0000 7f4816d9d640 1 -- 172.21.15.67:0/647079020 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481815d3f0 con 0x7f4818071b10 2024-04-25T15:53:25.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f4816d9d640 1 --2- 172.21.15.67:0/647079020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818071f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4810009d90 tx=0x7f481001c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e6fee17136e428 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f4815d9b640 1 -- 172.21.15.67:0/647079020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48100b2070 con 0x7f4818071b10 2024-04-25T15:53:25.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f4815d9b640 1 -- 172.21.15.67:0/647079020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48100a5820 con 0x7f4818071b10 2024-04-25T15:53:25.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f4815d9b640 1 -- 172.21.15.67:0/647079020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48100a5bf0 con 0x7f4818071b10 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 -- 172.21.15.67:0/647079020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 msgr2=0x7f4818071f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 --2- 172.21.15.67:0/647079020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818071f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4810009d90 tx=0x7f481001c800 comp rx=0 tx=0).stop 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 -- 172.21.15.67:0/647079020 shutdown_connections 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 --2- 172.21.15.67:0/647079020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818071f20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 --2- 172.21.15.67:0/647079020 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 -- 172.21.15.67:0/647079020 >> 172.21.15.67:0/647079020 conn(0x7f48180700b0 msgr2=0x7f48180704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 -- 172.21.15.67:0/647079020 shutdown_connections 2024-04-25T15:53:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.390+0000 7f481c901640 1 -- 172.21.15.67:0/647079020 wait complete. 2024-04-25T15:53:25.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 Processor -- start 2024-04-25T15:53:25.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 -- start start 2024-04-25T15:53:25.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181e7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f481815c880 0x7f481814f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4818160410 con 0x7f4818071b10 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f481815fe50 con 0x7f481815c880 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4818160de0 con 0x7f4818156c00 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40570/0 (socket says 172.21.15.67:40570) 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 -- 172.21.15.67:0/1223801886 learned_addr learned my addr 172.21.15.67:0/1223801886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f480ffff640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181e7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f4816d9d640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f481815c880 0x7f481814f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 -- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 msgr2=0x7f48181e7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181e7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 -- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f481815c880 msgr2=0x7f481814f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f481815c880 0x7f481814f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 -- 172.21.15.67:0/1223801886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48181e7a60 con 0x7f4818071b10 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f4816d9d640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f481815c880 0x7f481814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481659c640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818151220 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f480800d5f0 tx=0x7f480800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f480effd640 1 -- 172.21.15.67:0/1223801886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48080125f0 con 0x7f4818071b10 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f480effd640 1 -- 172.21.15.67:0/1223801886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4808019070 con 0x7f4818071b10 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f480effd640 1 -- 172.21.15.67:0/1223801886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48080129b0 con 0x7f4818071b10 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48181e6d60 con 0x7f4818071b10 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.394+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f481815a510 con 0x7f4818071b10 2024-04-25T15:53:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.398+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48181e7c30 con 0x7f4818071b10 2024-04-25T15:53:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.398+0000 7f480effd640 1 -- 172.21.15.67:0/1223801886 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4808012b50 con 0x7f4818071b10 2024-04-25T15:53:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.398+0000 7f480effd640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f47f806f400 0x7f47f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.398+0000 7f480effd640 1 -- 172.21.15.67:0/1223801886 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5915+0+0 (secure 0 0 0) 0x7f4808090140 con 0x7f4818071b10 2024-04-25T15:53:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.398+0000 7f480ffff640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f47f806f400 0x7f47f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.398+0000 7f480ffff640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f47f806f400 0x7f47f8071910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f48180c9fb0 tx=0x7f480000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.402+0000 7f480effd640 1 -- 172.21.15.67:0/1223801886 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f480805d160 con 0x7f4818071b10 2024-04-25T15:53:25.427 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:25.427+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.498+0000 7f08e36b6640 1 Processor -- start 2024-04-25T15:53:25.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.498+0000 7f08e36b6640 1 -- start start 2024-04-25T15:53:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e36b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 0x7f08dc072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e36b6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f08dc0c96e0 0x7f08dc073300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e36b6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08dc06d8e0 con 0x7f08dc160800 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e36b6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f08dc06dbc0 con 0x7f08dc0c96e0 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e36b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f08dc073c70 con 0x7f08dc0c92d0 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e1452640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f08dc160800 0x7f08dc06d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39456/0 (socket says 172.21.15.67:39456) 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e1452640 1 -- 172.21.15.67:0/2368745561 learned_addr learned my addr 172.21.15.67:0/2368745561 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 0x7f08dc072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e1c53640 1 --2- 172.21.15.67:0/2368745561 >> v2:172.21.15.176:3300/0 conn(0x7f08dc0c96e0 0x7f08dc073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4128890750 0 0) 0x7f08dc06d8e0 con 0x7f08dc160800 2024-04-25T15:53:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08c8003530 con 0x7f08dc160800 2024-04-25T15:53:25.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2977758334 0 0) 0x7f08c8003530 con 0x7f08dc160800 2024-04-25T15:53:25.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 msgr2=0x7f08dc072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 0x7f08dc072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 >> v2:172.21.15.176:3300/0 conn(0x7f08dc0c96e0 msgr2=0x7f08dc073300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 --2- 172.21.15.67:0/2368745561 >> v2:172.21.15.176:3300/0 conn(0x7f08dc0c96e0 0x7f08dc073300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc073f50 con 0x7f08dc160800 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f08d8001680 con 0x7f08dc160800 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 0x7f08dc072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 >> v1:172.21.15.67:6789/0 conn(0x7f08dc160800 legacy=0x7f08dc06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c80039c0 0x7f08c8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08c8004810 0x7f08c8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 0x7f08c8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc06dbc0 con 0x7f08c8004810 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc073c70 con 0x7f08c80039c0 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f08c800ba80 con 0x7f08dc0c92d0 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08c8004810 0x7f08c8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e1c53640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 0x7f08c8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e1452640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c80039c0 0x7f08c8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 -- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 msgr2=0x7f08c8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc0c92d0 0x7f08c8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 -- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c80039c0 msgr2=0x7f08c8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c80039c0 0x7f08c8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e0c51640 1 -- 172.21.15.67:0/2368745561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c8003530 con 0x7f08c8004810 2024-04-25T15:53:25.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.502+0000 7f08e1452640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c80039c0 0x7f08c8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e0c51640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08c8004810 0x7f08c8004c20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009cb0 tx=0x7f08cc01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ff8f8eea596e7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc0b2070 con 0x7f08c8004810 2024-04-25T15:53:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08cc0a56d0 con 0x7f08c8004810 2024-04-25T15:53:25.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08d3fff640 1 -- 172.21.15.67:0/2368745561 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc009880 con 0x7f08c8004810 2024-04-25T15:53:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e36b6640 1 -- 172.21.15.67:0/2368745561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08c8004810 msgr2=0x7f08c8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e36b6640 1 --2- 172.21.15.67:0/2368745561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08c8004810 0x7f08c8004c20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f08cc009cb0 tx=0x7f08cc01cc30 comp rx=0 tx=0).stop 2024-04-25T15:53:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e1c53640 1 -- 172.21.15.67:0/2368745561 reap_dead start 2024-04-25T15:53:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e36b6640 1 -- 172.21.15.67:0/2368745561 shutdown_connections 2024-04-25T15:53:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e36b6640 1 -- 172.21.15.67:0/2368745561 >> 172.21.15.67:0/2368745561 conn(0x7f08dc070db0 msgr2=0x7f08dc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e36b6640 1 -- 172.21.15.67:0/2368745561 shutdown_connections 2024-04-25T15:53:25.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.506+0000 7f08e36b6640 1 -- 172.21.15.67:0/2368745561 wait complete. 2024-04-25T15:53:25.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 Processor -- start 2024-04-25T15:53:25.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 -- start start 2024-04-25T15:53:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c8004810 0x7f08dc1f9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08dc15f800 0x7f08dc0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc15f0a0 0x7f08dc14a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc160d90 con 0x7f08dc15f800 2024-04-25T15:53:25.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc160c10 con 0x7f08c8004810 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f08dc167990 con 0x7f08dc15f0a0 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08dc15f800 0x7f08dc0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08dc15f800 0x7f08dc0cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40600/0 (socket says 172.21.15.67:40600) 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 -- 172.21.15.67:0/1086199791 learned_addr learned my addr 172.21.15.67:0/1086199791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c8004810 0x7f08dc1f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc15f0a0 0x7f08dc14a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc15f0a0 msgr2=0x7f08dc14a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc15f0a0 0x7f08dc14a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c8004810 msgr2=0x7f08dc1f9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08c8004810 0x7f08dc1f9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc15a400 con 0x7f08dc15f800 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f08dc15f0a0 0x7f08dc14a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e0c51640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08dc15f800 0x7f08dc0cda10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f08cc01ca60 tx=0x7f08c8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc0b2040 con 0x7f08dc15f800 2024-04-25T15:53:25.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08cc009400 con 0x7f08dc15f800 2024-04-25T15:53:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08cc009880 con 0x7f08dc15f800 2024-04-25T15:53:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08dc161090 con 0x7f08dc15f800 2024-04-25T15:53:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.510+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08dc1f6d10 con 0x7f08dc15f800 2024-04-25T15:53:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f08dc06a480 con 0x7f08dc15f800 2024-04-25T15:53:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f08cc0c0030 con 0x7f08dc15f800 2024-04-25T15:53:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08d1ffb640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f08c006f400 0x7f08c0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5915+0+0 (secure 0 0 0) 0x7f08cc128a20 con 0x7f08dc15f800 2024-04-25T15:53:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v81) ==== 54+0+16 (secure 0 0 0) 0x7f08cc016940 con 0x7f08dc15f800 2024-04-25T15:53:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f08c006f400 0x7f08c0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f08dc154ec0 0x7f08dc162070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08dc161370 con 0x7f08dc154ec0 2024-04-25T15:53:25.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f08dc154ec0 0x7f08dc162070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f08dc154ec0 0x7f08dc162070 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.514+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f08c006f400 0x7f08c0071910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f08d8001930 tx=0x7f08d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.518+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08d4009dd0 con 0x7f08dc154ec0 2024-04-25T15:53:25.528 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:25.527+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:25.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.542+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc15ee80 con 0x7f08dc154ec0 2024-04-25T15:53:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.546+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc15ee80 con 0x7f08dc154ec0 2024-04-25T15:53:25.546 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:25.547 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:25.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.546+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f08dc162ac0 0x7f08dc063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.546+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f08dc162ed0 con 0x7f08dc162ac0 2024-04-25T15:53:25.547 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:25.547 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:25.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.546+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f08dc162ac0 0x7f08dc063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.546+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f08dc162ac0 0x7f08dc063090 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.546+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08dc162ed0 con 0x7f08dc162ac0 2024-04-25T15:53:25.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.562+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc0c8c50 con 0x7f08dc162ac0 2024-04-25T15:53:25.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:25.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.562+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc0c8c50 con 0x7f08dc162ac0 2024-04-25T15:53:25.564 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.562+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f08dc063ae0 0x7f08dc14b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.562+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f08dc0ca790 con 0x7f08dc063ae0 2024-04-25T15:53:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.562+0000 7f08e0c51640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f08dc063ae0 0x7f08dc14b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.562+0000 7f08e0c51640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f08dc063ae0 0x7f08dc14b220 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.562+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08dc0ca790 con 0x7f08dc063ae0 2024-04-25T15:53:25.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.578+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc06d1b0 con 0x7f08dc063ae0 2024-04-25T15:53:25.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.578+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc06d1b0 con 0x7f08dc063ae0 2024-04-25T15:53:25.580 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:25.581 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:25.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.578+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f08dc14bc70 0x7f08dc02c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.578+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f08dc14c080 con 0x7f08dc14bc70 2024-04-25T15:53:25.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.578+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f08dc14bc70 0x7f08dc02c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.582+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f08dc14bc70 0x7f08dc02c9e0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.582+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08dc14c080 con 0x7f08dc14bc70 2024-04-25T15:53:25.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.582+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f481806a480 con 0x7f4818071b10 2024-04-25T15:53:25.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.594+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc14c080 con 0x7f08dc14bc70 2024-04-25T15:53:25.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.594+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc14c080 con 0x7f08dc14bc70 2024-04-25T15:53:25.597 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:25.597 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:25.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.594+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f08dc02d430 0x7f08dc02d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.594+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f08dc14c080 con 0x7f08dc02d430 2024-04-25T15:53:25.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.598+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f08dc02d430 0x7f08dc02d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.598+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f08dc02d430 0x7f08dc02d840 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.598+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08dc14c080 con 0x7f08dc02d430 2024-04-25T15:53:25.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.614+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc14c080 con 0x7f08dc02d430 2024-04-25T15:53:25.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.618+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc14c080 con 0x7f08dc02d430 2024-04-25T15:53:25.618 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:25.619 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:25.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.618+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f08dc20ad40 0x7f08dc20d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.618+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f08dc02dd80 con 0x7f08dc20ad40 2024-04-25T15:53:25.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.618+0000 7f08e0c51640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f08dc20ad40 0x7f08dc20d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.618+0000 7f08e0c51640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f08dc20ad40 0x7f08dc20d160 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.622+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08dc02dd80 con 0x7f08dc20ad40 2024-04-25T15:53:25.629 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:25.630+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:25.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.634+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc02dd80 con 0x7f08dc20ad40 2024-04-25T15:53:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.638+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc02dd80 con 0x7f08dc20ad40 2024-04-25T15:53:25.639 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:25.640 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:25.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.638+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f08dc20dbb0 0x7f08dc20ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.638+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f08dc02def0 con 0x7f08dc20dbb0 2024-04-25T15:53:25.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.638+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f08dc20dbb0 0x7f08dc20ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.638+0000 7f08e1c53640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f08dc20dbb0 0x7f08dc20ffd0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.642+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08dc02def0 con 0x7f08dc20dbb0 2024-04-25T15:53:25.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.654+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc02def0 con 0x7f08dc20dbb0 2024-04-25T15:53:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.658+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc02def0 con 0x7f08dc20dbb0 2024-04-25T15:53:25.660 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.658+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f08dc210a20 0x7f08dc212e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.658+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f08dc215d60 con 0x7f08dc210a20 2024-04-25T15:53:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.662+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f08dc210a20 0x7f08dc212e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.662+0000 7f08e1452640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f08dc210a20 0x7f08dc212e40 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.662+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08dc215d60 con 0x7f08dc210a20 2024-04-25T15:53:25.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.674+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f08dc215d30 con 0x7f08dc210a20 2024-04-25T15:53:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08d1ffb640 1 -- 172.21.15.67:0/1086199791 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f08dc215d30 con 0x7f08dc210a20 2024-04-25T15:53:25.681 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f08dc154ec0 msgr2=0x7f08dc162070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f08dc154ec0 0x7f08dc162070 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f08dc162ac0 msgr2=0x7f08dc063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f08dc162ac0 0x7f08dc063090 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f08dc063ae0 msgr2=0x7f08dc14b220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f08dc063ae0 0x7f08dc14b220 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f08dc14bc70 msgr2=0x7f08dc02c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f08dc14bc70 0x7f08dc02c9e0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f08dc02d430 msgr2=0x7f08dc02d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e1c53640 1 -- 172.21.15.67:0/1086199791 reap_dead start 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f08dc02d430 0x7f08dc02d840 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e1c53640 1 -- 172.21.15.67:0/1086199791 reap_dead start 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f08dc20ad40 msgr2=0x7f08dc20d160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f08dc20ad40 0x7f08dc20d160 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f08dc20dbb0 msgr2=0x7f08dc20ffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f08dc20dbb0 0x7f08dc20ffd0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f08dc210a20 msgr2=0x7f08dc212e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f08dc210a20 0x7f08dc212e40 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f08c006f400 msgr2=0x7f08c0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f08c006f400 0x7f08c0071910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f08d8001930 tx=0x7f08d8009040 comp rx=0 tx=0).stop 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e1c53640 1 -- 172.21.15.67:0/1086199791 reap_dead start 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08dc15f800 msgr2=0x7f08dc0cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08dc15f800 0x7f08dc0cda10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f08cc01ca60 tx=0x7f08c8005840 comp rx=0 tx=0).stop 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 shutdown_connections 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 --2- 172.21.15.67:0/1086199791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08dc15f800 0x7f08dc0cda10 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 >> 172.21.15.67:0/1086199791 conn(0x7f08dc070db0 msgr2=0x7f08dc155e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 shutdown_connections 2024-04-25T15:53:25.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:25.682+0000 7f08e36b6640 1 -- 172.21.15.67:0/1086199791 wait complete. 2024-04-25T15:53:25.697 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:25.697 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:25.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:25.753 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:25.754+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:25.832 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:25.830+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:25.898 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:25.898 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:25.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:25.957 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:25.958+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:26.034 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:26.034+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:26.110 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:26.110 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.118+0000 7f480effd640 1 -- 172.21.15.67:0/1223801886 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v82) ==== 101+0+0 (secure 0 0 0) 0x7f480805ca60 con 0x7f4818071b10 2024-04-25T15:53:26.121 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f47f806f400 msgr2=0x7f47f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f47f806f400 0x7f47f8071910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f48180c9fb0 tx=0x7f480000b040 comp rx=0 tx=0).stop 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 msgr2=0x7f4818151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818151220 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f480800d5f0 tx=0x7f480800dab0 comp rx=0 tx=0).stop 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 shutdown_connections 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f47f806f400 0x7f47f8071910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f481815c880 0x7f481814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4818156c00 0x7f48181e7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 --2- 172.21.15.67:0/1223801886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4818071b10 0x7f4818151220 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:26.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 >> 172.21.15.67:0/1223801886 conn(0x7f48180700b0 msgr2=0x7f48180704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:26.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 shutdown_connections 2024-04-25T15:53:26.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:26.122+0000 7f481c901640 1 -- 172.21.15.67:0/1223801886 wait complete. 2024-04-25T15:53:26.128 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:26.148 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:26.146+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:26.237 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:26.238+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:26.338 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:26.338+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:26.340 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:26.340 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:26.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:26.445 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:26.446+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:26.540 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:26.538+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:26.549 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:26.549 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:26.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:26.641 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:26.639+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:26.742 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:26.742+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:26.756 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:26.756 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:26.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:26.861 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:26.862+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:26.944 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:26.943+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:26.973 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:26.973 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:26.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:27.046 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:27.043+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:27.108 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:27.109 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:27.109 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-04-25T15:53:27.109 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:53:27.109 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:53:27.109 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-04-25T15:53:27.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:27.146+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:27.214 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:27.214 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:27.247 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:27.246+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:27.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:27.314 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:27.314 INFO:teuthology.orchestra.run.smithi176.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T15:53:27.314 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:27.348 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:27.346+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:27.449 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:27.447+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:27.523 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:27.523 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:27.550 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:27.547+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:27.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:27.650 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:27.650+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:27.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:27.753 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:27.751+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:27.845 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:27.845 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:27.852 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:27.850+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:27.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:27.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.890+0000 7fd99d18a640 1 Processor -- start 2024-04-25T15:53:27.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.890+0000 7fd99d18a640 1 -- start start 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99d18a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd998153730 0x7fd99806f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99d18a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd99806fc50 0x7fd998164060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99d18a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9981597f0 con 0x7fd998071f00 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99d18a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd9980c2e10 con 0x7fd99806fc50 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99d18a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd998159b20 con 0x7fd998153730 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd99806fc50 0x7fd998164060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd99806fc50 0x7fd998164060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40616/0 (socket says 172.21.15.67:40616) 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99759e640 1 -- 172.21.15.67:0/416532566 learned_addr learned my addr 172.21.15.67:0/416532566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99759e640 1 -- 172.21.15.67:0/416532566 >> v2:172.21.15.176:3300/0 conn(0x7fd998153730 msgr2=0x7fd99806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99759e640 1 --2- 172.21.15.67:0/416532566 >> v2:172.21.15.176:3300/0 conn(0x7fd998153730 0x7fd99806f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99759e640 1 -- 172.21.15.67:0/416532566 >> v1:172.21.15.176:6789/0 conn(0x7fd998071f00 legacy=0x7fd998159fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:53:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.894+0000 7fd99759e640 1 -- 172.21.15.67:0/416532566 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9980c9cd0 con 0x7fd99806fc50 2024-04-25T15:53:27.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99759e640 1 --2- 172.21.15.67:0/416532566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd99806fc50 0x7fd998164060 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd998071d00 tx=0x7fd99001f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293fd80097fd5f41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:27.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd995d9b640 1 -- 172.21.15.67:0/416532566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9900b4070 con 0x7fd99806fc50 2024-04-25T15:53:27.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd995d9b640 1 -- 172.21.15.67:0/416532566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9900af400 con 0x7fd99806fc50 2024-04-25T15:53:27.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd995d9b640 1 -- 172.21.15.67:0/416532566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9900af7d0 con 0x7fd99806fc50 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 -- 172.21.15.67:0/416532566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd99806fc50 msgr2=0x7fd998164060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 --2- 172.21.15.67:0/416532566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd99806fc50 0x7fd998164060 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd998071d00 tx=0x7fd99001f1a0 comp rx=0 tx=0).stop 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 -- 172.21.15.67:0/416532566 shutdown_connections 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 --2- 172.21.15.67:0/416532566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd99806fc50 0x7fd998164060 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 --2- 172.21.15.67:0/416532566 >> v2:172.21.15.176:3300/0 conn(0x7fd998153730 0x7fd99806f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 -- 172.21.15.67:0/416532566 >> 172.21.15.67:0/416532566 conn(0x7fd9980cdd60 msgr2=0x7fd9980ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 -- 172.21.15.67:0/416532566 shutdown_connections 2024-04-25T15:53:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.898+0000 7fd99d18a640 1 -- 172.21.15.67:0/416532566 wait complete. 2024-04-25T15:53:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 Processor -- start 2024-04-25T15:53:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 -- start start 2024-04-25T15:53:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd99806fc50 0x7fd9980c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd998071f00 0x7fd9980c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd998153730 0x7fd9980c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd99815a6c0 con 0x7fd998153730 2024-04-25T15:53:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd9980ca230 con 0x7fd998071f00 2024-04-25T15:53:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99d18a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd9980c3340 con 0x7fd99806fc50 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd996d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd99806fc50 0x7fd9980c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd996d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd99806fc50 0x7fd9980c6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:56472/0 (socket says 172.21.15.67:56472) 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd996d9d640 1 -- 172.21.15.67:0/961893763 learned_addr learned my addr 172.21.15.67:0/961893763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.902+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd998071f00 0x7fd9980c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd996d9d640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd998071f00 msgr2=0x7fd9980c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd998071f00 0x7fd9980c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd996d9d640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd998153730 msgr2=0x7fd9980c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd998153730 0x7fd9980c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd996d9d640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9980c6770 con 0x7fd99806fc50 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd998071f00 0x7fd9980c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:53:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd99806fc50 0x7fd9980c6230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd98800b3f0 tx=0x7fd98800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988015020 con 0x7fd99806fc50 2024-04-25T15:53:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd988011850 con 0x7fd99806fc50 2024-04-25T15:53:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988011c40 con 0x7fd99806fc50 2024-04-25T15:53:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd99d18a640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9980c7200 con 0x7fd99806fc50 2024-04-25T15:53:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.906+0000 7fd99d18a640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99815b700 con 0x7fd99806fc50 2024-04-25T15:53:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd988018020 con 0x7fd99806fc50 2024-04-25T15:53:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fd99815d010 con 0x7fd99806fc50 2024-04-25T15:53:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd97f7fe640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd98406f430 0x7fd984071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(82..82 src has 1..82) ==== 5915+0+0 (secure 0 0 0) 0x7fd98808ee70 con 0x7fd99806fc50 2024-04-25T15:53:27.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v82) ==== 54+0+16 (secure 0 0 0) 0x7fd98801f030 con 0x7fd99806fc50 2024-04-25T15:53:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd98406f430 0x7fd984071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fd99815d820 0x7fd9981445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9980c5b70 con 0x7fd99815d820 2024-04-25T15:53:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fd99815d820 0x7fd9981445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fd99815d820 0x7fd9981445a0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:27.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.910+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd98406f430 0x7fd984071940 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd998159360 tx=0x7fd98c006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:27.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.914+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9980c5b70 con 0x7fd99815d820 2024-04-25T15:53:27.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.934+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd99806a480 con 0x7fd99815d820 2024-04-25T15:53:27.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.938+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd99806a480 con 0x7fd99815d820 2024-04-25T15:53:27.938 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.938+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd998063090 0x7fd9980634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.938+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fd998074970 con 0x7fd998063090 2024-04-25T15:53:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.938+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd998063090 0x7fd9980634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.938+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd998063090 0x7fd9980634a0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:27.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.938+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd998074970 con 0x7fd998063090 2024-04-25T15:53:27.954 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:27.954+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:27.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.958+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd9980ca420 con 0x7fd998063090 2024-04-25T15:53:27.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.962+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd9980ca420 con 0x7fd998063090 2024-04-25T15:53:27.964 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:27.964 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:27.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.962+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fd998063da0 0x7fd998148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.962+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fd9980761a0 con 0x7fd998063da0 2024-04-25T15:53:27.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.962+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fd998063da0 0x7fd998148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.962+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fd998063da0 0x7fd998148810 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:27.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.966+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9980761a0 con 0x7fd998063da0 2024-04-25T15:53:27.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.986+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd998148fa0 con 0x7fd998063da0 2024-04-25T15:53:27.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.990+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd998148fa0 con 0x7fd998063da0 2024-04-25T15:53:27.991 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:27.991 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:27.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.990+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fd99815bde0 0x7fd99815c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:27.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.990+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fd99815c730 con 0x7fd99815bde0 2024-04-25T15:53:27.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.990+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fd99815bde0 0x7fd99815c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:27.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.994+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fd99815bde0 0x7fd99815c1f0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:27.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:27.994+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd99815c730 con 0x7fd99815bde0 2024-04-25T15:53:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.014+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd99815cc40 con 0x7fd99815bde0 2024-04-25T15:53:28.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.014+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd99815cc40 con 0x7fd99815bde0 2024-04-25T15:53:28.016 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:28.016 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:28.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.014+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fd998205e50 0x7fd998208270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.014+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fd99820b040 con 0x7fd998205e50 2024-04-25T15:53:28.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.014+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fd998205e50 0x7fd998208270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.018+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fd998205e50 0x7fd998208270 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.018+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd99820b040 con 0x7fd998205e50 2024-04-25T15:53:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.034+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd9982087b0 con 0x7fd998205e50 2024-04-25T15:53:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.038+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd9982087b0 con 0x7fd998205e50 2024-04-25T15:53:28.041 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.042+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fd998208e30 0x7fd99820b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.042+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fd99820e020 con 0x7fd998208e30 2024-04-25T15:53:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.042+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fd998208e30 0x7fd99820b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.042+0000 7fd99659c640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fd998208e30 0x7fd99820b250 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.046+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd99820e020 con 0x7fd998208e30 2024-04-25T15:53:28.056 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:28.054+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.066+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd99820b810 con 0x7fd998208e30 2024-04-25T15:53:28.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.070+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd99820b810 con 0x7fd998208e30 2024-04-25T15:53:28.073 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:28.073 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:28.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.074+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 0x7fd99820c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.074+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fd998211490 con 0x7fd99820bf30 2024-04-25T15:53:28.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.074+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 0x7fd99820c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.074+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 0x7fd99820c360 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.074+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd998211490 con 0x7fd99820bf30 2024-04-25T15:53:28.090 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:28.090 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:28.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.090 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:930 3.9 3:90d768d7:::100000001f7.0000009e:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:28.090 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.077538+0000", 2024-04-25T15:53:28.090 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.013107592, 2024-04-25T15:53:28.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013136579000000001, 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 930 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.077538+0000", 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.077538+0000", 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.077540+0000", 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.65e-06 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.082428+0000", 2024-04-25T15:53:28.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048879199999999996 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.082466+0000", 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.731e-05 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.082484+0000", 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8848999999999999e-05 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.082567+0000", 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2744000000000006e-05 2024-04-25T15:53:28.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.082652+0000", 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4726000000000002e-05 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.082765+0000", 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113317 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.088867+0000", 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061021870000000002 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.094 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:931 3.a e82/76 3:5122fcae:::100000001f7.0000009f:head v 82'8, mlcod=81'6)", 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.088682+0000", 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.001963092, 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002049169, 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.088682+0000", 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.088682+0000", 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.088684+0000", 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0200000000000001e-06 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.090524+0000", 2024-04-25T15:53:28.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001839937 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.090544+0000", 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9423e-05 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.090560+0000", 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6555999999999999e-05 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.090639+0000", 2024-04-25T15:53:28.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8566999999999995e-05 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.090659+0000", 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0553000000000001e-05 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 2 2024-04-25T15:53:28.098 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd99820c9b0 con 0x7fd99820bf30 2024-04-25T15:53:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd99759e640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 msgr2=0x7fd99820c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 27 2024-04-25T15:53:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd99759e640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 msgr2=0x7fd99820c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:53:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 0x7fd99820c360 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:53:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 0x7fd99820c360 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 msgr2=0x7fd99820c360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:53:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd97f7fe640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd99820bf30 0x7fd99820c360 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd97f7fe640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd984079770 0x7fd98407bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd998148fa0 con 0x7fd984079770 2024-04-25T15:53:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=83}) -- 0x7fd98407c0f0 con 0x7fd99806fc50 2024-04-25T15:53:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd984079770 0x7fd98407bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.102+0000 7fd99759e640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd984079770 0x7fd98407bb90 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.106+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd998148fa0 con 0x7fd984079770 2024-04-25T15:53:28.107 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:28.107 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:28.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.106+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fd99820cb60 0x7fd99820cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.106+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fd99820df90 con 0x7fd99820cb60 2024-04-25T15:53:28.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.106+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fd99820cb60 0x7fd99820cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.110+0000 7fd996d9d640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fd99820cb60 0x7fd99820cf70 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.110+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9880643f0 con 0x7fd99820cb60 2024-04-25T15:53:28.120 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:28.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.126+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fd99820d540 con 0x7fd99820cb60 2024-04-25T15:53:28.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.130+0000 7fd97f7fe640 1 -- 172.21.15.67:0/961893763 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fd99820d540 con 0x7fd99820cb60 2024-04-25T15:53:28.133 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fd99815d820 msgr2=0x7fd9981445a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fd99815d820 0x7fd9981445a0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd998063090 msgr2=0x7fd9980634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd998063090 0x7fd9980634a0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fd998063da0 msgr2=0x7fd998148810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fd998063da0 0x7fd998148810 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fd99815bde0 msgr2=0x7fd99815c1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd99759e640 1 -- 172.21.15.67:0/961893763 reap_dead start 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fd99815bde0 0x7fd99815c1f0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fd998205e50 msgr2=0x7fd998208270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fd998205e50 0x7fd998208270 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fd998208e30 msgr2=0x7fd99820b250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fd998208e30 0x7fd99820b250 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd984079770 msgr2=0x7fd98407bb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fd984079770 0x7fd98407bb90 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fd99820cb60 msgr2=0x7fd99820cf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fd99820cb60 0x7fd99820cf70 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd99759e640 1 -- 172.21.15.67:0/961893763 reap_dead start 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd98406f430 msgr2=0x7fd984071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd98406f430 0x7fd984071940 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd998159360 tx=0x7fd98c006f90 comp rx=0 tx=0).stop 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd99806fc50 msgr2=0x7fd9980c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd99806fc50 0x7fd9980c6230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd98800b3f0 tx=0x7fd98800b8b0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 shutdown_connections 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd98406f430 0x7fd984071940 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 --2- 172.21.15.67:0/961893763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd99806fc50 0x7fd9980c6230 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 >> 172.21.15.67:0/961893763 conn(0x7fd9980cdd60 msgr2=0x7fd9980c3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 shutdown_connections 2024-04-25T15:53:28.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.134+0000 7fd97d7fa640 1 -- 172.21.15.67:0/961893763 wait complete. 2024-04-25T15:53:28.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:53:28.167 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:28.167+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:28.254 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:28.254+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:28.313 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:28.313 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:28.313 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:28.313 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:28.313 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:28.347 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:28.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f796560d640 1 Processor -- start 2024-04-25T15:53:28.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f796560d640 1 -- start start 2024-04-25T15:53:28.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f796560d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7960156330 0x7f796006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f796560d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f796006cb80 0x7f79600c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f796560d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796006fa10 con 0x7f79600c4f30 2024-04-25T15:53:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f796560d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f79600ca6c0 con 0x7f7960156330 2024-04-25T15:53:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f796560d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f796006cf90 con 0x7f796006cb80 2024-04-25T15:53:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795f7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79600c4f30 0x7f7960158d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39460/0 (socket says 172.21.15.67:39460) 2024-04-25T15:53:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795f7fe640 1 -- 172.21.15.67:0/2513976844 learned_addr learned my addr 172.21.15.67:0/2513976844 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795effd640 1 --2- 172.21.15.67:0/2513976844 >> v2:172.21.15.67:3300/0 conn(0x7f7960156330 0x7f796006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3304205637 0 0) 0x7f796006fa10 con 0x7f79600c4f30 2024-04-25T15:53:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f794c003530 con 0x7f79600c4f30 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 966713828 0 0) 0x7f794c003530 con 0x7f79600c4f30 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 >> v2:172.21.15.176:3300/0 conn(0x7f796006cb80 msgr2=0x7f79600c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 --2- 172.21.15.67:0/2513976844 >> v2:172.21.15.176:3300/0 conn(0x7f796006cb80 0x7f79600c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 >> v2:172.21.15.67:3300/0 conn(0x7f7960156330 msgr2=0x7f796006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 --2- 172.21.15.67:0/2513976844 >> v2:172.21.15.67:3300/0 conn(0x7f7960156330 0x7f796006c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79600ca8f0 con 0x7f79600c4f30 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f7958001a00 con 0x7f79600c4f30 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 >> v1:172.21.15.67:6789/0 conn(0x7f79600c4f30 legacy=0x7f7960158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c0039f0 0x7f794c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794c004810 0x7f794c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c005360 0x7f794c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f794c00c200 con 0x7f794c005360 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f794c00c3b0 con 0x7f794c004810 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f794c00c5b0 con 0x7f794c0039f0 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.354+0000 7f795f7fe640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c005360 0x7f794c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795f7fe640 1 -- 172.21.15.67:0/2513976844 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c0039f0 msgr2=0x7f794c003e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795f7fe640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c0039f0 0x7f794c003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795f7fe640 1 -- 172.21.15.67:0/2513976844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794c004810 msgr2=0x7f794c004c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:53:28.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795f7fe640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f794c004810 0x7f794c004c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795f7fe640 1 -- 172.21.15.67:0/2513976844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c003530 con 0x7f794c005360 2024-04-25T15:53:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795f7fe640 1 -- 172.21.15.67:0/2513976844 reap_dead start 2024-04-25T15:53:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795f7fe640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c005360 0x7f794c00b9d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7960072070 tx=0x7f795801d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bee8c2244ad6a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795800a4a0 con 0x7f794c005360 2024-04-25T15:53:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79580ac640 con 0x7f794c005360 2024-04-25T15:53:28.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.358+0000 7f795e7fc640 1 -- 172.21.15.67:0/2513976844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580aca30 con 0x7f794c005360 2024-04-25T15:53:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 -- 172.21.15.67:0/2513976844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c005360 msgr2=0x7f794c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c005360 0x7f794c00b9d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7960072070 tx=0x7f795801d320 comp rx=0 tx=0).stop 2024-04-25T15:53:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 -- 172.21.15.67:0/2513976844 shutdown_connections 2024-04-25T15:53:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 --2- 172.21.15.67:0/2513976844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f794c005360 0x7f794c00b9d0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 -- 172.21.15.67:0/2513976844 >> 172.21.15.67:0/2513976844 conn(0x7f79600ce1c0 msgr2=0x7f79600ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:28.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 -- 172.21.15.67:0/2513976844 shutdown_connections 2024-04-25T15:53:28.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 -- 172.21.15.67:0/2513976844 wait complete. 2024-04-25T15:53:28.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 Processor -- start 2024-04-25T15:53:28.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.362+0000 7f796560d640 1 -- start start 2024-04-25T15:53:28.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c005360 0x7f7960158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79601598e0 0x7f79601ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79600cd620 0x7f79600c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79600c3d80 con 0x7f79601598e0 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f79600c43d0 con 0x7f79600cd620 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f79600c4060 con 0x7f794c005360 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79601598e0 0x7f79601ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79601598e0 0x7f79601ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40652/0 (socket says 172.21.15.67:40652) 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 -- 172.21.15.67:0/1482326669 learned_addr learned my addr 172.21.15.67:0/1482326669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f795effd640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c005360 0x7f7960158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f795f7fe640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79600cd620 0x7f79600c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 -- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c005360 msgr2=0x7f7960158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c005360 0x7f7960158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 -- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79600cd620 msgr2=0x7f79600c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79600cd620 0x7f79600c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 -- 172.21.15.67:0/1482326669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79601eedd0 con 0x7f79601598e0 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f795f7fe640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79600cd620 0x7f79600c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:28.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f7957fff640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79601598e0 0x7f79601ee890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f794800b410 tx=0x7f794800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f79577fe640 1 -- 172.21.15.67:0/1482326669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948015030 con 0x7f79601598e0 2024-04-25T15:53:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f79577fe640 1 -- 172.21.15.67:0/1482326669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7948005980 con 0x7f79601598e0 2024-04-25T15:53:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79601f1b10 con 0x7f79601598e0 2024-04-25T15:53:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f79577fe640 1 -- 172.21.15.67:0/1482326669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948005d90 con 0x7f79601598e0 2024-04-25T15:53:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79601f20b0 con 0x7f79601598e0 2024-04-25T15:53:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.366+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796006a480 con 0x7f79601598e0 2024-04-25T15:53:28.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.370+0000 7f79577fe640 1 -- 172.21.15.67:0/1482326669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7948018070 con 0x7f79601598e0 2024-04-25T15:53:28.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.370+0000 7f79577fe640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f794006f430 0x7f7940071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.370+0000 7f79577fe640 1 -- 172.21.15.67:0/1482326669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5915+0+0 (secure 0 0 0) 0x7f794808f620 con 0x7f79601598e0 2024-04-25T15:53:28.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.370+0000 7f795effd640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f794006f430 0x7f7940071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:28.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.374+0000 7f795effd640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f794006f430 0x7f7940071940 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f795000b9b0 tx=0x7f7950008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:28.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.378+0000 7f79577fe640 1 -- 172.21.15.67:0/1482326669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f794805c4b0 con 0x7f79601598e0 2024-04-25T15:53:28.396 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:28.395+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:28.456 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:28.455+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:28.557 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:28.554+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:597 3.e e81/76 3:72b88a04:::100000001f7.00000051:head v 81'6, mlcod=69'3)", 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.2310211090000003, 2024-04-25T15:53:28.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038266363999999997, 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374533+0000", 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5060000000000001e-06 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376869+0000", 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023357830000000001 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376914+0000", 2024-04-25T15:53:28.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4317000000000003e-05 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376941+0000", 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6965999999999999e-05 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377065+0000", 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012408300000000001 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377098+0000", 2024-04-25T15:53:28.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2943e-05 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412723+0000", 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035625742000000002 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412798+0000", 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5024000000000005e-05 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:599 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.22673188, 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039983414000000002, 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 599 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378825+0000", 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7189999999999999e-06 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384734+0000", 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059089859999999998 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384776+0000", 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2166999999999998e-05 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384795+0000", 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9307999999999999e-05 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384894+0000", 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9112999999999998e-05 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385064+0000", 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169524 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385212+0000", 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147849 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409088+0000", 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023875850000000001 2024-04-25T15:53:28.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418680+0000", 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095925639999999996 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418742+0000", 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1379999999999998e-05 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418805+0000", 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2953999999999994e-05 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.616 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:598 3.11 e81/76 3:89a80ce8:::100000001f7.00000052:head v 81'21, mlcod=80'19)", 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.2241925570000003, 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037275751000000003, 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381362+0000", 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.782e-06 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382748+0000", 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001385488 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382769+0000", 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0854000000000001e-05 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382786+0000", 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7807000000000001e-05 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382876+0000", 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9821999999999995e-05 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382901+0000", 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4845999999999999e-05 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418526+0000", 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035624426000000001 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418636+0000", 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110726 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:606 3.31 3:8d161915:::100000001f7.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.1756205839999998, 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031627436000000002, 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 606 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429935+0000", 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9610000000000001e-06 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435340+0000", 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054048719999999998 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435375+0000", 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4558999999999999e-05 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435394+0000", 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8819000000000001e-05 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435493+0000", 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8927999999999999e-05 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435806+0000", 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031338500000000001 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.436016+0000", 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000209586 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.459476+0000", 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023460547000000002 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461366+0000", 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001890206 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461433+0000", 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6206999999999998e-05 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461560+0000", 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012736600000000001 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:654 3.11 e81/76 3:892e96a3:::100000001f7.0000008a:head v 81'22, mlcod=80'20)", 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.7487924320000001, 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044455910000000001, 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.626 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856762+0000", 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.669e-06 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859651+0000", 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002889015 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859675+0000", 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3346999999999999e-05 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859696+0000", 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1021000000000001e-05 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859816+0000", 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120809 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859853+0000", 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6242999999999998e-05 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901127+0000", 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041274245000000001 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901217+0000", 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9561000000000005e-05 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:668 3.1 3:83ba1de1:::100000001f7.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.7454208060000003, 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062400662000000003, 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 668 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860140+0000", 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1559999999999995e-06 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869131+0000", 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089909859999999994 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869188+0000", 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6539000000000002e-05 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869212+0000", 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4501000000000001e-05 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869417+0000", 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020473999999999999 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869612+0000", 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000194668 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871230+0000", 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016182029999999999 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.917654+0000", 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046423786000000002 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922361+0000", 2024-04-25T15:53:28.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00470704 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922442+0000", 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.153e-05 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922533+0000", 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0513000000000006e-05 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:669 3.f e81/76 3:f0675d7f:::100000001f7.00000099:head v 81'10, mlcod=70'8)", 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.7193842879999996, 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080177950999999997, 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886171+0000", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8470000000000001e-06 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891613+0000", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054420349999999996 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891657+0000", 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4155e-05 2024-04-25T15:53:28.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891689+0000", 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1822999999999997e-05 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891773+0000", 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4602000000000001e-05 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891806+0000", 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2842999999999997e-05 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966243+0000", 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074437173999999995 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966347+0000", 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103472 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:647 3.a e81/76 3:524e89bd:::100000001f7.00000083:head v 81'3, mlcod=61'1)", 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.7009496659999996, 2024-04-25T15:53:28.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072551145999999997, 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904605+0000", 2024-04-25T15:53:28.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.421e-06 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906854+0000", 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022493249999999999 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906880+0000", 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5374000000000001e-05 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906897+0000", 2024-04-25T15:53:28.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7419999999999999e-05 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906994+0000", 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6633999999999997e-05 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.907027+0000", 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3336000000000001e-05 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977067+0000", 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.070040033000000002 2024-04-25T15:53:28.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977155+0000", 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7602999999999995e-05 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:674 3.9 3:90d768d7:::100000001f7.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.6932827350000004, 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065083915000000006, 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.643 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 674 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912273+0000", 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6929999999999999e-06 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921147+0000", 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088735250000000002 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921185+0000", 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8098e-05 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921200+0000", 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5543999999999999e-05 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921281+0000", 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0758000000000002e-05 2024-04-25T15:53:28.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921426+0000", 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014529000000000001 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921540+0000", 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113465 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.942585+0000", 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021045735999999999 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977215+0000", 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034629659 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977282+0000", 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6796000000000002e-05 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977354+0000", 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2350999999999999e-05 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:675 3.a e81/76 3:5122fcae:::100000001f7.0000009f:head v 81'4, mlcod=61'1)", 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.6757225890000003, 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047589770000000003, 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.649 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929833+0000", 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2850000000000001e-06 2024-04-25T15:53:28.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931776+0000", 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001943443 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931806+0000", 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0074e-05 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931829+0000", 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3027000000000002e-05 2024-04-25T15:53:28.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931927+0000", 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7540000000000005e-05 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931959+0000", 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2265999999999999e-05 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977374+0000", 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045414643999999997 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977420+0000", 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6490999999999999e-05 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:676 3.18 e81/76 3:191fe66d:::100000001f7.000000a0:head v 81'8, mlcod=71'6)", 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.6738780210000002, 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050752358999999997, 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.653 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931678+0000", 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.542e-06 2024-04-25T15:53:28.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935891+0000", 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042133930000000002 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935917+0000", 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5624000000000001e-05 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935935+0000", 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8258999999999999e-05 2024-04-25T15:53:28.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936019+0000", 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4538999999999998e-05 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936060+0000", 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1060000000000003e-05 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982347+0000", 2024-04-25T15:53:28.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046286542999999999 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982427+0000", 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0399000000000001e-05 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:678 3.1b 3:da348c2f:::100000001f7.000000a2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.6625099529999998, 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039665855999999999, 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.657 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 678 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943045+0000", 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9630000000000001e-06 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947273+0000", 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042283470000000004 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947307+0000", 2024-04-25T15:53:28.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3794000000000002e-05 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947322+0000", 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5184999999999999e-05 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947403+0000", 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1076999999999996e-05 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947533+0000", 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012934399999999999 2024-04-25T15:53:28.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947646+0000", 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113163 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.964158+0000", 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016512526 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982578+0000", 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018419333 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982646+0000", 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8010999999999996e-05 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982709+0000", 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:679 3.24 e81/76 3:27dcc3f5:::100000001f7.000000a3:head v 81'12, mlcod=81'10)", 2024-04-25T15:53:28.662 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.652964388, 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030189264, 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:28.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952590+0000", 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.381e-06 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956607+0000", 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040170140000000002 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956623+0000", 2024-04-25T15:53:28.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6025e-05 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956636+0000", 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2595000000000001e-05 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956708+0000", 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2402000000000002e-05 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956741+0000", 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3386999999999997e-05 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982717+0000", 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025975446999999999 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982778+0000", 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1013e-05 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:720 3.1a 3:59a21752:::100000001f7.000000cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3807206499999998, 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031267504000000002, 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 720 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224835+0000", 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.661e-06 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230311+0000", 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054762220000000002 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230341+0000", 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9972999999999999e-05 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230357+0000", 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.558e-05 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230440+0000", 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3030999999999998e-05 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230610+0000", 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016995500000000001 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230782+0000", 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000172079 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.247511+0000", 2024-04-25T15:53:28.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016729067 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255803+0000", 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082918499999999999 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.256001+0000", 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019800300000000001 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.256100+0000", 2024-04-25T15:53:28.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9083e-05 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:723 3.9 3:9221849e:::100000001f7.000000cf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3624881240000004, 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031588667000000001, 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.672 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 723 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243070+0000", 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0200000000000002e-06 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248284+0000", 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052135619999999997 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248318+0000", 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4072000000000002e-05 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248337+0000", 2024-04-25T15:53:28.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9867000000000001e-05 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248421+0000", 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3541000000000005e-05 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248550+0000", 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128768 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248782+0000", 2024-04-25T15:53:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023251200000000001 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.265337+0000", 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016554484000000001 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274315+0000", 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089778259999999995 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274497+0000", 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018229699999999999 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274654+0000", 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000156718 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:762 3.3 3:c0da3165:::100000001f7.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.0905084650000001, 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033743224000000002, 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.677 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 762 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515046+0000", 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.578e-06 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523433+0000", 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083871290000000001 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523458+0000", 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4732e-05 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523469+0000", 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0882e-05 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523588+0000", 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119361 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523741+0000", 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152311 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523908+0000", 2024-04-25T15:53:28.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167472 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.535424+0000", 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011516337 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548681+0000", 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013256274 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548736+0000", 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5306999999999997e-05 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548788+0000", 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1841000000000001e-05 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:957 3.11 e82/76 3:897f0f98:::100000001f7.000000b9:head v 82'38, mlcod=82'36)", 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.36773189899999997, 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030634000000000002, 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.682 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237823+0000", 2024-04-25T15:53:28.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9429999999999999e-06 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239142+0000", 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013188130000000001 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239168+0000", 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6132e-05 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239183+0000", 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5068999999999999e-05 2024-04-25T15:53:28.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239267+0000", 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4227000000000005e-05 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239290+0000", 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2209999999999999e-05 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268394+0000", 2024-04-25T15:53:28.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029104512999999999 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268455+0000", 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1093000000000006e-05 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1004 3.1d e82/76 3:b93a1278:::100000001f7.000000e8:head v 82'29, mlcod=82'27)", 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.081570166, 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044544315000000001, 2024-04-25T15:53:28.686 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523985+0000", 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9999999999999999e-06 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527244+0000", 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032587330000000002 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527284+0000", 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0031999999999999e-05 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527309+0000", 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527437+0000", 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128384 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527480+0000", 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2357999999999997e-05 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568424+0000", 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040944332 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568527+0000", 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103319 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1008 3.2c e82/76 3:3798f9d7:::100000001f7.000000ec:head v 82'14, mlcod=82'12)", 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.067155308999999996, 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042823282999999997, 2024-04-25T15:53:28.690 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538400+0000", 2024-04-25T15:53:28.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0650000000000001e-06 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541298+0000", 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028986440000000001 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541324+0000", 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5176000000000001e-05 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541339+0000", 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5821999999999999e-05 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541469+0000", 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541521+0000", 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1882999999999998e-05 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581098+0000", 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039576885999999999 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581221+0000", 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012303699999999999 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1012 3.1b 3:db4e25ba:::100000001f7.000000f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.056316353, 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032333898999999999, 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:28.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1012 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549239+0000", 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7700000000000002e-06 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555878+0000", 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066383470000000002 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555906+0000", 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8147e-05 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555923+0000", 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7385999999999999e-05 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556004+0000", 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0836999999999999e-05 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556110+0000", 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105487 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556220+0000", 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110053 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568508+0000", 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012288566000000001 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581445+0000", 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012937059000000001 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581507+0000", 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1703000000000002e-05 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581571+0000", 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3544e-05 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:28.699 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:28.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:28.646+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f79600ca4a0 con 0x7f79601598e0 2024-04-25T15:53:28.701 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:28.701 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 rewriters = 665193.56 kB/sec 2024-04-25T15:53:28.701 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 rewriters = 664935.45 kB/sec 2024-04-25T15:53:28.701 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 665193.56 kB/sec 2024-04-25T15:53:28.701 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 665193.56 kB/sec 2024-04-25T15:53:28.701 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 665193.56 kB/sec 2024-04-25T15:53:28.701 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:53:28.702 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:28.702 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:28.698+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:28.801 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:28.802+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:28.895 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:28.895 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:28.895 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:28.895 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:28.902 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:28.902+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:28.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:28.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:53:28.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:53:29.003 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:29.002+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:29.062 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:29.062 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:29.062 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:29.062 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:29.062 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:29.079 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:29.123 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:29.123+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.138+0000 7f79577fe640 1 -- 172.21.15.67:0/1482326669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v83) ==== 87+0+0 (secure 0 0 0) 0x7f794805bdb0 con 0x7f79601598e0 2024-04-25T15:53:29.138 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:53:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f794006f430 msgr2=0x7f7940071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f794006f430 0x7f7940071940 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f795000b9b0 tx=0x7f7950008040 comp rx=0 tx=0).stop 2024-04-25T15:53:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79601598e0 msgr2=0x7f79601ee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79601598e0 0x7f79601ee890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f794800b410 tx=0x7f794800b8d0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 shutdown_connections 2024-04-25T15:53:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f794006f430 0x7f7940071940 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f79600cd620 0x7f79600c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79601598e0 0x7f79601ee890 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 --2- 172.21.15.67:0/1482326669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f794c005360 0x7f7960158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 >> 172.21.15.67:0/1482326669 conn(0x7f79600ce1c0 msgr2=0x7f79600cab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 shutdown_connections 2024-04-25T15:53:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:29.142+0000 7f796560d640 1 -- 172.21.15.67:0/1482326669 wait complete. 2024-04-25T15:53:29.209 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:29.210+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:29.278 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:29.278 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:29.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:29.278 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:29.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.055744659999998, 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:29.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:594 3.16 3:6a58e86c:::100000001f7.0000004e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.9275672129999997, 2024-04-25T15:53:29.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034501508, 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 594 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:29.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347207+0000", 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8660000000000002e-06 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353453+0000", 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062464160000000003 2024-04-25T15:53:29.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353489+0000", 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5283000000000001e-05 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353501+0000", 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2860999999999999e-05 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353629+0000", 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000127632 2024-04-25T15:53:29.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353808+0000", 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017904800000000001 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353996+0000", 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018766099999999999 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.380705+0000", 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026708789 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381525+0000", 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00082074100000000005 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381619+0000", 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3153999999999996e-05 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381706+0000", 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7057000000000004e-05 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:596 3.16 3:6941c83d:::100000001f7.00000050:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.9128527310000001, 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049183123000000002, 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 596 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361922+0000", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367484+0000", 2024-04-25T15:53:29.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055618409999999997 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367522+0000", 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8260000000000003e-05 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367543+0000", 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0795999999999999e-05 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367646+0000", 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102913 2024-04-25T15:53:29.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367857+0000", 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021190400000000001 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368091+0000", 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023323699999999999 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.405516+0000", 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037425424999999998 2024-04-25T15:53:29.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410955+0000", 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054389859999999998 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411034+0000", 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8421e-05 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411102+0000", 2024-04-25T15:53:29.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8137000000000002e-05 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:600 3.5 3:a2751c53:::100000001f7.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.8893692250000003, 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035430837, 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.296 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 600 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385405+0000", 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8959999999999999e-06 2024-04-25T15:53:29.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390062+0000", 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046572949999999997 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390104+0000", 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1943000000000001e-05 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390123+0000", 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9147000000000002e-05 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390225+0000", 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101983 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390407+0000", 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181819 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390593+0000", 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018578900000000001 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409139+0000", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018546360000000001 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420728+0000", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011589083 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420783+0000", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5025000000000003e-05 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420833+0000", 2024-04-25T15:53:29.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9496999999999997e-05 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:649 3.3c 3:3e99caf7:::100000001f7.00000085:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5400286620000001, 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031048148000000001, 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 649 2024-04-25T15:53:29.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734744+0000", 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.694e-06 2024-04-25T15:53:29.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740013+0000", 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052687469999999998 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740078+0000", 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5402999999999997e-05 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740096+0000", 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7393000000000001e-05 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740206+0000", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109789 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740427+0000", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022180100000000001 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740653+0000", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022509799999999999 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.748988+0000", 2024-04-25T15:53:29.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083355619999999995 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765614+0000", 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016625788999999998 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765712+0000", 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8183999999999994e-05 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765791+0000", 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8688000000000003e-05 2024-04-25T15:53:29.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:665 3.1d 3:b87c769f:::100000001f7.00000095:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.4337476909999998, 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045554887000000002, 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 665 2024-04-25T15:53:29.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841029+0000", 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9359999999999998e-06 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851995+0000", 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010966115 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852062+0000", 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6975000000000002e-05 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852084+0000", 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2382999999999999e-05 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852218+0000", 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013397699999999999 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852418+0000", 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000200161 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852550+0000", 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013199300000000001 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878731+0000", 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026181157 2024-04-25T15:53:29.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886405+0000", 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076735090000000002 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886503+0000", 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8504000000000004e-05 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886578+0000", 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5177000000000002e-05 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:663 3.d e81/18 3:b1139e77:::100000001f7.00000093:head v 81'7, mlcod=80'5)", 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.4163098160000001, 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032554081999999998, 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858463+0000", 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8169999999999999e-06 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862335+0000", 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003871462 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862369+0000", 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4635999999999998e-05 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862395+0000", 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5698e-05 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862753+0000", 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035778999999999997 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862879+0000", 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012587500000000001 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.890905+0000", 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028025912 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891016+0000", 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011089199999999999 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:664 3.d e81/18 3:b353c7f2:::100000001f7.00000094:head v 81'8, mlcod=80'5)", 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.4054487909999995, 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051411299000000001, 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869325+0000", 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1689999999999999e-06 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872141+0000", 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028160759999999998 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872176+0000", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4996999999999997e-05 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872202+0000", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5944000000000001e-05 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872312+0000", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010998200000000001 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872351+0000", 2024-04-25T15:53:29.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9360999999999998e-05 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920645+0000", 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048293989000000002 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920734+0000", 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8781000000000003e-05 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:671 3.38 e81/18 3:1c0639bf:::100000001f7.0000009b:head v 81'8, mlcod=81'6)", 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.3881245829999997, 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069441491999999994, 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886668+0000", 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0876000000000001e-05 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893242+0000", 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065744849999999997 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893267+0000", 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5321e-05 2024-04-25T15:53:29.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893283+0000", 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5454000000000001e-05 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893362+0000", 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8770000000000006e-05 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893382+0000", 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0585999999999999e-05 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.955934+0000", 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062552041000000003 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956088+0000", 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.322 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:673 3.28 3:153e0bf4:::100000001f7.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.3698352939999996, 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063190614000000006, 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 673 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904939+0000", 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8739999999999999e-06 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911908+0000", 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069695449999999997 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911944+0000", 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5973000000000003e-05 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911960+0000", 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5319999999999999e-05 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912057+0000", 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7427000000000001e-05 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912221+0000", 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016433600000000001 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912375+0000", 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015359299999999999 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.941806+0000", 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029430523 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967997+0000", 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026191589000000001 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968063+0000", 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6321000000000004e-05 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968127+0000", 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:652 3.36 e81/18 3:6c98fff9:::100000001f7.00000088:head v 81'12, mlcod=69'10)", 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.3586385329999997, 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052156028, 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:29.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916135+0000", 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7850000000000001e-06 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918034+0000", 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018996099999999999 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918080+0000", 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5469000000000001e-05 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918098+0000", 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8726999999999999e-05 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918176+0000", 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8125000000000002e-05 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918198+0000", 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1531000000000001e-05 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968240+0000", 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050041583000000001 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968289+0000", 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9197999999999999e-05 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:677 3.5 3:a07f2395:::100000001f7.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.3392568100000002, 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041485695000000003, 2024-04-25T15:53:29.331 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 677 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:29.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935657+0000", 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014239100000000001 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940003+0000", 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043458239999999999 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940055+0000", 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2021999999999998e-05 2024-04-25T15:53:29.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940071+0000", 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6528000000000002e-05 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940148+0000", 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7179000000000001e-05 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940294+0000", 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014605799999999999 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940418+0000", 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012356499999999999 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956756+0000", 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016338149999999999 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976852+0000", 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020095572999999999 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976925+0000", 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3351999999999998e-05 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977000+0000", 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5053000000000001e-05 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:719 3.1d 3:b9e367d5:::100000001f7.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.0558817349999998, 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037012611000000001, 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.336 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 719 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218892+0000", 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.182e-06 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226010+0000", 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071184320000000001 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226050+0000", 2024-04-25T15:53:29.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0167999999999999e-05 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226070+0000", 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9256999999999999e-05 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226209+0000", 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000139226 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226360+0000", 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015169700000000001 2024-04-25T15:53:29.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226574+0000", 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021339199999999999 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.253457+0000", 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026883383 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255765+0000", 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023077409999999999 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255837+0000", 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1987e-05 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255902+0000", 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5146000000000003e-05 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:729 3.5 3:a1e9b24d:::100000001f7.000000d5:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.9886446470000001, 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033306524999999997, 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 729 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286128+0000", 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292385+0000", 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006257589 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292423+0000", 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7756999999999998e-05 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292441+0000", 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7954000000000001e-05 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292521+0000", 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0171000000000002e-05 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292651+0000", 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130025 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292787+0000", 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013599099999999999 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.302313+0000", 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095255389999999995 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319295+0000", 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016982391999999999 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319373+0000", 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7571000000000007e-05 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319433+0000", 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0306000000000002e-05 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:745 3.d e81/18 3:b225ffa8:::100000001f7.000000e5:head v 81'10, mlcod=81'8)", 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.8727164030000001, 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034113420999999998, 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.347 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402056+0000", 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3459999999999999e-06 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405220+0000", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031635439999999999 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405240+0000", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0695000000000001e-05 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405253+0000", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2897e-05 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405323+0000", 2024-04-25T15:53:29.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9719000000000001e-05 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405377+0000", 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3653000000000003e-05 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436064+0000", 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030687076000000001 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436168+0000", 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104491 2024-04-25T15:53:29.350 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:748 3.1d 3:b93a1278:::100000001f7.000000e8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.8668430230000004, 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033655512999999998, 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 748 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.351 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407929+0000", 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9899999999999999e-07 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419777+0000", 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011847952 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419801+0000", 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4049999999999998e-05 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419810+0000", 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3039999999999994e-06 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419874+0000", 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3467999999999996e-05 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.420010+0000", 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013569600000000001 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.420161+0000", 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015180900000000001 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.438066+0000", 2024-04-25T15:53:29.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017904571000000001 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441466+0000", 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033998959999999999 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441529+0000", 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3124000000000001e-05 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441584+0000", 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4744000000000003e-05 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.355 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1002 3.1f 3:fb90ae0e:::100000001f7.000000e6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.77607194099999999, 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035318779000000002, 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1002 2024-04-25T15:53:29.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498702+0000", 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2330000000000001e-06 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507556+0000", 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088549340000000001 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508591+0000", 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010341459999999999 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508610+0000", 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9490000000000001e-05 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508716+0000", 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105468 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508836+0000", 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120627 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.509012+0000", 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017549199999999999 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521191+0000", 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012179339000000001 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533885+0000", 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01269438 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533944+0000", 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8971000000000002e-05 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534018+0000", 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3699e-05 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1003 3.28 3:1719c8f9:::100000001f7.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.76720447599999997, 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062620618000000003, 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1003 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507569+0000", 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2110000000000001e-06 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519064+0000", 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011495182 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521838+0000", 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027741129999999999 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521856+0000", 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521967+0000", 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110913 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522100+0000", 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013309899999999999 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522235+0000", 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013518199999999999 2024-04-25T15:53:29.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.543023+0000", 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020788025000000002 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570076+0000", 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027052208000000001 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570138+0000", 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2376e-05 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570187+0000", 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9379000000000003e-05 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1004 3.1d 3:b93a1278:::100000001f7.000000e8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.75566802200000005, 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049745623000000003, 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:29.366 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1004 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519106+0000", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.464e-06 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523205+0000", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040996339999999996 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523267+0000", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1660000000000003e-05 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523292+0000", 2024-04-25T15:53:29.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5194999999999998e-05 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523428+0000", 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013594 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523595+0000", 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016733300000000001 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523819+0000", 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022358999999999999 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.566513+0000", 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042694339999999997 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568719+0000", 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022057230000000001 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568799+0000", 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0370000000000005e-05 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568849+0000", 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9373999999999999e-05 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1007 3.36 e82/18 3:6de076d9:::100000001f7.000000eb:head v 82'23, mlcod=82'21)", 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.74090090600000003, 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040044785999999999, 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:29.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533871+0000", 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3799999999999996e-07 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535899+0000", 2024-04-25T15:53:29.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020273069999999999 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535926+0000", 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7803000000000001e-05 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535945+0000", 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8555000000000001e-05 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.536054+0000", 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109441 2024-04-25T15:53:29.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.536089+0000", 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4411e-05 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.573843+0000", 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037753789000000003 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.573915+0000", 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2541999999999998e-05 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.375 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:29.375 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:29.375 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:29.376 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:29.376 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:29.375+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:29.476 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:29.474+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:29.577 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:29.575+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:29.584 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:29.584 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:29.584 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:29.584 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:29.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:29.678 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:29.678+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:29.779 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:29.775+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:29.791 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:29.791 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:29.791 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:29.792 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:29.792 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:29.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:29.879 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:29.878+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:29.980 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:29.978+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.3432599999999999, 2024-04-25T15:53:30.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.26243838600000002, 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920373+0000", 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039448372000000002 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921649+0000", 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012757210000000001 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921724+0000", 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5549000000000005e-05 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921802+0000", 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8150000000000002e-05 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:640 3.10 3:092a6a2c:::100000001f7.0000007c:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.1242287190000004, 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065130602999999995, 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 640 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878397+0000", 2024-04-25T15:53:30.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.198e-06 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894140+0000", 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015742477000000001 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894240+0000", 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010061299999999999 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894258+0000", 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7779999999999999e-05 2024-04-25T15:53:30.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894341+0000", 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3168000000000001e-05 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894526+0000", 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018456399999999999 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894632+0000", 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010651500000000001 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.923409+0000", 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028776380000000001 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943396+0000", 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019987393999999999 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943459+0000", 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2389000000000002e-05 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943526+0000", 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7125000000000005e-05 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:88 2.14 2:2e2fa760:::200.00000001:head [write 97242~3394 [fadvise_dontneed] in=3394b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.1111968409999999, 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052104437000000003, 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.017 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891429+0000", 2024-04-25T15:53:30.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8190000000000001e-06 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891440+0000", 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1029999999999999e-05 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891499+0000", 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8961e-05 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891530+0000", 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1049e-05 2024-04-25T15:53:30.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891630+0000", 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010053899999999999 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891787+0000", 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015704200000000001 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:30.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891996+0000", 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020865400000000001 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920805+0000", 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028808585000000001 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943394+0000", 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022589115 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943465+0000", 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1254000000000006e-05 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943531+0000", 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:647 3.a 3:524e89bd:::100000001f7.00000083:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.1084740430000002, 2024-04-25T15:53:30.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083393058000000006, 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 647 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894152+0000", 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.968e-06 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902694+0000", 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085420719999999995 2024-04-25T15:53:30.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903482+0000", 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078797999999999997 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903499+0000", 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7082e-05 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903593+0000", 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4452999999999998e-05 2024-04-25T15:53:30.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903774+0000", 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180803 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903953+0000", 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179106 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.965763+0000", 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061810057000000002 2024-04-25T15:53:30.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977390+0000", 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01162697 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977474+0000", 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3812000000000003e-05 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977543+0000", 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8755000000000002e-05 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:652 3.36 3:6c98fff9:::100000001f7.00000088:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.0999187380000004, 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065992254, 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.028 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 652 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902707+0000", 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5209999999999999e-06 2024-04-25T15:53:30.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910035+0000", 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073280000000000003 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910344+0000", 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030890699999999998 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910417+0000", 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3900999999999996e-05 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910547+0000", 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012940400000000001 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910876+0000", 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000329181 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911062+0000", 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018587000000000001 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966069+0000", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.055007038000000001 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968579+0000", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002509598 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968649+0000", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0189000000000002e-05 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968697+0000", 2024-04-25T15:53:30.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8644999999999999e-05 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:660 3.26 3:65a7570a:::100000001f7.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.0925753589999996, 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058617787999999997, 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 660 2024-04-25T15:53:30.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910051+0000", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7599999999999998e-06 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925001+0000", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014949405000000001 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925210+0000", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020949799999999999 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925223+0000", 2024-04-25T15:53:30.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3086e-05 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925337+0000", 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011333899999999999 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925525+0000", 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018789300000000001 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925638+0000", 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011354 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967535+0000", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041897074999999999 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968553+0000", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010179480000000001 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968609+0000", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5653999999999999e-05 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968666+0000", 2024-04-25T15:53:30.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7590000000000003e-05 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:675 3.a 3:5122fcae:::100000001f7.0000009f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.0776135120000001, 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052715010999999999, 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 675 2024-04-25T15:53:30.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925013+0000", 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3089999999999998e-06 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929333+0000", 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043198170000000001 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929365+0000", 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2323000000000002e-05 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929380+0000", 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5421999999999999e-05 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929460+0000", 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0148000000000006e-05 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929572+0000", 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011171 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929688+0000", 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011547 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966801+0000", 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037113896 2024-04-25T15:53:30.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977633+0000", 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010831294999999999 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977681+0000", 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7926000000000002e-05 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977725+0000", 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4694999999999998e-05 2024-04-25T15:53:30.042 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:750 3.10 3:08911dbf:::100000001f7.000000ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5805973079999998, 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029714553000000001, 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 750 2024-04-25T15:53:30.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422031+0000", 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5050000000000001e-06 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.427886+0000", 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058545810000000002 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433892+0000", 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060058660000000003 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433903+0000", 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1769e-05 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433975+0000", 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1872000000000006e-05 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.434165+0000", 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018962300000000001 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.434345+0000", 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180034 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441345+0000", 2024-04-25T15:53:30.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070004730000000001 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451615+0000", 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010269276000000001 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451679+0000", 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4369000000000007e-05 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451741+0000", 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2185000000000002e-05 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.047 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:751 3.36 3:6de076d9:::100000001f7.000000eb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5740228419999998, 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035860229, 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 751 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428603+0000", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5820000000000001e-06 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433862+0000", 2024-04-25T15:53:30.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052591870000000002 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442450+0000", 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085886669999999995 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442460+0000", 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3160000000000003e-06 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442541+0000", 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1217000000000005e-05 2024-04-25T15:53:30.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442706+0000", 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016477000000000001 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442865+0000", 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015895899999999999 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.460454+0000", 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017589370999999999 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464395+0000", 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039411139999999999 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464433+0000", 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7710999999999999e-05 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464461+0000", 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8334999999999999e-05 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.052 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:752 3.2c 3:3798f9d7:::100000001f7.000000ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.566292936, 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071715859000000007, 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 752 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436332+0000", 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.429e-06 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442426+0000", 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060933589999999996 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486527+0000", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044101711000000002 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486535+0000", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1749999999999999e-06 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486862+0000", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032728400000000001 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.487028+0000", 2024-04-25T15:53:30.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016650399999999999 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.487190+0000", 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016182699999999999 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.494699+0000", 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007509033 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507938+0000", 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013238689 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507980+0000", 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2268999999999998e-05 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.508047+0000", 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6579000000000001e-05 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:354 3.10 3:0ba0dc78:::10000000002.000000fd:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.4033072100000004, 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56691576200000005, 2024-04-25T15:53:30.057 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599318+0000", 2024-04-25T15:53:30.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599326+0000", 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9459999999999998e-06 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161759+0000", 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56243287799999997 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161783+0000", 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4057e-05 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161899+0000", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011615799999999999 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162007+0000", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107823 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162110+0000", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010235500000000001 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.163904+0000", 2024-04-25T15:53:30.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017941470000000001 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166138+0000", 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022346190000000002 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166186+0000", 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7902999999999999e-05 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166232+0000", 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6220999999999997e-05 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:953 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.7910682040000001, 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032562766999999999, 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.062 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 953 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211570+0000", 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3920999999999999e-05 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216848+0000", 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052779589999999996 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216884+0000", 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6423e-05 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216901+0000", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7033000000000001e-05 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217010+0000", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108792 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217136+0000", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125943 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217349+0000", 2024-04-25T15:53:30.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021362099999999999 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.240219+0000", 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022869490999999999 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.243777+0000", 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003557639 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.243864+0000", 2024-04-25T15:53:30.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7899000000000007e-05 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.244118+0000", 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025404600000000001 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:958 3.20 3:062e7a7c:::100000001f7.000000ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.7658998809999999, 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045110945999999999, 2024-04-25T15:53:30.067 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 958 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236727+0000", 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9780000000000001e-06 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242191+0000", 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054636880000000004 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242231+0000", 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9881e-05 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242250+0000", 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9837e-05 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242334+0000", 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3515000000000003e-05 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242412+0000", 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8503000000000004e-05 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242517+0000", 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010504499999999999 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.256116+0000", 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013598152000000001 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281714+0000", 2024-04-25T15:53:30.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025598415999999999 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281776+0000", 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2267000000000005e-05 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281835+0000", 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8663999999999999e-05 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:959 3.33 e82/18 3:cf81bd10:::100000001f7.000000bb:head v 82'15, mlcod=82'13)", 2024-04-25T15:53:30.072 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.757208697, 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039531930999999999, 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245416+0000", 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7100000000000011e-07 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249269+0000", 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038524689999999999 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249288+0000", 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9480999999999999e-05 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249301+0000", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2753e-05 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249366+0000", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5593999999999996e-05 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249388+0000", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1248000000000001e-05 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.284892+0000", 2024-04-25T15:53:30.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035504292999999999 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.284947+0000", 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5121999999999998e-05 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:960 3.1e e82/18 3:796b994c:::100000001f7.000000bc:head v 82'17, mlcod=81'15)", 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.7509016590000002, 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033456649999999998, 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.076 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251724+0000", 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5510000000000001e-06 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255631+0000", 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039072890000000004 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255650+0000", 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9082000000000001e-05 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255667+0000", 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6518e-05 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255746+0000", 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9821999999999996e-05 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255777+0000", 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0465999999999999e-05 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.285143+0000", 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029366040999999999 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.285179+0000", 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5880999999999997e-05 2024-04-25T15:53:30.079 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1006 3.10 3:08911dbf:::100000001f7.000000ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.486454854, 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051228383000000002, 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1006 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516173+0000", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0740000000000003e-06 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524942+0000", 2024-04-25T15:53:30.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087685709999999993 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.525634+0000", 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069202800000000002 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.525655+0000", 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1053e-05 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526010+0000", 2024-04-25T15:53:30.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035545200000000002 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526297+0000", 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028670600000000002 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526604+0000", 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030718299999999999 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.542597+0000", 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01599283 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567212+0000", 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024615298000000001 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567317+0000", 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105215 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567397+0000", 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9973000000000002e-05 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1007 3.36 3:6de076d9:::100000001f7.000000eb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.4776626290000001, 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052343149999999998, 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1007 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524964+0000", 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0419999999999998e-06 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531174+0000", 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006209661 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531738+0000", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00056402199999999996 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531765+0000", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6630000000000001e-05 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531886+0000", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121853 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.532033+0000", 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014617699999999999 2024-04-25T15:53:30.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.532238+0000", 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000205711 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.574143+0000", 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041904503000000003 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577154+0000", 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030110950000000001 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577224+0000", 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9806000000000002e-05 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577304+0000", 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0649999999999995e-05 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1008 3.2c 3:3798f9d7:::100000001f7.000000ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.4714360150000001, 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050443494999999998, 2024-04-25T15:53:30.089 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1008 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531190+0000", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6110000000000001e-06 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537725+0000", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006534949 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537770+0000", 2024-04-25T15:53:30.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4190000000000002e-05 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537789+0000", 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9355e-05 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537894+0000", 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104838 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538018+0000", 2024-04-25T15:53:30.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124463 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538199+0000", 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180638 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577390+0000", 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039191194999999998 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581497+0000", 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041064769999999999 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581571+0000", 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4548999999999994e-05 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581631+0000", 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0229999999999998e-05 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1009 3.3e e82/73 3:7fa01700:::100000001f7.000000ed:head v 82'15, mlcod=82'13)", 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.4679704089999999, 2024-04-25T15:53:30.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.043010400999999997, 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534655+0000", 2024-04-25T15:53:30.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4929999999999999e-06 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540839+0000", 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061837719999999997 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540872+0000", 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2777999999999997e-05 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540893+0000", 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1599000000000001e-05 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540996+0000", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102604 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541028+0000", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1986999999999997e-05 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577609+0000", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036581626999999999 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577664+0000", 2024-04-25T15:53:30.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4540999999999998e-05 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1010 3.33 e82/18 3:cf8599b8:::100000001f7.000000ee:head v 82'16, mlcod=82'14)", 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.460725491, 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041229029, 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:30.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541900+0000", 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3459999999999999e-06 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545834+0000", 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003934585 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545851+0000", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6923999999999999e-05 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545867+0000", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5715000000000001e-05 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545936+0000", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9429000000000002e-05 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545960+0000", 2024-04-25T15:53:30.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3342000000000001e-05 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.583050+0000", 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037089816999999997 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.583127+0000", 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7871e-05 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:30.101 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:30.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:30.103 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:30.099+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:30.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:30.209 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:30.210+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:30.299 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:30.299 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:30.299 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:30.299 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:30.304 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:30.302+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:30.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:30.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dc0e3f640 1 Processor -- start 2024-04-25T15:53:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dc0e3f640 1 -- start start 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dc0e3f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 0x7f3dbc0725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dc0e3f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dbc15a0b0 0x7f3dbc0746a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dc0e3f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dbc0c3bc0 con 0x7f3dbc153a00 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dc0e3f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3dbc153e10 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dc0e3f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3dbc068c40 con 0x7f3dbc0c4010 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dbad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dbc15a0b0 0x7f3dbc0746a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dbad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dbc15a0b0 0x7f3dbc0746a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40666/0 (socket says 172.21.15.67:40666) 2024-04-25T15:53:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3db9d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 0x7f3dbc0725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dbad9d640 1 -- 172.21.15.67:0/1528781330 learned_addr learned my addr 172.21.15.67:0/1528781330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dbad9d640 1 -- 172.21.15.67:0/1528781330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 msgr2=0x7f3dbc0725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/1528781330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 0x7f3dbc0725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dbad9d640 1 -- 172.21.15.67:0/1528781330 >> v1:172.21.15.176:6789/0 conn(0x7f3dbc153a00 legacy=0x7f3dbc073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.326+0000 7f3dbad9d640 1 -- 172.21.15.67:0/1528781330 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dbc0c53a0 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/1528781330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0746a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009e50 tx=0x7f3db401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4de3ad7605bbd73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3db959a640 1 -- 172.21.15.67:0/1528781330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db40a5700 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3db959a640 1 -- 172.21.15.67:0/1528781330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db40a58a0 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3db959a640 1 -- 172.21.15.67:0/1528781330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db40a5c70 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/1528781330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 msgr2=0x7f3dbc0746a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/1528781330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0746a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009e50 tx=0x7f3db401ca30 comp rx=0 tx=0).stop 2024-04-25T15:53:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/1528781330 shutdown_connections 2024-04-25T15:53:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/1528781330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0746a0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/1528781330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 0x7f3dbc0725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/1528781330 >> 172.21.15.67:0/1528781330 conn(0x7f3dbc0ce1c0 msgr2=0x7f3dbc0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/1528781330 shutdown_connections 2024-04-25T15:53:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.330+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/1528781330 wait complete. 2024-04-25T15:53:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 Processor -- start 2024-04-25T15:53:30.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 -- start start 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 0x7f3dbc0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dbc153a00 0x7f3dbc1547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc0686c0 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc0c4420 con 0x7f3dbc153a00 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3dbc074be0 con 0x7f3dbc0c4010 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40678/0 (socket says 172.21.15.67:40678) 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 -- 172.21.15.67:0/3478823700 learned_addr learned my addr 172.21.15.67:0/3478823700 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 0x7f3dbc0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dbc153a00 0x7f3dbc1547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 msgr2=0x7f3dbc0b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dbc0c4010 0x7f3dbc0b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dbc153a00 msgr2=0x7f3dbc1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dbc153a00 0x7f3dbc1547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dbc0b3cf0 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dbc153a00 0x7f3dbc1547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0b1160 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3db401a660 tx=0x7f3db40197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db40ba070 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db40b4640 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db4009510 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dbc157230 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dbc156f50 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.334+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3dbc06a480 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3db40b4a30 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3daaffd640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d9c06f400 0x7f3d9c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5915+0+0 (secure 0 0 0) 0x7f3db412bc50 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v83) ==== 54+0+16 (secure 0 0 0) 0x7f3dbc0c53a0 con 0x7f3dbc15a0b0 2024-04-25T15:53:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d9c06f400 0x7f3d9c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3dbc0b3f00 0x7f3dbc0ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc0b3420 con 0x7f3dbc0b3f00 2024-04-25T15:53:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3dbc0b3f00 0x7f3dbc0ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3dbc0b3f00 0x7f3dbc0ce5d0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d9c06f400 0x7f3d9c071910 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3dac007fd0 tx=0x7f3dac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.338+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3db0009d00 con 0x7f3dbc0b3f00 2024-04-25T15:53:30.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.354+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc156a00 con 0x7f3dbc0b3f00 2024-04-25T15:53:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.354+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc156a00 con 0x7f3dbc0b3f00 2024-04-25T15:53:30.355 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.354+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3dbc15d520 0x7f3dbc14cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.354+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc14d4c0 con 0x7f3dbc15d520 2024-04-25T15:53:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.354+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3dbc15d520 0x7f3dbc14cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.354+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3dbc15d520 0x7f3dbc14cf80 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.354+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc14d4c0 con 0x7f3dbc15d520 2024-04-25T15:53:30.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.370+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc0b0780 con 0x7f3dbc15d520 2024-04-25T15:53:30.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.374+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc0b0780 con 0x7f3dbc15d520 2024-04-25T15:53:30.373 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:30.374 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:30.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.374+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3dbc1f68f0 0x7f3dbc06c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.374+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc0c32b0 con 0x7f3dbc1f68f0 2024-04-25T15:53:30.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.374+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3dbc1f68f0 0x7f3dbc06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.374+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3dbc1f68f0 0x7f3dbc06c640 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.374+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc0c32b0 con 0x7f3dbc1f68f0 2024-04-25T15:53:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.386+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc0cd9a0 con 0x7f3dbc1f68f0 2024-04-25T15:53:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.390+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc0cd9a0 con 0x7f3dbc1f68f0 2024-04-25T15:53:30.390 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.390+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3dbc063090 0x7f3dbc0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.390+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc0cd6c0 con 0x7f3dbc063090 2024-04-25T15:53:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.390+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3dbc063090 0x7f3dbc0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.390+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3dbc063090 0x7f3dbc0634a0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.390+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc0cd6c0 con 0x7f3dbc063090 2024-04-25T15:53:30.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.402+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc0cd6c0 con 0x7f3dbc063090 2024-04-25T15:53:30.405 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:30.406+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:30.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.406+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc0cd6c0 con 0x7f3dbc063090 2024-04-25T15:53:30.409 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:30.409 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:30.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.410+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3dbc14e740 0x7f3dbc14eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.410+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc0cd9a0 con 0x7f3dbc14e740 2024-04-25T15:53:30.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.410+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3dbc14e740 0x7f3dbc14eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.410+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3dbc14e740 0x7f3dbc14eb50 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.410+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc0cd9a0 con 0x7f3dbc14e740 2024-04-25T15:53:30.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.422+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc0cd9a0 con 0x7f3dbc14e740 2024-04-25T15:53:30.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.430+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc0cd9a0 con 0x7f3dbc14e740 2024-04-25T15:53:30.429 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:30.430 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:30.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.430+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3dbc02c9e0 0x7f3dbc02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.430+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc14f5a0 con 0x7f3dbc02c9e0 2024-04-25T15:53:30.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.430+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3dbc02c9e0 0x7f3dbc02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.430+0000 7f3db9d9b640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3dbc02c9e0 0x7f3dbc02cdf0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.430+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc14f5a0 con 0x7f3dbc02c9e0 2024-04-25T15:53:30.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.446+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc14f5a0 con 0x7f3dbc02c9e0 2024-04-25T15:53:30.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.450+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc14f5a0 con 0x7f3dbc02c9e0 2024-04-25T15:53:30.451 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.450+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3dbc02d840 0x7f3dbc20b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.450+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc02dcd0 con 0x7f3dbc02d840 2024-04-25T15:53:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.450+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3dbc02d840 0x7f3dbc20b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.450+0000 7f3dbad9d640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3dbc02d840 0x7f3dbc20b3c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.454+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc02dcd0 con 0x7f3dbc02d840 2024-04-25T15:53:30.461 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:30.461 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:30.461 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:30.461 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:30.461 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:30.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.466+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc0008e0 con 0x7f3dbc02d840 2024-04-25T15:53:30.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.470+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc0008e0 con 0x7f3dbc02d840 2024-04-25T15:53:30.471 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.470+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3dbc20be10 0x7f3dbc20e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.470+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3dbc211000 con 0x7f3dbc20be10 2024-04-25T15:53:30.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.470+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3dbc20be10 0x7f3dbc20e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:30.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.474+0000 7f3dba59c640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3dbc20be10 0x7f3dbc20e230 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:30.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.474+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dbc211000 con 0x7f3dbc20be10 2024-04-25T15:53:30.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.490+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3dbc20e770 con 0x7f3dbc20be10 2024-04-25T15:53:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.490+0000 7f3daaffd640 1 -- 172.21.15.67:0/3478823700 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3dbc20e770 con 0x7f3dbc20be10 2024-04-25T15:53:30.492 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3dbc0b3f00 msgr2=0x7f3dbc0ce5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3dbc0b3f00 0x7f3dbc0ce5d0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3dbc15d520 msgr2=0x7f3dbc14cf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3dbc15d520 0x7f3dbc14cf80 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3dbc1f68f0 msgr2=0x7f3dbc06c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3dbc1f68f0 0x7f3dbc06c640 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3dbc063090 msgr2=0x7f3dbc0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3dbc063090 0x7f3dbc0634a0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3dbc14e740 msgr2=0x7f3dbc14eb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3dbc14e740 0x7f3dbc14eb50 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dbad9d640 1 -- 172.21.15.67:0/3478823700 reap_dead start 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3dbc02c9e0 msgr2=0x7f3dbc02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3dbc02c9e0 0x7f3dbc02cdf0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dbad9d640 1 -- 172.21.15.67:0/3478823700 reap_dead start 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3dbc02d840 msgr2=0x7f3dbc20b3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3dbc02d840 0x7f3dbc20b3c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3dbc20be10 msgr2=0x7f3dbc20e230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3dbc20be10 0x7f3dbc20e230 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dbad9d640 1 -- 172.21.15.67:0/3478823700 reap_dead start 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d9c06f400 msgr2=0x7f3d9c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d9c06f400 0x7f3d9c071910 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3dac007fd0 tx=0x7f3dac009040 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 msgr2=0x7f3dbc0b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0b1160 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3db401a660 tx=0x7f3db40197d0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 shutdown_connections 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d9c06f400 0x7f3d9c071910 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 --2- 172.21.15.67:0/3478823700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc15a0b0 0x7f3dbc0b1160 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 >> 172.21.15.67:0/3478823700 conn(0x7f3dbc0ce1c0 msgr2=0x7f3dbc1590d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:30.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 shutdown_connections 2024-04-25T15:53:30.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:30.494+0000 7f3dc0e3f640 1 -- 172.21.15.67:0/3478823700 wait complete. 2024-04-25T15:53:30.506 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:30.506+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:30.606 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:30.606+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:30.628 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:30.628 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:30.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:30.708 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:30.706+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:30.801 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:30.801 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:30.809 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:30.810+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:30.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:30.910 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:30.910+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:31.004 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:31.004 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:31.011 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:31.010+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:31.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:31.112 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:31.111+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:31.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:53:31.213 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:31.214+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:31.219 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:31.219 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 readers = 7118598.00 kB/sec 2024-04-25T15:53:31.219 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 readers = 7082008.85 kB/sec 2024-04-25T15:53:31.219 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 7118598.00 kB/sec 2024-04-25T15:53:31.219 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 7118598.00 kB/sec 2024-04-25T15:53:31.219 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 7118598.00 kB/sec 2024-04-25T15:53:31.219 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:53:31.225 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:31.225 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:31.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c58327640 1 Processor -- start 2024-04-25T15:53:31.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c58327640 1 -- start start 2024-04-25T15:53:31.317 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c58327640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 0x7f5c5015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c58327640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c5015de10 con 0x7f5c50071bc0 2024-04-25T15:53:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c58327640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c50072390 con 0x7f5c500ca540 2024-04-25T15:53:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c58327640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c5015ebb0 con 0x7f5c5015a720 2024-04-25T15:53:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c560c3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5c50071bc0 0x7f5c500cbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39462/0 (socket says 172.21.15.67:39462) 2024-04-25T15:53:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c560c3640 1 -- 172.21.15.67:0/2286509823 learned_addr learned my addr 172.21.15.67:0/2286509823 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:31.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c558c2640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 0x7f5c5015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2498820621 0 0) 0x7f5c5015de10 con 0x7f5c50071bc0 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c3c003530 con 0x7f5c50071bc0 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 847998725 0 0) 0x7f5c3c003530 con 0x7f5c50071bc0 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 msgr2=0x7f5c5015ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 0x7f5c5015ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 >> v1:172.21.15.176:6789/0 conn(0x7f5c500ca540 legacy=0x7f5c50164fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5015e0d0 con 0x7f5c50071bc0 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f5c4c001680 con 0x7f5c50071bc0 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 >> v1:172.21.15.67:6789/0 conn(0x7f5c50071bc0 legacy=0x7f5c500cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c3c0039f0 0x7f5c3c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 0x7f5c3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3c004d50 0x7f5c3c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5015ebb0 con 0x7f5c3c004d50 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c00bb00 con 0x7f5c3c0039f0 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3c010110 con 0x7f5c5015a720 2024-04-25T15:53:31.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c568c4640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3c004d50 0x7f5c3c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c560c3640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c3c0039f0 0x7f5c3c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c558c2640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 0x7f5c3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c568c4640 1 -- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 msgr2=0x7f5c3c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c568c4640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c5015a720 0x7f5c3c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c568c4640 1 -- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c3c0039f0 msgr2=0x7f5c3c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c568c4640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c3c0039f0 0x7f5c3c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.318+0000 7f5c568c4640 1 -- 172.21.15.67:0/2286509823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c003530 con 0x7f5c3c004d50 2024-04-25T15:53:31.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c568c4640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3c004d50 0x7f5c3c005160 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5c48001e40 tx=0x7f5c4801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3180c9353668628 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48001a90 con 0x7f5c3c004d50 2024-04-25T15:53:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c48003b80 con 0x7f5c3c004d50 2024-04-25T15:53:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c550c1640 1 -- 172.21.15.67:0/2286509823 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48005080 con 0x7f5c3c004d50 2024-04-25T15:53:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c58327640 1 -- 172.21.15.67:0/2286509823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3c004d50 msgr2=0x7f5c3c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c58327640 1 --2- 172.21.15.67:0/2286509823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c3c004d50 0x7f5c3c005160 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5c48001e40 tx=0x7f5c4801c760 comp rx=0 tx=0).stop 2024-04-25T15:53:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c568c4640 1 -- 172.21.15.67:0/2286509823 reap_dead start 2024-04-25T15:53:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c58327640 1 -- 172.21.15.67:0/2286509823 shutdown_connections 2024-04-25T15:53:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c58327640 1 -- 172.21.15.67:0/2286509823 >> 172.21.15.67:0/2286509823 conn(0x7f5c500700b0 msgr2=0x7f5c500704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c58327640 1 -- 172.21.15.67:0/2286509823 shutdown_connections 2024-04-25T15:53:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.322+0000 7f5c58327640 1 -- 172.21.15.67:0/2286509823 wait complete. 2024-04-25T15:53:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 Processor -- start 2024-04-25T15:53:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 -- start start 2024-04-25T15:53:31.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c5014ddf0 0x7f5c5014e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c501f23a0 0x7f5c501f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c50149960 0x7f5c50149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5c500c93b0 con 0x7f5c501f23a0 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5015e720 con 0x7f5c5014ddf0 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c500cc2d0 con 0x7f5c50149960 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c501f23a0 0x7f5c501f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c501f23a0 0x7f5c501f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40690/0 (socket says 172.21.15.67:40690) 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 -- 172.21.15.67:0/4196119003 learned_addr learned my addr 172.21.15.67:0/4196119003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c568c4640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c50149960 0x7f5c50149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c560c3640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c5014ddf0 0x7f5c5014e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 -- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c50149960 msgr2=0x7f5c50149d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c50149960 0x7f5c50149d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 -- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c5014ddf0 msgr2=0x7f5c5014e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c5014ddf0 0x7f5c5014e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:31.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 -- 172.21.15.67:0/4196119003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c500b5c80 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c568c4640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c50149960 0x7f5c50149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c558c2640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c501f23a0 0x7f5c501f27b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5c400115c0 tx=0x7f5c40011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c46ffd640 1 -- 172.21.15.67:0/4196119003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c40004220 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c46ffd640 1 -- 172.21.15.67:0/4196119003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c400043c0 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c50159a40 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c46ffd640 1 -- 172.21.15.67:0/4196119003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000f8f0 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c5015a000 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.326+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c5006a480 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.330+0000 7f5c46ffd640 1 -- 172.21.15.67:0/4196119003 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5c4000fa90 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.330+0000 7f5c46ffd640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3406f430 0x7f5c34071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.330+0000 7f5c46ffd640 1 -- 172.21.15.67:0/4196119003 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5915+0+0 (secure 0 0 0) 0x7f5c4008fd80 con 0x7f5c501f23a0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.330+0000 7f5c560c3640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3406f430 0x7f5c34071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:31.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.330+0000 7f5c560c3640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3406f430 0x7f5c34071940 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c001ec0 tx=0x7f5c4c0019b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:31.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.334+0000 7f5c46ffd640 1 -- 172.21.15.67:0/4196119003 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5c40019030 con 0x7f5c501f23a0 2024-04-25T15:53:31.415 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:31.415+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:31.436 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:31.436 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:31.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:31.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:31.506+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f5c5006f740 con 0x7f5c501f23a0 2024-04-25T15:53:31.516 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:31.515+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:31.617 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:31.618+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:31.645 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:31.645 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:31.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:31.718 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:31.715+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:31.819 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:31.818+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:31.858 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:31.858 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:31.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:31.919 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:31.918+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:32.020 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:32.019+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:32.065 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:32.065 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:32.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:32.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:32.126+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:32.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.166+0000 7f5c46ffd640 1 -- 172.21.15.67:0/4196119003 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v84) ==== 97+0+0 (secure 0 0 0) 0x7f5c4005cd10 con 0x7f5c501f23a0 2024-04-25T15:53:32.167 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:53:32.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3406f430 msgr2=0x7f5c34071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3406f430 0x7f5c34071940 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c001ec0 tx=0x7f5c4c0019b0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c501f23a0 msgr2=0x7f5c501f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c501f23a0 0x7f5c501f27b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5c400115c0 tx=0x7f5c40011a80 comp rx=0 tx=0).stop 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 shutdown_connections 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5c3406f430 0x7f5c34071940 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c50149960 0x7f5c50149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c501f23a0 0x7f5c501f27b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 --2- 172.21.15.67:0/4196119003 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c5014ddf0 0x7f5c5014e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 >> 172.21.15.67:0/4196119003 conn(0x7f5c500700b0 msgr2=0x7f5c500c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 shutdown_connections 2024-04-25T15:53:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.170+0000 7f5c58327640 1 -- 172.21.15.67:0/4196119003 wait complete. 2024-04-25T15:53:32.222 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:32.219+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:32.228 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:32.229 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:32.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:32.329 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:32.330+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:32.338 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:32.338 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:32.338 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T15:53:32.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:53:32.423 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:32.422+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:32.430 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:32.430 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:32.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.482+0000 7f2726db9640 1 Processor -- start 2024-04-25T15:53:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.482+0000 7f2726db9640 1 -- start start 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f2726db9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720160510 0x7f272006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f2726db9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272006d780 con 0x7f27200b5d70 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f2726db9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2720160920 con 0x7f272006ca60 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f2726db9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f272015d510 con 0x7f2720160510 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27200b5d70 0x7f272015fe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39464/0 (socket says 172.21.15.67:39464) 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 -- 172.21.15.67:0/105766410 learned_addr learned my addr 172.21.15.67:0/105766410 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f2725356640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720160510 0x7f272006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922091730 0 0) 0x7f272006d780 con 0x7f27200b5d70 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2708003530 con 0x7f27200b5d70 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2230598912 0 0) 0x7f2708003530 con 0x7f27200b5d70 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720160510 msgr2=0x7f272006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720160510 0x7f272006ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 >> v1:172.21.15.176:6789/0 conn(0x7f272006ca60 legacy=0x7f27200b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2720160be0 con 0x7f27200b5d70 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f27140016a0 con 0x7f27200b5d70 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 >> v1:172.21.15.67:6789/0 conn(0x7f27200b5d70 legacy=0x7f272015fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27080039f0 0x7f2708003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708004810 0x7f2708004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720160510 0x7f2708005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f272015d510 con 0x7f2708004810 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f270800bbc0 con 0x7f27080039f0 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f270800bdc0 con 0x7f2720160510 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708004810 0x7f2708004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f2724b55640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27080039f0 0x7f2708003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 -- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720160510 msgr2=0x7f2708005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720160510 0x7f2708005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 -- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27080039f0 msgr2=0x7f2708003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27080039f0 0x7f2708003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.486+0000 7f271ffff640 1 -- 172.21.15.67:0/105766410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2708003530 con 0x7f2708004810 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f271ffff640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708004810 0x7f2708004c20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f27200c7150 tx=0x7f271401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e01eda0a9297a91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714007d20 con 0x7f2708004810 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27140a9020 con 0x7f2708004810 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f271f7fe640 1 -- 172.21.15.67:0/105766410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27140a5b40 con 0x7f2708004810 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 -- 172.21.15.67:0/105766410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708004810 msgr2=0x7f2708004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 --2- 172.21.15.67:0/105766410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2708004810 0x7f2708004c20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f27200c7150 tx=0x7f271401cc00 comp rx=0 tx=0).stop 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2725356640 1 -- 172.21.15.67:0/105766410 reap_dead start 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 -- 172.21.15.67:0/105766410 shutdown_connections 2024-04-25T15:53:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 -- 172.21.15.67:0/105766410 >> 172.21.15.67:0/105766410 conn(0x7f272006e420 msgr2=0x7f272006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 -- 172.21.15.67:0/105766410 shutdown_connections 2024-04-25T15:53:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 -- 172.21.15.67:0/105766410 wait complete. 2024-04-25T15:53:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 Processor -- start 2024-04-25T15:53:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.490+0000 7f2726db9640 1 -- start start 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2708004810 0x7f27200d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720154fd0 0x7f27201558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27200d12c0 0x7f27201527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f272015d1a0 con 0x7f27200d12c0 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f272015cec0 con 0x7f2708004810 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f272015c890 con 0x7f2720154fd0 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27200d12c0 0x7f27201527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27200d12c0 0x7f27201527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40696/0 (socket says 172.21.15.67:40696) 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 -- 172.21.15.67:0/583681112 learned_addr learned my addr 172.21.15.67:0/583681112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f271ffff640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720154fd0 0x7f27201558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2724b55640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2708004810 0x7f27200d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 -- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720154fd0 msgr2=0x7f27201558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720154fd0 0x7f27201558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 -- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2708004810 msgr2=0x7f27200d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2708004810 0x7f27200d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 -- 172.21.15.67:0/583681112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27200d26f0 con 0x7f27200d12c0 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2725356640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27200d12c0 0x7f27201527b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f272006c6f0 tx=0x7f271800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f271d7fa640 1 -- 172.21.15.67:0/583681112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271800bd10 con 0x7f27200d12c0 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f271d7fa640 1 -- 172.21.15.67:0/583681112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f271801a070 con 0x7f27200d12c0 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2720154040 con 0x7f27200d12c0 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f271d7fa640 1 -- 172.21.15.67:0/583681112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718014640 con 0x7f27200d12c0 2024-04-25T15:53:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27201542e0 con 0x7f27200d12c0 2024-04-25T15:53:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27200d28c0 con 0x7f27200d12c0 2024-04-25T15:53:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f271d7fa640 1 -- 172.21.15.67:0/583681112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2718013070 con 0x7f27200d12c0 2024-04-25T15:53:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f271d7fa640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f270c06f430 0x7f270c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.494+0000 7f271d7fa640 1 -- 172.21.15.67:0/583681112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5915+0+0 (secure 0 0 0) 0x7f271808f5a0 con 0x7f27200d12c0 2024-04-25T15:53:32.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.498+0000 7f2724b55640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f270c06f430 0x7f270c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.498+0000 7f2724b55640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f270c06f430 0x7f270c071940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2710000c10 tx=0x7f2710009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.502+0000 7f271d7fa640 1 -- 172.21.15.67:0/583681112 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f27200d28c0 con 0x7f27200d12c0 2024-04-25T15:53:32.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:32.557 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:32.558+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:32.623 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:32.623 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:32.625 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:32.623+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:32.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77f5551640 1 Processor -- start 2024-04-25T15:53:32.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77f5551640 1 -- start start 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77f5551640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77f006ca60 0x7f77f006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77f5551640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 0x7f77f0160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77f5551640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77f006d780 con 0x7f77f00b5660 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77f5551640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f77f015d510 con 0x7f77f006ca60 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77f5551640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77f0160be0 con 0x7f77f015fe00 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77eeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77f006ca60 0x7f77f006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77eeffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77f006ca60 0x7f77f006ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38304/0 (socket says 172.21.15.67:38304) 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77e67fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f77f00b5660 0x7f77f00b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36282/0 (socket says 172.21.15.67:36282) 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77eeffd640 1 -- 172.21.15.67:0/1373898580 learned_addr learned my addr 172.21.15.67:0/1373898580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/1373898580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 0x7f77f0160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77eeffd640 1 -- 172.21.15.67:0/1373898580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 msgr2=0x7f77f0160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:32.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77eeffd640 1 --2- 172.21.15.67:0/1373898580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 0x7f77f0160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77eeffd640 1 -- 172.21.15.67:0/1373898580 >> v1:172.21.15.67:6789/0 conn(0x7f77f00b5660 legacy=0x7f77f00b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77ee7fc640 1 -- 172.21.15.67:0/1373898580 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744628404 0 0) 0x7f77f006d780 con 0x7f77f00b5660 2024-04-25T15:53:32.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.646+0000 7f77eeffd640 1 -- 172.21.15.67:0/1373898580 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f015cbe0 con 0x7f77f006ca60 2024-04-25T15:53:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77eeffd640 1 --2- 172.21.15.67:0/1373898580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f006ce70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f77e00098a0 tx=0x7f77e001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84e0557d34cafbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77ee7fc640 1 -- 172.21.15.67:0/1373898580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e00b5070 con 0x7f77f006ca60 2024-04-25T15:53:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77ee7fc640 1 -- 172.21.15.67:0/1373898580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e00a8a30 con 0x7f77f006ca60 2024-04-25T15:53:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77ee7fc640 1 -- 172.21.15.67:0/1373898580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e00a8e00 con 0x7f77f006ca60 2024-04-25T15:53:32.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 -- 172.21.15.67:0/1373898580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 msgr2=0x7f77f006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 --2- 172.21.15.67:0/1373898580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f006ce70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f77e00098a0 tx=0x7f77e001ed30 comp rx=0 tx=0).stop 2024-04-25T15:53:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 -- 172.21.15.67:0/1373898580 shutdown_connections 2024-04-25T15:53:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 --2- 172.21.15.67:0/1373898580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 0x7f77f0160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 --2- 172.21.15.67:0/1373898580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f006ce70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 -- 172.21.15.67:0/1373898580 >> 172.21.15.67:0/1373898580 conn(0x7f77f006e420 msgr2=0x7f77f006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 -- 172.21.15.67:0/1373898580 shutdown_connections 2024-04-25T15:53:32.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.650+0000 7f77f5551640 1 -- 172.21.15.67:0/1373898580 wait complete. 2024-04-25T15:53:32.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 Processor -- start 2024-04-25T15:53:32.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 -- start start 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f01537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77f00b5660 0x7f77f015a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 0x7f77f014f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77f015c890 con 0x7f77f006ca60 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77f015d1a0 con 0x7f77f00b5660 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77f00b6e90 con 0x7f77f015fe00 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f01537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f01537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38308/0 (socket says 172.21.15.67:38308) 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 -- 172.21.15.67:0/3635664024 learned_addr learned my addr 172.21.15.67:0/3635664024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:32.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77e67fc640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77f00b5660 0x7f77f015a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 0x7f77f014f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 msgr2=0x7f77f014f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77f015fe00 0x7f77f014f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77f00b5660 msgr2=0x7f77f015a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77f00b5660 0x7f77f015a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f0153d20 con 0x7f77f006ca60 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f01537e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f77e001c960 tx=0x7f77e001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e00b5070 con 0x7f77f006ca60 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e00b0c20 con 0x7f77f006ca60 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f00c7d00 con 0x7f77f006ca60 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e00bb410 con 0x7f77f006ca60 2024-04-25T15:53:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f01f3610 con 0x7f77f006ca60 2024-04-25T15:53:32.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.654+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f77f006a480 con 0x7f77f006ca60 2024-04-25T15:53:32.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f77e00c2060 con 0x7f77f006ca60 2024-04-25T15:53:32.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77e7fff640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f77d006f400 0x7f77d0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5915+0+0 (secure 0 0 0) 0x7f77e012c9d0 con 0x7f77f006ca60 2024-04-25T15:53:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v84) ==== 54+0+16 (secure 0 0 0) 0x7f77e0018890 con 0x7f77f006ca60 2024-04-25T15:53:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77e67fc640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f77d006f400 0x7f77d0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f77f0159b10 0x7f77f014fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77f01f3f40 con 0x7f77f0159b10 2024-04-25T15:53:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f77f0159b10 0x7f77f014fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f77f0159b10 0x7f77f014fac0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.658+0000 7f77e67fc640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f77d006f400 0x7f77d0071910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f77f00d2b30 tx=0x7f77d8007c30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.662+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f01f3f40 con 0x7f77f0159b10 2024-04-25T15:53:32.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.670+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f27201544b0 con 0x7f27200d12c0 2024-04-25T15:53:32.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.670+0000 7f271d7fa640 1 -- 172.21.15.67:0/583681112 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v84) ==== 81+0+35368 (secure 0 0 0) 0x7f271808de20 con 0x7f27200d12c0 2024-04-25T15:53:32.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:53:32.673 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:32.673 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 84, 2024-04-25T15:53:32.673 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:53:32.673 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:53:32.160148+0000", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:53:14.990625+0000", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5806080, 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:53:32.674 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:32.675 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:32.676 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:32.677 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:32.678 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:32.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:32.681 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:32.682 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:32.683 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:32.684 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.75, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.75, 2024-04-25T15:53:32.685 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:32.686 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:32.688 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.689 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.690 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.3200000524520874, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.3200000524520874, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.5, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.5, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:32.691 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:53:32.693 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:53:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:53:32.696 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0.2953948974609375, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:53:32.700 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 73, 2024-04-25T15:53:32.701 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:53:32.704 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.706 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.708 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:53:32.709 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.710 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:53:32.712 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.713 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:32.714 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.717 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.719 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.721 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.723 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:32.724 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.726 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.727 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:53:32.728 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:53:32.729 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.730 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.731 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.732 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:32.733 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.18", 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: 4 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:53:32.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.3e", 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 0, 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:53:32.737 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:53:32.738 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:53:32.739 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:53:32.739 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:53:32.739 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.739 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f270c06f430 msgr2=0x7f270c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f270c06f430 0x7f270c071940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2710000c10 tx=0x7f2710009040 comp rx=0 tx=0).stop 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27200d12c0 msgr2=0x7f27201527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27200d12c0 0x7f27201527b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f272006c6f0 tx=0x7f271800b480 comp rx=0 tx=0).stop 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 shutdown_connections 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f270c06f430 0x7f270c071940 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27200d12c0 0x7f27201527b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2720154fd0 0x7f27201558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 --2- 172.21.15.67:0/583681112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2708004810 0x7f27200d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 >> 172.21.15.67:0/583681112 conn(0x7f272006e420 msgr2=0x7f27200c7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:32.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 shutdown_connections 2024-04-25T15:53:32.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f2726db9640 1 -- 172.21.15.67:0/583681112 wait complete. 2024-04-25T15:53:32.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.674+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f0150000 con 0x7f77f0159b10 2024-04-25T15:53:32.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.678+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f0150000 con 0x7f77f0159b10 2024-04-25T15:53:32.741 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:32.742 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:32.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.678+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f77f0157200 0x7f77f01f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.678+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f77f006dd20 con 0x7f77f0157200 2024-04-25T15:53:32.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.678+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f77f0157200 0x7f77f01f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.678+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f77f0157200 0x7f77f01f6590 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.678+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f006dd20 con 0x7f77f0157200 2024-04-25T15:53:32.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.694+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f00cd380 con 0x7f77f0157200 2024-04-25T15:53:32.744 INFO:tasks.thrashosds.thrasher:j is {'epoch': 84, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:53:32.160148+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:53:14.990625+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 0.2953948974609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [1, 4]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '3.3e', 'mappings': [{'from': 0, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:53:32.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:53:32.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.694+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f00cd380 con 0x7f77f0157200 2024-04-25T15:53:32.745 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.694+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f77f0155040 0x7f77f01f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.694+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f77f00b5ca0 con 0x7f77f0155040 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.694+0000 7f77e67fc640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f77f0155040 0x7f77f01f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.694+0000 7f77e67fc640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f77f0155040 0x7f77f01f6fe0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.694+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f00b5ca0 con 0x7f77f0155040 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.710+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f00b5ca0 con 0x7f77f0155040 2024-04-25T15:53:32.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.710+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f00b5ca0 con 0x7f77f0155040 2024-04-25T15:53:32.747 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:32.747 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.710+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f77f0063c90 0x7f77f015de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.710+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f77f00b5ca0 con 0x7f77f0063c90 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.710+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f77f0063c90 0x7f77f015de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.710+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f77f0063c90 0x7f77f015de40 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.714+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f00b5ca0 con 0x7f77f0063c90 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.726+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f00b5ca0 con 0x7f77f0063c90 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.726+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f00b5ca0 con 0x7f77f0063c90 2024-04-25T15:53:32.748 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:32.749 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:32.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.726+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f77f015eb40 0x7f77f002c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.726+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f77f00b5ca0 con 0x7f77f015eb40 2024-04-25T15:53:32.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.730+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f77f015eb40 0x7f77f002c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.730+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f77f015eb40 0x7f77f002c9e0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.730+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f00b5ca0 con 0x7f77f015eb40 2024-04-25T15:53:32.750 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:32.738+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.746+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f00b5ca0 con 0x7f77f015eb40 2024-04-25T15:53:32.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.750+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f00b5ca0 con 0x7f77f015eb40 2024-04-25T15:53:32.751 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:32.751 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:32.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.750+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f77f002d430 0x7f77f002d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.750+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f77f00b6330 con 0x7f77f002d430 2024-04-25T15:53:32.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.750+0000 7f77e67fc640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f77f002d430 0x7f77f002d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.750+0000 7f77e67fc640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f77f002d430 0x7f77f002d840 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.750+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f00b6330 con 0x7f77f002d430 2024-04-25T15:53:32.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.766+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f00b6330 con 0x7f77f002d430 2024-04-25T15:53:32.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.770+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f00b6330 con 0x7f77f002d430 2024-04-25T15:53:32.770 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:32.771 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:32.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.770+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f77f0209e30 0x7f77f020c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.770+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f77f002dd80 con 0x7f77f0209e30 2024-04-25T15:53:32.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.770+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f77f0209e30 0x7f77f020c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.770+0000 7f77ef7fe640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f77f0209e30 0x7f77f020c250 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.774+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f002dd80 con 0x7f77f0209e30 2024-04-25T15:53:32.785 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:32.786 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:32.786 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:32.786 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:32.786 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.786+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f002dd80 con 0x7f77f0209e30 2024-04-25T15:53:32.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.790+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f002dd80 con 0x7f77f0209e30 2024-04-25T15:53:32.792 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.790+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f77f020cca0 0x7f77f020f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.790+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f77f0211fd0 con 0x7f77f020cca0 2024-04-25T15:53:32.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.794+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f77f020cca0 0x7f77f020f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.794+0000 7f77eeffd640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f77f020cca0 0x7f77f020f0c0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.794+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77f0211fd0 con 0x7f77f020cca0 2024-04-25T15:53:32.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.810+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f77f020f600 con 0x7f77f020cca0 2024-04-25T15:53:32.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77e7fff640 1 -- 172.21.15.67:0/3635664024 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f77f020f600 con 0x7f77f020cca0 2024-04-25T15:53:32.815 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f77f0159b10 msgr2=0x7f77f014fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f77f0159b10 0x7f77f014fac0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f77f0157200 msgr2=0x7f77f01f6590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f77f0157200 0x7f77f01f6590 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f77f0155040 msgr2=0x7f77f01f6fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f77f0155040 0x7f77f01f6fe0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f77f0063c90 msgr2=0x7f77f015de40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f77f0063c90 0x7f77f015de40 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f77f015eb40 msgr2=0x7f77f002c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f77f015eb40 0x7f77f002c9e0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f77f002d430 msgr2=0x7f77f002d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f77f002d430 0x7f77f002d840 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f77f0209e30 msgr2=0x7f77f020c250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f77f0209e30 0x7f77f020c250 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77ef7fe640 1 -- 172.21.15.67:0/3635664024 reap_dead start 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f77f020cca0 msgr2=0x7f77f020f0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f77f020cca0 0x7f77f020f0c0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77ef7fe640 1 -- 172.21.15.67:0/3635664024 reap_dead start 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77ef7fe640 1 -- 172.21.15.67:0/3635664024 reap_dead start 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77ef7fe640 1 -- 172.21.15.67:0/3635664024 reap_dead start 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f77d006f400 msgr2=0x7f77d0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f77d006f400 0x7f77d0071910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f77f00d2b30 tx=0x7f77d8007c30 comp rx=0 tx=0).stop 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 msgr2=0x7f77f01537e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f01537e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f77e001c960 tx=0x7f77e001bad0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 shutdown_connections 2024-04-25T15:53:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f77d006f400 0x7f77d0071910 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 --2- 172.21.15.67:0/3635664024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f006ca60 0x7f77f01537e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 >> 172.21.15.67:0/3635664024 conn(0x7f77f006e420 msgr2=0x7f77f006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 shutdown_connections 2024-04-25T15:53:32.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.814+0000 7f77f5551640 1 -- 172.21.15.67:0/3635664024 wait complete. 2024-04-25T15:53:32.841 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:32.839+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff19dd1f640 1 Processor -- start 2024-04-25T15:53:32.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff19dd1f640 1 -- start start 2024-04-25T15:53:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff19dd1f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff19806c640 0x7ff198154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff19dd1f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1980749d0 con 0x7ff19815e8b0 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff19dd1f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19806e510 con 0x7ff19815e4a0 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff19dd1f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff19815dca0 con 0x7ff19806c640 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff1977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff19806c640 0x7ff198154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff1977fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff19806c640 0x7ff198154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38322/0 (socket says 172.21.15.67:38322) 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff1977fe640 1 -- 172.21.15.67:0/1799293652 learned_addr learned my addr 172.21.15.67:0/1799293652 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff1977fe640 1 -- 172.21.15.67:0/1799293652 >> v1:172.21.15.176:6789/0 conn(0x7ff19815e4a0 legacy=0x7ff1980c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff1977fe640 1 -- 172.21.15.67:0/1799293652 >> v1:172.21.15.67:6789/0 conn(0x7ff19815e8b0 legacy=0x7ff198164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff1977fe640 1 -- 172.21.15.67:0/1799293652 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19815df80 con 0x7ff19806c640 2024-04-25T15:53:32.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.886+0000 7ff1967fc640 1 -- 172.21.15.67:0/1799293652 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3541172433 0 0) 0x7ff1980749d0 con 0x7ff19815e8b0 2024-04-25T15:53:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff1977fe640 1 --2- 172.21.15.67:0/1799293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff198154a60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff188009870 tx=0x7ff18801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d584b5c6a16e076 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff1967fc640 1 -- 172.21.15.67:0/1799293652 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1880ab620 con 0x7ff19806c640 2024-04-25T15:53:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff1967fc640 1 -- 172.21.15.67:0/1799293652 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1880b0070 con 0x7ff19806c640 2024-04-25T15:53:32.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff1967fc640 1 -- 172.21.15.67:0/1799293652 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1880ab9e0 con 0x7ff19806c640 2024-04-25T15:53:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff19dd1f640 1 -- 172.21.15.67:0/1799293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 msgr2=0x7ff198154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/1799293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff198154a60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ff188009870 tx=0x7ff18801c490 comp rx=0 tx=0).stop 2024-04-25T15:53:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff19dd1f640 1 -- 172.21.15.67:0/1799293652 shutdown_connections 2024-04-25T15:53:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/1799293652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff198154a60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff19dd1f640 1 -- 172.21.15.67:0/1799293652 >> 172.21.15.67:0/1799293652 conn(0x7ff198070db0 msgr2=0x7ff1980ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:32.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff19dd1f640 1 -- 172.21.15.67:0/1799293652 shutdown_connections 2024-04-25T15:53:32.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.890+0000 7ff19dd1f640 1 -- 172.21.15.67:0/1799293652 wait complete. 2024-04-25T15:53:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 Processor -- start 2024-04-25T15:53:32.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 -- start start 2024-04-25T15:53:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff1980ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff19815e4a0 0x7ff198155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff19815e8b0 0x7ff1981485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff1980cad60 con 0x7ff19806c640 2024-04-25T15:53:32.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff1980c86d0 con 0x7ff19815e4a0 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff1980cd0a0 con 0x7ff19815e8b0 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff1980ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff1980ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38330/0 (socket says 172.21.15.67:38330) 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 -- 172.21.15.67:0/186583632 learned_addr learned my addr 172.21.15.67:0/186583632 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff196ffd640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff19815e4a0 0x7ff198155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff197fff640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff19815e8b0 0x7ff1981485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 -- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff19815e8b0 msgr2=0x7ff1981485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff19815e8b0 0x7ff1981485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 -- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff19815e4a0 msgr2=0x7ff198155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff19815e4a0 0x7ff198155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 -- 172.21.15.67:0/186583632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff198155340 con 0x7ff19806c640 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff197fff640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff19815e8b0 0x7ff1981485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff1977fe640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff1980ccb60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff18801a0c0 tx=0x7ff18801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff18ffff640 1 -- 172.21.15.67:0/186583632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1880ba070 con 0x7ff19806c640 2024-04-25T15:53:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff18ffff640 1 -- 172.21.15.67:0/186583632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1880abbe0 con 0x7ff19806c640 2024-04-25T15:53:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19814f540 con 0x7ff19806c640 2024-04-25T15:53:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff18ffff640 1 -- 172.21.15.67:0/186583632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188004030 con 0x7ff19806c640 2024-04-25T15:53:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.894+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19814c4e0 con 0x7ff19806c640 2024-04-25T15:53:32.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.898+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff19806a480 con 0x7ff19806c640 2024-04-25T15:53:32.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.898+0000 7ff18ffff640 1 -- 172.21.15.67:0/186583632 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff1880041d0 con 0x7ff19806c640 2024-04-25T15:53:32.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.898+0000 7ff18ffff640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff17806f430 0x7ff178071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:32.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.898+0000 7ff18ffff640 1 -- 172.21.15.67:0/186583632 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5915+0+0 (secure 0 0 0) 0x7ff1881294f0 con 0x7ff19806c640 2024-04-25T15:53:32.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.898+0000 7ff196ffd640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff17806f430 0x7ff178071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:32.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.898+0000 7ff196ffd640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff17806f430 0x7ff178071940 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff19814d110 tx=0x7ff180008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:32.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:32.902+0000 7ff18ffff640 1 -- 172.21.15.67:0/186583632 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff188016110 con 0x7ff19806c640 2024-04-25T15:53:32.942 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:32.939+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:32.950 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:180 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.001754+0000", 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 42.948637626, 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031518879, 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.951 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 180 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.001754+0000", 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.001754+0000", 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.001757+0000", 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8590000000000001e-06 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003128+0000", 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013710160000000001 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003169+0000", 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0994000000000001e-05 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003188+0000", 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9419000000000002e-05 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003286+0000", 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8097000000000006e-05 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003436+0000", 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014946600000000001 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003559+0000", 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012318 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.009738+0000", 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061791939999999998 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.033190+0000", 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023451560999999999 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.033233+0000", 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3417000000000001e-05 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.033273+0000", 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9675999999999998e-05 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:343 3.8 3:1115f3ca:::10000000002.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.943524797999999, 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.46089677899999998, 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:32.957 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 343 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:32.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006870+0000", 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9280000000000002e-06 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008363+0000", 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001493288 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008398+0000", 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5067000000000002e-05 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008416+0000", 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7782e-05 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008522+0000", 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106327 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008535+0000", 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2883e-05 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.387477+0000", 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.37894188200000001 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.392562+0000", 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050850870000000003 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445264+0000", 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052702037 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445382+0000", 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118433 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445603+0000", 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022077500000000001 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.465250+0000", 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019646976 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467658+0000", 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024073850000000002 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467695+0000", 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7234999999999998e-05 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467763+0000", 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8694000000000003e-05 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:352 3.7 3:e0f5f483:::10000000002.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.338588+0000", 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.611803471000002, 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031524609000000002, 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.338588+0000", 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.338588+0000", 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.338591+0000", 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.339937+0000", 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013461759999999999 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.339976+0000", 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8631000000000003e-05 2024-04-25T15:53:32.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.339991+0000", 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5557999999999999e-05 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.340065+0000", 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3096000000000006e-05 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.340144+0000", 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9138e-05 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.340292+0000", 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000147873 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.351095+0000", 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010803708 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.370012+0000", 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018916397000000001 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.370062+0000", 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0300999999999998e-05 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.370113+0000", 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0664999999999997e-05 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:601 3.38 3:1c31e34b:::100000001f7.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.393712+0000", 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.5566791210000002, 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028807679999999999, 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 601 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.393712+0000", 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.393712+0000", 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.393715+0000", 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.112e-06 2024-04-25T15:53:32.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.395997+0000", 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002281295 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396043+0000", 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6440999999999998e-05 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396060+0000", 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6758000000000001e-05 2024-04-25T15:53:32.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396175+0000", 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115059 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396373+0000", 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019780399999999999 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396509+0000", 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136019 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.410556+0000", 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014046984 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.422383+0000", 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011826872 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.422459+0000", 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6304000000000007e-05 2024-04-25T15:53:32.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.422520+0000", 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1032000000000001e-05 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:670 3.8 3:1323134e:::100000001f7.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.0764546349999993, 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094982472999999998, 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.975 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 670 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873941+0000", 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0840000000000002e-06 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876488+0000", 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025470290000000001 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876647+0000", 2024-04-25T15:53:32.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015912000000000001 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876664+0000", 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7215e-05 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876782+0000", 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118082 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877222+0000", 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000439596 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877596+0000", 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037371700000000003 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.913106+0000", 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035509813000000001 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:32.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968792+0000", 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.055686739999999998 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968850+0000", 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8032000000000003e-05 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968919+0000", 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9045e-05 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:666 3.15 e81/18 3:a92ac213:::100000001f7.00000096:head v 81'11, mlcod=81'9)", 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.075582399, 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046403201999999998, 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874810+0000", 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882291+0000", 2024-04-25T15:53:32.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074801800000000003 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882310+0000", 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9809999999999998e-05 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882330+0000", 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0026000000000001e-05 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882471+0000", 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014052199999999999 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882509+0000", 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7756999999999998e-05 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921136+0000", 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038627364999999997 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921212+0000", 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6291999999999994e-05 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:671 3.38 3:1c0639bf:::100000001f7.0000009b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.0662896009999994, 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073197039000000005, 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.985 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 671 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884106+0000", 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9709999999999998e-06 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886679+0000", 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025726770000000002 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886716+0000", 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.735e-05 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886733+0000", 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6697999999999998e-05 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886843+0000", 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011074299999999999 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887031+0000", 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018804800000000001 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887215+0000", 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018384599999999999 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921326+0000", 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034110486000000002 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957180+0000", 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035854051999999997 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957227+0000", 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7290000000000003e-05 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957299+0000", 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1878000000000005e-05 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.990 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:88 2.14 e81/17 2:2e2fa760:::200.00000001:head v 81'51, mlcod=81'49)", 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.892986+0000", 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.0574054289999992, 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028378470999999999, 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.892986+0000", 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.892986+0000", 2024-04-25T15:53:32.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.892988+0000", 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893008+0000", 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0134000000000001e-05 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893031+0000", 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2486000000000001e-05 2024-04-25T15:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893089+0000", 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8686e-05 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893200+0000", 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111214 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893233+0000", 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2382999999999998e-05 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921296+0000", 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028063088999999999 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921365+0000", 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8606e-05 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.994 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:672 3.23 3:c45db235:::100000001f7.0000009c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.896255+0000", 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.0541366019999998, 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027004725, 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 672 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.896255+0000", 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.896255+0000", 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.896257+0000", 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3309999999999998e-06 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898207+0000", 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019496489999999999 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898257+0000", 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0340999999999998e-05 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898279+0000", 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1416e-05 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898423+0000", 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014452200000000001 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898715+0000", 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029176700000000001 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898979+0000", 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000263727 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921866+0000", 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022886969 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923116+0000", 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012504860000000001 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923193+0000", 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6632999999999996e-05 2024-04-25T15:53:32.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923260+0000", 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6884000000000005e-05 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:679 3.24 3:27dcc3f5:::100000001f7.000000a3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.950691+0000", 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.9997009030000008, 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033201485000000003, 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.000 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 679 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.950691+0000", 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.950691+0000", 2024-04-25T15:53:33.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.950693+0000", 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1830000000000001e-06 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952770+0000", 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020768869999999999 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952809+0000", 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9662000000000001e-05 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952823+0000", 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3737000000000001e-05 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952925+0000", 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010165100000000001 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.953095+0000", 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017042899999999999 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.953217+0000", 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121672 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.964557+0000", 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011340283 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983805+0000", 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019247543999999998 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983850+0000", 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5639000000000001e-05 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983892+0000", 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1798000000000002e-05 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:730 3.38 3:1c05d319:::100000001f7.000000d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.285762+0000", 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.6646299080000002, 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038281437000000001, 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 730 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.006 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.285762+0000", 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.285762+0000", 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.285769+0000", 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1910000000000003e-06 2024-04-25T15:53:33.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287695+0000", 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001926002 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287732+0000", 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6837e-05 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287749+0000", 2024-04-25T15:53:33.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.738e-05 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287850+0000", 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010052000000000001 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287988+0000", 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138176 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.288116+0000", 2024-04-25T15:53:33.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012841999999999999 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.295377+0000", 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072610230000000001 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.323938+0000", 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028561287000000001 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.323984+0000", 2024-04-25T15:53:33.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5379e-05 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.324043+0000", 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9222000000000003e-05 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:732 3.15 e81/18 3:aadcba4a:::100000001f7.000000d8:head v 81'13, mlcod=81'11)", 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.6427646120000006, 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056638899999999999, 2024-04-25T15:53:33.011 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307628+0000", 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4529999999999999e-06 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.312965+0000", 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367110000000001 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.312985+0000", 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9470000000000002e-05 2024-04-25T15:53:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313001+0000", 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6840000000000001e-05 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313091+0000", 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9770000000000003e-05 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313122+0000", 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364214+0000", 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051091996000000001 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364266+0000", 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1613000000000002e-05 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:733 3.22 e81/45 3:44586a4d:::100000001f7.000000d9:head v 81'6, mlcod=81'4)", 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.310476+0000", 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.639915426, 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034633787999999999, 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310476+0000", 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310476+0000", 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310478+0000", 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.485e-06 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313574+0000", 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003096264 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313592+0000", 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8006e-05 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313607+0000", 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5311e-05 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313685+0000", 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8116999999999998e-05 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313711+0000", 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5745000000000002e-05 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345018+0000", 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031306542999999999 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345110+0000", 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2317000000000004e-05 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:736 3.25 3:a5356222:::100000001f7.000000dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.6236925010000007, 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041194891999999997, 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 736 2024-04-25T15:53:33.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326702+0000", 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5500000000000001e-06 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328063+0000", 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013610849999999999 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328097+0000", 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4638000000000003e-05 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328111+0000", 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3896999999999999e-05 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328183+0000", 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1859000000000004e-05 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328336+0000", 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015279700000000001 2024-04-25T15:53:33.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328465+0000", 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012961200000000001 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.346379+0000", 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017913097999999999 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367792+0000", 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021413926 2024-04-25T15:53:33.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367838+0000", 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5541999999999999e-05 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367894+0000", 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5887999999999998e-05 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:738 3.23 3:c7b191fb:::100000001f7.000000de:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.338982+0000", 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.6114093139999994, 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034696241000000003, 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 738 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.338982+0000", 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.338982+0000", 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.338985+0000", 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.582e-06 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340200+0000", 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012147939999999999 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340233+0000", 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2957999999999999e-05 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340248+0000", 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4996e-05 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340325+0000", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7132999999999995e-05 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340447+0000", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012218900000000001 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340573+0000", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012636900000000001 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364265+0000", 2024-04-25T15:53:33.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023691727999999999 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.373549+0000", 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092843230000000006 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.373616+0000", 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7000000000000002e-05 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.373678+0000", 2024-04-25T15:53:33.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2168999999999994e-05 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:955 3.7 3:e0d97e51:::100000001f7.000000b7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.222517+0000", 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7278748940000002, 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036962529000000001, 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 955 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.222517+0000", 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.222517+0000", 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.222519+0000", 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4140000000000001e-06 2024-04-25T15:53:33.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223838+0000", 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001319188 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223874+0000", 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6216e-05 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223891+0000", 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6623e-05 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223967+0000", 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.224044+0000", 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7134999999999999e-05 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.224146+0000", 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102032 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.229620+0000", 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054742319999999999 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259348+0000", 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029727334000000001 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259412+0000", 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4789999999999995e-05 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259479+0000", 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6736000000000006e-05 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:959 3.33 3:cf81bd10:::100000001f7.000000bb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7065197850000002, 2024-04-25T15:53:33.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042315856999999998, 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 959 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243874+0000", 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.95e-06 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245749+0000", 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018754080000000001 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245791+0000", 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2200000000000003e-05 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245807+0000", 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5379999999999998e-05 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245884+0000", 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-05 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245972+0000", 2024-04-25T15:53:33.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8459000000000001e-05 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.246076+0000", 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103861 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252170+0000", 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060943439999999998 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.040 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286068+0000", 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033897570000000002 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286116+0000", 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8374999999999997e-05 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286188+0000", 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.127e-05 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:960 3.1e 3:796b994c:::100000001f7.000000bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.249999+0000", 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.70039271, 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036397816, 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 960 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.042 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.249999+0000", 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.249999+0000", 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.250002+0000", 2024-04-25T15:53:33.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.18e-06 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252017+0000", 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020151689999999998 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252052+0000", 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4754e-05 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252070+0000", 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252161+0000", 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252251+0000", 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9963000000000006e-05 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252395+0000", 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144864 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259647+0000", 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072518180000000002 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286270+0000", 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026622845999999999 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286341+0000", 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1069999999999995e-05 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286397+0000", 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5396999999999998e-05 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1010 3.33 3:cf8599b8:::100000001f7.000000ee:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4096711290000004, 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043954280999999998, 2024-04-25T15:53:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1010 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:33.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540723+0000", 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6029999999999999e-06 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542109+0000", 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013856610000000001 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542152+0000", 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3353999999999999e-05 2024-04-25T15:53:33.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542167+0000", 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4779e-05 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542259+0000", 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2472999999999994e-05 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542384+0000", 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012520200000000001 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542571+0000", 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018691000000000001 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.551607+0000", 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090354900000000002 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584560+0000", 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032952759999999998 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584623+0000", 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3305999999999993e-05 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584675+0000", 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1743000000000002e-05 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1011 3.7 3:e14869e0:::100000001f7.000000ef:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:33.052 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.404624965, 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038908115, 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1011 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545769+0000", 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.587e-06 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547305+0000", 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015358590000000001 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:33.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547334+0000", 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8538e-05 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547346+0000", 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2449000000000001e-05 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547459+0000", 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112759 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547581+0000", 2024-04-25T15:53:33.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012228900000000001 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547754+0000", 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017338 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.555402+0000", 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076472600000000003 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584550+0000", 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029148484999999998 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584616+0000", 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5528e-05 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584675+0000", 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8981000000000003e-05 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:33.057 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:33.059 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.058+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff198155620 con 0x7ff17806f430 2024-04-25T15:53:33.060 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:33.055+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:33.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.078+0000 7ff18ffff640 1 -- 172.21.15.67:0/186583632 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253728 (secure 0 0 0) 0x7ff198155620 con 0x7ff17806f430 2024-04-25T15:53:33.081 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:53:33.083 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:53:33.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff17806f430 msgr2=0x7ff178071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff17806f430 0x7ff178071940 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff19814d110 tx=0x7ff180008040 comp rx=0 tx=0).stop 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 msgr2=0x7ff1980ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff1980ccb60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff18801a0c0 tx=0x7ff18801acd0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 shutdown_connections 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff17806f430 0x7ff178071940 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff19815e8b0 0x7ff1981485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff19815e4a0 0x7ff198155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 --2- 172.21.15.67:0/186583632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff19806c640 0x7ff1980ccb60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 >> 172.21.15.67:0/186583632 conn(0x7ff198070db0 msgr2=0x7ff198152d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 shutdown_connections 2024-04-25T15:53:33.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.082+0000 7ff19dd1f640 1 -- 172.21.15.67:0/186583632 wait complete. 2024-04-25T15:53:33.101 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-04-25T15:53:32.169592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1074447512,"num_objects":357,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":67,"num_read_kb":56,"num_write":1520,"num_write_kb":3634975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":541072643,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1462,"ondisk_log_size":1462,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2368040,"kb_used_data":2106336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":752606680,"statfs":{"total":773094113280,"available":770669240320,"internally_reserved":0,"allocated":2156888064,"data_stored":2152497672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":81,"apply_latency_ms":81,"commit_latency_ns":81000000,"apply_latency_ns":81000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-50312197,"num_objects":-11,"num_object_clones":0,"num_object_copies":-22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":635,"num_write_kb":1486873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.310783"},"pg_stats":[{"pgid":"3.3f","version":"82'17","reported_seq":152,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.715148+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:53:27.715148+0000","last_peered":"2024-04-25T15:53:27.715148+0000","last_clean":"2024-04-25T15:53:27.715148+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:53:27.715148+0000","last_undegraded":"2024-04-25T15:53:27.715148+0000","last_fullsized":"2024-04-25T15:53:27.715148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"81'12","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570854+0000","last_change":"2024-04-25T15:53:13.069367+0000","last_active":"2024-04-25T15:53:26.570854+0000","last_peered":"2024-04-25T15:53:26.570854+0000","last_clean":"2024-04-25T15:53:26.570854+0000","last_became_active":"2024-04-25T15:53:12.990829+0000","last_became_peered":"2024-04-25T15:53:12.990829+0000","last_unstale":"2024-04-25T15:53:26.570854+0000","last_undegraded":"2024-04-25T15:53:26.570854+0000","last_fullsized":"2024-04-25T15:53:26.570854+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:46.276338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"82'16","reported_seq":108,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.555832+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:53:27.555832+0000","last_peered":"2024-04-25T15:53:27.555832+0000","last_clean":"2024-04-25T15:53:27.555832+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:53:27.555832+0000","last_undegraded":"2024-04-25T15:53:27.555832+0000","last_fullsized":"2024-04-25T15:53:27.555832+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"81'11","reported_seq":148,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570826+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:53:26.570826+0000","last_peered":"2024-04-25T15:53:26.570826+0000","last_clean":"2024-04-25T15:53:26.570826+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:53:26.570826+0000","last_undegraded":"2024-04-25T15:53:26.570826+0000","last_fullsized":"2024-04-25T15:53:26.570826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"82'20","reported_seq":155,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337918+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:53:26.337918+0000","last_peered":"2024-04-25T15:53:26.337918+0000","last_clean":"2024-04-25T15:53:26.337918+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:53:26.337918+0000","last_undegraded":"2024-04-25T15:53:26.337918+0000","last_fullsized":"2024-04-25T15:53:26.337918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"82'14","reported_seq":151,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.430320+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:53:27.430320+0000","last_peered":"2024-04-25T15:53:27.430320+0000","last_clean":"2024-04-25T15:53:27.430320+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:53:27.430320+0000","last_undegraded":"2024-04-25T15:53:27.430320+0000","last_fullsized":"2024-04-25T15:53:27.430320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"81'22","reported_seq":157,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337430+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:53:26.337430+0000","last_peered":"2024-04-25T15:53:26.337430+0000","last_clean":"2024-04-25T15:53:26.337430+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:53:26.337430+0000","last_undegraded":"2024-04-25T15:53:26.337430+0000","last_fullsized":"2024-04-25T15:53:26.337430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"82'13","reported_seq":150,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.323006+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:53:27.323006+0000","last_peered":"2024-04-25T15:53:27.323006+0000","last_clean":"2024-04-25T15:53:27.323006+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:53:27.323006+0000","last_undegraded":"2024-04-25T15:53:27.323006+0000","last_fullsized":"2024-04-25T15:53:27.323006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"82'19","reported_seq":148,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.210792+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:53:27.210792+0000","last_peered":"2024-04-25T15:53:27.210792+0000","last_clean":"2024-04-25T15:53:27.210792+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:53:27.210792+0000","last_undegraded":"2024-04-25T15:53:27.210792+0000","last_fullsized":"2024-04-25T15:53:27.210792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"82'20","reported_seq":157,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.127821+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:53:26.127821+0000","last_peered":"2024-04-25T15:53:26.127821+0000","last_clean":"2024-04-25T15:53:26.127821+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:53:26.127821+0000","last_undegraded":"2024-04-25T15:53:26.127821+0000","last_fullsized":"2024-04-25T15:53:26.127821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"82'14","reported_seq":151,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.233276+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:53:27.233276+0000","last_peered":"2024-04-25T15:53:27.233276+0000","last_clean":"2024-04-25T15:53:27.233276+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:53:27.233276+0000","last_undegraded":"2024-04-25T15:53:27.233276+0000","last_fullsized":"2024-04-25T15:53:27.233276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"82'26","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.128921+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:53:26.128921+0000","last_peered":"2024-04-25T15:53:26.128921+0000","last_clean":"2024-04-25T15:53:26.128921+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:53:26.128921+0000","last_undegraded":"2024-04-25T15:53:26.128921+0000","last_fullsized":"2024-04-25T15:53:26.128921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"82'14","reported_seq":151,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.430481+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:53:27.430481+0000","last_peered":"2024-04-25T15:53:27.430481+0000","last_clean":"2024-04-25T15:53:27.430481+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:53:27.430481+0000","last_undegraded":"2024-04-25T15:53:27.430481+0000","last_fullsized":"2024-04-25T15:53:27.430481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"82'20","reported_seq":158,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.568737+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:53:26.568737+0000","last_peered":"2024-04-25T15:53:26.568737+0000","last_clean":"2024-04-25T15:53:26.568737+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:53:26.568737+0000","last_undegraded":"2024-04-25T15:53:26.568737+0000","last_fullsized":"2024-04-25T15:53:26.568737+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"82'18","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.635271+0000","last_change":"2024-04-25T15:53:17.148839+0000","last_active":"2024-04-25T15:53:27.635271+0000","last_peered":"2024-04-25T15:53:27.635271+0000","last_clean":"2024-04-25T15:53:27.635271+0000","last_became_active":"2024-04-25T15:53:16.064416+0000","last_became_peered":"2024-04-25T15:53:16.064416+0000","last_unstale":"2024-04-25T15:53:27.635271+0000","last_undegraded":"2024-04-25T15:53:27.635271+0000","last_fullsized":"2024-04-25T15:53:27.635271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:58.798689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"82'24","reported_seq":161,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.739903+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:53:27.739903+0000","last_peered":"2024-04-25T15:53:27.739903+0000","last_clean":"2024-04-25T15:53:27.739903+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:53:27.739903+0000","last_undegraded":"2024-04-25T15:53:27.739903+0000","last_fullsized":"2024-04-25T15:53:27.739903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"81'12","reported_seq":82,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.571142+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:53:26.571142+0000","last_peered":"2024-04-25T15:53:26.571142+0000","last_clean":"2024-04-25T15:53:26.571142+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:53:26.571142+0000","last_undegraded":"2024-04-25T15:53:26.571142+0000","last_fullsized":"2024-04-25T15:53:26.571142+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"82'11","reported_seq":147,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.432825+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:53:27.432825+0000","last_peered":"2024-04-25T15:53:27.432825+0000","last_clean":"2024-04-25T15:53:27.432825+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:53:27.432825+0000","last_undegraded":"2024-04-25T15:53:27.432825+0000","last_fullsized":"2024-04-25T15:53:27.432825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"82'12","reported_seq":147,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.233096+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:53:27.233096+0000","last_peered":"2024-04-25T15:53:27.233096+0000","last_clean":"2024-04-25T15:53:27.233096+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:53:27.233096+0000","last_undegraded":"2024-04-25T15:53:27.233096+0000","last_fullsized":"2024-04-25T15:53:27.233096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"82'12","reported_seq":165,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.209546+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:53:27.209546+0000","last_peered":"2024-04-25T15:53:27.209546+0000","last_clean":"2024-04-25T15:53:27.209546+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:53:27.209546+0000","last_undegraded":"2024-04-25T15:53:27.209546+0000","last_fullsized":"2024-04-25T15:53:27.209546+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"82'20","reported_seq":111,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337811+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:53:26.337811+0000","last_peered":"2024-04-25T15:53:26.337811+0000","last_clean":"2024-04-25T15:53:26.337811+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:53:26.337811+0000","last_undegraded":"2024-04-25T15:53:26.337811+0000","last_fullsized":"2024-04-25T15:53:26.337811+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"82'21","reported_seq":151,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.529881+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:53:27.529881+0000","last_peered":"2024-04-25T15:53:27.529881+0000","last_clean":"2024-04-25T15:53:27.529881+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:53:27.529881+0000","last_undegraded":"2024-04-25T15:53:27.529881+0000","last_fullsized":"2024-04-25T15:53:27.529881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"82'11","reported_seq":164,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570188+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:53:26.570188+0000","last_peered":"2024-04-25T15:53:26.570188+0000","last_clean":"2024-04-25T15:53:26.570188+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:53:26.570188+0000","last_undegraded":"2024-04-25T15:53:26.570188+0000","last_fullsized":"2024-04-25T15:53:26.570188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"82'19","reported_seq":156,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.277777+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:53:27.277777+0000","last_peered":"2024-04-25T15:53:27.277777+0000","last_clean":"2024-04-25T15:53:27.277777+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:53:27.277777+0000","last_undegraded":"2024-04-25T15:53:27.277777+0000","last_fullsized":"2024-04-25T15:53:27.277777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"82'11","reported_seq":164,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570739+0000","last_change":"2024-04-25T15:53:17.119870+0000","last_active":"2024-04-25T15:53:26.570739+0000","last_peered":"2024-04-25T15:53:26.570739+0000","last_clean":"2024-04-25T15:53:26.570739+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:53:26.570739+0000","last_undegraded":"2024-04-25T15:53:26.570739+0000","last_fullsized":"2024-04-25T15:53:26.570739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:05:09.853949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"82'32","reported_seq":169,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.127919+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:53:26.127919+0000","last_peered":"2024-04-25T15:53:26.127919+0000","last_clean":"2024-04-25T15:53:26.127919+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:53:26.127919+0000","last_undegraded":"2024-04-25T15:53:26.127919+0000","last_fullsized":"2024-04-25T15:53:26.127919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"81'12","reported_seq":129,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570558+0000","last_change":"2024-04-25T15:53:03.808515+0000","last_active":"2024-04-25T15:53:26.570558+0000","last_peered":"2024-04-25T15:53:26.570558+0000","last_clean":"2024-04-25T15:53:26.570558+0000","last_became_active":"2024-04-25T15:53:03.367253+0000","last_became_peered":"2024-04-25T15:53:03.367253+0000","last_unstale":"2024-04-25T15:53:26.570558+0000","last_undegraded":"2024-04-25T15:53:26.570558+0000","last_fullsized":"2024-04-25T15:53:26.570558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:05.644251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"82'18","reported_seq":190,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.376169+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:53:27.376169+0000","last_peered":"2024-04-25T15:53:27.376169+0000","last_clean":"2024-04-25T15:53:27.376169+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:53:27.376169+0000","last_undegraded":"2024-04-25T15:53:27.376169+0000","last_fullsized":"2024-04-25T15:53:27.376169+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"82'17","reported_seq":110,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.650972+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:53:27.650972+0000","last_peered":"2024-04-25T15:53:27.650972+0000","last_clean":"2024-04-25T15:53:27.650972+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:53:27.650972+0000","last_undegraded":"2024-04-25T15:53:27.650972+0000","last_fullsized":"2024-04-25T15:53:27.650972+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"82'9","reported_seq":90,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.343682+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:53:27.343682+0000","last_peered":"2024-04-25T15:53:27.343682+0000","last_clean":"2024-04-25T15:53:27.343682+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:53:27.343682+0000","last_undegraded":"2024-04-25T15:53:27.343682+0000","last_fullsized":"2024-04-25T15:53:27.343682+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":36864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"82'19","reported_seq":165,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569997+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:53:26.569997+0000","last_peered":"2024-04-25T15:53:26.569997+0000","last_clean":"2024-04-25T15:53:26.569997+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:53:26.569997+0000","last_undegraded":"2024-04-25T15:53:26.569997+0000","last_fullsized":"2024-04-25T15:53:26.569997+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"82'28","reported_seq":165,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.692956+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:53:27.692956+0000","last_peered":"2024-04-25T15:53:27.692956+0000","last_clean":"2024-04-25T15:53:27.692956+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:53:27.692956+0000","last_undegraded":"2024-04-25T15:53:27.692956+0000","last_fullsized":"2024-04-25T15:53:27.692956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"81'14","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.128280+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:53:26.128280+0000","last_peered":"2024-04-25T15:53:26.128280+0000","last_clean":"2024-04-25T15:53:26.128280+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:53:26.128280+0000","last_undegraded":"2024-04-25T15:53:26.128280+0000","last_fullsized":"2024-04-25T15:53:26.128280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":145,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570322+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:53:26.570322+0000","last_peered":"2024-04-25T15:53:26.570322+0000","last_clean":"2024-04-25T15:53:26.570322+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:53:26.570322+0000","last_undegraded":"2024-04-25T15:53:26.570322+0000","last_fullsized":"2024-04-25T15:53:26.570322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"82'16","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.509219+0000","last_change":"2024-04-25T15:53:16.510837+0000","last_active":"2024-04-25T15:53:27.509219+0000","last_peered":"2024-04-25T15:53:27.509219+0000","last_clean":"2024-04-25T15:53:27.509219+0000","last_became_active":"2024-04-25T15:53:16.041557+0000","last_became_peered":"2024-04-25T15:53:16.041557+0000","last_unstale":"2024-04-25T15:53:27.509219+0000","last_undegraded":"2024-04-25T15:53:27.509219+0000","last_fullsized":"2024-04-25T15:53:27.509219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:06.600570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":172,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.126868+0000","last_change":"2024-04-25T15:53:16.102712+0000","last_active":"2024-04-25T15:53:26.126868+0000","last_peered":"2024-04-25T15:53:26.126868+0000","last_clean":"2024-04-25T15:53:26.126868+0000","last_became_active":"2024-04-25T15:53:16.052729+0000","last_became_peered":"2024-04-25T15:53:16.052729+0000","last_unstale":"2024-04-25T15:53:26.126868+0000","last_undegraded":"2024-04-25T15:53:26.126868+0000","last_fullsized":"2024-04-25T15:53:26.126868+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:53.161289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":142,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569179+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:53:26.569179+0000","last_peered":"2024-04-25T15:53:26.569179+0000","last_clean":"2024-04-25T15:53:26.569179+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:53:26.569179+0000","last_undegraded":"2024-04-25T15:53:26.569179+0000","last_fullsized":"2024-04-25T15:53:26.569179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"82'14","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.568283+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:53:26.568283+0000","last_peered":"2024-04-25T15:53:26.568283+0000","last_clean":"2024-04-25T15:53:26.568283+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:53:26.568283+0000","last_undegraded":"2024-04-25T15:53:26.568283+0000","last_fullsized":"2024-04-25T15:53:26.568283+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":160,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569682+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:53:26.569682+0000","last_peered":"2024-04-25T15:53:26.569682+0000","last_clean":"2024-04-25T15:53:26.569682+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:53:26.569682+0000","last_undegraded":"2024-04-25T15:53:26.569682+0000","last_fullsized":"2024-04-25T15:53:26.569682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"82'14","reported_seq":141,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.210334+0000","last_change":"2024-04-25T15:53:16.206656+0000","last_active":"2024-04-25T15:53:27.210334+0000","last_peered":"2024-04-25T15:53:27.210334+0000","last_clean":"2024-04-25T15:53:27.210334+0000","last_became_active":"2024-04-25T15:53:16.049908+0000","last_became_peered":"2024-04-25T15:53:16.049908+0000","last_unstale":"2024-04-25T15:53:27.210334+0000","last_undegraded":"2024-04-25T15:53:27.210334+0000","last_fullsized":"2024-04-25T15:53:27.210334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.217418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":145,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.127447+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:53:26.127447+0000","last_peered":"2024-04-25T15:53:26.127447+0000","last_clean":"2024-04-25T15:53:26.127447+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:53:26.127447+0000","last_undegraded":"2024-04-25T15:53:26.127447+0000","last_fullsized":"2024-04-25T15:53:26.127447+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"82'23","reported_seq":160,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.637966+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:53:27.637966+0000","last_peered":"2024-04-25T15:53:27.637966+0000","last_clean":"2024-04-25T15:53:27.637966+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:53:27.637966+0000","last_undegraded":"2024-04-25T15:53:27.637966+0000","last_fullsized":"2024-04-25T15:53:27.637966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":143,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569248+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:53:26.569248+0000","last_peered":"2024-04-25T15:53:26.569248+0000","last_clean":"2024-04-25T15:53:26.569248+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:53:26.569248+0000","last_undegraded":"2024-04-25T15:53:26.569248+0000","last_fullsized":"2024-04-25T15:53:26.569248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"81'19","reported_seq":112,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.571718+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:53:26.571718+0000","last_peered":"2024-04-25T15:53:26.571718+0000","last_clean":"2024-04-25T15:53:26.571718+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:53:26.571718+0000","last_undegraded":"2024-04-25T15:53:26.571718+0000","last_fullsized":"2024-04-25T15:53:26.571718+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"81'7","reported_seq":97,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.571334+0000","last_change":"2024-04-25T15:53:03.435546+0000","last_active":"2024-04-25T15:53:26.571334+0000","last_peered":"2024-04-25T15:53:26.571334+0000","last_clean":"2024-04-25T15:53:26.571334+0000","last_became_active":"2024-04-25T15:53:03.387235+0000","last_became_peered":"2024-04-25T15:53:03.387235+0000","last_unstale":"2024-04-25T15:53:26.571334+0000","last_undegraded":"2024-04-25T15:53:26.571334+0000","last_fullsized":"2024-04-25T15:53:26.571334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:30.444213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1344,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"82'18","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.128978+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:53:26.128978+0000","last_peered":"2024-04-25T15:53:26.128978+0000","last_clean":"2024-04-25T15:53:26.128978+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:53:26.128978+0000","last_undegraded":"2024-04-25T15:53:26.128978+0000","last_fullsized":"2024-04-25T15:53:26.128978+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"82'55","reported_seq":194,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.127770+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:53:26.127770+0000","last_peered":"2024-04-25T15:53:26.127770+0000","last_clean":"2024-04-25T15:53:26.127770+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:53:26.127770+0000","last_undegraded":"2024-04-25T15:53:26.127770+0000","last_fullsized":"2024-04-25T15:53:26.127770+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107474,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"82'20","reported_seq":155,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.496691+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:53:27.496691+0000","last_peered":"2024-04-25T15:53:27.496691+0000","last_clean":"2024-04-25T15:53:27.496691+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:53:27.496691+0000","last_undegraded":"2024-04-25T15:53:27.496691+0000","last_fullsized":"2024-04-25T15:53:27.496691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.128779+0000","last_change":"2024-04-25T15:53:16.094342+0000","last_active":"2024-04-25T15:53:26.128779+0000","last_peered":"2024-04-25T15:53:26.128779+0000","last_clean":"2024-04-25T15:53:26.128779+0000","last_became_active":"2024-04-25T15:53:16.041484+0000","last_became_peered":"2024-04-25T15:53:16.041484+0000","last_unstale":"2024-04-25T15:53:26.128779+0000","last_undegraded":"2024-04-25T15:53:26.128779+0000","last_fullsized":"2024-04-25T15:53:26.128779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:42.531801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"82'25","reported_seq":116,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.527215+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:53:27.527215+0000","last_peered":"2024-04-25T15:53:27.527215+0000","last_clean":"2024-04-25T15:53:27.527215+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:53:27.527215+0000","last_undegraded":"2024-04-25T15:53:27.527215+0000","last_fullsized":"2024-04-25T15:53:27.527215+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":144,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570227+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:53:26.570227+0000","last_peered":"2024-04-25T15:53:26.570227+0000","last_clean":"2024-04-25T15:53:26.570227+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:53:26.570227+0000","last_undegraded":"2024-04-25T15:53:26.570227+0000","last_fullsized":"2024-04-25T15:53:26.570227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"82'14","reported_seq":169,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.338396+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:53:26.338396+0000","last_peered":"2024-04-25T15:53:26.338396+0000","last_clean":"2024-04-25T15:53:26.338396+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:53:26.338396+0000","last_undegraded":"2024-04-25T15:53:26.338396+0000","last_fullsized":"2024-04-25T15:53:26.338396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":184,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.126163+0000","last_change":"2024-04-25T15:53:16.115440+0000","last_active":"2024-04-25T15:53:26.126163+0000","last_peered":"2024-04-25T15:53:26.126163+0000","last_clean":"2024-04-25T15:53:26.126163+0000","last_became_active":"2024-04-25T15:53:16.053229+0000","last_became_peered":"2024-04-25T15:53:16.053229+0000","last_unstale":"2024-04-25T15:53:26.126163+0000","last_undegraded":"2024-04-25T15:53:26.126163+0000","last_fullsized":"2024-04-25T15:53:26.126163+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:13.860370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3087,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"82'25","reported_seq":162,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.743914+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:53:27.743914+0000","last_peered":"2024-04-25T15:53:27.743914+0000","last_clean":"2024-04-25T15:53:27.743914+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:53:27.743914+0000","last_undegraded":"2024-04-25T15:53:27.743914+0000","last_fullsized":"2024-04-25T15:53:27.743914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":224,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569764+0000","last_change":"2024-04-25T15:53:03.421455+0000","last_active":"2024-04-25T15:53:26.569764+0000","last_peered":"2024-04-25T15:53:26.569764+0000","last_clean":"2024-04-25T15:53:26.569764+0000","last_became_active":"2024-04-25T15:53:03.385909+0000","last_became_peered":"2024-04-25T15:53:03.385909+0000","last_unstale":"2024-04-25T15:53:26.569764+0000","last_undegraded":"2024-04-25T15:53:26.569764+0000","last_fullsized":"2024-04-25T15:53:26.569764+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:54.677309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"82'36","reported_seq":199,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.334914+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:53:27.334914+0000","last_peered":"2024-04-25T15:53:27.334914+0000","last_clean":"2024-04-25T15:53:27.334914+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:53:27.334914+0000","last_undegraded":"2024-04-25T15:53:27.334914+0000","last_fullsized":"2024-04-25T15:53:27.334914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":26,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.129063+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:53:26.129063+0000","last_peered":"2024-04-25T15:53:26.129063+0000","last_clean":"2024-04-25T15:53:26.129063+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:53:26.129063+0000","last_undegraded":"2024-04-25T15:53:26.129063+0000","last_fullsized":"2024-04-25T15:53:26.129063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"82'12","reported_seq":163,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.641953+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:53:27.641953+0000","last_peered":"2024-04-25T15:53:27.641953+0000","last_clean":"2024-04-25T15:53:27.641953+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:53:27.641953+0000","last_undegraded":"2024-04-25T15:53:27.641953+0000","last_fullsized":"2024-04-25T15:53:27.641953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":143,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.571157+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:53:26.571157+0000","last_peered":"2024-04-25T15:53:26.571157+0000","last_clean":"2024-04-25T15:53:26.571157+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:53:26.571157+0000","last_undegraded":"2024-04-25T15:53:26.571157+0000","last_fullsized":"2024-04-25T15:53:26.571157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"82'15","reported_seq":170,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.339237+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:53:26.339237+0000","last_peered":"2024-04-25T15:53:26.339237+0000","last_clean":"2024-04-25T15:53:26.339237+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:53:26.339237+0000","last_undegraded":"2024-04-25T15:53:26.339237+0000","last_fullsized":"2024-04-25T15:53:26.339237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.128999+0000","last_change":"2024-04-25T15:53:16.098410+0000","last_active":"2024-04-25T15:53:26.128999+0000","last_peered":"2024-04-25T15:53:26.128999+0000","last_clean":"2024-04-25T15:53:26.128999+0000","last_became_active":"2024-04-25T15:53:16.053979+0000","last_became_peered":"2024-04-25T15:53:16.053979+0000","last_unstale":"2024-04-25T15:53:26.128999+0000","last_undegraded":"2024-04-25T15:53:26.128999+0000","last_fullsized":"2024-04-25T15:53:26.128999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:24.859180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"82'17","reported_seq":154,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.209292+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:53:27.209292+0000","last_peered":"2024-04-25T15:53:27.209292+0000","last_clean":"2024-04-25T15:53:27.209292+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:53:27.209292+0000","last_undegraded":"2024-04-25T15:53:27.209292+0000","last_fullsized":"2024-04-25T15:53:27.209292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":144,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569927+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:53:26.569927+0000","last_peered":"2024-04-25T15:53:26.569927+0000","last_clean":"2024-04-25T15:53:26.569927+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:53:26.569927+0000","last_undegraded":"2024-04-25T15:53:26.569927+0000","last_fullsized":"2024-04-25T15:53:26.569927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"82'13","reported_seq":148,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.367372+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:53:27.367372+0000","last_peered":"2024-04-25T15:53:27.367372+0000","last_clean":"2024-04-25T15:53:27.367372+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:53:27.367372+0000","last_undegraded":"2024-04-25T15:53:27.367372+0000","last_fullsized":"2024-04-25T15:53:27.367372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":233,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337688+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:53:26.337688+0000","last_peered":"2024-04-25T15:53:26.337688+0000","last_clean":"2024-04-25T15:53:26.337688+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:53:26.337688+0000","last_undegraded":"2024-04-25T15:53:26.337688+0000","last_fullsized":"2024-04-25T15:53:26.337688+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":145,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569678+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:53:26.569678+0000","last_peered":"2024-04-25T15:53:26.569678+0000","last_clean":"2024-04-25T15:53:26.569678+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:53:26.569678+0000","last_undegraded":"2024-04-25T15:53:26.569678+0000","last_fullsized":"2024-04-25T15:53:26.569678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"81'16","reported_seq":153,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570362+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:53:26.570362+0000","last_peered":"2024-04-25T15:53:26.570362+0000","last_clean":"2024-04-25T15:53:26.570362+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:53:26.570362+0000","last_undegraded":"2024-04-25T15:53:26.570362+0000","last_fullsized":"2024-04-25T15:53:26.570362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":168,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337562+0000","last_change":"2024-04-25T15:53:03.424512+0000","last_active":"2024-04-25T15:53:26.337562+0000","last_peered":"2024-04-25T15:53:26.337562+0000","last_clean":"2024-04-25T15:53:26.337562+0000","last_became_active":"2024-04-25T15:53:03.385988+0000","last_became_peered":"2024-04-25T15:53:03.385988+0000","last_unstale":"2024-04-25T15:53:26.337562+0000","last_undegraded":"2024-04-25T15:53:26.337562+0000","last_fullsized":"2024-04-25T15:53:26.337562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:55.224996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'24","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.613182+0000","last_change":"2024-04-25T15:53:16.680938+0000","last_active":"2024-04-25T15:53:27.613182+0000","last_peered":"2024-04-25T15:53:27.613182+0000","last_clean":"2024-04-25T15:53:27.613182+0000","last_became_active":"2024-04-25T15:53:16.041368+0000","last_became_peered":"2024-04-25T15:53:16.041368+0000","last_unstale":"2024-04-25T15:53:27.613182+0000","last_undegraded":"2024-04-25T15:53:27.613182+0000","last_fullsized":"2024-04-25T15:53:27.613182+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:49.681769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"81'8","reported_seq":147,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570021+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:53:26.570021+0000","last_peered":"2024-04-25T15:53:26.570021+0000","last_clean":"2024-04-25T15:53:26.570021+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:53:26.570021+0000","last_undegraded":"2024-04-25T15:53:26.570021+0000","last_fullsized":"2024-04-25T15:53:26.570021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"82'17","reported_seq":133,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.234607+0000","last_change":"2024-04-25T15:53:03.530328+0000","last_active":"2024-04-25T15:53:27.234607+0000","last_peered":"2024-04-25T15:53:27.234607+0000","last_clean":"2024-04-25T15:53:27.234607+0000","last_became_active":"2024-04-25T15:53:03.382259+0000","last_became_peered":"2024-04-25T15:53:03.382259+0000","last_unstale":"2024-04-25T15:53:27.234607+0000","last_undegraded":"2024-04-25T15:53:27.234607+0000","last_fullsized":"2024-04-25T15:53:27.234607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:43.788748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":166,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570386+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:53:26.570386+0000","last_peered":"2024-04-25T15:53:26.570386+0000","last_clean":"2024-04-25T15:53:26.570386+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:53:26.570386+0000","last_undegraded":"2024-04-25T15:53:26.570386+0000","last_fullsized":"2024-04-25T15:53:26.570386+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"82'13","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.129220+0000","last_change":"2024-04-25T15:53:16.788986+0000","last_active":"2024-04-25T15:53:26.129220+0000","last_peered":"2024-04-25T15:53:26.129220+0000","last_clean":"2024-04-25T15:53:26.129220+0000","last_became_active":"2024-04-25T15:53:16.046426+0000","last_became_peered":"2024-04-25T15:53:16.046426+0000","last_unstale":"2024-04-25T15:53:26.129220+0000","last_undegraded":"2024-04-25T15:53:26.129220+0000","last_fullsized":"2024-04-25T15:53:26.129220+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:33.326656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337852+0000","last_change":"2024-04-25T15:53:03.381919+0000","last_active":"2024-04-25T15:53:26.337852+0000","last_peered":"2024-04-25T15:53:26.337852+0000","last_clean":"2024-04-25T15:53:26.337852+0000","last_became_active":"2024-04-25T15:53:03.381463+0000","last_became_peered":"2024-04-25T15:53:03.381463+0000","last_unstale":"2024-04-25T15:53:26.337852+0000","last_undegraded":"2024-04-25T15:53:26.337852+0000","last_fullsized":"2024-04-25T15:53:26.337852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:09.993891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"82'20","reported_seq":157,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.708548+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:53:27.708548+0000","last_peered":"2024-04-25T15:53:27.708548+0000","last_clean":"2024-04-25T15:53:27.708548+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:53:27.708548+0000","last_undegraded":"2024-04-25T15:53:27.708548+0000","last_fullsized":"2024-04-25T15:53:27.708548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":141,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.339406+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:53:26.339406+0000","last_peered":"2024-04-25T15:53:26.339406+0000","last_clean":"2024-04-25T15:53:26.339406+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:53:26.339406+0000","last_undegraded":"2024-04-25T15:53:26.339406+0000","last_fullsized":"2024-04-25T15:53:26.339406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"82'12","reported_seq":114,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.450715+0000","last_change":"2024-04-25T15:53:03.480191+0000","last_active":"2024-04-25T15:53:27.450715+0000","last_peered":"2024-04-25T15:53:27.450715+0000","last_clean":"2024-04-25T15:53:27.450715+0000","last_became_active":"2024-04-25T15:53:03.380081+0000","last_became_peered":"2024-04-25T15:53:03.380081+0000","last_unstale":"2024-04-25T15:53:27.450715+0000","last_undegraded":"2024-04-25T15:53:27.450715+0000","last_fullsized":"2024-04-25T15:53:27.450715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:50.114177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"81'17","reported_seq":149,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.127984+0000","last_change":"2024-04-25T15:53:16.083270+0000","last_active":"2024-04-25T15:53:26.127984+0000","last_peered":"2024-04-25T15:53:26.127984+0000","last_clean":"2024-04-25T15:53:26.127984+0000","last_became_active":"2024-04-25T15:53:16.045200+0000","last_became_peered":"2024-04-25T15:53:16.045200+0000","last_unstale":"2024-04-25T15:53:26.127984+0000","last_undegraded":"2024-04-25T15:53:26.127984+0000","last_fullsized":"2024-04-25T15:53:26.127984+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:38.382894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"82'31","reported_seq":168,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.560712+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:53:27.560712+0000","last_peered":"2024-04-25T15:53:27.560712+0000","last_clean":"2024-04-25T15:53:27.560712+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:53:27.560712+0000","last_undegraded":"2024-04-25T15:53:27.560712+0000","last_fullsized":"2024-04-25T15:53:27.560712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":146,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.127944+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:53:26.127944+0000","last_peered":"2024-04-25T15:53:26.127944+0000","last_clean":"2024-04-25T15:53:26.127944+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:53:26.127944+0000","last_undegraded":"2024-04-25T15:53:26.127944+0000","last_fullsized":"2024-04-25T15:53:26.127944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"82'15","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.308788+0000","last_change":"2024-04-25T15:53:16.863394+0000","last_active":"2024-04-25T15:53:27.308788+0000","last_peered":"2024-04-25T15:53:27.308788+0000","last_clean":"2024-04-25T15:53:27.308788+0000","last_became_active":"2024-04-25T15:53:16.048669+0000","last_became_peered":"2024-04-25T15:53:16.048669+0000","last_unstale":"2024-04-25T15:53:27.308788+0000","last_undegraded":"2024-04-25T15:53:27.308788+0000","last_fullsized":"2024-04-25T15:53:27.308788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:53.132724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":139,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337351+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:53:26.337351+0000","last_peered":"2024-04-25T15:53:26.337351+0000","last_clean":"2024-04-25T15:53:26.337351+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:53:26.337351+0000","last_undegraded":"2024-04-25T15:53:26.337351+0000","last_fullsized":"2024-04-25T15:53:26.337351+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"82'28","reported_seq":50,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.723708+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:53:27.723708+0000","last_peered":"2024-04-25T15:53:27.723708+0000","last_clean":"2024-04-25T15:53:27.723708+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:53:27.723708+0000","last_undegraded":"2024-04-25T15:53:27.723708+0000","last_fullsized":"2024-04-25T15:53:27.723708+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":111,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.568200+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:53:26.568200+0000","last_peered":"2024-04-25T15:53:26.568200+0000","last_clean":"2024-04-25T15:53:26.568200+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:53:26.568200+0000","last_undegraded":"2024-04-25T15:53:26.568200+0000","last_fullsized":"2024-04-25T15:53:26.568200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"82'27","reported_seq":195,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.569640+0000","last_change":"2024-04-25T15:53:03.761415+0000","last_active":"2024-04-25T15:53:26.569640+0000","last_peered":"2024-04-25T15:53:26.569640+0000","last_clean":"2024-04-25T15:53:26.569640+0000","last_became_active":"2024-04-25T15:53:03.384666+0000","last_became_peered":"2024-04-25T15:53:03.384666+0000","last_unstale":"2024-04-25T15:53:26.569640+0000","last_undegraded":"2024-04-25T15:53:26.569640+0000","last_fullsized":"2024-04-25T15:53:26.569640+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:14.524576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":85,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570196+0000","last_change":"2024-04-25T15:53:03.409633+0000","last_active":"2024-04-25T15:53:26.570196+0000","last_peered":"2024-04-25T15:53:26.570196+0000","last_clean":"2024-04-25T15:53:26.570196+0000","last_became_active":"2024-04-25T15:53:03.384646+0000","last_became_peered":"2024-04-25T15:53:03.384646+0000","last_unstale":"2024-04-25T15:53:26.570196+0000","last_undegraded":"2024-04-25T15:53:26.570196+0000","last_fullsized":"2024-04-25T15:53:26.570196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:43.442571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"82'21","reported_seq":151,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.692779+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:53:27.692779+0000","last_peered":"2024-04-25T15:53:27.692779+0000","last_clean":"2024-04-25T15:53:27.692779+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:53:27.692779+0000","last_undegraded":"2024-04-25T15:53:27.692779+0000","last_fullsized":"2024-04-25T15:53:27.692779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":164,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.338291+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:53:26.338291+0000","last_peered":"2024-04-25T15:53:26.338291+0000","last_clean":"2024-04-25T15:53:26.338291+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:53:26.338291+0000","last_undegraded":"2024-04-25T15:53:26.338291+0000","last_fullsized":"2024-04-25T15:53:26.338291+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"81'6","reported_seq":169,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.125991+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:53:26.125991+0000","last_peered":"2024-04-25T15:53:26.125991+0000","last_clean":"2024-04-25T15:53:26.125991+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:53:26.125991+0000","last_undegraded":"2024-04-25T15:53:26.125991+0000","last_fullsized":"2024-04-25T15:53:26.125991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"81'25","reported_seq":188,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570989+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:53:26.570989+0000","last_peered":"2024-04-25T15:53:26.570989+0000","last_clean":"2024-04-25T15:53:26.570989+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:53:26.570989+0000","last_undegraded":"2024-04-25T15:53:26.570989+0000","last_fullsized":"2024-04-25T15:53:26.570989+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":142,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.338215+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:53:26.338215+0000","last_peered":"2024-04-25T15:53:26.338215+0000","last_clean":"2024-04-25T15:53:26.338215+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:53:26.338215+0000","last_undegraded":"2024-04-25T15:53:26.338215+0000","last_fullsized":"2024-04-25T15:53:26.338215+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"82'24","reported_seq":204,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:27.309250+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:53:27.309250+0000","last_peered":"2024-04-25T15:53:27.309250+0000","last_clean":"2024-04-25T15:53:27.309250+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:53:27.309250+0000","last_undegraded":"2024-04-25T15:53:27.309250+0000","last_fullsized":"2024-04-25T15:53:27.309250+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":150,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.337190+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:53:26.337190+0000","last_peered":"2024-04-25T15:53:26.337190+0000","last_clean":"2024-04-25T15:53:26.337190+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:53:26.337190+0000","last_undegraded":"2024-04-25T15:53:26.337190+0000","last_fullsized":"2024-04-25T15:53:26.337190+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"81'11","reported_seq":104,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.571561+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:53:26.571561+0000","last_peered":"2024-04-25T15:53:26.571561+0000","last_clean":"2024-04-25T15:53:26.571561+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:53:26.571561+0000","last_undegraded":"2024-04-25T15:53:26.571561+0000","last_fullsized":"2024-04-25T15:53:26.571561+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":141,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570998+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:53:26.570998+0000","last_peered":"2024-04-25T15:53:26.570998+0000","last_clean":"2024-04-25T15:53:26.570998+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:53:26.570998+0000","last_undegraded":"2024-04-25T15:53:26.570998+0000","last_fullsized":"2024-04-25T15:53:26.570998+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"81'15","reported_seq":152,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.570986+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:53:26.570986+0000","last_peered":"2024-04-25T15:53:26.570986+0000","last_clean":"2024-04-25T15:53:26.570986+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:53:26.570986+0000","last_undegraded":"2024-04-25T15:53:26.570986+0000","last_fullsized":"2024-04-25T15:53:26.570986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":115,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-25T15:53:26.339373+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:53:26.339373+0000","last_peered":"2024-04-25T15:53:26.339373+0000","last_clean":"2024-04-25T15:53:26.339373+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:53:26.339373+0000","last_undegraded":"2024-04-25T15:53:26.339373+0000","last_fullsized":"2024-04-25T15:53:26.339373+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1073741824,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1143,"num_write_kb":3633152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":541065216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2147483648,"data_stored":2147483648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1144,"ondisk_log_size":1144,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":115320,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":7427,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":230640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":246,"ondisk_log_size":246,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":335840,"kb_used_data":300568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94036000,"statfs":{"total":96636764160,"available":96292864000,"internally_reserved":0,"allocated":307781632,"data_stored":307227432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278012,"kb_used_data":246836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94093828,"statfs":{"total":96636764160,"available":96352079872,"internally_reserved":0,"allocated":252760064,"data_stored":252217276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":372136,"kb_used_data":336864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93999704,"statfs":{"total":96636764160,"available":96255696896,"internally_reserved":0,"allocated":344948736,"data_stored":344387632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":376208,"kb_used_data":345032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93995632,"statfs":{"total":96636764160,"available":96251527168,"internally_reserved":0,"allocated":353312768,"data_stored":352773638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":359944,"kb_used_data":324672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94011896,"statfs":{"total":96636764160,"available":96268181504,"internally_reserved":0,"allocated":332464128,"data_stored":331909257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":245148,"kb_used_data":213972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94126692,"statfs":{"total":96636764160,"available":96385732608,"internally_reserved":0,"allocated":219107328,"data_stored":218556143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85380,"kb_used_data":54200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286460,"statfs":{"total":96636764160,"available":96549335040,"internally_reserved":0,"allocated":55500800,"data_stored":54976961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315372,"kb_used_data":284192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94056468,"statfs":{"total":96636764160,"available":96313823232,"internally_reserved":0,"allocated":291012608,"data_stored":290449333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":108910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":108705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289406976,"data_stored":289406976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54525952,"data_stored":54525952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218103808,"data_stored":218103808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331350016,"data_stored":331350016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352321536,"data_stored":352321536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":343932928,"data_stored":343932928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":251658240,"data_stored":251658240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":306184192,"data_stored":306184192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:53:33.105 INFO:tasks.thrashosds.thrasher:Setting 3.27 to [7, 6] 2024-04-25T15:53:33.105 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.27', '7', '6'] 2024-04-25T15:53:33.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.27 7 6 2024-04-25T15:53:33.160 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:33.159+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:33.218 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:33.218 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:33.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:33.283 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:33.283+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:33.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.294+0000 7f83038c8640 1 Processor -- start 2024-04-25T15:53:33.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f83038c8640 1 -- start start 2024-04-25T15:53:33.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f83038c8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f82fc152b00 0x7f82fc0c4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:33.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f83038c8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82fc06c640 0x7f82fc06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:33.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f83038c8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82fc158d20 con 0x7f82fc0c8680 2024-04-25T15:53:33.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f83038c8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f82fc0722c0 con 0x7f82fc06c640 2024-04-25T15:53:33.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f83038c8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f82fc158fe0 con 0x7f82fc152b00 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8300e63640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82fc06c640 0x7f82fc06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8300e63640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82fc06c640 0x7f82fc06ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38342/0 (socket says 172.21.15.67:38342) 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8300e63640 1 -- 172.21.15.67:0/746030132 learned_addr learned my addr 172.21.15.67:0/746030132 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8301664640 1 --2- 172.21.15.67:0/746030132 >> v2:172.21.15.176:3300/0 conn(0x7f82fc152b00 0x7f82fc0c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8300e63640 1 -- 172.21.15.67:0/746030132 >> v2:172.21.15.176:3300/0 conn(0x7f82fc152b00 msgr2=0x7f82fc0c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8300e63640 1 --2- 172.21.15.67:0/746030132 >> v2:172.21.15.176:3300/0 conn(0x7f82fc152b00 0x7f82fc0c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8300e63640 1 -- 172.21.15.67:0/746030132 >> v1:172.21.15.176:6789/0 conn(0x7f82fc0c8680 legacy=0x7f82fc0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:33.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.298+0000 7f8300e63640 1 -- 172.21.15.67:0/746030132 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc1592c0 con 0x7f82fc06c640 2024-04-25T15:53:33.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f8300e63640 1 --2- 172.21.15.67:0/746030132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc06c640 0x7f82fc06ca50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f82fc06d490 tx=0x7f82ec01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4934c9612c7abc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:33.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f82f3fff640 1 -- 172.21.15.67:0/746030132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0b0070 con 0x7f82fc06c640 2024-04-25T15:53:33.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f82f3fff640 1 -- 172.21.15.67:0/746030132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ec0ab600 con 0x7f82fc06c640 2024-04-25T15:53:33.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f82f3fff640 1 -- 172.21.15.67:0/746030132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0ab9d0 con 0x7f82fc06c640 2024-04-25T15:53:33.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 -- 172.21.15.67:0/746030132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc06c640 msgr2=0x7f82fc06ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:33.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 --2- 172.21.15.67:0/746030132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc06c640 0x7f82fc06ca50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f82fc06d490 tx=0x7f82ec01c520 comp rx=0 tx=0).stop 2024-04-25T15:53:33.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 -- 172.21.15.67:0/746030132 shutdown_connections 2024-04-25T15:53:33.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 --2- 172.21.15.67:0/746030132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc06c640 0x7f82fc06ca50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 --2- 172.21.15.67:0/746030132 >> v2:172.21.15.176:3300/0 conn(0x7f82fc152b00 0x7f82fc0c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 -- 172.21.15.67:0/746030132 >> 172.21.15.67:0/746030132 conn(0x7f82fc06fe70 msgr2=0x7f82fc0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:33.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 -- 172.21.15.67:0/746030132 shutdown_connections 2024-04-25T15:53:33.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.302+0000 7f83038c8640 1 -- 172.21.15.67:0/746030132 wait complete. 2024-04-25T15:53:33.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 Processor -- start 2024-04-25T15:53:33.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 -- start start 2024-04-25T15:53:33.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82fc06c640 0x7f82fc1f9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc0c8680 0x7f82fc14e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82fc152b00 0x7f82fc147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc0c4960 con 0x7f82fc0c8680 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc152f10 con 0x7f82fc152b00 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f82fc0c4ae0 con 0x7f82fc06c640 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc0c8680 0x7f82fc14e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc0c8680 0x7f82fc14e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38354/0 (socket says 172.21.15.67:38354) 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 -- 172.21.15.67:0/4177724620 learned_addr learned my addr 172.21.15.67:0/4177724620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8301664640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82fc06c640 0x7f82fc1f9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8301e65640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82fc152b00 0x7f82fc147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 -- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82fc06c640 msgr2=0x7f82fc1f9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82fc06c640 0x7f82fc1f9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 -- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82fc152b00 msgr2=0x7f82fc147f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82fc152b00 0x7f82fc147f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 -- 172.21.15.67:0/4177724620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc1f9740 con 0x7f82fc0c8680 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8301664640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82fc06c640 0x7f82fc1f9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8301e65640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82fc152b00 0x7f82fc147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f8300e63640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc0c8680 0x7f82fc14e130 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0ae040 tx=0x7f82ec0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f82f1ffb640 1 -- 172.21.15.67:0/4177724620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0b0040 con 0x7f82fc0c8680 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f82f1ffb640 1 -- 172.21.15.67:0/4177724620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ec0b9070 con 0x7f82fc0c8680 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82fc1655d0 con 0x7f82fc0c8680 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f82f1ffb640 1 -- 172.21.15.67:0/4177724620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0b4610 con 0x7f82fc0c8680 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82fc156370 con 0x7f82fc0c8680 2024-04-25T15:53:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.306+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82fc076730 con 0x7f82fc0c8680 2024-04-25T15:53:33.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.310+0000 7f82f1ffb640 1 -- 172.21.15.67:0/4177724620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f82ec0b47b0 con 0x7f82fc0c8680 2024-04-25T15:53:33.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.310+0000 7f82f1ffb640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82e006f430 0x7f82e0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:33.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.310+0000 7f82f1ffb640 1 -- 172.21.15.67:0/4177724620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5915+0+0 (secure 0 0 0) 0x7f82ec128ff0 con 0x7f82fc0c8680 2024-04-25T15:53:33.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.310+0000 7f8301664640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82e006f430 0x7f82e0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:33.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.310+0000 7f8301664640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82e006f430 0x7f82e0071940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b9f0 tx=0x7f82f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.314+0000 7f82f1ffb640 1 -- 172.21.15.67:0/4177724620 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f82ec0a4e20 con 0x7f82fc0c8680 2024-04-25T15:53:33.362 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:33.362+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:33.434 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:33.435 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:33.435 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:33.435 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:33.435 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:33.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:33.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:33.482+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "3.27", "id": [7, 6]} v 0) -- 0x7f82fc0c96b0 con 0x7f82fc0c8680 2024-04-25T15:53:33.499 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:33.499+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:33.564 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:33.563+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:33.571 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:33.571 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 re-readers = 6365697.50 kB/sec 2024-04-25T15:53:33.571 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 re-readers = 6340369.09 kB/sec 2024-04-25T15:53:33.571 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 6365697.50 kB/sec 2024-04-25T15:53:33.571 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 6365697.50 kB/sec 2024-04-25T15:53:33.571 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 6365697.50 kB/sec 2024-04-25T15:53:33.571 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 1048576.00 kB 2024-04-25T15:53:33.603 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:33.603 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:33.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:33.603 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:33.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.603 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:597 3.e e81/76 3:72b88a04:::100000001f7.00000051:head v 81'6, mlcod=69'3)", 2024-04-25T15:53:33.603 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.227426997, 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038266363999999997, 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374533+0000", 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5060000000000001e-06 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376869+0000", 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023357830000000001 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376914+0000", 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4317000000000003e-05 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376941+0000", 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6965999999999999e-05 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377065+0000", 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012408300000000001 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377098+0000", 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2943e-05 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412723+0000", 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035625742000000002 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412798+0000", 2024-04-25T15:53:33.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5024000000000005e-05 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:599 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.223137768000001, 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039983414000000002, 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 599 2024-04-25T15:53:33.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378825+0000", 2024-04-25T15:53:33.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7189999999999999e-06 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384734+0000", 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059089859999999998 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384776+0000", 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2166999999999998e-05 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384795+0000", 2024-04-25T15:53:33.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9307999999999999e-05 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384894+0000", 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9112999999999998e-05 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385064+0000", 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169524 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385212+0000", 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147849 2024-04-25T15:53:33.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409088+0000", 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023875850000000001 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418680+0000", 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095925639999999996 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418742+0000", 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1379999999999998e-05 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418805+0000", 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2953999999999994e-05 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:598 3.11 e81/76 3:89a80ce8:::100000001f7.00000052:head v 81'21, mlcod=80'19)", 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.220598445, 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037275751000000003, 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.613 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381362+0000", 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.782e-06 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382748+0000", 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001385488 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382769+0000", 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0854000000000001e-05 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382786+0000", 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7807000000000001e-05 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382876+0000", 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9821999999999995e-05 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382901+0000", 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4845999999999999e-05 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418526+0000", 2024-04-25T15:53:33.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035624426000000001 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418636+0000", 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110726 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:606 3.31 3:8d161915:::100000001f7.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.172026472000001, 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031627436000000002, 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 606 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429935+0000", 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9610000000000001e-06 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435340+0000", 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054048719999999998 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435375+0000", 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4558999999999999e-05 2024-04-25T15:53:33.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435394+0000", 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8819000000000001e-05 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435493+0000", 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8927999999999999e-05 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435806+0000", 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031338500000000001 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.436016+0000", 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000209586 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.459476+0000", 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023460547000000002 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461366+0000", 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001890206 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461433+0000", 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6206999999999998e-05 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461560+0000", 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012736600000000001 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:654 3.11 e81/76 3:892e96a3:::100000001f7.0000008a:head v 81'22, mlcod=80'20)", 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7451983200000001, 2024-04-25T15:53:33.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044455910000000001, 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856762+0000", 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.669e-06 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859651+0000", 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002889015 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859675+0000", 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3346999999999999e-05 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859696+0000", 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1021000000000001e-05 2024-04-25T15:53:33.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859816+0000", 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120809 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859853+0000", 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6242999999999998e-05 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901127+0000", 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041274245000000001 2024-04-25T15:53:33.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901217+0000", 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9561000000000005e-05 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:668 3.1 3:83ba1de1:::100000001f7.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7418266940000002, 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062400662000000003, 2024-04-25T15:53:33.626 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 668 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860140+0000", 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1559999999999995e-06 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869131+0000", 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089909859999999994 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869188+0000", 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6539000000000002e-05 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869212+0000", 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4501000000000001e-05 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869417+0000", 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020473999999999999 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869612+0000", 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000194668 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871230+0000", 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016182029999999999 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.917654+0000", 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046423786000000002 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922361+0000", 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00470704 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922442+0000", 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.153e-05 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922533+0000", 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0513000000000006e-05 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.631 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:669 3.f e81/76 3:f0675d7f:::100000001f7.00000099:head v 81'10, mlcod=70'8)", 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7157901760000005, 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080177950999999997, 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886171+0000", 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8470000000000001e-06 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891613+0000", 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054420349999999996 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891657+0000", 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4155e-05 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891689+0000", 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1822999999999997e-05 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891773+0000", 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4602000000000001e-05 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891806+0000", 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2842999999999997e-05 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966243+0000", 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074437173999999995 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966347+0000", 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103472 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:647 3.a e81/76 3:524e89bd:::100000001f7.00000083:head v 81'3, mlcod=61'1)", 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6973555539999996, 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072551145999999997, 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.636 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904605+0000", 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.421e-06 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906854+0000", 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022493249999999999 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906880+0000", 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5374000000000001e-05 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906897+0000", 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7419999999999999e-05 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906994+0000", 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6633999999999997e-05 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.907027+0000", 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3336000000000001e-05 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977067+0000", 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.070040033000000002 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977155+0000", 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7602999999999995e-05 2024-04-25T15:53:33.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:674 3.9 3:90d768d7:::100000001f7.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6896886230000003, 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065083915000000006, 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 674 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912273+0000", 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6929999999999999e-06 2024-04-25T15:53:33.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921147+0000", 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088735250000000002 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921185+0000", 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8098e-05 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921200+0000", 2024-04-25T15:53:33.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5543999999999999e-05 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921281+0000", 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0758000000000002e-05 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921426+0000", 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014529000000000001 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921540+0000", 2024-04-25T15:53:33.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113465 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.942585+0000", 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021045735999999999 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977215+0000", 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034629659 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977282+0000", 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6796000000000002e-05 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977354+0000", 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2350999999999999e-05 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:675 3.a e81/76 3:5122fcae:::100000001f7.0000009f:head v 81'4, mlcod=61'1)", 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6721284769999993, 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047589770000000003, 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.645 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929833+0000", 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2850000000000001e-06 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931776+0000", 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001943443 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931806+0000", 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0074e-05 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931829+0000", 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3027000000000002e-05 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931927+0000", 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7540000000000005e-05 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931959+0000", 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2265999999999999e-05 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977374+0000", 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045414643999999997 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977420+0000", 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6490999999999999e-05 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:676 3.18 e81/76 3:191fe66d:::100000001f7.000000a0:head v 81'8, mlcod=71'6)", 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:33.649 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6702839090000001, 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050752358999999997, 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:33.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931678+0000", 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.542e-06 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935891+0000", 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042133930000000002 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935917+0000", 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5624000000000001e-05 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935935+0000", 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8258999999999999e-05 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936019+0000", 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4538999999999998e-05 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936060+0000", 2024-04-25T15:53:33.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1060000000000003e-05 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982347+0000", 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046286542999999999 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982427+0000", 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0399000000000001e-05 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:678 3.1b 3:da348c2f:::100000001f7.000000a2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.653 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6589158410000007, 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039665855999999999, 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 678 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943045+0000", 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9630000000000001e-06 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947273+0000", 2024-04-25T15:53:33.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042283470000000004 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947307+0000", 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3794000000000002e-05 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947322+0000", 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5184999999999999e-05 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947403+0000", 2024-04-25T15:53:33.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1076999999999996e-05 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947533+0000", 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012934399999999999 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947646+0000", 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113163 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.964158+0000", 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016512526 2024-04-25T15:53:33.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982578+0000", 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018419333 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982646+0000", 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8010999999999996e-05 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982709+0000", 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:33.658 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:679 3.24 e81/76 3:27dcc3f5:::100000001f7.000000a3:head v 81'12, mlcod=81'10)", 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.6493702759999991, 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030189264, 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:33.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952590+0000", 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.381e-06 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956607+0000", 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040170140000000002 2024-04-25T15:53:33.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956623+0000", 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6025e-05 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956636+0000", 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2595000000000001e-05 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956708+0000", 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2402000000000002e-05 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956741+0000", 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3386999999999997e-05 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982717+0000", 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025975446999999999 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982778+0000", 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1013e-05 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:720 3.1a 3:59a21752:::100000001f7.000000cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.3771265380000006, 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031267504000000002, 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 720 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224835+0000", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.661e-06 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230311+0000", 2024-04-25T15:53:33.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054762220000000002 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230341+0000", 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9972999999999999e-05 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230357+0000", 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.558e-05 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230440+0000", 2024-04-25T15:53:33.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3030999999999998e-05 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230610+0000", 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016995500000000001 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230782+0000", 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000172079 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.247511+0000", 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016729067 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255803+0000", 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082918499999999999 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.256001+0000", 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019800300000000001 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.256100+0000", 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9083e-05 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:723 3.9 3:9221849e:::100000001f7.000000cf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.3588940120000004, 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031588667000000001, 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.668 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 723 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243070+0000", 2024-04-25T15:53:33.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0200000000000002e-06 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248284+0000", 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052135619999999997 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248318+0000", 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4072000000000002e-05 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248337+0000", 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9867000000000001e-05 2024-04-25T15:53:33.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248421+0000", 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3541000000000005e-05 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248550+0000", 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128768 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248782+0000", 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023251200000000001 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.265337+0000", 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016554484000000001 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274315+0000", 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089778259999999995 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274497+0000", 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018229699999999999 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274654+0000", 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000156718 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:762 3.3 3:c0da3165:::100000001f7.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.0869143529999992, 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033743224000000002, 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 762 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515046+0000", 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.578e-06 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523433+0000", 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083871290000000001 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523458+0000", 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4732e-05 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523469+0000", 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0882e-05 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523588+0000", 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119361 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523741+0000", 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152311 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523908+0000", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167472 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.535424+0000", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011516337 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548681+0000", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013256274 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548736+0000", 2024-04-25T15:53:33.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5306999999999997e-05 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548788+0000", 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1841000000000001e-05 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:957 3.11 e82/76 3:897f0f98:::100000001f7.000000b9:head v 82'38, mlcod=82'36)", 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.3641377869999998, 2024-04-25T15:53:33.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030634000000000002, 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237823+0000", 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9429999999999999e-06 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239142+0000", 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013188130000000001 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239168+0000", 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6132e-05 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239183+0000", 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5068999999999999e-05 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239267+0000", 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4227000000000005e-05 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239290+0000", 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2209999999999999e-05 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268394+0000", 2024-04-25T15:53:33.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029104512999999999 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268455+0000", 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1093000000000006e-05 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1004 3.1d e82/76 3:b93a1278:::100000001f7.000000e8:head v 82'29, mlcod=82'27)", 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.0779760539999996, 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044544315000000001, 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.682 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523985+0000", 2024-04-25T15:53:33.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9999999999999999e-06 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527244+0000", 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032587330000000002 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527284+0000", 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0031999999999999e-05 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527309+0000", 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527437+0000", 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128384 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527480+0000", 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2357999999999997e-05 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568424+0000", 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040944332 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568527+0000", 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103319 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1008 3.2c e82/76 3:3798f9d7:::100000001f7.000000ec:head v 82'14, mlcod=82'12)", 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.0635611970000003, 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042823282999999997, 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.686 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538400+0000", 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0650000000000001e-06 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541298+0000", 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028986440000000001 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541324+0000", 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5176000000000001e-05 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541339+0000", 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5821999999999999e-05 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541469+0000", 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541521+0000", 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1882999999999998e-05 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581098+0000", 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039576885999999999 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581221+0000", 2024-04-25T15:53:33.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012303699999999999 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1012 3.1b 3:db4e25ba:::100000001f7.000000f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.0527222409999997, 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032333898999999999, 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:33.690 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1012 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549239+0000", 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7700000000000002e-06 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555878+0000", 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066383470000000002 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555906+0000", 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8147e-05 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555923+0000", 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7385999999999999e-05 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556004+0000", 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0836999999999999e-05 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556110+0000", 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105487 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556220+0000", 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110053 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568508+0000", 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012288566000000001 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581445+0000", 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012937059000000001 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581507+0000", 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1703000000000002e-05 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581571+0000", 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3544e-05 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:33.695 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:33.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:33.697 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:33.698+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:33.797 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:33.798+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:33.893 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:33.893 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:33.893 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:33.893 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:33.898 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:33.898+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:33.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:33.999 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:33.998+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:34.055 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:34.055 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:34.055 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:34.055 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:34.055 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:34.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:34.123 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:34.123+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.178+0000 7f82f1ffb640 1 -- 172.21.15.67:0/4177724620 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "3.27", "id": [7, 6]}]=0 set 3.27 pg_upmap mapping to [7,6] v85) ==== 124+0+0 (secure 0 0 0) 0x7f82ec016110 con 0x7f82fc0c8680 2024-04-25T15:53:34.179 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.27 pg_upmap mapping to [7,6] 2024-04-25T15:53:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82e006f430 msgr2=0x7f82e0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82e006f430 0x7f82e0071940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b9f0 tx=0x7f82f8008040 comp rx=0 tx=0).stop 2024-04-25T15:53:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc0c8680 msgr2=0x7f82fc14e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc0c8680 0x7f82fc14e130 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0ae040 tx=0x7f82ec0192c0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 shutdown_connections 2024-04-25T15:53:34.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82e006f430 0x7f82e0071940 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82fc152b00 0x7f82fc147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82fc0c8680 0x7f82fc14e130 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 --2- 172.21.15.67:0/4177724620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82fc06c640 0x7f82fc1f9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 >> 172.21.15.67:0/4177724620 conn(0x7f82fc06fe70 msgr2=0x7f82fc14ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 shutdown_connections 2024-04-25T15:53:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.182+0000 7f83038c8640 1 -- 172.21.15.67:0/4177724620 wait complete. 2024-04-25T15:53:34.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:53:34.213 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:34.210+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 31.000989886999999, 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.223 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:53:34.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:53:34.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:53:34.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:594 3.16 3:6a58e86c:::100000001f7.0000004e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.872812440000001, 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034501508, 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 594 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347207+0000", 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8660000000000002e-06 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353453+0000", 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062464160000000003 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353489+0000", 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5283000000000001e-05 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353501+0000", 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2860999999999999e-05 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353629+0000", 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000127632 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353808+0000", 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017904800000000001 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353996+0000", 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018766099999999999 2024-04-25T15:53:34.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.380705+0000", 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026708789 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381525+0000", 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00082074100000000005 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381619+0000", 2024-04-25T15:53:34.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3153999999999996e-05 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381706+0000", 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7057000000000004e-05 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:596 3.16 3:6941c83d:::100000001f7.00000050:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.858097958, 2024-04-25T15:53:34.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049183123000000002, 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 596 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361922+0000", 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367484+0000", 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055618409999999997 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367522+0000", 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8260000000000003e-05 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367543+0000", 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0795999999999999e-05 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367646+0000", 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102913 2024-04-25T15:53:34.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367857+0000", 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021190400000000001 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368091+0000", 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023323699999999999 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.405516+0000", 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037425424999999998 2024-04-25T15:53:34.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410955+0000", 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054389859999999998 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411034+0000", 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8421e-05 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411102+0000", 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8137000000000002e-05 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:600 3.5 3:a2751c53:::100000001f7.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.834614452, 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035430837, 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.241 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 600 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385405+0000", 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8959999999999999e-06 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390062+0000", 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046572949999999997 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390104+0000", 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1943000000000001e-05 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390123+0000", 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9147000000000002e-05 2024-04-25T15:53:34.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390225+0000", 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101983 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390407+0000", 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181819 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390593+0000", 2024-04-25T15:53:34.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018578900000000001 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409139+0000", 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018546360000000001 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420728+0000", 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011589083 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420783+0000", 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5025000000000003e-05 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420833+0000", 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9496999999999997e-05 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:649 3.3c 3:3e99caf7:::100000001f7.00000085:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.246 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.485273889, 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031048148000000001, 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 649 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734744+0000", 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.694e-06 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740013+0000", 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052687469999999998 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740078+0000", 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5402999999999997e-05 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740096+0000", 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7393000000000001e-05 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740206+0000", 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109789 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740427+0000", 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022180100000000001 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740653+0000", 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022509799999999999 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.748988+0000", 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083355619999999995 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765614+0000", 2024-04-25T15:53:34.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016625788999999998 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765712+0000", 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8183999999999994e-05 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765791+0000", 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8688000000000003e-05 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.251 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:665 3.1d 3:b87c769f:::100000001f7.00000095:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.378992918, 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045554887000000002, 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 665 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841029+0000", 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9359999999999998e-06 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851995+0000", 2024-04-25T15:53:34.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010966115 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852062+0000", 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6975000000000002e-05 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852084+0000", 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2382999999999999e-05 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852218+0000", 2024-04-25T15:53:34.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013397699999999999 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852418+0000", 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000200161 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852550+0000", 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013199300000000001 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878731+0000", 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026181157 2024-04-25T15:53:34.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886405+0000", 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076735090000000002 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886503+0000", 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8504000000000004e-05 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886578+0000", 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5177000000000002e-05 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.256 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:663 3.d e81/18 3:b1139e77:::100000001f7.00000093:head v 81'7, mlcod=80'5)", 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.361555042999999, 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032554081999999998, 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858463+0000", 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8169999999999999e-06 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862335+0000", 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003871462 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862369+0000", 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4635999999999998e-05 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862395+0000", 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5698e-05 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862753+0000", 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035778999999999997 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862879+0000", 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012587500000000001 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.890905+0000", 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028025912 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891016+0000", 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011089199999999999 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:664 3.d e81/18 3:b353c7f2:::100000001f7.00000094:head v 81'8, mlcod=80'5)", 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.350694018, 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051411299000000001, 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869325+0000", 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1689999999999999e-06 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872141+0000", 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028160759999999998 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872176+0000", 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4996999999999997e-05 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872202+0000", 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5944000000000001e-05 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872312+0000", 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010998200000000001 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872351+0000", 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9360999999999998e-05 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920645+0000", 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048293989000000002 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920734+0000", 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8781000000000003e-05 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:671 3.38 e81/18 3:1c0639bf:::100000001f7.0000009b:head v 81'8, mlcod=81'6)", 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.333369810000001, 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069441491999999994, 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886668+0000", 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0876000000000001e-05 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893242+0000", 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065744849999999997 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893267+0000", 2024-04-25T15:53:34.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5321e-05 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893283+0000", 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5454000000000001e-05 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893362+0000", 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8770000000000006e-05 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893382+0000", 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0585999999999999e-05 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.955934+0000", 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062552041000000003 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956088+0000", 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:673 3.28 3:153e0bf4:::100000001f7.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.315080521000001, 2024-04-25T15:53:34.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063190614000000006, 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 673 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904939+0000", 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8739999999999999e-06 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911908+0000", 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069695449999999997 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911944+0000", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5973000000000003e-05 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911960+0000", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5319999999999999e-05 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912057+0000", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7427000000000001e-05 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912221+0000", 2024-04-25T15:53:34.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016433600000000001 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912375+0000", 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015359299999999999 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.941806+0000", 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029430523 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967997+0000", 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026191589000000001 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968063+0000", 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6321000000000004e-05 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968127+0000", 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:652 3.36 e81/18 3:6c98fff9:::100000001f7.00000088:head v 81'12, mlcod=69'10)", 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.30388376, 2024-04-25T15:53:34.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052156028, 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916135+0000", 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7850000000000001e-06 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918034+0000", 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018996099999999999 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918080+0000", 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5469000000000001e-05 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918098+0000", 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8726999999999999e-05 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918176+0000", 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8125000000000002e-05 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918198+0000", 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1531000000000001e-05 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968240+0000", 2024-04-25T15:53:34.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050041583000000001 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968289+0000", 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9197999999999999e-05 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:677 3.5 3:a07f2395:::100000001f7.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.284502036999999, 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041485695000000003, 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.277 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 677 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935657+0000", 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014239100000000001 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940003+0000", 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043458239999999999 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940055+0000", 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2021999999999998e-05 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940071+0000", 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6528000000000002e-05 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940148+0000", 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7179000000000001e-05 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940294+0000", 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014605799999999999 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940418+0000", 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012356499999999999 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956756+0000", 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016338149999999999 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976852+0000", 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020095572999999999 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976925+0000", 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3351999999999998e-05 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977000+0000", 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5053000000000001e-05 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.282 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:719 3.1d 3:b9e367d5:::100000001f7.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.001126962000001, 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037012611000000001, 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 719 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:34.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218892+0000", 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.182e-06 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226010+0000", 2024-04-25T15:53:34.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071184320000000001 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226050+0000", 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0167999999999999e-05 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226070+0000", 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9256999999999999e-05 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226209+0000", 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000139226 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226360+0000", 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015169700000000001 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226574+0000", 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021339199999999999 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.253457+0000", 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026883383 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255765+0000", 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023077409999999999 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255837+0000", 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1987e-05 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255902+0000", 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5146000000000003e-05 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:729 3.5 3:a1e9b24d:::100000001f7.000000d5:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.9338898740000001, 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033306524999999997, 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 729 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286128+0000", 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292385+0000", 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006257589 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292423+0000", 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7756999999999998e-05 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292441+0000", 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7954000000000001e-05 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292521+0000", 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0171000000000002e-05 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292651+0000", 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130025 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292787+0000", 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013599099999999999 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.302313+0000", 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095255389999999995 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319295+0000", 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016982391999999999 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319373+0000", 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7571000000000007e-05 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319433+0000", 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0306000000000002e-05 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:745 3.d e81/18 3:b225ffa8:::100000001f7.000000e5:head v 81'10, mlcod=81'8)", 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:34.293 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.8179616299999992, 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034113420999999998, 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402056+0000", 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3459999999999999e-06 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405220+0000", 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031635439999999999 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405240+0000", 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0695000000000001e-05 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405253+0000", 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2897e-05 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405323+0000", 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9719000000000001e-05 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405377+0000", 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3653000000000003e-05 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436064+0000", 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030687076000000001 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436168+0000", 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104491 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:748 3.1d 3:b93a1278:::100000001f7.000000e8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.297 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.8120882500000004, 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033655512999999998, 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 748 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407929+0000", 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9899999999999999e-07 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419777+0000", 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011847952 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419801+0000", 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4049999999999998e-05 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419810+0000", 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3039999999999994e-06 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419874+0000", 2024-04-25T15:53:34.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3467999999999996e-05 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.420010+0000", 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013569600000000001 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.420161+0000", 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015180900000000001 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.438066+0000", 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017904571000000001 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441466+0000", 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033998959999999999 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441529+0000", 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3124000000000001e-05 2024-04-25T15:53:34.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441584+0000", 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4744000000000003e-05 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1002 3.1f 3:fb90ae0e:::100000001f7.000000e6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.7213171679999997, 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035318779000000002, 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.303 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1002 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:34.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498702+0000", 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2330000000000001e-06 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507556+0000", 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088549340000000001 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508591+0000", 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010341459999999999 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508610+0000", 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9490000000000001e-05 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508716+0000", 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105468 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508836+0000", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120627 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.509012+0000", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017549199999999999 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521191+0000", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012179339000000001 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533885+0000", 2024-04-25T15:53:34.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01269438 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533944+0000", 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8971000000000002e-05 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534018+0000", 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3699e-05 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1003 3.28 3:1719c8f9:::100000001f7.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.308 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.7124497029999999, 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062620618000000003, 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1003 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:34.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507569+0000", 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2110000000000001e-06 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519064+0000", 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011495182 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521838+0000", 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027741129999999999 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521856+0000", 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521967+0000", 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110913 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522100+0000", 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013309899999999999 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522235+0000", 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013518199999999999 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.543023+0000", 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020788025000000002 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570076+0000", 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027052208000000001 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570138+0000", 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2376e-05 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570187+0000", 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9379000000000003e-05 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.313 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1004 3.1d 3:b93a1278:::100000001f7.000000e8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.7009132490000001, 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049745623000000003, 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1004 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519106+0000", 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.464e-06 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523205+0000", 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040996339999999996 2024-04-25T15:53:34.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523267+0000", 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1660000000000003e-05 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523292+0000", 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5194999999999998e-05 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523428+0000", 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013594 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523595+0000", 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016733300000000001 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523819+0000", 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022358999999999999 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.566513+0000", 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042694339999999997 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568719+0000", 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022057230000000001 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568799+0000", 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0370000000000005e-05 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568849+0000", 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9373999999999999e-05 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1007 3.36 e82/18 3:6de076d9:::100000001f7.000000eb:head v 82'23, mlcod=82'21)", 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6861461330000003, 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040044785999999999, 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:34.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533871+0000", 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3799999999999996e-07 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535899+0000", 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020273069999999999 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535926+0000", 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7803000000000001e-05 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535945+0000", 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8555000000000001e-05 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.536054+0000", 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109441 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.536089+0000", 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4411e-05 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.573843+0000", 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037753789000000003 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.573915+0000", 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2541999999999998e-05 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.322 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:34.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:34.324 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:34.323+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d2f28640 1 Processor -- start 2024-04-25T15:53:34.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d2f28640 1 -- start start 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d2f28640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66cc06c640 0x7f66cc06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d2f28640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d2f28640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66cc158d20 con 0x7f66cc152b00 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d2f28640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f66cc0722c0 con 0x7f66cc06c640 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d2f28640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f66cc158fe0 con 0x7f66cc0c8680 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66cbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66cc06c640 0x7f66cc06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d0cc4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66cc152b00 0x7f66cc0c4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36292/0 (socket says 172.21.15.67:36292) 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.354+0000 7f66d0cc4640 1 -- 172.21.15.67:0/3586457729 learned_addr learned my addr 172.21.15.67:0/3586457729 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d14c5640 1 --2- 172.21.15.67:0/3586457729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:34.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cbfff640 1 -- 172.21.15.67:0/3586457729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 msgr2=0x7f66cc0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cbfff640 1 --2- 172.21.15.67:0/3586457729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cbfff640 1 -- 172.21.15.67:0/3586457729 >> v1:172.21.15.67:6789/0 conn(0x7f66cc152b00 legacy=0x7f66cc0c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cb7fe640 1 -- 172.21.15.67:0/3586457729 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761558946 0 0) 0x7f66cc158d20 con 0x7f66cc152b00 2024-04-25T15:53:34.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cbfff640 1 -- 172.21.15.67:0/3586457729 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66cc1592c0 con 0x7f66cc06c640 2024-04-25T15:53:34.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cbfff640 1 --2- 172.21.15.67:0/3586457729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc06c640 0x7f66cc06ca50 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f66c000a960 tx=0x7f66c001d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672a6996864c3b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:34.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cb7fe640 1 -- 172.21.15.67:0/3586457729 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c000b040 con 0x7f66cc06c640 2024-04-25T15:53:34.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cb7fe640 1 -- 172.21.15.67:0/3586457729 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c00a76d0 con 0x7f66cc06c640 2024-04-25T15:53:34.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66cb7fe640 1 -- 172.21.15.67:0/3586457729 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c00a7aa0 con 0x7f66cc06c640 2024-04-25T15:53:34.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 -- 172.21.15.67:0/3586457729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc06c640 msgr2=0x7f66cc06ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:34.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3586457729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc06c640 0x7f66cc06ca50 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f66c000a960 tx=0x7f66c001d870 comp rx=0 tx=0).stop 2024-04-25T15:53:34.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 -- 172.21.15.67:0/3586457729 shutdown_connections 2024-04-25T15:53:34.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3586457729 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3586457729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc06c640 0x7f66cc06ca50 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 -- 172.21.15.67:0/3586457729 >> 172.21.15.67:0/3586457729 conn(0x7f66cc06fe70 msgr2=0x7f66cc0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 -- 172.21.15.67:0/3586457729 shutdown_connections 2024-04-25T15:53:34.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.358+0000 7f66d2f28640 1 -- 172.21.15.67:0/3586457729 wait complete. 2024-04-25T15:53:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 Processor -- start 2024-04-25T15:53:34.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 -- start start 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66cc06c640 0x7f66cc0747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc152b00 0x7f66cc1f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66cc152f10 con 0x7f66cc152b00 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f66cc0ca100 con 0x7f66cc06c640 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f66cc1654c0 con 0x7f66cc0c8680 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc152b00 0x7f66cc1f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc152b00 0x7f66cc1f6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38380/0 (socket says 172.21.15.67:38380) 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 -- 172.21.15.67:0/3421148041 learned_addr learned my addr 172.21.15.67:0/3421148041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:34.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66cbfff640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d0cc4640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66cc06c640 0x7f66cc0747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 -- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 msgr2=0x7f66cc0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 -- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66cc06c640 msgr2=0x7f66cc0747e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66cc06c640 0x7f66cc0747e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 -- 172.21.15.67:0/3421148041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66cc073280 con 0x7f66cc152b00 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d14c5640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc152b00 0x7f66cc1f6db0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f66c4007da0 tx=0x7f66c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66c97fa640 1 -- 172.21.15.67:0/3421148041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4014420 con 0x7f66cc152b00 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66c97fa640 1 -- 172.21.15.67:0/3421148041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c4019070 con 0x7f66cc152b00 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66cc1f72f0 con 0x7f66cc152b00 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66c97fa640 1 -- 172.21.15.67:0/3421148041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4014820 con 0x7f66cc152b00 2024-04-25T15:53:34.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.362+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66cc0cd420 con 0x7f66cc152b00 2024-04-25T15:53:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.366+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66cc072ff0 con 0x7f66cc152b00 2024-04-25T15:53:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.366+0000 7f66c97fa640 1 -- 172.21.15.67:0/3421148041 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f66c40149c0 con 0x7f66cc152b00 2024-04-25T15:53:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.366+0000 7f66c97fa640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66b806f430 0x7f66b8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.366+0000 7f66c97fa640 1 -- 172.21.15.67:0/3421148041 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5944+0+0 (secure 0 0 0) 0x7f66c408f860 con 0x7f66cc152b00 2024-04-25T15:53:34.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.366+0000 7f66d0cc4640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66b806f430 0x7f66b8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:34.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.366+0000 7f66d0cc4640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66b806f430 0x7f66b8071940 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000f70 tx=0x7f66bc007df0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:34.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.370+0000 7f66c97fa640 1 -- 172.21.15.67:0/3421148041 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f66cc073280 con 0x7f66cc152b00 2024-04-25T15:53:34.424 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:34.422+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:34.515 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:34.515 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:34.515 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:34.515 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:34.525 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:34.523+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:34.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:34.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.542+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f66cc06a480 con 0x7f66cc152b00 2024-04-25T15:53:34.626 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:34.626+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:34.717 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:34.718 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:34.718 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:34.718 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:34.718 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:34.727 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:34.726+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:34.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:34.828 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:34.827+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:34.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.221205596000001, 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.26243838600000002, 2024-04-25T15:53:34.884 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920373+0000", 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039448372000000002 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921649+0000", 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012757210000000001 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921724+0000", 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5549000000000005e-05 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921802+0000", 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8150000000000002e-05 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.889 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:640 3.10 3:092a6a2c:::100000001f7.0000007c:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.002174315, 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065130602999999995, 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.890 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 640 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878397+0000", 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.198e-06 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894140+0000", 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015742477000000001 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894240+0000", 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010061299999999999 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894258+0000", 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7779999999999999e-05 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894341+0000", 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3168000000000001e-05 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894526+0000", 2024-04-25T15:53:34.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018456399999999999 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894632+0000", 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010651500000000001 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.923409+0000", 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028776380000000001 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943396+0000", 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019987393999999999 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943459+0000", 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2389000000000002e-05 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943526+0000", 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7125000000000005e-05 2024-04-25T15:53:34.895 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:88 2.14 2:2e2fa760:::200.00000001:head [write 97242~3394 [fadvise_dontneed] in=3394b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.989142437, 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052104437000000003, 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891429+0000", 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8190000000000001e-06 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891440+0000", 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1029999999999999e-05 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891499+0000", 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8961e-05 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891530+0000", 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1049e-05 2024-04-25T15:53:34.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891630+0000", 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010053899999999999 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891787+0000", 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015704200000000001 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891996+0000", 2024-04-25T15:53:34.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020865400000000001 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920805+0000", 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028808585000000001 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943394+0000", 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022589115 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943465+0000", 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1254000000000006e-05 2024-04-25T15:53:34.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943531+0000", 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:647 3.a 3:524e89bd:::100000001f7.00000083:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.986419638999999, 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083393058000000006, 2024-04-25T15:53:34.901 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 647 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894152+0000", 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.968e-06 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902694+0000", 2024-04-25T15:53:34.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085420719999999995 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903482+0000", 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078797999999999997 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903499+0000", 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7082e-05 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903593+0000", 2024-04-25T15:53:34.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4452999999999998e-05 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903774+0000", 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180803 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903953+0000", 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179106 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.965763+0000", 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061810057000000002 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977390+0000", 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01162697 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977474+0000", 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3812000000000003e-05 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977543+0000", 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8755000000000002e-05 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:652 3.36 3:6c98fff9:::100000001f7.00000088:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.977864334, 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065992254, 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.907 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 652 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902707+0000", 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5209999999999999e-06 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910035+0000", 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073280000000000003 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910344+0000", 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030890699999999998 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910417+0000", 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3900999999999996e-05 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910547+0000", 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012940400000000001 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910876+0000", 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000329181 2024-04-25T15:53:34.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911062+0000", 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018587000000000001 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966069+0000", 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.055007038000000001 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968579+0000", 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002509598 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968649+0000", 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0189000000000002e-05 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968697+0000", 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8644999999999999e-05 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:660 3.26 3:65a7570a:::100000001f7.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:34.912 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.970520955, 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058617787999999997, 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 660 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910051+0000", 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7599999999999998e-06 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925001+0000", 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014949405000000001 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925210+0000", 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020949799999999999 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925223+0000", 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3086e-05 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925337+0000", 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011333899999999999 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925525+0000", 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018789300000000001 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925638+0000", 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011354 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967535+0000", 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041897074999999999 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968553+0000", 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010179480000000001 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968609+0000", 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5653999999999999e-05 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968666+0000", 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7590000000000003e-05 2024-04-25T15:53:34.917 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:675 3.a 3:5122fcae:::100000001f7.0000009f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.955559107999999, 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052715010999999999, 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.918 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 675 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925013+0000", 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3089999999999998e-06 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929333+0000", 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043198170000000001 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929365+0000", 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2323000000000002e-05 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929380+0000", 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5421999999999999e-05 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929460+0000", 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0148000000000006e-05 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929572+0000", 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011171 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929688+0000", 2024-04-25T15:53:34.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011547 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966801+0000", 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037113896 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977633+0000", 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010831294999999999 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977681+0000", 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7926000000000002e-05 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977725+0000", 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4694999999999998e-05 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:750 3.10 3:08911dbf:::100000001f7.000000ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.458542904, 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029714553000000001, 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.923 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 750 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422031+0000", 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5050000000000001e-06 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.427886+0000", 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058545810000000002 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433892+0000", 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060058660000000003 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433903+0000", 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1769e-05 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433975+0000", 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1872000000000006e-05 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.434165+0000", 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018962300000000001 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.434345+0000", 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180034 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441345+0000", 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070004730000000001 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451615+0000", 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010269276000000001 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451679+0000", 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4369000000000007e-05 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451741+0000", 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2185000000000002e-05 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:751 3.36 3:6de076d9:::100000001f7.000000eb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.451968438, 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035860229, 2024-04-25T15:53:34.928 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 751 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428603+0000", 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5820000000000001e-06 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433862+0000", 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052591870000000002 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442450+0000", 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085886669999999995 2024-04-25T15:53:34.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442460+0000", 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3160000000000003e-06 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442541+0000", 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1217000000000005e-05 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442706+0000", 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016477000000000001 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442865+0000", 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015895899999999999 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.460454+0000", 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017589370999999999 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464395+0000", 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039411139999999999 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464433+0000", 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7710999999999999e-05 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464461+0000", 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8334999999999999e-05 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:752 3.2c 3:3798f9d7:::100000001f7.000000ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.444238532, 2024-04-25T15:53:34.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071715859000000007, 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 752 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436332+0000", 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.429e-06 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442426+0000", 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060933589999999996 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486527+0000", 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044101711000000002 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486535+0000", 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1749999999999999e-06 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486862+0000", 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032728400000000001 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.487028+0000", 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016650399999999999 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.487190+0000", 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016182699999999999 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.494699+0000", 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007509033 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507938+0000", 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013238689 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507980+0000", 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2268999999999998e-05 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.508047+0000", 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6579000000000001e-05 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.938 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:354 3.10 3:0ba0dc78:::10000000002.000000fd:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.281252805999999, 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56691576200000005, 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599318+0000", 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599326+0000", 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9459999999999998e-06 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161759+0000", 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56243287799999997 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161783+0000", 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4057e-05 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161899+0000", 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011615799999999999 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162007+0000", 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107823 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162110+0000", 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010235500000000001 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.163904+0000", 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017941470000000001 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166138+0000", 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022346190000000002 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166186+0000", 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7902999999999999e-05 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166232+0000", 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6220999999999997e-05 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:953 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.6690138000000001, 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032562766999999999, 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 953 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211570+0000", 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3920999999999999e-05 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216848+0000", 2024-04-25T15:53:34.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052779589999999996 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216884+0000", 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6423e-05 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216901+0000", 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7033000000000001e-05 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217010+0000", 2024-04-25T15:53:34.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108792 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217136+0000", 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125943 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217349+0000", 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021362099999999999 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.240219+0000", 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022869490999999999 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.243777+0000", 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003557639 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.243864+0000", 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7899000000000007e-05 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.244118+0000", 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025404600000000001 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.948 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:958 3.20 3:062e7a7c:::100000001f7.000000ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.6438454770000002, 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045110945999999999, 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 958 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236727+0000", 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9780000000000001e-06 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242191+0000", 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054636880000000004 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242231+0000", 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9881e-05 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242250+0000", 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9837e-05 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242334+0000", 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3515000000000003e-05 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242412+0000", 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8503000000000004e-05 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242517+0000", 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010504499999999999 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.256116+0000", 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013598152000000001 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281714+0000", 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025598415999999999 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281776+0000", 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2267000000000005e-05 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281835+0000", 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8663999999999999e-05 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:959 3.33 e82/18 3:cf81bd10:::100000001f7.000000bb:head v 82'15, mlcod=82'13)", 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.6351542930000003, 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039531930999999999, 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245416+0000", 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7100000000000011e-07 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249269+0000", 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038524689999999999 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249288+0000", 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9480999999999999e-05 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249301+0000", 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2753e-05 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249366+0000", 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5593999999999996e-05 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249388+0000", 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1248000000000001e-05 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.284892+0000", 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035504292999999999 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.284947+0000", 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5121999999999998e-05 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:960 3.1e e82/18 3:796b994c:::100000001f7.000000bc:head v 82'17, mlcod=81'15)", 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.6288472550000002, 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033456649999999998, 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.958 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251724+0000", 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5510000000000001e-06 2024-04-25T15:53:34.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255631+0000", 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039072890000000004 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255650+0000", 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9082000000000001e-05 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255667+0000", 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6518e-05 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255746+0000", 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9821999999999996e-05 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255777+0000", 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0465999999999999e-05 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.285143+0000", 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029366040999999999 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.285179+0000", 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5880999999999997e-05 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1006 3.10 3:08911dbf:::100000001f7.000000ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.3644004499999998, 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051228383000000002, 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.962 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1006 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516173+0000", 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0740000000000003e-06 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524942+0000", 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087685709999999993 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.525634+0000", 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069202800000000002 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.525655+0000", 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1053e-05 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526010+0000", 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035545200000000002 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526297+0000", 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028670600000000002 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:34.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526604+0000", 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030718299999999999 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.542597+0000", 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01599283 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567212+0000", 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024615298000000001 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567317+0000", 2024-04-25T15:53:34.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105215 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567397+0000", 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9973000000000002e-05 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1007 3.36 3:6de076d9:::100000001f7.000000eb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.3556082250000001, 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052343149999999998, 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.967 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1007 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524964+0000", 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0419999999999998e-06 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531174+0000", 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006209661 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531738+0000", 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00056402199999999996 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531765+0000", 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6630000000000001e-05 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531886+0000", 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121853 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.532033+0000", 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014617699999999999 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.532238+0000", 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000205711 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.574143+0000", 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041904503000000003 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577154+0000", 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030110950000000001 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577224+0000", 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9806000000000002e-05 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577304+0000", 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0649999999999995e-05 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1008 3.2c 3:3798f9d7:::100000001f7.000000ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.3493816110000001, 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050443494999999998, 2024-04-25T15:53:34.972 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1008 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:34.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531190+0000", 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6110000000000001e-06 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537725+0000", 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006534949 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537770+0000", 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4190000000000002e-05 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537789+0000", 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9355e-05 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537894+0000", 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104838 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538018+0000", 2024-04-25T15:53:34.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124463 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538199+0000", 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180638 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577390+0000", 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039191194999999998 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581497+0000", 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041064769999999999 2024-04-25T15:53:34.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581571+0000", 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4548999999999994e-05 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581631+0000", 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0229999999999998e-05 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1009 3.3e e82/73 3:7fa01700:::100000001f7.000000ed:head v 82'15, mlcod=82'13)", 2024-04-25T15:53:34.977 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.3459160050000003, 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.043010400999999997, 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534655+0000", 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4929999999999999e-06 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540839+0000", 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061837719999999997 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540872+0000", 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2777999999999997e-05 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540893+0000", 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1599000000000001e-05 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540996+0000", 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102604 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541028+0000", 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1986999999999997e-05 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577609+0000", 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036581626999999999 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577664+0000", 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4540999999999998e-05 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1010 3.33 e82/18 3:cf8599b8:::100000001f7.000000ee:head v 82'16, mlcod=82'14)", 2024-04-25T15:53:34.981 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.3386710869999998, 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041229029, 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:34.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541900+0000", 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3459999999999999e-06 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545834+0000", 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003934585 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545851+0000", 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6923999999999999e-05 2024-04-25T15:53:34.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545867+0000", 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5715000000000001e-05 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545936+0000", 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9429000000000002e-05 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545960+0000", 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3342000000000001e-05 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.583050+0000", 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037089816999999997 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.583127+0000", 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7871e-05 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:34.985 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:34.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:34.987 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:34.986+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:34.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f09a10bb640 1 Processor -- start 2024-04-25T15:53:34.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f09a10bb640 1 -- start start 2024-04-25T15:53:35.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f09a10bb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f099c068830 0x7f099c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f09a10bb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f099c068c40 con 0x7f099c06cb80 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f09a10bb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f099c160990 con 0x7f099c15dd00 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f09a10bb640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f099c15e110 con 0x7f099c068830 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f099b59e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f099c06cb80 0x7f099c167190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36296/0 (socket says 172.21.15.67:36296) 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f099b59e640 1 -- 172.21.15.67:0/1217731312 learned_addr learned my addr 172.21.15.67:0/1217731312 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f099a59c640 1 --2- 172.21.15.67:0/1217731312 >> v2:172.21.15.67:3300/0 conn(0x7f099c068830 0x7f099c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f0999d9b640 1 -- 172.21.15.67:0/1217731312 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241706213 0 0) 0x7f0994001120 con 0x7f099c06cb80 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f0999d9b640 1 -- 172.21.15.67:0/1217731312 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0988003530 con 0x7f099c06cb80 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f099a59c640 1 -- 172.21.15.67:0/1217731312 >> v1:172.21.15.176:6789/0 conn(0x7f099c15dd00 legacy=0x7f099c1569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f099a59c640 1 -- 172.21.15.67:0/1217731312 >> v1:172.21.15.67:6789/0 conn(0x7f099c06cb80 legacy=0x7f099c167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:35.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:34.998+0000 7f099a59c640 1 -- 172.21.15.67:0/1217731312 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c160c70 con 0x7f099c068830 2024-04-25T15:53:35.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f099a59c640 1 --2- 172.21.15.67:0/1217731312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c06c640 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f09840097a0 tx=0x7f098401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cf01f543d948b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:35.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f0999d9b640 1 -- 172.21.15.67:0/1217731312 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09840b0070 con 0x7f099c068830 2024-04-25T15:53:35.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f0999d9b640 1 -- 172.21.15.67:0/1217731312 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09840ab600 con 0x7f099c068830 2024-04-25T15:53:35.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f0999d9b640 1 -- 172.21.15.67:0/1217731312 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09840ab9b0 con 0x7f099c068830 2024-04-25T15:53:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f09a10bb640 1 -- 172.21.15.67:0/1217731312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 msgr2=0x7f099c06c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1217731312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c06c640 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f09840097a0 tx=0x7f098401c480 comp rx=0 tx=0).stop 2024-04-25T15:53:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f09a10bb640 1 -- 172.21.15.67:0/1217731312 shutdown_connections 2024-04-25T15:53:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1217731312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c06c640 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f09a10bb640 1 -- 172.21.15.67:0/1217731312 >> 172.21.15.67:0/1217731312 conn(0x7f099c0ce1c0 msgr2=0x7f099c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f09a10bb640 1 -- 172.21.15.67:0/1217731312 shutdown_connections 2024-04-25T15:53:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.002+0000 7f09a10bb640 1 -- 172.21.15.67:0/1217731312 wait complete. 2024-04-25T15:53:35.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 Processor -- start 2024-04-25T15:53:35.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 -- start start 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c15be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099c06cb80 0x7f099c158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f099c15dd00 0x7f099c1597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f099c074a80 con 0x7f099c068830 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f099c074900 con 0x7f099c15dd00 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f099c06d660 con 0x7f099c06cb80 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c15be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c15be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38392/0 (socket says 172.21.15.67:38392) 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 -- 172.21.15.67:0/1706578004 learned_addr learned my addr 172.21.15.67:0/1706578004 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099c06cb80 0x7f099c158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f099c15dd00 0x7f099c1597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099c06cb80 msgr2=0x7f099c158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099c06cb80 0x7f099c158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f099c15dd00 msgr2=0x7f099c1597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f099c15dd00 0x7f099c1597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c15c3b0 con 0x7f099c068830 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099c06cb80 0x7f099c158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f099c15dd00 0x7f099c1597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c15be70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f098c00c7d0 tx=0x7f098c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c003b80 con 0x7f099c068830 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098c003d20 con 0x7f099c068830 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c00aa90 con 0x7f099c068830 2024-04-25T15:53:35.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099c157380 con 0x7f099c068830 2024-04-25T15:53:35.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f099c15b390 con 0x7f099c068830 2024-04-25T15:53:35.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.006+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f099c15baa0 con 0x7f099c068830 2024-04-25T15:53:35.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f098c00ac30 con 0x7f099c068830 2024-04-25T15:53:35.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f09937fe640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f097c06f400 0x7f097c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5944+0+0 (secure 0 0 0) 0x7f098c08fc20 con 0x7f099c068830 2024-04-25T15:53:35.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v85) ==== 54+0+16 (secure 0 0 0) 0x7f098c0900b0 con 0x7f099c068830 2024-04-25T15:53:35.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f097c06f400 0x7f097c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f099c158ed0 0x7f099c1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f099c15b180 con 0x7f099c158ed0 2024-04-25T15:53:35.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f099c158ed0 0x7f099c1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f099c158ed0 0x7f099c1554f0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:35.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f097c06f400 0x7f097c071910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f09840ad000 tx=0x7f09840a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:35.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.010+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c15b180 con 0x7f099c158ed0 2024-04-25T15:53:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.026+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c06a480 con 0x7f099c158ed0 2024-04-25T15:53:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.026+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c06a480 con 0x7f099c158ed0 2024-04-25T15:53:35.029 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.030+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f099c1f9900 0x7f099c1f9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.030+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f099c1fa250 con 0x7f099c1f9900 2024-04-25T15:53:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.030+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f099c1f9900 0x7f099c1f9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.030+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f099c1f9900 0x7f099c1f9d10 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.030+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c1fa250 con 0x7f099c1f9900 2024-04-25T15:53:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.042+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c15f0b0 con 0x7f099c1f9900 2024-04-25T15:53:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.046+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c15f0b0 con 0x7f099c1f9900 2024-04-25T15:53:35.046 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.046+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f099c146a40 0x7f099c146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.046+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f099c147390 con 0x7f099c146a40 2024-04-25T15:53:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.046+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f099c146a40 0x7f099c146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:35.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.046+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f099c146a40 0x7f099c146e50 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:35.087 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:35.086+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:35.188 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:35.186+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:35.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.186+0000 7f66c97fa640 1 -- 172.21.15.67:0/3421148041 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v86) ==== 91+0+0 (secure 0 0 0) 0x7f66c405c130 con 0x7f66cc152b00 2024-04-25T15:53:35.189 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:53:35.190 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:35.191 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:35.191 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:35.191 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66b806f430 msgr2=0x7f66b8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66b806f430 0x7f66b8071940 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000f70 tx=0x7f66bc007df0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc152b00 msgr2=0x7f66cc1f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc152b00 0x7f66cc1f6db0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f66c4007da0 tx=0x7f66c400e4b0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 shutdown_connections 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66b806f430 0x7f66b8071940 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66cc152b00 0x7f66cc1f6db0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66cc0c8680 0x7f66cc0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 --2- 172.21.15.67:0/3421148041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66cc06c640 0x7f66cc0747e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 >> 172.21.15.67:0/3421148041 conn(0x7f66cc06fe70 msgr2=0x7f66cc0c5130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 shutdown_connections 2024-04-25T15:53:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:35.194+0000 7f66d2f28640 1 -- 172.21.15.67:0/3421148041 wait complete. 2024-04-25T15:53:35.214 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:35.289 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:35.287+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:35.371 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:35.371 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:35.371 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:35.389 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:35.387+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:35.392 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:35.491 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:35.490+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:35.589 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 196.48773833999999, 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.590 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:35.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:35.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.595 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.368497859000001, 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:35.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:35.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.273863806999998, 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:35.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:35.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:35.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:597 3.e 3:72b88a04:::100000001f7.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.218085885000001, 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045014089, 2024-04-25T15:53:35.607 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 597 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368119+0000", 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.311e-06 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372689+0000", 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004569666 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372724+0000", 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5407000000000002e-05 2024-04-25T15:53:35.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372735+0000", 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1528e-05 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373941+0000", 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001205788 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374137+0000", 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195725 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374337+0000", 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020006099999999999 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412835+0000", 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038497525999999997 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413004+0000", 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016924900000000001 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413078+0000", 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4159000000000006e-05 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413131+0000", 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2669e-05 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:596 3.16 e81/18 3:6941c83d:::100000001f7.00000050:head v 81'12, mlcod=81'9)", 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:35.612 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.217499444, 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042032427999999997, 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368705+0000", 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8419999999999999e-06 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.371969+0000", 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032644509999999998 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372007+0000", 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7419000000000002e-05 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372032+0000", 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4905999999999998e-05 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372147+0000", 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115716 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372181+0000", 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3578999999999998e-05 2024-04-25T15:53:35.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410659+0000", 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038477931 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410735+0000", 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6583999999999997e-05 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:598 3.11 3:89a80ce8:::100000001f7.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.616 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.212871973, 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045571093999999999, 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 598 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373332+0000", 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.035e-06 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379271+0000", 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059388929999999998 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379309+0000", 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7287999999999999e-05 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379327+0000", 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8508e-05 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379451+0000", 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012354999999999999 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379662+0000", 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000211277 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379850+0000", 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018827000000000001 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413014+0000", 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033163661999999997 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418816+0000", 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058018840000000002 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418863+0000", 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6910000000000003e-05 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418902+0000", 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8816999999999998e-05 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.003185855, 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.622 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:53:35.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:635 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.772399334999999, 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044093436999999999, 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 635 2024-04-25T15:53:35.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813805+0000", 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.66e-06 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832121+0000", 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018315765000000001 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832272+0000", 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015122099999999999 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832299+0000", 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7013e-05 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832408+0000", 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109121 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832613+0000", 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020534899999999999 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832814+0000", 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020120399999999999 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.848197+0000", 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015382290999999999 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857706+0000", 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095096690000000001 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857804+0000", 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8047999999999994e-05 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857897+0000", 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2095999999999994e-05 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:654 3.11 3:892e96a3:::100000001f7.0000008a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.633 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.745625258, 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061060665, 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 654 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840579+0000", 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.894e-06 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849327+0000", 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087475109999999995 2024-04-25T15:53:35.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851420+0000", 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020932490000000002 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851444+0000", 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4652999999999999e-05 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851555+0000", 2024-04-25T15:53:35.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110671 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851751+0000", 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195921 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851947+0000", 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019585699999999999 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877957+0000", 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02600968 2024-04-25T15:53:35.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901479+0000", 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023522661 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901570+0000", 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1168000000000005e-05 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901638+0000", 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7399999999999998e-05 2024-04-25T15:53:35.638 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:663 3.d 3:b1139e77:::100000001f7.00000093:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.736860601, 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042145086999999998, 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.639 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 663 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849344+0000", 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854735+0000", 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053907210000000002 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857691+0000", 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002955877 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857704+0000", 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2668e-05 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857835+0000", 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131786 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858065+0000", 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022928099999999999 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858248+0000", 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183124 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891161+0000", 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032912917 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891323+0000", 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016235699999999999 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891410+0000", 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7183999999999997e-05 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891487+0000", 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6607999999999996e-05 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:664 3.d 3:b353c7f2:::100000001f7.00000094:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.644 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.731452964000001, 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079861626000000005, 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 664 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:35.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854752+0000", 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.762e-06 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865333+0000", 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581248999999999 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865758+0000", 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042434800000000001 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865779+0000", 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1209999999999999e-05 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868028+0000", 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022485339999999999 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868222+0000", 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000193986 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868340+0000", 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118163 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920976+0000", 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052636792000000002 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934415+0000", 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013438662000000001 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934516+0000", 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010058199999999999 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934611+0000", 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5338000000000001e-05 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:666 3.15 3:a92ac213:::100000001f7.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.720854489000001, 2024-04-25T15:53:35.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10217435, 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 666 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865350+0000", 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9360000000000002e-06 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871962+0000", 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006612349 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872932+0000", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097010200000000003 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872950+0000", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7193999999999998e-05 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873133+0000", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183265 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873494+0000", 2024-04-25T15:53:35.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00036074300000000001 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873747+0000", 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025347100000000001 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920781+0000", 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047033951999999997 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967388+0000", 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046606748000000003 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967458+0000", 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999994e-05 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967522+0000", 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4250000000000003e-05 2024-04-25T15:53:35.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:667 3.2d 3:b6615a40:::100000001f7.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.714224577, 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095661866999999998, 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 667 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871981+0000", 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0759999999999999e-06 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877165+0000", 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051845709999999998 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878247+0000", 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001081448 2024-04-25T15:53:35.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878262+0000", 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5196e-05 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885690+0000", 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074279159999999997 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886266+0000", 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057628900000000001 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886495+0000", 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022871600000000001 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910525+0000", 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024029452 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967541+0000", 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.057016940000000002 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967590+0000", 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8585999999999999e-05 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967640+0000", 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9676999999999999e-05 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:670 3.8 e81/68 3:1323134e:::100000001f7.0000009a:head v 81'17, mlcod=80'15)", 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.709082336, 2024-04-25T15:53:35.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090558897999999999, 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877122+0000", 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.883982+0000", 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068602339999999998 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884003+0000", 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0418999999999999e-05 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884024+0000", 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1225000000000001e-05 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884154+0000", 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977099999999999 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884218+0000", 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3830000000000004e-05 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967623+0000", 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083405807999999998 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967679+0000", 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5559000000000001e-05 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:669 3.f 3:f0675d7f:::100000001f7.00000099:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.70902673, 2024-04-25T15:53:35.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090729281999999994, 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 669 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877178+0000", 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.88e-06 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884938+0000", 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077602569999999996 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884975+0000", 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7157999999999999e-05 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884991+0000", 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5637999999999998e-05 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885075+0000", 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4696000000000004e-05 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885226+0000", 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015028799999999999 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885331+0000", 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010580400000000001 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967662+0000", 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.082330667999999996 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967792+0000", 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130259 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967861+0000", 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9002000000000001e-05 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967905+0000", 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3631999999999998e-05 2024-04-25T15:53:35.670 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:660 3.26 e81/18 3:65a7570a:::100000001f7.00000090:head v 81'20, mlcod=70'18)", 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.660409631, 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042555325999999997, 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:35.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925794+0000", 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4950000000000001e-06 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.927977+0000", 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002183192 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928011+0000", 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3037000000000002e-05 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928031+0000", 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0863999999999999e-05 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928165+0000", 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013332700000000001 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928204+0000", 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.888e-05 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968300+0000", 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040096738999999999 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968348+0000", 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7791999999999999e-05 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:732 3.15 3:aadcba4a:::100000001f7.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.287267962, 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.064849247999999998, 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.675 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 732 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298937+0000", 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8930000000000001e-06 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306019+0000", 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070820759999999996 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306065+0000", 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5235999999999999e-05 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306086+0000", 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0883e-05 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306275+0000", 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018911500000000001 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306470+0000", 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019556900000000001 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306646+0000", 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017527999999999999 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319323+0000", 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012677034 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363674+0000", 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044351773999999997 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363724+0000", 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0016999999999999e-05 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363784+0000", 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9370999999999998e-05 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.680 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:745 3.d 3:b225ffa8:::100000001f7.000000e5:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.197022402, 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047290008000000001, 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 745 2024-04-25T15:53:35.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389181+0000", 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1500000000000003e-07 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401220+0000", 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012039525000000001 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401248+0000", 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7922000000000001e-05 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401261+0000", 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2224e-05 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401480+0000", 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021909300000000001 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401643+0000", 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016358 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401799+0000", 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015555699999999999 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.412210+0000", 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010410777 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436343+0000", 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024133884000000001 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436416+0000", 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2268000000000007e-05 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436470+0000", 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4262999999999999e-05 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:761 3.3b 3:df080f3c:::100000001f7.000000f5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.078498263, 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045918457000000003, 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 761 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507707+0000", 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.717e-06 2024-04-25T15:53:35.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.524752+0000", 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017044996999999999 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525517+0000", 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076479300000000005 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525526+0000", 2024-04-25T15:53:35.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3829999999999998e-06 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525610+0000", 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3758999999999995e-05 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525797+0000", 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187302 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:35.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525954+0000", 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015707000000000001 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.536701+0000", 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01074656 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553463+0000", 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016761804000000002 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553540+0000", 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7026000000000004e-05 2024-04-25T15:53:35.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553623+0000", 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3046000000000004e-05 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:957 3.11 3:897f0f98:::100000001f7.000000b9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.231904+0000", 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 7.3542984149999997, 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046297234, 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:35.692 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 957 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.231904+0000", 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.231904+0000", 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:35.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.231905+0000", 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3769999999999999e-06 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236484+0000", 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045787220000000003 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236519+0000", 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4835000000000001e-05 2024-04-25T15:53:35.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236534+0000", 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5437000000000001e-05 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236640+0000", 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010555799999999999 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236763+0000", 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012274700000000001 2024-04-25T15:53:35.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236935+0000", 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017239499999999999 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268629+0000", 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031693741999999997 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.278006+0000", 2024-04-25T15:53:35.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00937757 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.278096+0000", 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9899000000000001e-05 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.278201+0000", 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010495199999999999 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:35.697 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:35.699 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:35.695+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:35.699 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:35.799 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:35.798+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:35.861 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:35.862 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:35.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:35.921 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:35.922+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:36.001 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:35.999+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:36.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.018+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c147390 con 0x7f099c146a40 2024-04-25T15:53:36.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.034+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c063360 con 0x7f099c146a40 2024-04-25T15:53:36.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.038+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c063360 con 0x7f099c146a40 2024-04-25T15:53:36.038 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.038+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f099c0cc200 0x7f099c0cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.038+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f099c15edd0 con 0x7f099c0cc200 2024-04-25T15:53:36.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.038+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f099c0cc200 0x7f099c0cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:36.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.038+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f099c0cc200 0x7f099c0cc610 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:36.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.038+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c15edd0 con 0x7f099c0cc200 2024-04-25T15:53:36.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.062+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c15edd0 con 0x7f099c0cc200 2024-04-25T15:53:36.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.066+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c15edd0 con 0x7f099c0cc200 2024-04-25T15:53:36.065 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:36.066 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:36.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.066+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f099c02c9e0 0x7f099c02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:36.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.066+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f099c0cd060 con 0x7f099c02c9e0 2024-04-25T15:53:36.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.066+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f099c02c9e0 0x7f099c02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:36.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.066+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f099c02c9e0 0x7f099c02cdf0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:36.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.066+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c0cd060 con 0x7f099c02c9e0 2024-04-25T15:53:36.075 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:36.075 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:36.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:36.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.094+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c0cd060 con 0x7f099c02c9e0 2024-04-25T15:53:36.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.098+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c0cd060 con 0x7f099c02c9e0 2024-04-25T15:53:36.099 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:36.100 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:36.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.098+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f099c02d840 0x7f099c20c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:36.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.098+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f099c02dcd0 con 0x7f099c02d840 2024-04-25T15:53:36.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.098+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f099c02d840 0x7f099c20c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:36.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.102+0000 7f099a59c640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f099c02d840 0x7f099c20c990 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:36.101 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:36.099+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:36.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.102+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c02dcd0 con 0x7f099c02d840 2024-04-25T15:53:36.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.122+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c0008e0 con 0x7f099c02d840 2024-04-25T15:53:36.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.126+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c0008e0 con 0x7f099c02d840 2024-04-25T15:53:36.126 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:36.127 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:36.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.126+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f099c20d3e0 0x7f099c20f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:36.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.126+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f099c212650 con 0x7f099c20d3e0 2024-04-25T15:53:36.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.126+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f099c20d3e0 0x7f099c20f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:36.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.126+0000 7f099b59e640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f099c20d3e0 0x7f099c20f800 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:36.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.130+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c212650 con 0x7f099c20d3e0 2024-04-25T15:53:36.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.146+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c210010 con 0x7f099c20d3e0 2024-04-25T15:53:36.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.150+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c210010 con 0x7f099c20d3e0 2024-04-25T15:53:36.153 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:36.153 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:36.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.154+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f099c210730 0x7f099c210b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:36.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.154+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f099c215a20 con 0x7f099c210730 2024-04-25T15:53:36.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.154+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f099c210730 0x7f099c210b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:36.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.154+0000 7f099ad9d640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f099c210730 0x7f099c210b40 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:36.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.154+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f099c215a20 con 0x7f099c210730 2024-04-25T15:53:36.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.174+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f099c2148a0 con 0x7f099c210730 2024-04-25T15:53:36.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09937fe640 1 -- 172.21.15.67:0/1706578004 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f099c2148a0 con 0x7f099c210730 2024-04-25T15:53:36.180 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:36.181 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:36.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f099c158ed0 msgr2=0x7f099c1554f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f099c158ed0 0x7f099c1554f0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f099c1f9900 msgr2=0x7f099c1f9d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f099c1f9900 0x7f099c1f9d10 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f099c146a40 msgr2=0x7f099c146e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f099c146a40 0x7f099c146e50 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f099c0cc200 msgr2=0x7f099c0cc610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f099c0cc200 0x7f099c0cc610 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f099b59e640 1 -- 172.21.15.67:0/1706578004 reap_dead start 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f099c02c9e0 msgr2=0x7f099c02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f099c02c9e0 0x7f099c02cdf0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f099b59e640 1 -- 172.21.15.67:0/1706578004 reap_dead start 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f099c02d840 msgr2=0x7f099c20c990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f099c02d840 0x7f099c20c990 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f099c20d3e0 msgr2=0x7f099c20f800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f099c20d3e0 0x7f099c20f800 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f099c210730 msgr2=0x7f099c210b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f099c210730 0x7f099c210b40 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f097c06f400 msgr2=0x7f097c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f097c06f400 0x7f097c071910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f09840ad000 tx=0x7f09840a6000 comp rx=0 tx=0).stop 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 msgr2=0x7f099c15be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f099b59e640 1 -- 172.21.15.67:0/1706578004 reap_dead start 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c15be70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f098c00c7d0 tx=0x7f098c00cc90 comp rx=0 tx=0).stop 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 shutdown_connections 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 --2- 172.21.15.67:0/1706578004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099c068830 0x7f099c15be70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.178+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 >> 172.21.15.67:0/1706578004 conn(0x7f099c0ce1c0 msgr2=0x7f099c154de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.182+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 shutdown_connections 2024-04-25T15:53:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:36.182+0000 7f09a10bb640 1 -- 172.21.15.67:0/1706578004 wait complete. 2024-04-25T15:53:36.202 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:36.199+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:36.241 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:36.241 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:36.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:36.303 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:36.299+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:36.403 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:36.402+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:36.452 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:36.453 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:36.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:36.513 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:36.514+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:36.605 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:36.603+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:36.665 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:36.665 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:36.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:36.706 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:36.703+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:36.806 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:36.803+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:36.837 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:36.837 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:36.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:36.907 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:36.906+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:37.009 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:37.010+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:37.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:37.113 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:37.114+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:37.210 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:37.210+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:37.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:53:37.224 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:37.224 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:37.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:37.333 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:37.334+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.370+0000 7fb88a131640 1 Processor -- start 2024-04-25T15:53:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.370+0000 7fb88a131640 1 -- start start 2024-04-25T15:53:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb88a131640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb884153a00 0x7fb8840c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb88a131640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb88415a0b0 0x7fb8840746a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb88a131640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8840c3bc0 con 0x7fb8840725c0 2024-04-25T15:53:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb88a131640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb884153e10 con 0x7fb88415a0b0 2024-04-25T15:53:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb88a131640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb884068c40 con 0x7fb884153a00 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb8837fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb884153a00 0x7fb8840c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb8837fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb884153a00 0x7fb8840c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:60642/0 (socket says 172.21.15.67:60642) 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb8837fe640 1 -- 172.21.15.67:0/3564849609 learned_addr learned my addr 172.21.15.67:0/3564849609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb883fff640 1 --2- 172.21.15.67:0/3564849609 >> v2:172.21.15.176:3300/0 conn(0x7fb88415a0b0 0x7fb8840746a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb8837fe640 1 -- 172.21.15.67:0/3564849609 >> v2:172.21.15.176:3300/0 conn(0x7fb88415a0b0 msgr2=0x7fb8840746a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb8837fe640 1 --2- 172.21.15.67:0/3564849609 >> v2:172.21.15.176:3300/0 conn(0x7fb88415a0b0 0x7fb8840746a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb8837fe640 1 -- 172.21.15.67:0/3564849609 >> v1:172.21.15.176:6789/0 conn(0x7fb8840725c0 legacy=0x7fb884073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.374+0000 7fb8837fe640 1 -- 172.21.15.67:0/3564849609 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8840c53a0 con 0x7fb884153a00 2024-04-25T15:53:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb8837fe640 1 --2- 172.21.15.67:0/3564849609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb884153a00 0x7fb8840c4010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb878009870 tx=0x7fb87801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38cabc1028d78bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb8827fc640 1 -- 172.21.15.67:0/3564849609 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8780b0070 con 0x7fb884153a00 2024-04-25T15:53:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb8827fc640 1 -- 172.21.15.67:0/3564849609 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8780ab600 con 0x7fb884153a00 2024-04-25T15:53:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb8827fc640 1 -- 172.21.15.67:0/3564849609 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8780ab9d0 con 0x7fb884153a00 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 -- 172.21.15.67:0/3564849609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb884153a00 msgr2=0x7fb8840c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 --2- 172.21.15.67:0/3564849609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb884153a00 0x7fb8840c4010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb878009870 tx=0x7fb87801c4d0 comp rx=0 tx=0).stop 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 -- 172.21.15.67:0/3564849609 shutdown_connections 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 --2- 172.21.15.67:0/3564849609 >> v2:172.21.15.176:3300/0 conn(0x7fb88415a0b0 0x7fb8840746a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 --2- 172.21.15.67:0/3564849609 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb884153a00 0x7fb8840c4010 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 -- 172.21.15.67:0/3564849609 >> 172.21.15.67:0/3564849609 conn(0x7fb8840ce1c0 msgr2=0x7fb8840ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 -- 172.21.15.67:0/3564849609 shutdown_connections 2024-04-25T15:53:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 -- 172.21.15.67:0/3564849609 wait complete. 2024-04-25T15:53:37.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 Processor -- start 2024-04-25T15:53:37.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.378+0000 7fb88a131640 1 -- start start 2024-04-25T15:53:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8840725c0 0x7fb884162d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:37.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb884153a00 0x7fb8841f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb88415a0b0 0x7fb88406c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb884074be0 con 0x7fb88415a0b0 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb8840686c0 con 0x7fb884153a00 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb8840729d0 con 0x7fb8840725c0 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb88415a0b0 0x7fb88406c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb88415a0b0 0x7fb88406c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38394/0 (socket says 172.21.15.67:38394) 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 -- 172.21.15.67:0/634792024 learned_addr learned my addr 172.21.15.67:0/634792024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb8837fe640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8840725c0 0x7fb884162d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb882ffd640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb884153a00 0x7fb8841f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 -- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8840725c0 msgr2=0x7fb884162d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8840725c0 0x7fb884162d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 -- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb884153a00 msgr2=0x7fb8841f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb884153a00 0x7fb8841f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 -- 172.21.15.67:0/634792024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8841f70c0 con 0x7fb88415a0b0 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb8837fe640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8840725c0 0x7fb884162d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:37.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb883fff640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb88415a0b0 0x7fb88406c810 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009cb0 tx=0x7fb87c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb86bfff640 1 -- 172.21.15.67:0/634792024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c018020 con 0x7fb88415a0b0 2024-04-25T15:53:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb86bfff640 1 -- 172.21.15.67:0/634792024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c014480 con 0x7fb88415a0b0 2024-04-25T15:53:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8840c5d10 con 0x7fb88415a0b0 2024-04-25T15:53:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb86bfff640 1 -- 172.21.15.67:0/634792024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c014870 con 0x7fb88415a0b0 2024-04-25T15:53:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb884162520 con 0x7fb88415a0b0 2024-04-25T15:53:37.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.382+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb88406a480 con 0x7fb88415a0b0 2024-04-25T15:53:37.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.386+0000 7fb86bfff640 1 -- 172.21.15.67:0/634792024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb87c01b020 con 0x7fb88415a0b0 2024-04-25T15:53:37.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.386+0000 7fb86bfff640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb87006f430 0x7fb870071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:37.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.386+0000 7fb86bfff640 1 -- 172.21.15.67:0/634792024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5944+0+0 (secure 0 0 0) 0x7fb87c090560 con 0x7fb88415a0b0 2024-04-25T15:53:37.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.386+0000 7fb8837fe640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb87006f430 0x7fb870071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:37.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.386+0000 7fb8837fe640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb87006f430 0x7fb870071940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb8780ad000 tx=0x7fb8780a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:37.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.394+0000 7fb86bfff640 1 -- 172.21.15.67:0/634792024 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb87c022030 con 0x7fb88415a0b0 2024-04-25T15:53:37.412 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:37.411+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:37.452 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:37.452 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:37.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:37.513 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:37.511+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:37.566+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fb8840c5aa0 con 0x7fb88415a0b0 2024-04-25T15:53:37.614 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:37.611+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:37.657 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:37.658 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:37.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:37.715 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:37.715+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:37.815 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:37.815+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:37.828 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:37.828 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:37.844 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:37.917 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:37.914+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:38.000 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:38.000 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:38.017 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:38.018+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:38.018 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:38.119 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:38.115+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:38.176 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:38.176 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:38.194 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:38.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:38.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.210+0000 7fb86bfff640 1 -- 172.21.15.67:0/634792024 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v87) ==== 101+0+0 (secure 0 0 0) 0x7fb87c010040 con 0x7fb88415a0b0 2024-04-25T15:53:38.210 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb87006f430 msgr2=0x7fb870071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb87006f430 0x7fb870071940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb8780ad000 tx=0x7fb8780a6000 comp rx=0 tx=0).stop 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb88415a0b0 msgr2=0x7fb88406c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb88415a0b0 0x7fb88406c810 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009cb0 tx=0x7fb87c00c490 comp rx=0 tx=0).stop 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 shutdown_connections 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb87006f430 0x7fb870071940 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb88415a0b0 0x7fb88406c810 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb884153a00 0x7fb8841f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 --2- 172.21.15.67:0/634792024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8840725c0 0x7fb884162d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 >> 172.21.15.67:0/634792024 conn(0x7fb8840ce1c0 msgr2=0x7fb8840c9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:38.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 shutdown_connections 2024-04-25T15:53:38.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.214+0000 7fb88a131640 1 -- 172.21.15.67:0/634792024 wait complete. 2024-04-25T15:53:38.235 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:38.235+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:38.320 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:38.318+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:38.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f40700f4640 1 Processor -- start 2024-04-25T15:53:38.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f40700f4640 1 -- start start 2024-04-25T15:53:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f40700f4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f406815dd00 0x7f4068068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f40700f4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4068160990 con 0x7f4068156db0 2024-04-25T15:53:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f40700f4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4068160c70 con 0x7f40681569a0 2024-04-25T15:53:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f40700f4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f406815e110 con 0x7f406815dd00 2024-04-25T15:53:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f406e691640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4068156db0 0x7f4068167190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36306/0 (socket says 172.21.15.67:36306) 2024-04-25T15:53:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.362+0000 7f406e691640 1 -- 172.21.15.67:0/2114485960 learned_addr learned my addr 172.21.15.67:0/2114485960 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:38.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406de90640 1 --2- 172.21.15.67:0/2114485960 >> v2:172.21.15.176:3300/0 conn(0x7f406815dd00 0x7f4068068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781847869 0 0) 0x7f4068160990 con 0x7f4068156db0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4050003530 con 0x7f4068156db0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3361276823 0 0) 0x7f4050003530 con 0x7f4068156db0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 >> v2:172.21.15.176:3300/0 conn(0x7f406815dd00 msgr2=0x7f4068068830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 --2- 172.21.15.67:0/2114485960 >> v2:172.21.15.176:3300/0 conn(0x7f406815dd00 0x7f4068068830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 >> v1:172.21.15.176:6789/0 conn(0x7f40681569a0 legacy=0x7f406806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40680c3bf0 con 0x7f4068156db0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f4060001680 con 0x7f4068156db0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 >> v1:172.21.15.67:6789/0 conn(0x7f4068156db0 legacy=0x7f4068167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500039c0 0x7f4050003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4050004810 0x7f4050004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500051d0 0x7f405000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f406815e110 con 0x7f40500051d0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f405000c170 con 0x7f40500039c0 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f405000c370 con 0x7f4050004810 2024-04-25T15:53:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500051d0 0x7f405000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406de90640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500039c0 0x7f4050003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406d68f640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4050004810 0x7f4050004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 -- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4050004810 msgr2=0x7f4050004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4050004810 0x7f4050004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 -- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500039c0 msgr2=0x7f4050003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500039c0 0x7f4050003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 -- 172.21.15.67:0/2114485960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4050003530 con 0x7f40500051d0 2024-04-25T15:53:38.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406de90640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500039c0 0x7f4050003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:38.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 -- 172.21.15.67:0/2114485960 reap_dead start 2024-04-25T15:53:38.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406e691640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500051d0 0x7f405000b9d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f406000c190 tx=0x7f406001cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73199aa55f8b22a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060007d00 con 0x7f40500051d0 2024-04-25T15:53:38.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40600a9070 con 0x7f40500051d0 2024-04-25T15:53:38.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.366+0000 7f406ce8e640 1 -- 172.21.15.67:0/2114485960 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4060005d40 con 0x7f40500051d0 2024-04-25T15:53:38.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 -- 172.21.15.67:0/2114485960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500051d0 msgr2=0x7f405000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500051d0 0x7f405000b9d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f406000c190 tx=0x7f406001cb80 comp rx=0 tx=0).stop 2024-04-25T15:53:38.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 -- 172.21.15.67:0/2114485960 shutdown_connections 2024-04-25T15:53:38.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 --2- 172.21.15.67:0/2114485960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40500051d0 0x7f405000b9d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 -- 172.21.15.67:0/2114485960 >> 172.21.15.67:0/2114485960 conn(0x7f40680ce1c0 msgr2=0x7f40680ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:38.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 -- 172.21.15.67:0/2114485960 shutdown_connections 2024-04-25T15:53:38.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 -- 172.21.15.67:0/2114485960 wait complete. 2024-04-25T15:53:38.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 Processor -- start 2024-04-25T15:53:38.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.370+0000 7f40700f4640 1 -- start start 2024-04-25T15:53:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500051d0 0x7f40680cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40681e7100 0x7f40681e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f406814a9b0 0x7f406814adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40680c4240 con 0x7f40681e7100 2024-04-25T15:53:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4068160810 con 0x7f40500051d0 2024-04-25T15:53:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f406806d660 con 0x7f406814a9b0 2024-04-25T15:53:38.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40681e7100 0x7f40681e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40681e7100 0x7f40681e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38420/0 (socket says 172.21.15.67:38420) 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 -- 172.21.15.67:0/3899525737 learned_addr learned my addr 172.21.15.67:0/3899525737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500051d0 0x7f40680cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406e691640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f406814a9b0 0x7f406814adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f406814a9b0 msgr2=0x7f406814adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f406814a9b0 0x7f406814adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500051d0 msgr2=0x7f40680cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40500051d0 0x7f40680cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406814b300 con 0x7f40681e7100 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406d68f640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40681e7100 0x7f40681e5e30 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f405c00e440 tx=0x7f405c00e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c019070 con 0x7f40681e7100 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c0052e0 con 0x7f40681e7100 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40681e6940 con 0x7f40681e7100 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c003650 con 0x7f40681e7100 2024-04-25T15:53:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40680ccf30 con 0x7f40681e7100 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f406806a480 con 0x7f40681e7100 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f405c01c070 con 0x7f40681e7100 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40667fc640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5944+0+0 (secure 0 0 0) 0x7f405c08e770 con 0x7f40681e7100 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v87) ==== 54+0+16 (secure 0 0 0) 0x7f405c094170 con 0x7f40681e7100 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40681f5ac0 0x7f40681f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40680cc710 con 0x7f40681f5ac0 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.374+0000 7f406e691640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40681f5ac0 0x7f40681f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.378+0000 7f406e691640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40681f5ac0 0x7f40681f5ed0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.378+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4058007440 tx=0x7f405800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.378+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4060001440 con 0x7f40681f5ac0 2024-04-25T15:53:38.386 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:38.386 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:38.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.390+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f40680c3330 con 0x7f40681f5ac0 2024-04-25T15:53:38.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.394+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f40680c3330 con 0x7f40681f5ac0 2024-04-25T15:53:38.395 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.394+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4068063090 0x7f40680634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.394+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f40681f6920 con 0x7f4068063090 2024-04-25T15:53:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.394+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4068063090 0x7f40680634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.394+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4068063090 0x7f40680634a0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.394+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40681f6920 con 0x7f4068063090 2024-04-25T15:53:38.404 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:38.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.410+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406814d250 con 0x7f4068063090 2024-04-25T15:53:38.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.410+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f406814d250 con 0x7f4068063090 2024-04-25T15:53:38.411 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.410+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4068154070 0x7f4068154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.410+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f40680b41c0 con 0x7f4068154070 2024-04-25T15:53:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.410+0000 7f406d68f640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4068154070 0x7f4068154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.410+0000 7f406d68f640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4068154070 0x7f4068154480 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.414+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40680b41c0 con 0x7f4068154070 2024-04-25T15:53:38.421 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:38.422+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:38.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.426+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f40680b41c0 con 0x7f4068154070 2024-04-25T15:53:38.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.426+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f40680b41c0 con 0x7f4068154070 2024-04-25T15:53:38.428 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.426+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f406802c9e0 0x7f406802cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.426+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f40680b41c0 con 0x7f406802c9e0 2024-04-25T15:53:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.426+0000 7f406e691640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f406802c9e0 0x7f406802cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.430+0000 7f406e691640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f406802c9e0 0x7f406802cdf0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.430+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40680b41c0 con 0x7f406802c9e0 2024-04-25T15:53:38.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.442+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f40680b41c0 con 0x7f406802c9e0 2024-04-25T15:53:38.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.446+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f40680b41c0 con 0x7f406802c9e0 2024-04-25T15:53:38.445 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:38.445 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:38.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.446+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f406802d840 0x7f4068206e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.446+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f40681e6370 con 0x7f406802d840 2024-04-25T15:53:38.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.446+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f406802d840 0x7f4068206e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.446+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f406802d840 0x7f4068206e40 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.446+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40681e6370 con 0x7f406802d840 2024-04-25T15:53:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.458+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f40681e6370 con 0x7f406802d840 2024-04-25T15:53:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.462+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f40681e6370 con 0x7f406802d840 2024-04-25T15:53:38.463 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.462+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f406802dc50 0x7f40682098a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.462+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f40681e75c0 con 0x7f406802dc50 2024-04-25T15:53:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.462+0000 7f406d68f640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f406802dc50 0x7f40682098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.462+0000 7f406d68f640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f406802dc50 0x7f40682098a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.466+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40681e75c0 con 0x7f406802dc50 2024-04-25T15:53:38.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.478+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f40681e7510 con 0x7f406802dc50 2024-04-25T15:53:38.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.482+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f40681e7510 con 0x7f406802dc50 2024-04-25T15:53:38.482 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:38.483 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:38.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.482+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f406820a2f0 0x7f406820c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.482+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f406820f680 con 0x7f406820a2f0 2024-04-25T15:53:38.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.482+0000 7f406e691640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f406820a2f0 0x7f406820c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.482+0000 7f406e691640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f406820a2f0 0x7f406820c710 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.486+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f406820f680 con 0x7f406820a2f0 2024-04-25T15:53:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.498+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406820f650 con 0x7f406820a2f0 2024-04-25T15:53:38.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.502+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f406820f650 con 0x7f406820a2f0 2024-04-25T15:53:38.504 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:38.505 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:38.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.506+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f406820e8d0 0x7f406820ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:38.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.506+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4068212a50 con 0x7f406820e8d0 2024-04-25T15:53:38.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.506+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f406820e8d0 0x7f406820ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.506+0000 7f406de90640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f406820e8d0 0x7f406820ece0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:38.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.506+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4068212a50 con 0x7f406820e8d0 2024-04-25T15:53:38.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.522+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f406820cc80 con 0x7f406820e8d0 2024-04-25T15:53:38.522 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:38.522+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40667fc640 1 -- 172.21.15.67:0/3899525737 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f406820cc80 con 0x7f406820e8d0 2024-04-25T15:53:38.526 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40681f5ac0 msgr2=0x7f40681f5ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f40681f5ac0 0x7f40681f5ed0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4068063090 msgr2=0x7f40680634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4068063090 0x7f40680634a0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4068154070 msgr2=0x7f4068154480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4068154070 0x7f4068154480 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f406802c9e0 msgr2=0x7f406802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f406802c9e0 0x7f406802cdf0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f406802d840 msgr2=0x7f4068206e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f406802d840 0x7f4068206e40 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f406802dc50 msgr2=0x7f40682098a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f406802dc50 0x7f40682098a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f406820a2f0 msgr2=0x7f406820c710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f406820a2f0 0x7f406820c710 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f406820e8d0 msgr2=0x7f406820ece0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f406820e8d0 0x7f406820ece0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f406e691640 1 -- 172.21.15.67:0/3899525737 reap_dead start 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f406e691640 1 -- 172.21.15.67:0/3899525737 reap_dead start 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f406e691640 1 -- 172.21.15.67:0/3899525737 reap_dead start 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f406e691640 1 -- 172.21.15.67:0/3899525737 reap_dead start 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f406e691640 1 -- 172.21.15.67:0/3899525737 reap_dead start 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 msgr2=0x7f4054071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4058007440 tx=0x7f405800b040 comp rx=0 tx=0).stop 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40681e7100 msgr2=0x7f40681e5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40681e7100 0x7f40681e5e30 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f405c00e440 tx=0x7f405c00e900 comp rx=0 tx=0).stop 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 shutdown_connections 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f405406f400 0x7f4054071910 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 --2- 172.21.15.67:0/3899525737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40681e7100 0x7f40681e5e30 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:38.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 >> 172.21.15.67:0/3899525737 conn(0x7f40680ce1c0 msgr2=0x7f406815eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:38.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 shutdown_connections 2024-04-25T15:53:38.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:38.526+0000 7f40700f4640 1 -- 172.21.15.67:0/3899525737 wait complete. 2024-04-25T15:53:38.560 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:38.560 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:38.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:38.560 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:38.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:597 3.e e81/76 3:72b88a04:::100000001f7.00000051:head v 81'6, mlcod=69'3)", 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.182865934000001, 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038266363999999997, 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374533+0000", 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5060000000000001e-06 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376869+0000", 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023357830000000001 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376914+0000", 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4317000000000003e-05 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376941+0000", 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6965999999999999e-05 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377065+0000", 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012408300000000001 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377098+0000", 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2943e-05 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412723+0000", 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035625742000000002 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412798+0000", 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5024000000000005e-05 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:599 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.178576704999999, 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039983414000000002, 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 599 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378825+0000", 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7189999999999999e-06 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384734+0000", 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059089859999999998 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384776+0000", 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2166999999999998e-05 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384795+0000", 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9307999999999999e-05 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384894+0000", 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9112999999999998e-05 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385064+0000", 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169524 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385212+0000", 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147849 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409088+0000", 2024-04-25T15:53:38.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023875850000000001 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418680+0000", 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095925639999999996 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418742+0000", 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1379999999999998e-05 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418805+0000", 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2953999999999994e-05 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.569 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:598 3.11 e81/76 3:89a80ce8:::100000001f7.00000052:head v 81'21, mlcod=80'19)", 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.176037382000001, 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037275751000000003, 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:38.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381362+0000", 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.782e-06 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382748+0000", 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001385488 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382769+0000", 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0854000000000001e-05 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382786+0000", 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7807000000000001e-05 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382876+0000", 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9821999999999995e-05 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382901+0000", 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4845999999999999e-05 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418526+0000", 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035624426000000001 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418636+0000", 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110726 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.573 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:606 3.31 3:8d161915:::100000001f7.0000005a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.127465408999999, 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031627436000000002, 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 606 2024-04-25T15:53:38.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429932+0000", 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.429935+0000", 2024-04-25T15:53:38.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9610000000000001e-06 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435340+0000", 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054048719999999998 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435375+0000", 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4558999999999999e-05 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435394+0000", 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8819000000000001e-05 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435493+0000", 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8927999999999999e-05 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.435806+0000", 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031338500000000001 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.436016+0000", 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000209586 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.459476+0000", 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023460547000000002 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461366+0000", 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001890206 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461433+0000", 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6206999999999998e-05 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.461560+0000", 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012736600000000001 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:654 3.11 e81/76 3:892e96a3:::100000001f7.0000008a:head v 81'22, mlcod=80'20)", 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.700637257, 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044455910000000001, 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856762+0000", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.669e-06 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859651+0000", 2024-04-25T15:53:38.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002889015 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859675+0000", 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3346999999999999e-05 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859696+0000", 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1021000000000001e-05 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859816+0000", 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120809 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859853+0000", 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6242999999999998e-05 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901127+0000", 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041274245000000001 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901217+0000", 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9561000000000005e-05 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.582 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:668 3.1 3:83ba1de1:::100000001f7.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.697265631, 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062400662000000003, 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 668 2024-04-25T15:53:38.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860140+0000", 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1559999999999995e-06 2024-04-25T15:53:38.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869131+0000", 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089909859999999994 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869188+0000", 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6539000000000002e-05 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869212+0000", 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4501000000000001e-05 2024-04-25T15:53:38.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869417+0000", 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020473999999999999 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869612+0000", 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000194668 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871230+0000", 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016182029999999999 2024-04-25T15:53:38.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.917654+0000", 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046423786000000002 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922361+0000", 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00470704 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922442+0000", 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.153e-05 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922533+0000", 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0513000000000006e-05 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:669 3.f e81/76 3:f0675d7f:::100000001f7.00000099:head v 81'10, mlcod=70'8)", 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.671229113000001, 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080177950999999997, 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.588 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886171+0000", 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8470000000000001e-06 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891613+0000", 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054420349999999996 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891657+0000", 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4155e-05 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891689+0000", 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1822999999999997e-05 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891773+0000", 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4602000000000001e-05 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891806+0000", 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2842999999999997e-05 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966243+0000", 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074437173999999995 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966347+0000", 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103472 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:647 3.a e81/76 3:524e89bd:::100000001f7.00000083:head v 81'3, mlcod=61'1)", 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.652794491, 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072551145999999997, 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904605+0000", 2024-04-25T15:53:38.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.421e-06 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906854+0000", 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022493249999999999 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906880+0000", 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5374000000000001e-05 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906897+0000", 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7419999999999999e-05 2024-04-25T15:53:38.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906994+0000", 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6633999999999997e-05 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.907027+0000", 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3336000000000001e-05 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977067+0000", 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.070040033000000002 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977155+0000", 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7602999999999995e-05 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:674 3.9 3:90d768d7:::100000001f7.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.645127560000001, 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065083915000000006, 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 674 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912273+0000", 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6929999999999999e-06 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921147+0000", 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088735250000000002 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921185+0000", 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8098e-05 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921200+0000", 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5543999999999999e-05 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921281+0000", 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0758000000000002e-05 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921426+0000", 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014529000000000001 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:38.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921540+0000", 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113465 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.942585+0000", 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021045735999999999 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977215+0000", 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034629659 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977282+0000", 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6796000000000002e-05 2024-04-25T15:53:38.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977354+0000", 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2350999999999999e-05 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:675 3.a e81/76 3:5122fcae:::100000001f7.0000009f:head v 81'4, mlcod=61'1)", 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.627567414, 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047589770000000003, 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.601 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929833+0000", 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2850000000000001e-06 2024-04-25T15:53:38.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931776+0000", 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001943443 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931806+0000", 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0074e-05 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931829+0000", 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3027000000000002e-05 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931927+0000", 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7540000000000005e-05 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931959+0000", 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2265999999999999e-05 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977374+0000", 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045414643999999997 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977420+0000", 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6490999999999999e-05 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:676 3.18 e81/76 3:191fe66d:::100000001f7.000000a0:head v 81'8, mlcod=71'6)", 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.625722846, 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050752358999999997, 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931678+0000", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.542e-06 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935891+0000", 2024-04-25T15:53:38.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042133930000000002 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935917+0000", 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5624000000000001e-05 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935935+0000", 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8258999999999999e-05 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936019+0000", 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4538999999999998e-05 2024-04-25T15:53:38.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936060+0000", 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1060000000000003e-05 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982347+0000", 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046286542999999999 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982427+0000", 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0399000000000001e-05 2024-04-25T15:53:38.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:678 3.1b 3:da348c2f:::100000001f7.000000a2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.614354777999999, 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039665855999999999, 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 678 2024-04-25T15:53:38.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943045+0000", 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9630000000000001e-06 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947273+0000", 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042283470000000004 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947307+0000", 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3794000000000002e-05 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947322+0000", 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5184999999999999e-05 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947403+0000", 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1076999999999996e-05 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947533+0000", 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012934399999999999 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:38.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947646+0000", 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113163 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.964158+0000", 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016512526 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982578+0000", 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018419333 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982646+0000", 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8010999999999996e-05 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982709+0000", 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:679 3.24 e81/76 3:27dcc3f5:::100000001f7.000000a3:head v 81'12, mlcod=81'10)", 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.604809212999999, 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030189264, 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.614 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952589+0000", 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.952590+0000", 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.381e-06 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956607+0000", 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040170140000000002 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956623+0000", 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6025e-05 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956636+0000", 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2595000000000001e-05 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956708+0000", 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2402000000000002e-05 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956741+0000", 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3386999999999997e-05 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982717+0000", 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025975446999999999 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982778+0000", 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1013e-05 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:720 3.1a 3:59a21752:::100000001f7.000000cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.332565474999999, 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031267504000000002, 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.618 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 720 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224832+0000", 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.224835+0000", 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.661e-06 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230311+0000", 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054762220000000002 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230341+0000", 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9972999999999999e-05 2024-04-25T15:53:38.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230357+0000", 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.558e-05 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230440+0000", 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3030999999999998e-05 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230610+0000", 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016995500000000001 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.230782+0000", 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000172079 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.247511+0000", 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016729067 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255803+0000", 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082918499999999999 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.256001+0000", 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019800300000000001 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.256100+0000", 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9083e-05 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:723 3.9 3:9221849e:::100000001f7.000000cf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.314332949000001, 2024-04-25T15:53:38.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031588667000000001, 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 723 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243065+0000", 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.243070+0000", 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0200000000000002e-06 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248284+0000", 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052135619999999997 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248318+0000", 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4072000000000002e-05 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248337+0000", 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9867000000000001e-05 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248421+0000", 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3541000000000005e-05 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248550+0000", 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128768 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.248782+0000", 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023251200000000001 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.265337+0000", 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016554484000000001 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274315+0000", 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089778259999999995 2024-04-25T15:53:38.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274497+0000", 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018229699999999999 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.274654+0000", 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000156718 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:762 3.3 3:c0da3165:::100000001f7.000000f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:38.628 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.042353289999999, 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033743224000000002, 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 762 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515045+0000", 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.515046+0000", 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.578e-06 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523433+0000", 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083871290000000001 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523458+0000", 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4732e-05 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523469+0000", 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0882e-05 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523588+0000", 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119361 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523741+0000", 2024-04-25T15:53:38.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152311 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.523908+0000", 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167472 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.535424+0000", 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011516337 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548681+0000", 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013256274 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548736+0000", 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5306999999999997e-05 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.548788+0000", 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1841000000000001e-05 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:957 3.11 e82/76 3:897f0f98:::100000001f7.000000b9:head v 82'38, mlcod=82'36)", 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:38.633 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.319576723999999, 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030634000000000002, 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237821+0000", 2024-04-25T15:53:38.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.237823+0000", 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9429999999999999e-06 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239142+0000", 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013188130000000001 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239168+0000", 2024-04-25T15:53:38.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6132e-05 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239183+0000", 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5068999999999999e-05 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239267+0000", 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4227000000000005e-05 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.239290+0000", 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2209999999999999e-05 2024-04-25T15:53:38.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268394+0000", 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029104512999999999 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268455+0000", 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1093000000000006e-05 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1004 3.1d e82/76 3:b93a1278:::100000001f7.000000e8:head v 82'29, mlcod=82'27)", 2024-04-25T15:53:38.637 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.033414991000001, 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044544315000000001, 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523985+0000", 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9999999999999999e-06 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527244+0000", 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032587330000000002 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527284+0000", 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0031999999999999e-05 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527309+0000", 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527437+0000", 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128384 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527480+0000", 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2357999999999997e-05 2024-04-25T15:53:38.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568424+0000", 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040944332 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568527+0000", 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103319 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1008 3.2c e82/76 3:3798f9d7:::100000001f7.000000ec:head v 82'14, mlcod=82'12)", 2024-04-25T15:53:38.641 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.019000134000001, 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042823282999999997, 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538400+0000", 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0650000000000001e-06 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541298+0000", 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028986440000000001 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541324+0000", 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5176000000000001e-05 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541339+0000", 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5821999999999999e-05 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541469+0000", 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541521+0000", 2024-04-25T15:53:38.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1882999999999998e-05 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581098+0000", 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039576885999999999 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581221+0000", 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012303699999999999 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.645 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1012 3.1b 3:db4e25ba:::100000001f7.000000f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.008161178, 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032333898999999999, 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1012 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:38.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549237+0000", 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.549239+0000", 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7700000000000002e-06 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555878+0000", 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066383470000000002 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555906+0000", 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8147e-05 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.555923+0000", 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7385999999999999e-05 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556004+0000", 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0836999999999999e-05 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556110+0000", 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105487 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.556220+0000", 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110053 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568508+0000", 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012288566000000001 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581445+0000", 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012937059000000001 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581507+0000", 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1703000000000002e-05 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:38.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581571+0000", 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3544e-05 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:38.651 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:38.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:38.653 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:38.651+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:38.754 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:38.751+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:38.849 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:38.849 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:38.855 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:38.851+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:38.866 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:38.955 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:38.955+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:39.055 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:39.055 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:39.056 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:39.056 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:39.055+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:39.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:39.157 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:39.158+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:39.213 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:39.213 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:39.214 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-25T15:53:39.214 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:53:39.214 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:53:39.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-25T15:53:39.261 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:39.262+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:39.265 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 36.044656093, 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:53:39.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:594 3.16 3:6a58e86c:::100000001f7.0000004e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.916478646, 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034501508, 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.272 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 594 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347204+0000", 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.347207+0000", 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8660000000000002e-06 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353453+0000", 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062464160000000003 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353489+0000", 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5283000000000001e-05 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353501+0000", 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2860999999999999e-05 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353629+0000", 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000127632 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353808+0000", 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017904800000000001 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.353996+0000", 2024-04-25T15:53:39.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018766099999999999 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.380705+0000", 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026708789 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381525+0000", 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00082074100000000005 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381619+0000", 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3153999999999996e-05 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381706+0000", 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7057000000000004e-05 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:596 3.16 3:6941c83d:::100000001f7.00000050:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:39.277 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.901764163999999, 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049183123000000002, 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 596 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:39.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361922+0000", 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367484+0000", 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055618409999999997 2024-04-25T15:53:39.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367522+0000", 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8260000000000003e-05 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367543+0000", 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0795999999999999e-05 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367646+0000", 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102913 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367857+0000", 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021190400000000001 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368091+0000", 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023323699999999999 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.405516+0000", 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037425424999999998 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410955+0000", 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054389859999999998 2024-04-25T15:53:39.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411034+0000", 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8421e-05 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411102+0000", 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8137000000000002e-05 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:600 3.5 3:a2751c53:::100000001f7.00000054:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.282 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.878280658, 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035430837, 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 600 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385402+0000", 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385405+0000", 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8959999999999999e-06 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390062+0000", 2024-04-25T15:53:39.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046572949999999997 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390104+0000", 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1943000000000001e-05 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390123+0000", 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9147000000000002e-05 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390225+0000", 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101983 2024-04-25T15:53:39.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390407+0000", 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000181819 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.390593+0000", 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018578900000000001 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409139+0000", 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018546360000000001 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420728+0000", 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011589083 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420783+0000", 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5025000000000003e-05 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.420833+0000", 2024-04-25T15:53:39.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9496999999999997e-05 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:649 3.3c 3:3e99caf7:::100000001f7.00000085:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.528940094999999, 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031048148000000001, 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 649 2024-04-25T15:53:39.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734743+0000", 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.734744+0000", 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.694e-06 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740013+0000", 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052687469999999998 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740078+0000", 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5402999999999997e-05 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740096+0000", 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7393000000000001e-05 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740206+0000", 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109789 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740427+0000", 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022180100000000001 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.740653+0000", 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022509799999999999 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.748988+0000", 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083355619999999995 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765614+0000", 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016625788999999998 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765712+0000", 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8183999999999994e-05 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.765791+0000", 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8688000000000003e-05 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:665 3.1d 3:b87c769f:::100000001f7.00000095:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.422659124000001, 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045554887000000002, 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.293 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 665 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841029+0000", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9359999999999998e-06 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851995+0000", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010966115 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852062+0000", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6975000000000002e-05 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852084+0000", 2024-04-25T15:53:39.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2382999999999999e-05 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852218+0000", 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013397699999999999 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852418+0000", 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000200161 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852550+0000", 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013199300000000001 2024-04-25T15:53:39.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878731+0000", 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026181157 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886405+0000", 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076735090000000002 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886503+0000", 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8504000000000004e-05 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886578+0000", 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5177000000000002e-05 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:663 3.d e81/18 3:b1139e77:::100000001f7.00000093:head v 81'7, mlcod=80'5)", 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.405221249, 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032554081999999998, 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.298 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858461+0000", 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858463+0000", 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8169999999999999e-06 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862335+0000", 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003871462 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862369+0000", 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4635999999999998e-05 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862395+0000", 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5698e-05 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862753+0000", 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035778999999999997 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.862879+0000", 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012587500000000001 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.890905+0000", 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028025912 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891016+0000", 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011089199999999999 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:664 3.d e81/18 3:b353c7f2:::100000001f7.00000094:head v 81'8, mlcod=80'5)", 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.394360224, 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051411299000000001, 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869325+0000", 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1689999999999999e-06 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872141+0000", 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028160759999999998 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872176+0000", 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4996999999999997e-05 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872202+0000", 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5944000000000001e-05 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872312+0000", 2024-04-25T15:53:39.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010998200000000001 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872351+0000", 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9360999999999998e-05 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920645+0000", 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048293989000000002 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920734+0000", 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8781000000000003e-05 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.305 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:671 3.38 e81/18 3:1c0639bf:::100000001f7.0000009b:head v 81'8, mlcod=81'6)", 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.377036016, 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069441491999999994, 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886668+0000", 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0876000000000001e-05 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893242+0000", 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065744849999999997 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893267+0000", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5321e-05 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893283+0000", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5454000000000001e-05 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893362+0000", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8770000000000006e-05 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893382+0000", 2024-04-25T15:53:39.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0585999999999999e-05 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.955934+0000", 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062552041000000003 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956088+0000", 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.309 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:673 3.28 3:153e0bf4:::100000001f7.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.358746727, 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063190614000000006, 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 673 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:39.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904939+0000", 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8739999999999999e-06 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911908+0000", 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069695449999999997 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911944+0000", 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5973000000000003e-05 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911960+0000", 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5319999999999999e-05 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912057+0000", 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7427000000000001e-05 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912221+0000", 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016433600000000001 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912375+0000", 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015359299999999999 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.941806+0000", 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029430523 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967997+0000", 2024-04-25T15:53:39.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026191589000000001 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968063+0000", 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6321000000000004e-05 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968127+0000", 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:652 3.36 e81/18 3:6c98fff9:::100000001f7.00000088:head v 81'12, mlcod=69'10)", 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.347549966000001, 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052156028, 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:39.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916135+0000", 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7850000000000001e-06 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918034+0000", 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018996099999999999 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918080+0000", 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5469000000000001e-05 2024-04-25T15:53:39.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918098+0000", 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8726999999999999e-05 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918176+0000", 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8125000000000002e-05 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918198+0000", 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1531000000000001e-05 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968240+0000", 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050041583000000001 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968289+0000", 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9197999999999999e-05 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:677 3.5 3:a07f2395:::100000001f7.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.318 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.328168243, 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041485695000000003, 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 677 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935657+0000", 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014239100000000001 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940003+0000", 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043458239999999999 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940055+0000", 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2021999999999998e-05 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940071+0000", 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6528000000000002e-05 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940148+0000", 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7179000000000001e-05 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940294+0000", 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014605799999999999 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940418+0000", 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012356499999999999 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956756+0000", 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016338149999999999 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976852+0000", 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020095572999999999 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976925+0000", 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3351999999999998e-05 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977000+0000", 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5053000000000001e-05 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.323 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:719 3.1d 3:b9e367d5:::100000001f7.000000cb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.044793168, 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037012611000000001, 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 719 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218890+0000", 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.218892+0000", 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.182e-06 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226010+0000", 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0071184320000000001 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226050+0000", 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0167999999999999e-05 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226070+0000", 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9256999999999999e-05 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226209+0000", 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000139226 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226360+0000", 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015169700000000001 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.226574+0000", 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021339199999999999 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.253457+0000", 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026883383 2024-04-25T15:53:39.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255765+0000", 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023077409999999999 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255837+0000", 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1987e-05 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.255902+0000", 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5146000000000003e-05 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.328 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:729 3.5 3:a1e9b24d:::100000001f7.000000d5:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.977556079999999, 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033306524999999997, 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 729 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.329 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286127+0000", 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.286128+0000", 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2300000000000001e-06 2024-04-25T15:53:39.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292385+0000", 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006257589 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292423+0000", 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7756999999999998e-05 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292441+0000", 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7954000000000001e-05 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292521+0000", 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0171000000000002e-05 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292651+0000", 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130025 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.292787+0000", 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013599099999999999 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.302313+0000", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095255389999999995 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319295+0000", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016982391999999999 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319373+0000", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7571000000000007e-05 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319433+0000", 2024-04-25T15:53:39.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0306000000000002e-05 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:745 3.d e81/18 3:b225ffa8:::100000001f7.000000e5:head v 81'10, mlcod=81'8)", 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.861627836, 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034113420999999998, 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402055+0000", 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.402056+0000", 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3459999999999999e-06 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405220+0000", 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031635439999999999 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405240+0000", 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0695000000000001e-05 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405253+0000", 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2897e-05 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405323+0000", 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9719000000000001e-05 2024-04-25T15:53:39.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.405377+0000", 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3653000000000003e-05 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436064+0000", 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030687076000000001 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436168+0000", 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104491 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.337 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:748 3.1d 3:b93a1278:::100000001f7.000000e8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.855754456, 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033655512999999998, 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.338 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 748 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407928+0000", 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.407929+0000", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9899999999999999e-07 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419777+0000", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011847952 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419801+0000", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4049999999999998e-05 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419810+0000", 2024-04-25T15:53:39.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3039999999999994e-06 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.419874+0000", 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3467999999999996e-05 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.420010+0000", 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013569600000000001 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.420161+0000", 2024-04-25T15:53:39.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015180900000000001 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.438066+0000", 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017904571000000001 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441466+0000", 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033998959999999999 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441529+0000", 2024-04-25T15:53:39.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3124000000000001e-05 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441584+0000", 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4744000000000003e-05 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1002 3.1f 3:fb90ae0e:::100000001f7.000000e6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.764983374, 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035318779000000002, 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.343 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1002 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498699+0000", 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.498702+0000", 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2330000000000001e-06 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507556+0000", 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088549340000000001 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508591+0000", 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010341459999999999 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508610+0000", 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9490000000000001e-05 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508716+0000", 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105468 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.508836+0000", 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120627 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.509012+0000", 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017549199999999999 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521191+0000", 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012179339000000001 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533885+0000", 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01269438 2024-04-25T15:53:39.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533944+0000", 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8971000000000002e-05 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534018+0000", 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3699e-05 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1003 3.28 3:1719c8f9:::100000001f7.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.756115909, 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062620618000000003, 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1003 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507569+0000", 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2110000000000001e-06 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519064+0000", 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011495182 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521838+0000", 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027741129999999999 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521856+0000", 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521967+0000", 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110913 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522100+0000", 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013309899999999999 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522235+0000", 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013518199999999999 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.543023+0000", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020788025000000002 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570076+0000", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027052208000000001 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570138+0000", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2376e-05 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570187+0000", 2024-04-25T15:53:39.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9379000000000003e-05 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1004 3.1d 3:b93a1278:::100000001f7.000000e8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.744579455, 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049745623000000003, 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.354 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1004 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519106+0000", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.464e-06 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523205+0000", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040996339999999996 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523267+0000", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1660000000000003e-05 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523292+0000", 2024-04-25T15:53:39.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5194999999999998e-05 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523428+0000", 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013594 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523595+0000", 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016733300000000001 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523819+0000", 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022358999999999999 2024-04-25T15:53:39.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.566513+0000", 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042694339999999997 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568719+0000", 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022057230000000001 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568799+0000", 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0370000000000005e-05 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568849+0000", 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9373999999999999e-05 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1007 3.36 e82/18 3:6de076d9:::100000001f7.000000eb:head v 82'23, mlcod=82'21)", 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.729812339, 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040044785999999999, 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.359 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533870+0000", 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.533871+0000", 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3799999999999996e-07 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535899+0000", 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020273069999999999 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535926+0000", 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7803000000000001e-05 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.535945+0000", 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8555000000000001e-05 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.536054+0000", 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109441 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.536089+0000", 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4411e-05 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.573843+0000", 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037753789000000003 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.573915+0000", 2024-04-25T15:53:39.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2541999999999998e-05 2024-04-25T15:53:39.363 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.363 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.363 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.363 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.363 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.363 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:39.364 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:39.365 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:39.362+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:39.375 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "" 2024-04-25T15:53:39.375 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:39.465 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:39.466+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:39.516 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:39.516 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:39.517 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:39.517 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:39.534 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:39.583 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:39.583+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:39.667 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:39.663+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:39.684 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:39.684 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:39.684 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:39.684 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:39.684 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:39.702 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:39.767 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:39.766+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:39.868 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:39.866+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:39.894 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:39.894 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:39.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:39.894 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:39.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.894 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.231640304999999, 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.26243838600000002, 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:53:39.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920373+0000", 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039448372000000002 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921649+0000", 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012757210000000001 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921724+0000", 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5549000000000005e-05 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921802+0000", 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8150000000000002e-05 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.899 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:640 3.10 3:092a6a2c:::100000001f7.0000007c:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.012609024, 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065130602999999995, 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 640 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878395+0000", 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878397+0000", 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.198e-06 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894140+0000", 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015742477000000001 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894240+0000", 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010061299999999999 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894258+0000", 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7779999999999999e-05 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894341+0000", 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3168000000000001e-05 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894526+0000", 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018456399999999999 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894632+0000", 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010651500000000001 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.923409+0000", 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028776380000000001 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943396+0000", 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019987393999999999 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943459+0000", 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2389000000000002e-05 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943526+0000", 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7125000000000005e-05 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:88 2.14 2:2e2fa760:::200.00000001:head [write 97242~3394 [fadvise_dontneed] in=3394b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.999577146, 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052104437000000003, 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891427+0000", 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891429+0000", 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8190000000000001e-06 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891440+0000", 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1029999999999999e-05 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891499+0000", 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8961e-05 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891530+0000", 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1049e-05 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891630+0000", 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010053899999999999 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891787+0000", 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015704200000000001 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891996+0000", 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020865400000000001 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920805+0000", 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028808585000000001 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943394+0000", 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022589115 2024-04-25T15:53:39.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943465+0000", 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1254000000000006e-05 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943531+0000", 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:647 3.a 3:524e89bd:::100000001f7.00000083:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:39.909 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.996854347999999, 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083393058000000006, 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 647 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894152+0000", 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.968e-06 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902694+0000", 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085420719999999995 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903482+0000", 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078797999999999997 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903499+0000", 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7082e-05 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903593+0000", 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4452999999999998e-05 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903774+0000", 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180803 2024-04-25T15:53:39.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903953+0000", 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179106 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.965763+0000", 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061810057000000002 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977390+0000", 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01162697 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977474+0000", 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3812000000000003e-05 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977543+0000", 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8755000000000002e-05 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:652 3.36 3:6c98fff9:::100000001f7.00000088:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.988299043, 2024-04-25T15:53:39.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065992254, 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 652 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902705+0000", 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902707+0000", 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5209999999999999e-06 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910035+0000", 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073280000000000003 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910344+0000", 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030890699999999998 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910417+0000", 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3900999999999996e-05 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910547+0000", 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012940400000000001 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910876+0000", 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000329181 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911062+0000", 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018587000000000001 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966069+0000", 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.055007038000000001 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968579+0000", 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002509598 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968649+0000", 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0189000000000002e-05 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968697+0000", 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8644999999999999e-05 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:660 3.26 3:65a7570a:::100000001f7.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.980955664, 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058617787999999997, 2024-04-25T15:53:39.919 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 660 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910049+0000", 2024-04-25T15:53:39.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910051+0000", 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7599999999999998e-06 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925001+0000", 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014949405000000001 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925210+0000", 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020949799999999999 2024-04-25T15:53:39.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925223+0000", 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3086e-05 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925337+0000", 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011333899999999999 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925525+0000", 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018789300000000001 2024-04-25T15:53:39.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925638+0000", 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011354 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967535+0000", 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041897074999999999 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968553+0000", 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010179480000000001 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968609+0000", 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5653999999999999e-05 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968666+0000", 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7590000000000003e-05 2024-04-25T15:53:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:675 3.a 3:5122fcae:::100000001f7.0000009f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.965993816999999, 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052715010999999999, 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 675 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925010+0000", 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925013+0000", 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3089999999999998e-06 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929333+0000", 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043198170000000001 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929365+0000", 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2323000000000002e-05 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929380+0000", 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5421999999999999e-05 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929460+0000", 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0148000000000006e-05 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929572+0000", 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011171 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929688+0000", 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011547 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966801+0000", 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037113896 2024-04-25T15:53:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977633+0000", 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010831294999999999 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977681+0000", 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7926000000000002e-05 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977725+0000", 2024-04-25T15:53:39.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4694999999999998e-05 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:750 3.10 3:08911dbf:::100000001f7.000000ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.468977613, 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029714553000000001, 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.931 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 750 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422027+0000", 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.422031+0000", 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5050000000000001e-06 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.427886+0000", 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058545810000000002 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433892+0000", 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060058660000000003 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433903+0000", 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1769e-05 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433975+0000", 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1872000000000006e-05 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.434165+0000", 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018962300000000001 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.434345+0000", 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180034 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.441345+0000", 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070004730000000001 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451615+0000", 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010269276000000001 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451679+0000", 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4369000000000007e-05 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.451741+0000", 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2185000000000002e-05 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:751 3.36 3:6de076d9:::100000001f7.000000eb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:39.936 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.462403147, 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035860229, 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 751 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428601+0000", 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.428603+0000", 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5820000000000001e-06 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.433862+0000", 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052591870000000002 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442450+0000", 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085886669999999995 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442460+0000", 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3160000000000003e-06 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442541+0000", 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1217000000000005e-05 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442706+0000", 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016477000000000001 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442865+0000", 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015895899999999999 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.460454+0000", 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017589370999999999 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464395+0000", 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039411139999999999 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464433+0000", 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7710999999999999e-05 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.464461+0000", 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8334999999999999e-05 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.941 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:752 3.2c 3:3798f9d7:::100000001f7.000000ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.454673241, 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071715859000000007, 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.942 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 752 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436331+0000", 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436332+0000", 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.429e-06 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.442426+0000", 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060933589999999996 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486527+0000", 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044101711000000002 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486535+0000", 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1749999999999999e-06 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.486862+0000", 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032728400000000001 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.487028+0000", 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016650399999999999 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.487190+0000", 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016182699999999999 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.494699+0000", 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007509033 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507938+0000", 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013238689 2024-04-25T15:53:39.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507980+0000", 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2268999999999998e-05 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.508047+0000", 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6579000000000001e-05 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:354 3.10 3:0ba0dc78:::10000000002.000000fd:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.291687515, 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56691576200000005, 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599318+0000", 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599326+0000", 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9459999999999998e-06 2024-04-25T15:53:39.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161759+0000", 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56243287799999997 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161783+0000", 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4057e-05 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161899+0000", 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011615799999999999 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162007+0000", 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107823 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162110+0000", 2024-04-25T15:53:39.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010235500000000001 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.163904+0000", 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017941470000000001 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166138+0000", 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022346190000000002 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166186+0000", 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7902999999999999e-05 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166232+0000", 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6220999999999997e-05 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.953 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:953 3.26 3:67e57578:::100000001f7.000000b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.679448509, 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032562766999999999, 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.954 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 953 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211556+0000", 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.211570+0000", 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3920999999999999e-05 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216848+0000", 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052779589999999996 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216884+0000", 2024-04-25T15:53:39.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6423e-05 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.216901+0000", 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7033000000000001e-05 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217010+0000", 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108792 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217136+0000", 2024-04-25T15:53:39.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125943 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.217349+0000", 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021362099999999999 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.240219+0000", 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022869490999999999 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.243777+0000", 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003557639 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.243864+0000", 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7899000000000007e-05 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.244118+0000", 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025404600000000001 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.959 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:958 3.20 3:062e7a7c:::100000001f7.000000ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.654280185999999, 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045110945999999999, 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 958 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.960 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236724+0000", 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236727+0000", 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9780000000000001e-06 2024-04-25T15:53:39.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242191+0000", 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054636880000000004 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242231+0000", 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9881e-05 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242250+0000", 2024-04-25T15:53:39.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9837e-05 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242334+0000", 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3515000000000003e-05 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242412+0000", 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8503000000000004e-05 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.242517+0000", 2024-04-25T15:53:39.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010504499999999999 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.256116+0000", 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013598152000000001 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281714+0000", 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025598415999999999 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281776+0000", 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2267000000000005e-05 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.281835+0000", 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8663999999999999e-05 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:959 3.33 e82/18 3:cf81bd10:::100000001f7.000000bb:head v 82'15, mlcod=82'13)", 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.645589001999999, 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039531930999999999, 2024-04-25T15:53:39.965 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245415+0000", 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.245416+0000", 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7100000000000011e-07 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249269+0000", 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038524689999999999 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249288+0000", 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9480999999999999e-05 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249301+0000", 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2753e-05 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249366+0000", 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5593999999999996e-05 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.249388+0000", 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1248000000000001e-05 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.284892+0000", 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035504292999999999 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.284947+0000", 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5121999999999998e-05 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:960 3.1e e82/18 3:796b994c:::100000001f7.000000bc:head v 82'17, mlcod=81'15)", 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.639281964, 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033456649999999998, 2024-04-25T15:53:39.969 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251722+0000", 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.251724+0000", 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5510000000000001e-06 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255631+0000", 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039072890000000004 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255650+0000", 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9082000000000001e-05 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255667+0000", 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6518e-05 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255746+0000", 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9821999999999996e-05 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.255777+0000", 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0465999999999999e-05 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.285143+0000", 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029366040999999999 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.285179+0000", 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5880999999999997e-05 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1006 3.10 3:08911dbf:::100000001f7.000000ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.374835159, 2024-04-25T15:53:39.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051228383000000002, 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1006 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516169+0000", 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.516173+0000", 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0740000000000003e-06 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524942+0000", 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087685709999999993 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.525634+0000", 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00069202800000000002 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.525655+0000", 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1053e-05 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526010+0000", 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035545200000000002 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526297+0000", 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028670600000000002 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.526604+0000", 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030718299999999999 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.542597+0000", 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01599283 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567212+0000", 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024615298000000001 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567317+0000", 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105215 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.567397+0000", 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9973000000000002e-05 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1007 3.36 3:6de076d9:::100000001f7.000000eb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.366042933999999, 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052343149999999998, 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1007 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.979 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524961+0000", 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.524964+0000", 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0419999999999998e-06 2024-04-25T15:53:39.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531174+0000", 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006209661 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531738+0000", 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00056402199999999996 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531765+0000", 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6630000000000001e-05 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.981 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531886+0000", 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121853 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.532033+0000", 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014617699999999999 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.532238+0000", 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000205711 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.574143+0000", 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041904503000000003 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577154+0000", 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030110950000000001 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577224+0000", 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9806000000000002e-05 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577304+0000", 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0649999999999995e-05 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1008 3.2c 3:3798f9d7:::100000001f7.000000ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.35981632, 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050443494999999998, 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.984 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1008 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531188+0000", 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.531190+0000", 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6110000000000001e-06 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537725+0000", 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006534949 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537770+0000", 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4190000000000002e-05 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537789+0000", 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9355e-05 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.537894+0000", 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104838 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538018+0000", 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124463 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538199+0000", 2024-04-25T15:53:39.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180638 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577390+0000", 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039191194999999998 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581497+0000", 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041064769999999999 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581571+0000", 2024-04-25T15:53:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4548999999999994e-05 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581631+0000", 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0229999999999998e-05 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1009 3.3e e82/73 3:7fa01700:::100000001f7.000000ed:head v 82'15, mlcod=82'13)", 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.356350714, 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.043010400999999997, 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.989 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534653+0000", 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.534655+0000", 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4929999999999999e-06 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540839+0000", 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061837719999999997 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540872+0000", 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2777999999999997e-05 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540893+0000", 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1599000000000001e-05 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.540996+0000", 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102604 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541028+0000", 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1986999999999997e-05 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577609+0000", 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036581626999999999 2024-04-25T15:53:39.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.577664+0000", 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4540999999999998e-05 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1010 3.33 e82/18 3:cf8599b8:::100000001f7.000000ee:head v 82'16, mlcod=82'14)", 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 11.349105796, 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041229029, 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:39.993 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541898+0000", 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541900+0000", 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3459999999999999e-06 2024-04-25T15:53:39.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545834+0000", 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003934585 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545851+0000", 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6923999999999999e-05 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545867+0000", 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5715000000000001e-05 2024-04-25T15:53:39.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545936+0000", 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9429000000000002e-05 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.545960+0000", 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3342000000000001e-05 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.583050+0000", 2024-04-25T15:53:39.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037089816999999997 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.583127+0000", 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7871e-05 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:39.997 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:39.998 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:39.999 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:39.998+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:40.100 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:40.099+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:40.156 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:40.157 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:40.157 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:40.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:40.215 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:40.215+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:40.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:53:40.301 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:40.299+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:40.313 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:40.313 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:40.313 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:40.313 INFO:tasks.workunit.client.0.smithi067.stdout:iozone test complete. 2024-04-25T15:53:40.315 INFO:tasks.workunit.client.0.smithi067.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-25T15:53:40.364 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:40.364 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:40.365 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:40.365 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:40.365 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:40.382 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:40.392 INFO:tasks.workunit.client.0.smithi067.stdout: Iozone: Performance Test of File I/O 2024-04-25T15:53:40.392 INFO:tasks.workunit.client.0.smithi067.stdout: Version $Revision: 3.489 $ 2024-04-25T15:53:40.392 INFO:tasks.workunit.client.0.smithi067.stdout: Compiled for 64 bit mode. 2024-04-25T15:53:40.392 INFO:tasks.workunit.client.0.smithi067.stdout: Build: linux-AMD64 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Run began: Thu Apr 25 15:53:40 2024 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Include close in write timing 2024-04-25T15:53:40.393 INFO:tasks.workunit.client.0.smithi067.stdout: Include fsync in write timing 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: File size set to 10485760 kB 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Record Size 1024 kB 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Output is in kBytes/sec 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Time Resolution = 0.000001 seconds. 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache size set to 1024 kBytes. 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Processor cache line size set to 32 bytes. 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: File stride size set to 17 * record size. 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Throughput test with 1 process 2024-04-25T15:53:40.394 INFO:tasks.workunit.client.0.smithi067.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-04-25T15:53:40.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f641b5f6640 1 Processor -- start 2024-04-25T15:53:40.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f641b5f6640 1 -- start start 2024-04-25T15:53:40.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f641b5f6640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6414068830 0x7f641406c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f641b5f6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f641406cb80 0x7f6414167190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f641b5f6640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6414068c40 con 0x7f641415dd00 2024-04-25T15:53:40.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f641b5f6640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f641415e110 con 0x7f6414068830 2024-04-25T15:53:40.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f641b5f6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6414160990 con 0x7f641406cb80 2024-04-25T15:53:40.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6418b91640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6414068830 0x7f641406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6419392640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f641415dd00 0x7f64141569a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36310/0 (socket says 172.21.15.67:36310) 2024-04-25T15:53:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6419392640 1 -- 172.21.15.67:0/2815460335 learned_addr learned my addr 172.21.15.67:0/2815460335 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6419b93640 1 --2- 172.21.15.67:0/2815460335 >> v2:172.21.15.176:3300/0 conn(0x7f641406cb80 0x7f6414167190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6418b91640 1 -- 172.21.15.67:0/2815460335 >> v2:172.21.15.176:3300/0 conn(0x7f641406cb80 msgr2=0x7f6414167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6418b91640 1 --2- 172.21.15.67:0/2815460335 >> v2:172.21.15.176:3300/0 conn(0x7f641406cb80 0x7f6414167190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6418b91640 1 -- 172.21.15.67:0/2815460335 >> v1:172.21.15.67:6789/0 conn(0x7f641415dd00 legacy=0x7f64141569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.406+0000 7f6418b91640 1 -- 172.21.15.67:0/2815460335 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6414160c70 con 0x7f6414068830 2024-04-25T15:53:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f6418b91640 1 --2- 172.21.15.67:0/2815460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6414068830 0x7f641406c640 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f64140cf1e0 tx=0x7f640401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594b3c744fb0795e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f640bfff640 1 -- 172.21.15.67:0/2815460335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040b0070 con 0x7f6414068830 2024-04-25T15:53:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f640bfff640 1 -- 172.21.15.67:0/2815460335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64040ab600 con 0x7f6414068830 2024-04-25T15:53:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f640bfff640 1 -- 172.21.15.67:0/2815460335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040ab9d0 con 0x7f6414068830 2024-04-25T15:53:40.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 -- 172.21.15.67:0/2815460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6414068830 msgr2=0x7f641406c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 --2- 172.21.15.67:0/2815460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6414068830 0x7f641406c640 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f64140cf1e0 tx=0x7f640401c460 comp rx=0 tx=0).stop 2024-04-25T15:53:40.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 -- 172.21.15.67:0/2815460335 shutdown_connections 2024-04-25T15:53:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 --2- 172.21.15.67:0/2815460335 >> v2:172.21.15.176:3300/0 conn(0x7f641406cb80 0x7f6414167190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 --2- 172.21.15.67:0/2815460335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6414068830 0x7f641406c640 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 -- 172.21.15.67:0/2815460335 >> 172.21.15.67:0/2815460335 conn(0x7f64140ce1c0 msgr2=0x7f64140ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 -- 172.21.15.67:0/2815460335 shutdown_connections 2024-04-25T15:53:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.410+0000 7f641b5f6640 1 -- 172.21.15.67:0/2815460335 wait complete. 2024-04-25T15:53:40.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 Processor -- start 2024-04-25T15:53:40.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 -- start start 2024-04-25T15:53:40.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6414068830 0x7f64140c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f641406cb80 0x7f641415cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641415dd00 0x7f641415d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6414074a80 con 0x7f641406cb80 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6414074900 con 0x7f641415dd00 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f641406d660 con 0x7f6414068830 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f641406cb80 0x7f641415cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f641406cb80 0x7f641415cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38428/0 (socket says 172.21.15.67:38428) 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 -- 172.21.15.67:0/3996576337 learned_addr learned my addr 172.21.15.67:0/3996576337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6419b93640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641415dd00 0x7f641415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6419392640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6414068830 0x7f64140c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 -- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6414068830 msgr2=0x7f64140c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6414068830 0x7f64140c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 -- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641415dd00 msgr2=0x7f641415d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641415dd00 0x7f641415d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 -- 172.21.15.67:0/3996576337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64140c6800 con 0x7f641406cb80 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6419b93640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641415dd00 0x7f641415d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6418b91640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f641406cb80 0x7f641415cdf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6404019070 tx=0x7f6404019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6409ffb640 1 -- 172.21.15.67:0/3996576337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040b0040 con 0x7f641406cb80 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6409ffb640 1 -- 172.21.15.67:0/3996576337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64040b9070 con 0x7f641406cb80 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641415b0f0 con 0x7f641406cb80 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f6409ffb640 1 -- 172.21.15.67:0/3996576337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040b4650 con 0x7f641406cb80 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64140c7d60 con 0x7f641406cb80 2024-04-25T15:53:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.414+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641406a480 con 0x7f641406cb80 2024-04-25T15:53:40.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.418+0000 7f6409ffb640 1 -- 172.21.15.67:0/3996576337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f64040b47f0 con 0x7f641406cb80 2024-04-25T15:53:40.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.418+0000 7f6409ffb640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f63f806f430 0x7f63f8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.418+0000 7f6409ffb640 1 -- 172.21.15.67:0/3996576337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5944+0+0 (secure 0 0 0) 0x7f6404129a40 con 0x7f641406cb80 2024-04-25T15:53:40.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.418+0000 7f6419392640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f63f806f430 0x7f63f8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.418+0000 7f6419392640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f63f806f430 0x7f63f8071940 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f640c0012f0 tx=0x7f640c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.422+0000 7f6409ffb640 1 -- 172.21.15.67:0/3996576337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6404016050 con 0x7f641406cb80 2024-04-25T15:53:40.427 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:40.427+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:40.503 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:40.499+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:40.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:40.603 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:40.602+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:40.622 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 201.52117809200001, 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.623 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:40.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:40.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:40.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:40.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.401937611000001, 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:40.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.307303558999998, 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.635 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:40.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:40.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:40.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:40.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:597 3.e 3:72b88a04:::100000001f7.00000051:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 17.251525637, 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045014089, 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.640 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 597 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368117+0000", 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368119+0000", 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.311e-06 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372689+0000", 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004569666 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372724+0000", 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5407000000000002e-05 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372735+0000", 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1528e-05 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373941+0000", 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001205788 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374137+0000", 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195725 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374337+0000", 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020006099999999999 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412835+0000", 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038497525999999997 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413004+0000", 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016924900000000001 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413078+0000", 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4159000000000006e-05 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413131+0000", 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2669e-05 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.645 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:596 3.16 e81/18 3:6941c83d:::100000001f7.00000050:head v 81'12, mlcod=81'9)", 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 17.250939196000001, 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042032427999999997, 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:40.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368703+0000", 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368705+0000", 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8419999999999999e-06 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.371969+0000", 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032644509999999998 2024-04-25T15:53:40.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372007+0000", 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7419000000000002e-05 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372032+0000", 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4905999999999998e-05 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372147+0000", 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115716 2024-04-25T15:53:40.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.372181+0000", 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3578999999999998e-05 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410659+0000", 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038477931 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410735+0000", 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6583999999999997e-05 2024-04-25T15:53:40.649 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:598 3.11 3:89a80ce8:::100000001f7.00000052:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 17.246311725000002, 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045571093999999999, 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 598 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373330+0000", 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.373332+0000", 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.035e-06 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379271+0000", 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059388929999999998 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379309+0000", 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7287999999999999e-05 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379327+0000", 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8508e-05 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379451+0000", 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012354999999999999 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379662+0000", 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000211277 2024-04-25T15:53:40.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.379850+0000", 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018827000000000001 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.413014+0000", 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033163661999999997 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418816+0000", 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058018840000000002 2024-04-25T15:53:40.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418863+0000", 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6910000000000003e-05 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418902+0000", 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8816999999999998e-05 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 17.036625607000001, 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:53:40.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:53:40.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:635 3.e 3:70183cd7:::100000001f7.00000077:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.805839086999999, 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044093436999999999, 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.661 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 635 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813803+0000", 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.813805+0000", 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.66e-06 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832121+0000", 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018315765000000001 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832272+0000", 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015122099999999999 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832299+0000", 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7013e-05 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832408+0000", 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109121 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832613+0000", 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020534899999999999 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.832814+0000", 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020120399999999999 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.848197+0000", 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015382290999999999 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857706+0000", 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095096690000000001 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857804+0000", 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8047999999999994e-05 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857897+0000", 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2095999999999994e-05 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.666 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:654 3.11 3:892e96a3:::100000001f7.0000008a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.77906501, 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061060665, 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.667 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 654 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840577+0000", 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.840579+0000", 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.894e-06 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849327+0000", 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087475109999999995 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851420+0000", 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020932490000000002 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851444+0000", 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4652999999999999e-05 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851555+0000", 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110671 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851751+0000", 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195921 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851947+0000", 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019585699999999999 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877957+0000", 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.02600968 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901479+0000", 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023522661 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901570+0000", 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1168000000000005e-05 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901638+0000", 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7399999999999998e-05 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:663 3.d 3:b1139e77:::100000001f7.00000093:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:40.672 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.770300353, 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042145086999999998, 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 663 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849342+0000", 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.849344+0000", 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5639999999999999e-06 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854735+0000", 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0053907210000000002 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857691+0000", 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002955877 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857704+0000", 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2668e-05 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.857835+0000", 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131786 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858065+0000", 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022928099999999999 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.858248+0000", 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183124 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891161+0000", 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032912917 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891323+0000", 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016235699999999999 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891410+0000", 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7183999999999997e-05 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891487+0000", 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6607999999999996e-05 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:664 3.d 3:b353c7f2:::100000001f7.00000094:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.764892715999999, 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079861626000000005, 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 664 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.678 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854749+0000", 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.854752+0000", 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.762e-06 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865333+0000", 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010581248999999999 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865758+0000", 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042434800000000001 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865779+0000", 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1209999999999999e-05 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868028+0000", 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022485339999999999 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868222+0000", 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000193986 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.868340+0000", 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118163 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920976+0000", 2024-04-25T15:53:40.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052636792000000002 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934415+0000", 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013438662000000001 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934516+0000", 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010058199999999999 2024-04-25T15:53:40.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.934611+0000", 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5338000000000001e-05 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:666 3.15 3:a92ac213:::100000001f7.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.754294241, 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10217435, 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.683 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 666 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865350+0000", 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9360000000000002e-06 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871962+0000", 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006612349 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872932+0000", 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097010200000000003 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872950+0000", 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7193999999999998e-05 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873133+0000", 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183265 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873494+0000", 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00036074300000000001 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873747+0000", 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025347100000000001 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920781+0000", 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047033951999999997 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967388+0000", 2024-04-25T15:53:40.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046606748000000003 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967458+0000", 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999994e-05 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967522+0000", 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4250000000000003e-05 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.688 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:667 3.2d 3:b6615a40:::100000001f7.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.747664328999999, 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095661866999999998, 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 667 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871981+0000", 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0759999999999999e-06 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877165+0000", 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051845709999999998 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878247+0000", 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001081448 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878262+0000", 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5196e-05 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885690+0000", 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074279159999999997 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886266+0000", 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057628900000000001 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:40.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886495+0000", 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022871600000000001 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910525+0000", 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024029452 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967541+0000", 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.057016940000000002 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967590+0000", 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8585999999999999e-05 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967640+0000", 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9676999999999999e-05 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:670 3.8 e81/68 3:1323134e:::100000001f7.0000009a:head v 81'17, mlcod=80'15)", 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.742522088000001, 2024-04-25T15:53:40.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090558897999999999, 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877122+0000", 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.883982+0000", 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068602339999999998 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884003+0000", 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0418999999999999e-05 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884024+0000", 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1225000000000001e-05 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884154+0000", 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977099999999999 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884218+0000", 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3830000000000004e-05 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967623+0000", 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083405807999999998 2024-04-25T15:53:40.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967679+0000", 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5559000000000001e-05 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:669 3.f 3:f0675d7f:::100000001f7.00000099:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.742466482000001, 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090729281999999994, 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.698 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 669 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877178+0000", 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.88e-06 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884938+0000", 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077602569999999996 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884975+0000", 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7157999999999999e-05 2024-04-25T15:53:40.700 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884991+0000", 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5637999999999998e-05 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885075+0000", 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4696000000000004e-05 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885226+0000", 2024-04-25T15:53:40.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015028799999999999 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885331+0000", 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010580400000000001 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967662+0000", 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.082330667999999996 2024-04-25T15:53:40.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967792+0000", 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130259 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967861+0000", 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9002000000000001e-05 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967905+0000", 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3631999999999998e-05 2024-04-25T15:53:40.703 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:660 3.26 e81/18 3:65a7570a:::100000001f7.00000090:head v 81'20, mlcod=70'18)", 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.693849383, 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042555325999999997, 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:40.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925793+0000", 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.925794+0000", 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4950000000000001e-06 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.927977+0000", 2024-04-25T15:53:40.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002183192 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928011+0000", 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3037000000000002e-05 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928031+0000", 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0863999999999999e-05 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928165+0000", 2024-04-25T15:53:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013332700000000001 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.928204+0000", 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.888e-05 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968300+0000", 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040096738999999999 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968348+0000", 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7791999999999999e-05 2024-04-25T15:53:40.707 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:732 3.15 3:aadcba4a:::100000001f7.000000d8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.320707714000001, 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.064849247999999998, 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 732 2024-04-25T15:53:40.708 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298934+0000", 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.298937+0000", 2024-04-25T15:53:40.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8930000000000001e-06 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306019+0000", 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070820759999999996 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306065+0000", 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5235999999999999e-05 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.710 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306086+0000", 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0883e-05 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306275+0000", 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018911500000000001 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306470+0000", 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019556900000000001 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.306646+0000", 2024-04-25T15:53:40.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017527999999999999 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.319323+0000", 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012677034 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363674+0000", 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044351773999999997 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363724+0000", 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0016999999999999e-05 2024-04-25T15:53:40.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.363784+0000", 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9370999999999998e-05 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:745 3.d 3:b225ffa8:::100000001f7.000000e5:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.230462154000001, 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047290008000000001, 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.713 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 745 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389180+0000", 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.714 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.389181+0000", 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1500000000000003e-07 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401220+0000", 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012039525000000001 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401248+0000", 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7922000000000001e-05 2024-04-25T15:53:40.715 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401261+0000", 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2224e-05 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401480+0000", 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021909300000000001 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401643+0000", 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016358 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.716 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.401799+0000", 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015555699999999999 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.412210+0000", 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010410777 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436343+0000", 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024133884000000001 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436416+0000", 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2268000000000007e-05 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.436470+0000", 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4262999999999999e-05 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:761 3.3b 3:df080f3c:::100000001f7.000000f5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:40.718 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.111938015, 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045918457000000003, 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 761 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507704+0000", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.507707+0000", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.717e-06 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.524752+0000", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017044996999999999 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525517+0000", 2024-04-25T15:53:40.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076479300000000005 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525526+0000", 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3829999999999998e-06 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525610+0000", 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3758999999999995e-05 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525797+0000", 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187302 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.525954+0000", 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015707000000000001 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.536701+0000", 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01074656 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553463+0000", 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016761804000000002 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553540+0000", 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7026000000000004e-05 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.553623+0000", 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3046000000000004e-05 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:957 3.11 3:897f0f98:::100000001f7.000000b9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.231904+0000", 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.387738167, 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046297234, 2024-04-25T15:53:40.723 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 957 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.231904+0000", 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:40.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.231904+0000", 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.231905+0000", 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3769999999999999e-06 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236484+0000", 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045787220000000003 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236519+0000", 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4835000000000001e-05 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236534+0000", 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5437000000000001e-05 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236640+0000", 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010555799999999999 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.726 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236763+0000", 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012274700000000001 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.236935+0000", 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017239499999999999 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.268629+0000", 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031693741999999997 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:40.727 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.278006+0000", 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00937757 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.278096+0000", 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9899000000000001e-05 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.278201+0000", 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010495199999999999 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:40.728 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:40.729 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:40.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.662+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f64140c3380 con 0x7f641406cb80 2024-04-25T15:53:40.730 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:40.727+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:40.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:40.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fd0e64640 1 Processor -- start 2024-04-25T15:53:40.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fd0e64640 1 -- start start 2024-04-25T15:53:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fd0e64640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fcc154700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fd0e64640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fcc0c57a0 con 0x7f0fcc0c4bf0 2024-04-25T15:53:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fd0e64640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fcc0c5a40 con 0x7f0fcc154c40 2024-04-25T15:53:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fd0e64640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0fcc0cae50 con 0x7f0fcc0c5000 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fca59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0fcc0c4bf0 0x7f0fcc0ca740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36324/0 (socket says 172.21.15.67:36324) 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fca59c640 1 -- 172.21.15.67:0/2201577336 learned_addr learned my addr 172.21.15.67:0/2201577336 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1748181220 0 0) 0x7f0fcc0c57a0 con 0x7f0fcc0c4bf0 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fb4003530 con 0x7f0fcc0c4bf0 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fcc154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3456062976 0 0) 0x7f0fb4003530 con 0x7f0fcc0c4bf0 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 msgr2=0x7f0fcc154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fcc154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 >> v1:172.21.15.176:6789/0 conn(0x7f0fcc154c40 legacy=0x7f0fcc15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fcc0c5d20 con 0x7f0fcc0c4bf0 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f0fbc001680 con 0x7f0fcc0c4bf0 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 >> v1:172.21.15.67:6789/0 conn(0x7f0fcc0c4bf0 legacy=0x7f0fcc0ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb40039f0 0x7f0fb4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fb4004810 0x7f0fb4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fb40052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc0cae50 con 0x7f0fb40039f0 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb400bbb0 con 0x7f0fb4004810 2024-04-25T15:53:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb400bdb0 con 0x7f0fcc0c5000 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fca59c640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb40039f0 0x7f0fb4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fb4004810 0x7f0fb4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.738+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fb40052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fcad9d640 1 -- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fb4004810 msgr2=0x7f0fb4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fb4004810 0x7f0fb4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fcad9d640 1 -- 172.21.15.67:0/2201577336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb40039f0 msgr2=0x7f0fb4003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fb40039f0 0x7f0fb4003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fcad9d640 1 -- 172.21.15.67:0/2201577336 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb4003530 con 0x7f0fcc0c5000 2024-04-25T15:53:40.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fb4004810 0x7f0fb4004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:53:40.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fb40052e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4001e40 tx=0x7f0fc401c780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0beb857ef79b80c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40af070 con 0x7f0fcc0c5000 2024-04-25T15:53:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc4003b80 con 0x7f0fcc0c5000 2024-04-25T15:53:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.742+0000 7f0fc959a640 1 -- 172.21.15.67:0/2201577336 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4001a90 con 0x7f0fcc0c5000 2024-04-25T15:53:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 -- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 msgr2=0x7f0fb40052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/2201577336 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fb40052e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4001e40 tx=0x7f0fc401c780 comp rx=0 tx=0).stop 2024-04-25T15:53:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fcad9d640 1 -- 172.21.15.67:0/2201577336 reap_dead start 2024-04-25T15:53:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 -- 172.21.15.67:0/2201577336 shutdown_connections 2024-04-25T15:53:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 -- 172.21.15.67:0/2201577336 >> 172.21.15.67:0/2201577336 conn(0x7f0fcc0cfb70 msgr2=0x7f0fcc0cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:40.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 -- 172.21.15.67:0/2201577336 shutdown_connections 2024-04-25T15:53:40.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 -- 172.21.15.67:0/2201577336 wait complete. 2024-04-25T15:53:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 Processor -- start 2024-04-25T15:53:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.746+0000 7f0fd0e64640 1 -- start start 2024-04-25T15:53:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fcc1f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fcc160a70 0x7f0fcc0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fcc157ea0 0x7f0fcc1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc15ea90 con 0x7f0fcc157ea0 2024-04-25T15:53:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0fcc0cc2b0 con 0x7f0fcc160a70 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0fcc15ec10 con 0x7f0fcc0c5000 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fcc157ea0 0x7f0fcc1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fcc157ea0 0x7f0fcc1582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38444/0 (socket says 172.21.15.67:38444) 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 -- 172.21.15.67:0/3708894132 learned_addr learned my addr 172.21.15.67:0/3708894132 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fcc160a70 0x7f0fcc0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fcc1f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 msgr2=0x7f0fcc1f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fcc1f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fcc160a70 msgr2=0x7f0fcc0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fcc160a70 0x7f0fcc0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fcc1516d0 con 0x7f0fcc157ea0 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0fcc160a70 0x7f0fcc0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:40.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0fcc0c5000 0x7f0fcc1f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:40.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fcc157ea0 0x7f0fcc1582b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4097f70 tx=0x7f0fb4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40af070 con 0x7f0fcc157ea0 2024-04-25T15:53:40.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc40bc070 con 0x7f0fcc157ea0 2024-04-25T15:53:40.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fcc1f7210 con 0x7f0fcc157ea0 2024-04-25T15:53:40.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc000b600 con 0x7f0fcc157ea0 2024-04-25T15:53:40.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.750+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fcc1f77d0 con 0x7f0fcc157ea0 2024-04-25T15:53:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.754+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0fc40a9070 con 0x7f0fcc157ea0 2024-04-25T15:53:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.754+0000 7f0fbaffd640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fac073840 0x7f0fac075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.754+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5944+0+0 (secure 0 0 0) 0x7f0fc4128ec0 con 0x7f0fcc157ea0 2024-04-25T15:53:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.754+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fac073840 0x7f0fac075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.754+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0fcc06a480 con 0x7f0fcc157ea0 2024-04-25T15:53:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.758+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v87) ==== 54+0+16 (secure 0 0 0) 0x7f0fc4016490 con 0x7f0fcc157ea0 2024-04-25T15:53:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.758+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fac073840 0x7f0fac075d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0015e0 tx=0x7f0fbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.758+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0fcc1658f0 0x7f0fcc150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.758+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc154190 con 0x7f0fcc1658f0 2024-04-25T15:53:40.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.758+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0fcc1658f0 0x7f0fcc150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.758+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0fcc1658f0 0x7f0fcc150aa0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.758+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc154190 con 0x7f0fcc1658f0 2024-04-25T15:53:40.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.782+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc0cb020 con 0x7f0fcc1658f0 2024-04-25T15:53:40.787 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:40.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.786+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc0cb020 con 0x7f0fcc1658f0 2024-04-25T15:53:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.786+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0fcc0c8730 0x7f0fcc0c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.786+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc154670 con 0x7f0fcc0c8730 2024-04-25T15:53:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.786+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0fcc0c8730 0x7f0fcc0c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.786+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0fcc0c8730 0x7f0fcc0c8b40 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.786+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc154670 con 0x7f0fcc0c8730 2024-04-25T15:53:40.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.806+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc0cd350 con 0x7f0fcc0c8730 2024-04-25T15:53:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.810+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc0cd350 con 0x7f0fcc0c8730 2024-04-25T15:53:40.811 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:40.811 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.810+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0fcc063090 0x7f0fcc0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.810+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc0c9590 con 0x7f0fcc063090 2024-04-25T15:53:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.810+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0fcc063090 0x7f0fcc0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.810+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0fcc063090 0x7f0fcc0634a0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.814+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc0c9590 con 0x7f0fcc063090 2024-04-25T15:53:40.830 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:40.827+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:40.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.834+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc0ceba0 con 0x7f0fcc063090 2024-04-25T15:53:40.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.838+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc0ceba0 con 0x7f0fcc063090 2024-04-25T15:53:40.839 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.838+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0fcc063da0 0x7f0fcc152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.838+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc1531c0 con 0x7f0fcc063da0 2024-04-25T15:53:40.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.838+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0fcc063da0 0x7f0fcc152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.838+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0fcc063da0 0x7f0fcc152c80 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.838+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc1531c0 con 0x7f0fcc063da0 2024-04-25T15:53:40.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.862+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc1536a0 con 0x7f0fcc063da0 2024-04-25T15:53:40.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.862+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc1536a0 con 0x7f0fcc063da0 2024-04-25T15:53:40.864 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:40.865 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:40.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.862+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0fcc02cef0 0x7f0fcc02d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.862+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc02d840 con 0x7f0fcc02cef0 2024-04-25T15:53:40.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.866+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0fcc02cef0 0x7f0fcc02d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.866+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0fcc02cef0 0x7f0fcc02d300 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.866+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc02d840 con 0x7f0fcc02cef0 2024-04-25T15:53:40.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.886+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc02d840 con 0x7f0fcc02cef0 2024-04-25T15:53:40.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.890+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc02d840 con 0x7f0fcc02cef0 2024-04-25T15:53:40.890 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.890+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0fcc206370 0x7f0fcc208790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.890+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc20b560 con 0x7f0fcc206370 2024-04-25T15:53:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.890+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0fcc206370 0x7f0fcc208790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.890+0000 7f0fc9d9b640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0fcc206370 0x7f0fcc208790 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.894+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc20b560 con 0x7f0fcc206370 2024-04-25T15:53:40.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.914+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc208cd0 con 0x7f0fcc206370 2024-04-25T15:53:40.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.922+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc208cd0 con 0x7f0fcc206370 2024-04-25T15:53:40.923 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.922+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0fcc209350 0x7f0fcc20b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.922+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc20e5c0 con 0x7f0fcc209350 2024-04-25T15:53:40.923 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:40.924 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.922+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0fcc209350 0x7f0fcc20b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.922+0000 7f0fcad9d640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0fcc209350 0x7f0fcc20b770 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.926+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc20e5c0 con 0x7f0fcc209350 2024-04-25T15:53:40.930 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:40.930+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:40.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.946+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc20bf80 con 0x7f0fcc209350 2024-04-25T15:53:40.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.950+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc20bf80 con 0x7f0fcc209350 2024-04-25T15:53:40.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:40.953 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:40.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.954+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0fcc20c6a0 0x7f0fcc20cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:40.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.954+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0fcc2119b0 con 0x7f0fcc20c6a0 2024-04-25T15:53:40.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.954+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0fcc20c6a0 0x7f0fcc20cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:40.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:40.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.954+0000 7f0fca59c640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0fcc20c6a0 0x7f0fcc20cab0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:40.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.958+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0fcc2119b0 con 0x7f0fcc20c6a0 2024-04-25T15:53:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.978+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0fcc210830 con 0x7f0fcc20c6a0 2024-04-25T15:53:40.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fbaffd640 1 -- 172.21.15.67:0/3708894132 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0fcc210830 con 0x7f0fcc20c6a0 2024-04-25T15:53:40.983 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:40.984 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:40.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0fcc1658f0 msgr2=0x7f0fcc150aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f0fcc1658f0 0x7f0fcc150aa0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0fcc0c8730 msgr2=0x7f0fcc0c8b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f0fcc0c8730 0x7f0fcc0c8b40 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0fcc063090 msgr2=0x7f0fcc0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f0fcc063090 0x7f0fcc0634a0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0fcc063da0 msgr2=0x7f0fcc152c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f0fcc063da0 0x7f0fcc152c80 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0fcc02cef0 msgr2=0x7f0fcc02d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f0fcc02cef0 0x7f0fcc02d300 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fcad9d640 1 -- 172.21.15.67:0/3708894132 reap_dead start 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0fcc206370 msgr2=0x7f0fcc208790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f0fcc206370 0x7f0fcc208790 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fcad9d640 1 -- 172.21.15.67:0/3708894132 reap_dead start 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0fcc209350 msgr2=0x7f0fcc20b770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f0fcc209350 0x7f0fcc20b770 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0fcc20c6a0 msgr2=0x7f0fcc20cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f0fcc20c6a0 0x7f0fcc20cab0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fcad9d640 1 -- 172.21.15.67:0/3708894132 reap_dead start 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fac073840 msgr2=0x7f0fac075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fac073840 0x7f0fac075d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0015e0 tx=0x7f0fbc009040 comp rx=0 tx=0).stop 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fcc157ea0 msgr2=0x7f0fcc1582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fcc157ea0 0x7f0fcc1582b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4097f70 tx=0x7f0fb4005840 comp rx=0 tx=0).stop 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 shutdown_connections 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0fac073840 0x7f0fac075d50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 --2- 172.21.15.67:0/3708894132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0fcc157ea0 0x7f0fcc1582b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 >> 172.21.15.67:0/3708894132 conn(0x7f0fcc0cfb70 msgr2=0x7f0fcc0cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 shutdown_connections 2024-04-25T15:53:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:40.982+0000 7f0fd0e64640 1 -- 172.21.15.67:0/3708894132 wait complete. 2024-04-25T15:53:41.065 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:41.066+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:41.132 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:41.130+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:41.163 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:41.163 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:41.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:41.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.218+0000 7f6409ffb640 1 -- 172.21.15.67:0/3996576337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v88) ==== 87+0+0 (secure 0 0 0) 0x7f64040f6a40 con 0x7f641406cb80 2024-04-25T15:53:41.219 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f63f806f430 msgr2=0x7f63f8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f63f806f430 0x7f63f8071940 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f640c0012f0 tx=0x7f640c009040 comp rx=0 tx=0).stop 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f641406cb80 msgr2=0x7f641415cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f641406cb80 0x7f641415cdf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6404019070 tx=0x7f6404019200 comp rx=0 tx=0).stop 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 shutdown_connections 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f63f806f430 0x7f63f8071940 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641415dd00 0x7f641415d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f641406cb80 0x7f641415cdf0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 --2- 172.21.15.67:0/3996576337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6414068830 0x7f64140c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:41.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 >> 172.21.15.67:0/3996576337 conn(0x7f64140ce1c0 msgr2=0x7f6414155650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:41.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 shutdown_connections 2024-04-25T15:53:41.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:41.222+0000 7f641b5f6640 1 -- 172.21.15.67:0/3996576337 wait complete. 2024-04-25T15:53:41.233 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:41.234+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:41.334 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:41.331+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:41.418 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:41.418 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:41.435 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:41.431+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:41.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:41.536 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:41.535+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:41.637 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:41.638+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:41.649 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:41.649 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:41.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:41.737 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:41.735+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:41.838 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:41.838+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:41.927 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:41.927 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:41.939 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:41.938+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:41.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:42.040 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:42.039+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:42.140 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:42.138+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:42.196 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:42.197 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:42.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:42.242 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:42.239+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:42.342 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:42.342+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:42.444 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:42.442+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:42.473 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:42.473 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:42.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:42.545 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:42.543+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:42.646 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:42.643+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:42.739 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:42.739 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:42.747 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:42.747+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:42.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:42.848 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:42.846+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:42.948 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:42.947+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:42.980 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:42.980 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:43.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:43.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:43.049 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:43.050+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:43.150 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:43.147+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:43.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.234+0000 7fc673cd9640 1 Processor -- start 2024-04-25T15:53:43.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.234+0000 7fc673cd9640 1 -- start start 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc673cd9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 0x7fc66c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc673cd9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc66c0c92d0 0x7fc66c072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc673cd9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc66c0c96e0 0x7fc66c073300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc673cd9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc66c06dbc0 con 0x7fc66c0c96e0 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc673cd9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc66c06d8e0 con 0x7fc66c0c92d0 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc673cd9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc66c073c70 con 0x7fc66c160800 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc66c0c96e0 0x7fc66c073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc66c0c96e0 0x7fc66c073300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43672/0 (socket says 172.21.15.67:43672) 2024-04-25T15:53:43.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 -- 172.21.15.67:0/2321735514 learned_addr learned my addr 172.21.15.67:0/2321735514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:43.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 -- 172.21.15.67:0/2321735514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 msgr2=0x7fc66c06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc671274640 1 --2- 172.21.15.67:0/2321735514 >> v2:172.21.15.176:3300/0 conn(0x7fc66c0c92d0 0x7fc66c072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc671a75640 1 --2- 172.21.15.67:0/2321735514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 0x7fc66c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 --2- 172.21.15.67:0/2321735514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 0x7fc66c06d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 -- 172.21.15.67:0/2321735514 >> v2:172.21.15.176:3300/0 conn(0x7fc66c0c92d0 msgr2=0x7fc66c072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 --2- 172.21.15.67:0/2321735514 >> v2:172.21.15.176:3300/0 conn(0x7fc66c0c92d0 0x7fc66c072dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc672276640 1 -- 172.21.15.67:0/2321735514 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66c073f50 con 0x7fc66c0c96e0 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc671274640 1 --2- 172.21.15.67:0/2321735514 >> v2:172.21.15.176:3300/0 conn(0x7fc66c0c92d0 0x7fc66c072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:43.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.238+0000 7fc671a75640 1 --2- 172.21.15.67:0/2321735514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 0x7fc66c06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc672276640 1 --2- 172.21.15.67:0/2321735514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c073300 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0cae30 tx=0x7fc66401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8746e57de1efe514 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc670a73640 1 -- 172.21.15.67:0/2321735514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6640a56d0 con 0x7fc66c0c96e0 2024-04-25T15:53:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc670a73640 1 -- 172.21.15.67:0/2321735514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6640a5870 con 0x7fc66c0c96e0 2024-04-25T15:53:43.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc670a73640 1 -- 172.21.15.67:0/2321735514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6640a5c40 con 0x7fc66c0c96e0 2024-04-25T15:53:43.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 -- 172.21.15.67:0/2321735514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 msgr2=0x7fc66c073300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2321735514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c073300 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0cae30 tx=0x7fc66401c960 comp rx=0 tx=0).stop 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 -- 172.21.15.67:0/2321735514 shutdown_connections 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2321735514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c073300 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2321735514 >> v2:172.21.15.176:3300/0 conn(0x7fc66c0c92d0 0x7fc66c072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2321735514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 0x7fc66c06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 -- 172.21.15.67:0/2321735514 >> 172.21.15.67:0/2321735514 conn(0x7fc66c070db0 msgr2=0x7fc66c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 -- 172.21.15.67:0/2321735514 shutdown_connections 2024-04-25T15:53:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.242+0000 7fc673cd9640 1 -- 172.21.15.67:0/2321735514 wait complete. 2024-04-25T15:53:43.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 Processor -- start 2024-04-25T15:53:43.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 -- start start 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc66c0c92d0 0x7fc66c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 0x7fc66c13ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c160c10 con 0x7fc66c0c96e0 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c06d570 con 0x7fc66c0c92d0 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.246+0000 7fc673cd9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc66c160d90 con 0x7fc66c160800 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c15ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43678/0 (socket says 172.21.15.67:43678) 2024-04-25T15:53:43.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 -- 172.21.15.67:0/2870673274 learned_addr learned my addr 172.21.15.67:0/2870673274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:43.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 msgr2=0x7fc66c13ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc66c160800 0x7fc66c13ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc66c0c92d0 msgr2=0x7fc66c154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc66c0c92d0 0x7fc66c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66c155080 con 0x7fc66c0c96e0 2024-04-25T15:53:43.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc671274640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c15ee10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00c5a0 tx=0x7fc65c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c015070 con 0x7fc66c0c96e0 2024-04-25T15:53:43.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65c004220 con 0x7fc66c0c96e0 2024-04-25T15:53:43.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66c155ab0 con 0x7fc66c0c96e0 2024-04-25T15:53:43.252 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:43.251+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:43.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c003620 con 0x7fc66c0c96e0 2024-04-25T15:53:43.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.250+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66c156010 con 0x7fc66c0c96e0 2024-04-25T15:53:43.253 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:43.253 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:43.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc66c06a480 con 0x7fc66c0c96e0 2024-04-25T15:53:43.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc65c0037c0 con 0x7fc66c0c96e0 2024-04-25T15:53:43.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc6627fc640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc65006f430 0x7fc650071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5944+0+0 (secure 0 0 0) 0x7fc65c08f0c0 con 0x7fc66c0c96e0 2024-04-25T15:53:43.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v88) ==== 54+0+16 (secure 0 0 0) 0x7fc65c091740 con 0x7fc66c0c96e0 2024-04-25T15:53:43.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc65006f430 0x7fc650071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc66c0c7b00 0x7fc66c140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc66c0ce370 con 0x7fc66c0c7b00 2024-04-25T15:53:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc672276640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc66c0c7b00 0x7fc66c140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.254+0000 7fc672276640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc66c0c7b00 0x7fc66c140480 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.258+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc65006f430 0x7fc650071940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc66800a890 tx=0x7fc668008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.258+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c0ce370 con 0x7fc66c0c7b00 2024-04-25T15:53:43.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:53:43.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.282+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c0008e0 con 0x7fc66c0c7b00 2024-04-25T15:53:43.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.282+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c0008e0 con 0x7fc66c0c7b00 2024-04-25T15:53:43.285 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.282+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc66c1f9100 0x7fc66c1f9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.282+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc66c1543d0 con 0x7fc66c1f9100 2024-04-25T15:53:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.282+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc66c1f9100 0x7fc66c1f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.286+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc66c1f9100 0x7fc66c1f9510 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.286+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c1543d0 con 0x7fc66c1f9100 2024-04-25T15:53:43.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.302+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c1546b0 con 0x7fc66c1f9100 2024-04-25T15:53:43.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.306+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c1546b0 con 0x7fc66c1f9100 2024-04-25T15:53:43.307 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:43.308 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:43.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.306+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc66c063090 0x7fc66c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.306+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc66c1f9f60 con 0x7fc66c063090 2024-04-25T15:53:43.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.306+0000 7fc671274640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc66c063090 0x7fc66c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.306+0000 7fc671274640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc66c063090 0x7fc66c0634a0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.310+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c1f9f60 con 0x7fc66c063090 2024-04-25T15:53:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.334+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c151eb0 con 0x7fc66c063090 2024-04-25T15:53:43.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.338+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c151eb0 con 0x7fc66c063090 2024-04-25T15:53:43.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:43.337 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.338+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc66c0cc730 0x7fc66c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.338+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc66c064060 con 0x7fc66c0cc730 2024-04-25T15:53:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.338+0000 7fc672276640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc66c0cc730 0x7fc66c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.338+0000 7fc672276640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc66c0cc730 0x7fc66c0ccb40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.338+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c064060 con 0x7fc66c0cc730 2024-04-25T15:53:43.352 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:43.351+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:43.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.370+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c0cd350 con 0x7fc66c0cc730 2024-04-25T15:53:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.370+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c0cd350 con 0x7fc66c0cc730 2024-04-25T15:53:43.373 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.374+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc66c02cef0 0x7fc66c02d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.374+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc66c02d840 con 0x7fc66c02cef0 2024-04-25T15:53:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.374+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc66c02cef0 0x7fc66c02d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.374+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc66c02cef0 0x7fc66c02d300 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.374+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c02d840 con 0x7fc66c02cef0 2024-04-25T15:53:43.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.406+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c02d840 con 0x7fc66c02cef0 2024-04-25T15:53:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.410+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c02d840 con 0x7fc66c02cef0 2024-04-25T15:53:43.411 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:43.411 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:43.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.410+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc66c20a6c0 0x7fc66c20cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.410+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc66c02d840 con 0x7fc66c20a6c0 2024-04-25T15:53:43.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.410+0000 7fc671274640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc66c20a6c0 0x7fc66c20cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.414+0000 7fc671274640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc66c20a6c0 0x7fc66c20cae0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.414+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c02d840 con 0x7fc66c20a6c0 2024-04-25T15:53:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.438+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c02d840 con 0x7fc66c20a6c0 2024-04-25T15:53:43.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.442+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c02d840 con 0x7fc66c20a6c0 2024-04-25T15:53:43.445 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:43.445 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:43.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.446+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc66c20d530 0x7fc66c20f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.446+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc66c212720 con 0x7fc66c20d530 2024-04-25T15:53:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.446+0000 7fc672276640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc66c20d530 0x7fc66c20f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.450+0000 7fc672276640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc66c20d530 0x7fc66c20f950 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.453 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:43.454+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:43.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c212720 con 0x7fc66c20d530 2024-04-25T15:53:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fef627d0640 1 Processor -- start 2024-04-25T15:53:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fef627d0640 1 -- start start 2024-04-25T15:53:43.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fef627d0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fef5c0c5620 0x7fef5c0c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fef627d0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef5c160c20 0x7fef5c161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fef627d0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef5c165040 con 0x7fef5c06c640 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fef627d0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fef5c1642c0 con 0x7fef5c160c20 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.454+0000 7fef627d0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fef5c0c6140 con 0x7fef5c0c5620 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef5c160c20 0x7fef5c161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef5c160c20 0x7fef5c161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43682/0 (socket says 172.21.15.67:43682) 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 -- 172.21.15.67:0/3056672049 learned_addr learned my addr 172.21.15.67:0/3056672049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 -- 172.21.15.67:0/3056672049 >> v2:172.21.15.176:3300/0 conn(0x7fef5c0c5620 msgr2=0x7fef5c0c4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:53:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/3056672049 >> v2:172.21.15.176:3300/0 conn(0x7fef5c0c5620 0x7fef5c0c4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 -- 172.21.15.67:0/3056672049 >> v1:172.21.15.176:6789/0 conn(0x7fef5c06c640 legacy=0x7fef5c0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 -- 172.21.15.67:0/3056672049 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c0c5e60 con 0x7fef5c160c20 2024-04-25T15:53:43.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/3056672049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c160c20 0x7fef5c161030 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0706d0 tx=0x7fef4401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49471fa593c2281a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5affd640 1 -- 172.21.15.67:0/3056672049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef440ab680 con 0x7fef5c160c20 2024-04-25T15:53:43.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5affd640 1 -- 172.21.15.67:0/3056672049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef440b0070 con 0x7fef5c160c20 2024-04-25T15:53:43.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.458+0000 7fef5affd640 1 -- 172.21.15.67:0/3056672049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef440aba40 con 0x7fef5c160c20 2024-04-25T15:53:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 -- 172.21.15.67:0/3056672049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c160c20 msgr2=0x7fef5c161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 --2- 172.21.15.67:0/3056672049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c160c20 0x7fef5c161030 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0706d0 tx=0x7fef4401c480 comp rx=0 tx=0).stop 2024-04-25T15:53:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 -- 172.21.15.67:0/3056672049 shutdown_connections 2024-04-25T15:53:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 --2- 172.21.15.67:0/3056672049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c160c20 0x7fef5c161030 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 --2- 172.21.15.67:0/3056672049 >> v2:172.21.15.176:3300/0 conn(0x7fef5c0c5620 0x7fef5c0c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 -- 172.21.15.67:0/3056672049 >> 172.21.15.67:0/3056672049 conn(0x7fef5c06fe70 msgr2=0x7fef5c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:43.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 -- 172.21.15.67:0/3056672049 shutdown_connections 2024-04-25T15:53:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.462+0000 7fef627d0640 1 -- 172.21.15.67:0/3056672049 wait complete. 2024-04-25T15:53:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 Processor -- start 2024-04-25T15:53:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 -- start start 2024-04-25T15:53:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c06c640 0x7fef5c0738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c5620 0x7fef5c151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c160c20 0x7fef5c151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c0c52e0 con 0x7fef5c06c640 2024-04-25T15:53:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c06ce60 con 0x7fef5c160c20 2024-04-25T15:53:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef627d0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fef5c0c5a30 con 0x7fef5c0c5620 2024-04-25T15:53:43.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c06c640 0x7fef5c0738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c06c640 0x7fef5c0738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43698/0 (socket says 172.21.15.67:43698) 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 -- 172.21.15.67:0/2399896412 learned_addr learned my addr 172.21.15.67:0/2399896412 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef60d6d640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c160c20 0x7fef5c151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c5620 0x7fef5c151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 -- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c5620 msgr2=0x7fef5c151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c5620 0x7fef5c151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 -- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c160c20 msgr2=0x7fef5c151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c160c20 0x7fef5c151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 -- 172.21.15.67:0/2399896412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c072e30 con 0x7fef5c06c640 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.466+0000 7fef5bfff640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c06c640 0x7fef5c0738b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fef5400ba50 tx=0x7fef5400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.470+0000 7fef58ff9640 1 -- 172.21.15.67:0/2399896412 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54019070 con 0x7fef5c06c640 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.470+0000 7fef58ff9640 1 -- 172.21.15.67:0/2399896412 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef5400cbb0 con 0x7fef5c06c640 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.470+0000 7fef58ff9640 1 -- 172.21.15.67:0/2399896412 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54014440 con 0x7fef5c06c640 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.470+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c073df0 con 0x7fef5c06c640 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.470+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c155f60 con 0x7fef5c06c640 2024-04-25T15:53:43.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.470+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef5c06a480 con 0x7fef5c06c640 2024-04-25T15:53:43.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.470+0000 7fef58ff9640 1 -- 172.21.15.67:0/2399896412 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fef5401c070 con 0x7fef5c06c640 2024-04-25T15:53:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.474+0000 7fef58ff9640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef3c06f400 0x7fef3c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.474+0000 7fef58ff9640 1 -- 172.21.15.67:0/2399896412 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5944+0+0 (secure 0 0 0) 0x7fef5408f450 con 0x7fef5c06c640 2024-04-25T15:53:43.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.474+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef3c06f400 0x7fef3c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.474+0000 7fef5b7fe640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef3c06f400 0x7fef3c071910 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fef440af000 tx=0x7fef440a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.478+0000 7fef58ff9640 1 -- 172.21.15.67:0/2399896412 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fef54023020 con 0x7fef5c06c640 2024-04-25T15:53:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.478+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c20fe90 con 0x7fc66c20d530 2024-04-25T15:53:43.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.482+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c20fe90 con 0x7fc66c20d530 2024-04-25T15:53:43.482 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.482+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc66c2105b0 0x7fc66c2109c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.482+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc66c215af0 con 0x7fc66c2105b0 2024-04-25T15:53:43.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.482+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc66c2105b0 0x7fc66c2109c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:43.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.482+0000 7fc671a75640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc66c2105b0 0x7fc66c2109c0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:43.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.486+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc66c215af0 con 0x7fc66c2105b0 2024-04-25T15:53:43.501 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:43.501 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:43.501 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:43.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.506+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc66c211010 con 0x7fc66c2105b0 2024-04-25T15:53:43.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.510+0000 7fc6627fc640 1 -- 172.21.15.67:0/2870673274 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc66c211010 con 0x7fc66c2105b0 2024-04-25T15:53:43.513 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc66c0c7b00 msgr2=0x7fc66c140480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fc66c0c7b00 0x7fc66c140480 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc66c1f9100 msgr2=0x7fc66c1f9510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fc66c1f9100 0x7fc66c1f9510 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc66c063090 msgr2=0x7fc66c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fc66c063090 0x7fc66c0634a0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc66c0cc730 msgr2=0x7fc66c0ccb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fc66c0cc730 0x7fc66c0ccb40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc672276640 1 -- 172.21.15.67:0/2870673274 reap_dead start 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc66c02cef0 msgr2=0x7fc66c02d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fc66c02cef0 0x7fc66c02d300 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc672276640 1 -- 172.21.15.67:0/2870673274 reap_dead start 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc66c20a6c0 msgr2=0x7fc66c20cae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc66c20a6c0 0x7fc66c20cae0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc66c20d530 msgr2=0x7fc66c20f950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fc66c20d530 0x7fc66c20f950 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc66c2105b0 msgr2=0x7fc66c2109c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fc66c2105b0 0x7fc66c2109c0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc65006f430 msgr2=0x7fc650071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc65006f430 0x7fc650071940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc66800a890 tx=0x7fc668008040 comp rx=0 tx=0).stop 2024-04-25T15:53:43.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 msgr2=0x7fc66c15ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 --2- 172.21.15.67:0/2870673274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc66c0c96e0 0x7fc66c15ee10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00c5a0 tx=0x7fc65c00ca60 comp rx=0 tx=0).stop 2024-04-25T15:53:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc672276640 1 -- 172.21.15.67:0/2870673274 reap_dead start 2024-04-25T15:53:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 shutdown_connections 2024-04-25T15:53:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 >> 172.21.15.67:0/2870673274 conn(0x7fc66c070db0 msgr2=0x7fc66c15b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 shutdown_connections 2024-04-25T15:53:43.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.514+0000 7fc673cd9640 1 -- 172.21.15.67:0/2870673274 wait complete. 2024-04-25T15:53:43.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:43.594 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:43.594+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:43.654 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:43.654+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:43.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:43.734+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fef5c0cb750 con 0x7fef5c06c640 2024-04-25T15:53:43.774 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:43.774+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:43.813 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:180 3.2 3:42ce5726:::10000000002.000000b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:50.001754+0000", 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 53.811109719999997, 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031518879, 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.814 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 180 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.001754+0000", 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.001754+0000", 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.001757+0000", 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8590000000000001e-06 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003128+0000", 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013710160000000001 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003169+0000", 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0994000000000001e-05 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003188+0000", 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9419000000000002e-05 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003286+0000", 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8097000000000006e-05 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003436+0000", 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014946600000000001 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.003559+0000", 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012318 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.009738+0000", 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061791939999999998 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.033190+0000", 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023451560999999999 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.033233+0000", 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3417000000000001e-05 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:50.033273+0000", 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9675999999999998e-05 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:343 3.8 3:1115f3ca:::10000000002.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:43.819 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.805996892000003, 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.46089677899999998, 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 343 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:43.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006870+0000", 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9280000000000002e-06 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008363+0000", 2024-04-25T15:53:43.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001493288 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008398+0000", 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5067000000000002e-05 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008416+0000", 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7782e-05 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008522+0000", 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106327 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008535+0000", 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2883e-05 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.387477+0000", 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.37894188200000001 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:43.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.392562+0000", 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050850870000000003 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445264+0000", 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052702037 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445382+0000", 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118433 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445603+0000", 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022077500000000001 2024-04-25T15:53:43.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.465250+0000", 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019646976 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467658+0000", 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024073850000000002 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467695+0000", 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7234999999999998e-05 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467763+0000", 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8694000000000003e-05 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:352 3.7 3:e0f5f483:::10000000002.0000005c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.338588+0000", 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.474275564999999, 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031524609000000002, 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.826 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 352 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.338588+0000", 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.338588+0000", 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.338591+0000", 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.066e-06 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.339937+0000", 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013461759999999999 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.339976+0000", 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8631000000000003e-05 2024-04-25T15:53:43.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.339991+0000", 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5557999999999999e-05 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.340065+0000", 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3096000000000006e-05 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.340144+0000", 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9138e-05 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.340292+0000", 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000147873 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.351095+0000", 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010803708 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.370012+0000", 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018916397000000001 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.370062+0000", 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0300999999999998e-05 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.370113+0000", 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0664999999999997e-05 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:601 3.38 3:1c31e34b:::100000001f7.00000055:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.393712+0000", 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.419151214999999, 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028807679999999999, 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 601 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.393712+0000", 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.393712+0000", 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.393715+0000", 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.112e-06 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.395997+0000", 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002281295 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396043+0000", 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6440999999999998e-05 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396060+0000", 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6758000000000001e-05 2024-04-25T15:53:43.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396175+0000", 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115059 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396373+0000", 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019780399999999999 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.396509+0000", 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000136019 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.410556+0000", 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014046984 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.422383+0000", 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011826872 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.422459+0000", 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6304000000000007e-05 2024-04-25T15:53:43.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.422520+0000", 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1032000000000001e-05 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:670 3.8 3:1323134e:::100000001f7.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.938926728999999, 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094982472999999998, 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 670 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:43.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873941+0000", 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0840000000000002e-06 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876488+0000", 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025470290000000001 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876647+0000", 2024-04-25T15:53:43.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015912000000000001 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876664+0000", 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7215e-05 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876782+0000", 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118082 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877222+0000", 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000439596 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877596+0000", 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037371700000000003 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.913106+0000", 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035509813000000001 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968792+0000", 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.055686739999999998 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968850+0000", 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8032000000000003e-05 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968919+0000", 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9045e-05 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:666 3.15 e81/18 3:a92ac213:::100000001f7.00000096:head v 81'11, mlcod=81'9)", 2024-04-25T15:53:43.842 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.938054492999999, 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046403201999999998, 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:43.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874810+0000", 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882291+0000", 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074801800000000003 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882310+0000", 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9809999999999998e-05 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882330+0000", 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0026000000000001e-05 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882471+0000", 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014052199999999999 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882509+0000", 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7756999999999998e-05 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921136+0000", 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038627364999999997 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921212+0000", 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6291999999999994e-05 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:671 3.38 3:1c0639bf:::100000001f7.0000009b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.928761694999999, 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073197039000000005, 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 671 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884106+0000", 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9709999999999998e-06 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886679+0000", 2024-04-25T15:53:43.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025726770000000002 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886716+0000", 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.735e-05 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886733+0000", 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6697999999999998e-05 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886843+0000", 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011074299999999999 2024-04-25T15:53:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887031+0000", 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018804800000000001 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887215+0000", 2024-04-25T15:53:43.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018384599999999999 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921326+0000", 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034110486000000002 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957180+0000", 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035854051999999997 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957227+0000", 2024-04-25T15:53:43.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7290000000000003e-05 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957299+0000", 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1878000000000005e-05 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(mds.0.4:88 2.14 e81/17 2:2e2fa760:::200.00000001:head v 81'51, mlcod=81'49)", 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.892986+0000", 2024-04-25T15:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.919877523, 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028378470999999999, 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.892986+0000", 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.892986+0000", 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.892988+0000", 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8729999999999999e-06 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893008+0000", 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0134000000000001e-05 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893031+0000", 2024-04-25T15:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2486000000000001e-05 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893089+0000", 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8686e-05 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893200+0000", 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111214 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.893233+0000", 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2382999999999998e-05 2024-04-25T15:53:43.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921296+0000", 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028063088999999999 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921365+0000", 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8606e-05 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:672 3.23 3:c45db235:::100000001f7.0000009c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.896255+0000", 2024-04-25T15:53:43.856 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.916608696000001, 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027004725, 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 672 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.896255+0000", 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.896255+0000", 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.896257+0000", 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3309999999999998e-06 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898207+0000", 2024-04-25T15:53:43.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019496489999999999 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898257+0000", 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0340999999999998e-05 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898279+0000", 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1416e-05 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898423+0000", 2024-04-25T15:53:43.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014452200000000001 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898715+0000", 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00029176700000000001 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.898979+0000", 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000263727 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921866+0000", 2024-04-25T15:53:43.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022886969 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923116+0000", 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012504860000000001 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923193+0000", 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6632999999999996e-05 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923260+0000", 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6884000000000005e-05 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:679 3.24 3:27dcc3f5:::100000001f7.000000a3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.950691+0000", 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.862172997000002, 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033201485000000003, 2024-04-25T15:53:43.862 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 679 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.950691+0000", 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.950691+0000", 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.950693+0000", 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1830000000000001e-06 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952770+0000", 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020768869999999999 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952809+0000", 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9662000000000001e-05 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952823+0000", 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3737000000000001e-05 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.952925+0000", 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010165100000000001 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.953095+0000", 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017042899999999999 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.953217+0000", 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121672 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.964557+0000", 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011340283 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983805+0000", 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019247543999999998 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983850+0000", 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5639000000000001e-05 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983892+0000", 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1798000000000002e-05 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:730 3.38 3:1c05d319:::100000001f7.000000d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.285762+0000", 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.527102001999999, 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038281437000000001, 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 730 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.285762+0000", 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.285762+0000", 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.285769+0000", 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1910000000000003e-06 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287695+0000", 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001926002 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287732+0000", 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6837e-05 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287749+0000", 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.738e-05 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287850+0000", 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010052000000000001 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.287988+0000", 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000138176 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.288116+0000", 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012841999999999999 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.295377+0000", 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072610230000000001 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.323938+0000", 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028561287000000001 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.323984+0000", 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5379e-05 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.324043+0000", 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9222000000000003e-05 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:732 3.15 e81/18 3:aadcba4a:::100000001f7.000000d8:head v 81'13, mlcod=81'11)", 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.505236706000002, 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056638899999999999, 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307628+0000", 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4529999999999999e-06 2024-04-25T15:53:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.312965+0000", 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367110000000001 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.312985+0000", 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9470000000000002e-05 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313001+0000", 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6840000000000001e-05 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.875 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313091+0000", 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9770000000000003e-05 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313122+0000", 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364214+0000", 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051091996000000001 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.876 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364266+0000", 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1613000000000002e-05 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:733 3.22 e81/45 3:44586a4d:::100000001f7.000000d9:head v 81'6, mlcod=81'4)", 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.310476+0000", 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.502387519999999, 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034633787999999999, 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.877 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310476+0000", 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310476+0000", 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310478+0000", 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.485e-06 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.878 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313574+0000", 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.003096264 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313592+0000", 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8006e-05 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313607+0000", 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5311e-05 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.879 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313685+0000", 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8116999999999998e-05 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313711+0000", 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5745000000000002e-05 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345018+0000", 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031306542999999999 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345110+0000", 2024-04-25T15:53:43.880 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2317000000000004e-05 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:736 3.25 3:a5356222:::100000001f7.000000dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.486164594999998, 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041194891999999997, 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.881 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 736 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.882 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326702+0000", 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5500000000000001e-06 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328063+0000", 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013610849999999999 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328097+0000", 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4638000000000003e-05 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.883 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328111+0000", 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3896999999999999e-05 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328183+0000", 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1859000000000004e-05 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328336+0000", 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015279700000000001 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:43.884 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328465+0000", 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012961200000000001 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.346379+0000", 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017913097999999999 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367792+0000", 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021413926 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.885 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367838+0000", 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5541999999999999e-05 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367894+0000", 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5887999999999998e-05 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:738 3.23 3:c7b191fb:::100000001f7.000000de:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.338982+0000", 2024-04-25T15:53:43.886 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.473881408, 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034696241000000003, 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 738 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.338982+0000", 2024-04-25T15:53:43.887 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.338982+0000", 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.338985+0000", 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.582e-06 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340200+0000", 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012147939999999999 2024-04-25T15:53:43.888 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340233+0000", 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2957999999999999e-05 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340248+0000", 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4996e-05 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.889 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340325+0000", 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7132999999999995e-05 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340447+0000", 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012218900000000001 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.340573+0000", 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012636900000000001 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.890 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364265+0000", 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023691727999999999 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.373549+0000", 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092843230000000006 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.373616+0000", 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7000000000000002e-05 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.891 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.373678+0000", 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2168999999999994e-05 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:955 3.7 3:e0d97e51:::100000001f7.000000b7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.222517+0000", 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.590346988, 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036962529000000001, 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.892 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 955 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.222517+0000", 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.222517+0000", 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.893 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.222519+0000", 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4140000000000001e-06 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223838+0000", 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001319188 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223874+0000", 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6216e-05 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.894 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223891+0000", 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6623e-05 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.223967+0000", 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5828999999999995e-05 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.224044+0000", 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7134999999999999e-05 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.895 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.224146+0000", 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102032 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.229620+0000", 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054742319999999999 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259348+0000", 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029727334000000001 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.896 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259412+0000", 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4789999999999995e-05 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259479+0000", 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6736000000000006e-05 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:959 3.33 3:cf81bd10:::100000001f7.000000bb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:43.897 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.568991879, 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042315856999999998, 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 959 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:43.898 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243874+0000", 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.95e-06 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245749+0000", 2024-04-25T15:53:43.899 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018754080000000001 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245791+0000", 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2200000000000003e-05 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245807+0000", 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5379999999999998e-05 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245884+0000", 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-05 2024-04-25T15:53:43.900 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245972+0000", 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8459000000000001e-05 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.246076+0000", 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103861 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252170+0000", 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060943439999999998 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286068+0000", 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033897570000000002 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286116+0000", 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8374999999999997e-05 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286188+0000", 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.127e-05 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.902 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:960 3.1e 3:796b994c:::100000001f7.000000bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.249999+0000", 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.562864804, 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036397816, 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 960 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.249999+0000", 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.249999+0000", 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.250002+0000", 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.18e-06 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252017+0000", 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020151689999999998 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252052+0000", 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4754e-05 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252070+0000", 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7776999999999999e-05 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.905 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252161+0000", 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0977999999999995e-05 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252251+0000", 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9963000000000006e-05 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252395+0000", 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144864 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.259647+0000", 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0072518180000000002 2024-04-25T15:53:43.906 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286270+0000", 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026622845999999999 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286341+0000", 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1069999999999995e-05 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286397+0000", 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5396999999999998e-05 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.907 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1010 3.33 3:cf8599b8:::100000001f7.000000ee:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.272143223, 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043954280999999998, 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.908 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1010 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540723+0000", 2024-04-25T15:53:43.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6029999999999999e-06 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542109+0000", 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013856610000000001 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542152+0000", 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3353999999999999e-05 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542167+0000", 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4779e-05 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542259+0000", 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2472999999999994e-05 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542384+0000", 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012520200000000001 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542571+0000", 2024-04-25T15:53:43.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018691000000000001 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.551607+0000", 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090354900000000002 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584560+0000", 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032952759999999998 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584623+0000", 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3305999999999993e-05 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584675+0000", 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1743000000000002e-05 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1011 3.7 3:e14869e0:::100000001f7.000000ef:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.267097058999999, 2024-04-25T15:53:43.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038908115, 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1011 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545769+0000", 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.587e-06 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547305+0000", 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015358590000000001 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547334+0000", 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8538e-05 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547346+0000", 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2449000000000001e-05 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547459+0000", 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112759 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547581+0000", 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012228900000000001 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547754+0000", 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017338 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.555402+0000", 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076472600000000003 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:43.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584550+0000", 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029148484999999998 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584616+0000", 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5528e-05 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584675+0000", 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8981000000000003e-05 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.918 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:43.919 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:43.919 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:43.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:43.920 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:43.919+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:44.021 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:44.019+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:44.107 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:44.107 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:44.107 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:44.107 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:44.121 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:44.122+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:44.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:44.223 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:44.222+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.266+0000 7fef58ff9640 1 -- 172.21.15.67:0/2399896412 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v89) ==== 97+0+0 (secure 0 0 0) 0x7fef5c072e30 con 0x7fef5c06c640 2024-04-25T15:53:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef3c06f400 msgr2=0x7fef3c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef3c06f400 0x7fef3c071910 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fef440af000 tx=0x7fef440a7000 comp rx=0 tx=0).stop 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c06c640 msgr2=0x7fef5c0738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c06c640 0x7fef5c0738b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fef5400ba50 tx=0x7fef5400bf10 comp rx=0 tx=0).stop 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 shutdown_connections 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fef3c06f400 0x7fef3c071910 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c160c20 0x7fef5c151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0c5620 0x7fef5c151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 --2- 172.21.15.67:0/2399896412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef5c06c640 0x7fef5c0738b0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 >> 172.21.15.67:0/2399896412 conn(0x7fef5c06fe70 msgr2=0x7fef5c075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 shutdown_connections 2024-04-25T15:53:44.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:44.270+0000 7fef627d0640 1 -- 172.21.15.67:0/2399896412 wait complete. 2024-04-25T15:53:44.345 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:44.346+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:44.363 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:44.363 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:44.363 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:44.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:44.394 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:44.394 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:44.394 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-04-25T15:53:44.394 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:53:44.394 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:53:44.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-04-25T15:53:44.439 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:44.439+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:44.525 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:44.526+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:44.582 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:44.582 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:44.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:44.582 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:44.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.582 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:597 3.e e81/76 3:72b88a04:::100000001f7.00000051:head v 81'6, mlcod=69'3)", 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.205782417000002, 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038266363999999997, 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374532+0000", 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.374533+0000", 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5060000000000001e-06 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376869+0000", 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023357830000000001 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376914+0000", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4317000000000003e-05 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.376941+0000", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6965999999999999e-05 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377065+0000", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012408300000000001 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.377098+0000", 2024-04-25T15:53:44.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2943e-05 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412723+0000", 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035625742000000002 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.412798+0000", 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5024000000000005e-05 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.586 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:599 3.1b 3:d81cefd1:::100000001f7.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.201493188000001, 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039983414000000002, 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 599 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378821+0000", 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.378825+0000", 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7189999999999999e-06 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384734+0000", 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059089859999999998 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384776+0000", 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2166999999999998e-05 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384795+0000", 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9307999999999999e-05 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.384894+0000", 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9112999999999998e-05 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385064+0000", 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000169524 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.385212+0000", 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147849 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.409088+0000", 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023875850000000001 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418680+0000", 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095925639999999996 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418742+0000", 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1379999999999998e-05 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418805+0000", 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2953999999999994e-05 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:598 3.11 e81/76 3:89a80ce8:::100000001f7.00000052:head v 81'21, mlcod=80'19)", 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.198953865, 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037275751000000003, 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381360+0000", 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.381362+0000", 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.782e-06 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382748+0000", 2024-04-25T15:53:44.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001385488 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382769+0000", 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0854000000000001e-05 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382786+0000", 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7807000000000001e-05 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382876+0000", 2024-04-25T15:53:44.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9821999999999995e-05 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.382901+0000", 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4845999999999999e-05 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418526+0000", 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035624426000000001 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.418636+0000", 2024-04-25T15:53:44.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110726 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:654 3.11 e81/76 3:892e96a3:::100000001f7.0000008a:head v 81'22, mlcod=80'20)", 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.72355374, 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044455910000000001, 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856761+0000", 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.856762+0000", 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.669e-06 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859651+0000", 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002889015 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859675+0000", 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3346999999999999e-05 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859696+0000", 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1021000000000001e-05 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859816+0000", 2024-04-25T15:53:44.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120809 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.859853+0000", 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6242999999999998e-05 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901127+0000", 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041274245000000001 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.901217+0000", 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9561000000000005e-05 2024-04-25T15:53:44.599 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:668 3.1 3:83ba1de1:::100000001f7.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.720182114, 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062400662000000003, 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.600 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 668 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860132+0000", 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.860140+0000", 2024-04-25T15:53:44.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1559999999999995e-06 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869131+0000", 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089909859999999994 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869188+0000", 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6539000000000002e-05 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869212+0000", 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4501000000000001e-05 2024-04-25T15:53:44.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869417+0000", 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020473999999999999 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869612+0000", 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000194668 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871230+0000", 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016182029999999999 2024-04-25T15:53:44.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.917654+0000", 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046423786000000002 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922361+0000", 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00470704 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922442+0000", 2024-04-25T15:53:44.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.153e-05 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.922533+0000", 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0513000000000006e-05 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:669 3.f e81/76 3:f0675d7f:::100000001f7.00000099:head v 81'10, mlcod=70'8)", 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.694145595999998, 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080177950999999997, 2024-04-25T15:53:44.605 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886169+0000", 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886171+0000", 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8470000000000001e-06 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891613+0000", 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054420349999999996 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891657+0000", 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4155e-05 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891689+0000", 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1822999999999997e-05 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891773+0000", 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4602000000000001e-05 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.891806+0000", 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2842999999999997e-05 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966243+0000", 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074437173999999995 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.966347+0000", 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103472 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:647 3.a e81/76 3:524e89bd:::100000001f7.00000083:head v 81'3, mlcod=61'1)", 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:44.609 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.675710974000001, 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072551145999999997, 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904603+0000", 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904605+0000", 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.421e-06 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906854+0000", 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022493249999999999 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906880+0000", 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5374000000000001e-05 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906897+0000", 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7419999999999999e-05 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.906994+0000", 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6633999999999997e-05 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.907027+0000", 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3336000000000001e-05 2024-04-25T15:53:44.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977067+0000", 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.070040033000000002 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977155+0000", 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7602999999999995e-05 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.613 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:674 3.9 3:90d768d7:::100000001f7.0000009e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.668044042999998, 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065083915000000006, 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 674 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.614 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912270+0000", 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912273+0000", 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6929999999999999e-06 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921147+0000", 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088735250000000002 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921185+0000", 2024-04-25T15:53:44.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8098e-05 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921200+0000", 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5543999999999999e-05 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921281+0000", 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0758000000000002e-05 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921426+0000", 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014529000000000001 2024-04-25T15:53:44.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921540+0000", 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113465 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.942585+0000", 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021045735999999999 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977215+0000", 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034629659 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977282+0000", 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6796000000000002e-05 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977354+0000", 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2350999999999999e-05 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.619 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:675 3.a e81/76 3:5122fcae:::100000001f7.0000009f:head v 81'4, mlcod=61'1)", 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.650483897000001, 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047589770000000003, 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929830+0000", 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.929833+0000", 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2850000000000001e-06 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931776+0000", 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001943443 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931806+0000", 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0074e-05 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931829+0000", 2024-04-25T15:53:44.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3027000000000002e-05 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931927+0000", 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7540000000000005e-05 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931959+0000", 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2265999999999999e-05 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977374+0000", 2024-04-25T15:53:44.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045414643999999997 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977420+0000", 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6490999999999999e-05 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:676 3.18 e81/76 3:191fe66d:::100000001f7.000000a0:head v 81'8, mlcod=71'6)", 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.648639329000002, 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050752358999999997, 2024-04-25T15:53:44.624 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931675+0000", 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.931678+0000", 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.542e-06 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935891+0000", 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042133930000000002 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935917+0000", 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5624000000000001e-05 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935935+0000", 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8258999999999999e-05 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936019+0000", 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4538999999999998e-05 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.936060+0000", 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1060000000000003e-05 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982347+0000", 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046286542999999999 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982427+0000", 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0399000000000001e-05 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:678 3.1b 3:da348c2f:::100000001f7.000000a2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:44.628 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.637271260999999, 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039665855999999999, 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 678 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:44.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943043+0000", 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.943045+0000", 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9630000000000001e-06 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947273+0000", 2024-04-25T15:53:44.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042283470000000004 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947307+0000", 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3794000000000002e-05 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947322+0000", 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5184999999999999e-05 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947403+0000", 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1076999999999996e-05 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947533+0000", 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012934399999999999 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:44.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.947646+0000", 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113163 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.964158+0000", 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016512526 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982578+0000", 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018419333 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982646+0000", 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8010999999999996e-05 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.982709+0000", 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1004 3.1d e82/76 3:b93a1278:::100000001f7.000000e8:head v 82'29, mlcod=82'27)", 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:44.634 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.056331474, 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.044544315000000001, 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523983+0000", 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523985+0000", 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9999999999999999e-06 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527244+0000", 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032587330000000002 2024-04-25T15:53:44.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527284+0000", 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0031999999999999e-05 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527309+0000", 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5157e-05 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527437+0000", 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128384 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.527480+0000", 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2357999999999997e-05 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568424+0000", 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040944332 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568527+0000", 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103319 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.638 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1008 3.2c e82/76 3:3798f9d7:::100000001f7.000000ec:head v 82'14, mlcod=82'12)", 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.041916616999998, 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042823282999999997, 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:44.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538398+0000", 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.538400+0000", 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0650000000000001e-06 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541298+0000", 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028986440000000001 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541324+0000", 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5176000000000001e-05 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541339+0000", 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5821999999999999e-05 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541469+0000", 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.541521+0000", 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1882999999999998e-05 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581098+0000", 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039576885999999999 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.581221+0000", 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012303699999999999 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1074 3.1b 3:dbd503eb:::100000001f8.0000002e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:40.791635+0000", 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.7886792529999997, 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041608973, 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.643 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1074 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.791635+0000", 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.791635+0000", 2024-04-25T15:53:44.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.791638+0000", 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4540000000000001e-06 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.811746+0000", 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020107979000000002 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.811808+0000", 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2932e-05 2024-04-25T15:53:44.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.811836+0000", 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7163000000000001e-05 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.811951+0000", 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115213 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.812153+0000", 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020230500000000001 2024-04-25T15:53:44.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.812327+0000", 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017395999999999999 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.824378+0000", 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01205091 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.833102+0000", 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087238750000000007 2024-04-25T15:53:44.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.833176+0000", 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3693000000000001e-05 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:40.833244+0000", 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8489000000000001e-05 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.648 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1466 3.14 3:28884a71:::100000001f8.000001b6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:43.441455+0000", 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.138859243, 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038117805999999997, 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1466 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.649 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.441455+0000", 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.441455+0000", 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.441459+0000", 2024-04-25T15:53:44.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.963e-06 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.450256+0000", 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087966499999999996 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.450317+0000", 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0995999999999997e-05 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.450341+0000", 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4443e-05 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.450493+0000", 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152275 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.450803+0000", 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030949800000000001 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.451066+0000", 2024-04-25T15:53:44.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026273799999999999 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.471702+0000", 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020635961000000001 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.479466+0000", 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077645179999999998 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.479523+0000", 2024-04-25T15:53:44.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7043999999999998e-05 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.479573+0000", 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9719999999999998e-05 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1517 3.31 3:8f866260:::100000001f8.000001e9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:43.795755+0000", 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.78455929899999999, 2024-04-25T15:53:44.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041031008000000001, 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1517 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.795755+0000", 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.795755+0000", 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.795758+0000", 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6630000000000002e-06 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.805627+0000", 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0098697060000000007 2024-04-25T15:53:44.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.809084+0000", 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034567700000000001 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.809100+0000", 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6240999999999999e-05 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.809186+0000", 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5332000000000002e-05 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.809353+0000", 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167231 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.809501+0000", 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014776599999999999 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.835396+0000", 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025895609 2024-04-25T15:53:44.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.836647+0000", 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012505019999999999 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.836723+0000", 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6440999999999995e-05 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.836786+0000", 2024-04-25T15:53:44.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2747000000000001e-05 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1530 3.34 3:2eaad5e5:::100000001f8.000001f6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:43.874853+0000", 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.70546173000000001, 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041429367000000002, 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.660 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1530 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.874853+0000", 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.874853+0000", 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.874855+0000", 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1330000000000002e-06 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.887725+0000", 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012870553 2024-04-25T15:53:44.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.887815+0000", 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9493000000000005e-05 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.887867+0000", 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1867999999999999e-05 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.887981+0000", 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114157 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.888217+0000", 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023647800000000001 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.888417+0000", 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019993800000000001 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.914601+0000", 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026183769999999999 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.916152+0000", 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001550889 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.916211+0000", 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9388000000000001e-05 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.916282+0000", 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0699999999999997e-05 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1541 3.9 3:9267f08d:::100000001f8.00000201:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:43.933183+0000", 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.64713140400000002, 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045729677000000003, 2024-04-25T15:53:44.666 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1541 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.933183+0000", 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.933183+0000", 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.933186+0000", 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7269999999999999e-06 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.939686+0000", 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065003500000000002 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.939743+0000", 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7377999999999999e-05 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.939772+0000", 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8520000000000001e-05 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.939877+0000", 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104781 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.940052+0000", 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017564300000000001 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.940245+0000", 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019228999999999999 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.948959+0000", 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0087143659999999994 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.978816+0000", 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029856673 2024-04-25T15:53:44.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.978867+0000", 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1616999999999997e-05 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:43.978913+0000", 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5331999999999999e-05 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1558 3.31 3:8df5fc08:::100000001f8.00000212:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:44.672 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.048140+0000", 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.53217441200000004, 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041902355000000002, 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1558 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.048140+0000", 2024-04-25T15:53:44.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.048140+0000", 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.048147+0000", 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5120000000000002e-06 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.055950+0000", 2024-04-25T15:53:44.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0078029559999999998 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.055986+0000", 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5463000000000002e-05 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.056001+0000", 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4754e-05 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.056091+0000", 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9883000000000007e-05 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.056269+0000", 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178321 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.056455+0000", 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000185737 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.078557+0000", 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.022102429999999999 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.089884+0000", 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01132738 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.089961+0000", 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6310999999999995e-05 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.090042+0000", 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1607999999999995e-05 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1589 3.1a 3:590c0ff7:::100000001f8.00000231:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.238979+0000", 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.34133519200000001, 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039420080000000003, 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:44.678 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1589 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.238979+0000", 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.238979+0000", 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:44.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.238981+0000", 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.564e-06 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.245243+0000", 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062625329999999998 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.245273+0000", 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0219999999999999e-05 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.245285+0000", 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2006e-05 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.245360+0000", 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4181e-05 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.245485+0000", 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012510000000000001 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.245592+0000", 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107599 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.251932+0000", 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063401170000000001 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.278274+0000", 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026341502999999999 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.278335+0000", 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1425000000000002e-05 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.278399+0000", 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3831999999999995e-05 2024-04-25T15:53:44.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.684 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.684 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.684 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:44.684 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:44.684 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:44.686 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:44.686 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "" 2024-04-25T15:53:44.686 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:44.686 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:44.687 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:44.683+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:44.787 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:44.783+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:44.887 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:44.887+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:44.928 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:44.928 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:44.928 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:44.928 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:44.964 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:44.988 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:44.986+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:45.089 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:45.087+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:45.189 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:45.189 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:45.189 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:45.189 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:45.189 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:45.190 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:45.187+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:45.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:45.290 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:45.290+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:45.390 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:45.390+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:45.455 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:45.455 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:45.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:45.455 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:45.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.455 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:45.455 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.233374992999998, 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:45.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:53:45.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:45.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:53:45.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:53:45.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:596 3.16 3:6941c83d:::100000001f7.00000050:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:45.462 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 22.090483064000001, 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049183123000000002, 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 596 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361919+0000", 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.361922+0000", 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2030000000000001e-06 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367484+0000", 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055618409999999997 2024-04-25T15:53:45.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367522+0000", 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8260000000000003e-05 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367543+0000", 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0795999999999999e-05 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367646+0000", 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102913 2024-04-25T15:53:45.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.367857+0000", 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021190400000000001 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.368091+0000", 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023323699999999999 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.405516+0000", 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037425424999999998 2024-04-25T15:53:45.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.410955+0000", 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054389859999999998 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411034+0000", 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8421e-05 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.411102+0000", 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8137000000000002e-05 2024-04-25T15:53:45.467 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:665 3.1d 3:b87c769f:::100000001f7.00000095:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.611378024, 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.045554887000000002, 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.468 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 665 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841024+0000", 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.841029+0000", 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9359999999999998e-06 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.851995+0000", 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010966115 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852062+0000", 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6975000000000002e-05 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852084+0000", 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2382999999999999e-05 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852218+0000", 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013397699999999999 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852418+0000", 2024-04-25T15:53:45.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000200161 2024-04-25T15:53:45.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:45.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.852550+0000", 2024-04-25T15:53:45.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013199300000000001 2024-04-25T15:53:45.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878731+0000", 2024-04-25T15:53:45.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026181157 2024-04-25T15:53:45.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886405+0000", 2024-04-25T15:53:45.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076735090000000002 2024-04-25T15:53:45.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886503+0000", 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8504000000000004e-05 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886578+0000", 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5177000000000002e-05 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:664 3.d e81/18 3:b353c7f2:::100000001f7.00000094:head v 81'8, mlcod=80'5)", 2024-04-25T15:53:45.475 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.583079124000001, 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051411299000000001, 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869322+0000", 2024-04-25T15:53:45.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.869325+0000", 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1689999999999999e-06 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872141+0000", 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028160759999999998 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872176+0000", 2024-04-25T15:53:45.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4996999999999997e-05 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872202+0000", 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5944000000000001e-05 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872312+0000", 2024-04-25T15:53:45.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010998200000000001 2024-04-25T15:53:45.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872351+0000", 2024-04-25T15:53:45.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9360999999999998e-05 2024-04-25T15:53:45.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920645+0000", 2024-04-25T15:53:45.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048293989000000002 2024-04-25T15:53:45.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920734+0000", 2024-04-25T15:53:45.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8781000000000003e-05 2024-04-25T15:53:45.481 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.481 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:671 3.38 e81/18 3:1c0639bf:::100000001f7.0000009b:head v 81'8, mlcod=81'6)", 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.565754915999999, 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069441491999999994, 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886668+0000", 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0876000000000001e-05 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893242+0000", 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065744849999999997 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893267+0000", 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5321e-05 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893283+0000", 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5454000000000001e-05 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893362+0000", 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8770000000000006e-05 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893382+0000", 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0585999999999999e-05 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.955934+0000", 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062552041000000003 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956088+0000", 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:673 3.28 3:153e0bf4:::100000001f7.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.547465627000001, 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063190614000000006, 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.486 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 673 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904939+0000", 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8739999999999999e-06 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911908+0000", 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069695449999999997 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911944+0000", 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5973000000000003e-05 2024-04-25T15:53:45.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911960+0000", 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5319999999999999e-05 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912057+0000", 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7427000000000001e-05 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912221+0000", 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016433600000000001 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912375+0000", 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015359299999999999 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.941806+0000", 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029430523 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967997+0000", 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026191589000000001 2024-04-25T15:53:45.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968063+0000", 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6321000000000004e-05 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968127+0000", 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:652 3.36 e81/18 3:6c98fff9:::100000001f7.00000088:head v 81'12, mlcod=69'10)", 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.536268866, 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052156028, 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916133+0000", 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.916135+0000", 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7850000000000001e-06 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918034+0000", 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018996099999999999 2024-04-25T15:53:45.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918080+0000", 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5469000000000001e-05 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918098+0000", 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8726999999999999e-05 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918176+0000", 2024-04-25T15:53:45.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8125000000000002e-05 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.918198+0000", 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1531000000000001e-05 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968240+0000", 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050041583000000001 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968289+0000", 2024-04-25T15:53:45.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9197999999999999e-05 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:677 3.5 3:a07f2395:::100000001f7.000000a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.516887143000002, 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041485695000000003, 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.496 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 677 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935514+0000", 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.935657+0000", 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014239100000000001 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940003+0000", 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043458239999999999 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940055+0000", 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2021999999999998e-05 2024-04-25T15:53:45.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940071+0000", 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6528000000000002e-05 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940148+0000", 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7179000000000001e-05 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940294+0000", 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014605799999999999 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.940418+0000", 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012356499999999999 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956756+0000", 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016338149999999999 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976852+0000", 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020095572999999999 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.976925+0000", 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3351999999999998e-05 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977000+0000", 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5053000000000001e-05 2024-04-25T15:53:45.501 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1003 3.28 3:1719c8f9:::100000001f7.000000e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.944834809, 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062620618000000003, 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.502 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1003 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507567+0000", 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.507569+0000", 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2110000000000001e-06 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519064+0000", 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011495182 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521838+0000", 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027741129999999999 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521856+0000", 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.521967+0000", 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110913 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522100+0000", 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013309899999999999 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:45.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.522235+0000", 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013518199999999999 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.543023+0000", 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020788025000000002 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570076+0000", 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027052208000000001 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570138+0000", 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2376e-05 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.570187+0000", 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9379000000000003e-05 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1004 3.1d 3:b93a1278:::100000001f7.000000e8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:45.507 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.933298355000002, 2024-04-25T15:53:45.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049745623000000003, 2024-04-25T15:53:45.508 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.508 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.508 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.508 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.508 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.509 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.509 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1004 2024-04-25T15:53:45.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.509 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:45.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519103+0000", 2024-04-25T15:53:45.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.519106+0000", 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.464e-06 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523205+0000", 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040996339999999996 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523267+0000", 2024-04-25T15:53:45.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1660000000000003e-05 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523292+0000", 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5194999999999998e-05 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523428+0000", 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013594 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523595+0000", 2024-04-25T15:53:45.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016733300000000001 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.523819+0000", 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022358999999999999 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.566513+0000", 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042694339999999997 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568719+0000", 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022057230000000001 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568799+0000", 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0370000000000005e-05 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:28.568849+0000", 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9373999999999999e-05 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1649 3.37 e89/21 3:ec163160:::100000001f8.0000026d:head v 89'38, mlcod=88'36)", 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.632298+0000", 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.82010399300000003, 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041752834000000003, 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.632298+0000", 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.632298+0000", 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.632303+0000", 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1370000000000001e-06 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.639365+0000", 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070626630000000003 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.639384+0000", 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8580000000000002e-05 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.639397+0000", 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2694000000000001e-05 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.639470+0000", 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2828000000000001e-05 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.639495+0000", 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5420000000000001e-05 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.673965+0000", 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034469939999999998 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.674050+0000", 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5572e-05 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.519 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1686 3.16 3:684e0030:::100000001f8.00000292:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.849621+0000", 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.60278075099999995, 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062057172000000001, 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1686 2024-04-25T15:53:45.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.849621+0000", 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.849621+0000", 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.849623+0000", 2024-04-25T15:53:45.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1299999999999999e-06 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.856379+0000", 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006755647 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.856413+0000", 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4100000000000002e-05 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.856429+0000", 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5820000000000001e-05 2024-04-25T15:53:45.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.856523+0000", 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4160999999999995e-05 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.856701+0000", 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017813700000000001 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.856857+0000", 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015621999999999999 2024-04-25T15:53:45.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.869519+0000", 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012661522 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.911565+0000", 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042046761000000002 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.911628+0000", 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2426000000000002e-05 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.911678+0000", 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0247999999999997e-05 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1690 3.3a 3:5d7dbe56:::100000001f8.00000296:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.876313+0000", 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.57608904599999999, 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041201847, 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.525 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1690 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.876313+0000", 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.876313+0000", 2024-04-25T15:53:45.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.876315+0000", 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.734e-06 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.882186+0000", 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058702629999999997 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.882235+0000", 2024-04-25T15:53:45.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9091000000000001e-05 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.882257+0000", 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2169999999999999e-05 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.882359+0000", 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010210299999999999 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.882560+0000", 2024-04-25T15:53:45.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201125 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.882726+0000", 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000166038 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.899008+0000", 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016281539000000001 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.917300+0000", 2024-04-25T15:53:45.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018292772999999998 2024-04-25T15:53:45.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.917390+0000", 2024-04-25T15:53:45.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0018000000000004e-05 2024-04-25T15:53:45.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.917514+0000", 2024-04-25T15:53:45.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123993 2024-04-25T15:53:45.532 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.532 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.532 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1699 3.2f 3:f7a50725:::100000001f8.0000029f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.934543+0000", 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.517858822, 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067346198999999995, 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1699 2024-04-25T15:53:45.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.934543+0000", 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.934543+0000", 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.934547+0000", 2024-04-25T15:53:45.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2270000000000004e-06 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939709+0000", 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051621649999999998 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939765+0000", 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5924999999999997e-05 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939789+0000", 2024-04-25T15:53:45.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4383e-05 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939913+0000", 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123738 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.940118+0000", 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020446799999999999 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.940324+0000", 2024-04-25T15:53:45.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206582 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.000408+0000", 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.060084151000000002 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.001776+0000", 2024-04-25T15:53:45.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013679359999999999 2024-04-25T15:53:45.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.001824+0000", 2024-04-25T15:53:45.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7366e-05 2024-04-25T15:53:45.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.001889+0000", 2024-04-25T15:53:45.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5258000000000004e-05 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1703 3.30 3:0f89dfab:::100000001f8.000002a3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.955530+0000", 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.49687205499999998, 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052387459999999997, 2024-04-25T15:53:45.540 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1703 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.955530+0000", 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.955530+0000", 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.955532+0000", 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.961008+0000", 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054759029999999998 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.961039+0000", 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0814e-05 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.961050+0000", 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0693000000000001e-05 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.961115+0000", 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5153000000000005e-05 2024-04-25T15:53:45.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.961296+0000", 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018147899999999999 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.961481+0000", 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018508500000000001 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.977107+0000", 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015625757000000001 2024-04-25T15:53:45.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.007790+0000", 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030683044 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.007857+0000", 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7041999999999999e-05 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.007917+0000", 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9670000000000003e-05 2024-04-25T15:53:45.545 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1702 3.36 e89/18 3:6ddcb6ef:::100000001f8.000002a2:head v 89'40, mlcod=88'38)", 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.957792+0000", 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.49460927199999999, 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049898221, 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.957792+0000", 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.957792+0000", 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.957795+0000", 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3700000000000002e-06 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.959907+0000", 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002112077 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.959931+0000", 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3849000000000002e-05 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.959945+0000", 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4817999999999999e-05 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.960028+0000", 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2905000000000007e-05 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.960071+0000", 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3105000000000001e-05 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.007608+0000", 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047536160000000001 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.007691+0000", 2024-04-25T15:53:45.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2936999999999996e-05 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1704 3.4 e89/68 3:230cd4d1:::100000001f8.000002a4:head v 89'22, mlcod=88'20)", 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.966412+0000", 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.48598939299999999, 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041560747000000002, 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.550 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.966412+0000", 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.966412+0000", 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.966413+0000", 2024-04-25T15:53:45.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2550000000000001e-06 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.970239+0000", 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003825429 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.970257+0000", 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7812000000000002e-05 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.970271+0000", 2024-04-25T15:53:45.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4447000000000001e-05 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.970368+0000", 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6537000000000001e-05 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.970401+0000", 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3114000000000002e-05 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.007912+0000", 2024-04-25T15:53:45.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.037511550999999997 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.007973+0000", 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0602e-05 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1764 3.2f 3:f60afc40:::100000001f8.000002e0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:45.328940+0000", 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.12346156599999999, 2024-04-25T15:53:45.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.049750859000000001, 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1764 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.328940+0000", 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.328940+0000", 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.328943+0000", 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2169999999999999e-06 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.333675+0000", 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047313779999999996 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.333706+0000", 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1343999999999997e-05 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.333719+0000", 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3372e-05 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.333810+0000", 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0493000000000002e-05 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.333945+0000", 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013565499999999999 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.334076+0000", 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130254 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.375467+0000", 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041391430999999999 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.378553+0000", 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030861579999999999 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.378626+0000", 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3034999999999994e-05 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.378691+0000", 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4522000000000003e-05 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.559 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1767 3.38 e89/18 3:1fae2fe4:::100000001f8.000002e3:head v 89'43, mlcod=89'41)", 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:45.346241+0000", 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.106160561, 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.043380136, 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.346241+0000", 2024-04-25T15:53:45.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.346241+0000", 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.346245+0000", 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9879999999999998e-06 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.351068+0000", 2024-04-25T15:53:45.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048233529999999998 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.351090+0000", 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1812999999999999e-05 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.351105+0000", 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4662e-05 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.351222+0000", 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116709 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.351246+0000", 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4227e-05 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.389388+0000", 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038142559999999999 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.389621+0000", 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023282399999999999 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:1769 3.35 e89/18 3:ae7a07fd:::100000001f8.000002e5:head v 89'35, mlcod=89'33)", 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:45.359383+0000", 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.093018201999999994, 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041599214000000002, 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:45.564 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.359383+0000", 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.359383+0000", 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.359385+0000", 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.64e-06 2024-04-25T15:53:45.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.362483+0000", 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030978020000000002 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.362518+0000", 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5061000000000002e-05 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.362537+0000", 2024-04-25T15:53:45.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9245999999999998e-05 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.362638+0000", 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100583 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.362677+0000", 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8810000000000003e-05 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.400925+0000", 2024-04-25T15:53:45.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038248580999999997 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.400983+0000", 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7491000000000002e-05 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:45.568 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:45.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:45.570 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:45.570+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:45.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:45.670 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:45.670+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:45.755 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:45.755 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:45.756 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:45.756 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:45.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.758+0000 7ffab468e640 1 Processor -- start 2024-04-25T15:53:45.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.758+0000 7ffab468e640 1 -- start start 2024-04-25T15:53:45.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab468e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ffaac06c640 0x7ffaac15b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab468e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffaac074780 0x7ffaac164f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab468e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaac0c92d0 con 0x7ffaac074370 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab468e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac074b90 con 0x7ffaac074780 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab468e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac15b560 con 0x7ffaac06c640 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab1c29640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffaac074370 0x7ffaac0c7fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35564/0 (socket says 172.21.15.67:35564) 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab1c29640 1 -- 172.21.15.67:0/301276650 learned_addr learned my addr 172.21.15.67:0/301276650 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/301276650 >> v2:172.21.15.67:3300/0 conn(0x7ffaac074780 0x7ffaac164f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab242a640 1 --2- 172.21.15.67:0/301276650 >> v2:172.21.15.176:3300/0 conn(0x7ffaac06c640 0x7ffaac15b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab1428640 1 -- 172.21.15.67:0/301276650 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562114989 0 0) 0x7ffaac0c92d0 con 0x7ffaac074370 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab1428640 1 -- 172.21.15.67:0/301276650 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa98003530 con 0x7ffaac074370 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab2c2b640 1 -- 172.21.15.67:0/301276650 >> v2:172.21.15.176:3300/0 conn(0x7ffaac06c640 msgr2=0x7ffaac15b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/301276650 >> v2:172.21.15.176:3300/0 conn(0x7ffaac06c640 0x7ffaac15b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab2c2b640 1 -- 172.21.15.67:0/301276650 >> v1:172.21.15.67:6789/0 conn(0x7ffaac074370 legacy=0x7ffaac0c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab2c2b640 1 -- 172.21.15.67:0/301276650 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac06e210 con 0x7ffaac074780 2024-04-25T15:53:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.762+0000 7ffab242a640 1 --2- 172.21.15.67:0/301276650 >> v2:172.21.15.176:3300/0 conn(0x7ffaac06c640 0x7ffaac15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/301276650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac074780 0x7ffaac164f50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0c8c00 tx=0x7ffaa401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c2bea111db3b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab1428640 1 -- 172.21.15.67:0/301276650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40b4070 con 0x7ffaac074780 2024-04-25T15:53:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab1428640 1 -- 172.21.15.67:0/301276650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa40af600 con 0x7ffaac074780 2024-04-25T15:53:45.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab1428640 1 -- 172.21.15.67:0/301276650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40af9b0 con 0x7ffaac074780 2024-04-25T15:53:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 -- 172.21.15.67:0/301276650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac074780 msgr2=0x7ffaac164f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 --2- 172.21.15.67:0/301276650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac074780 0x7ffaac164f50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0c8c00 tx=0x7ffaa401f110 comp rx=0 tx=0).stop 2024-04-25T15:53:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 -- 172.21.15.67:0/301276650 shutdown_connections 2024-04-25T15:53:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 --2- 172.21.15.67:0/301276650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac074780 0x7ffaac164f50 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 --2- 172.21.15.67:0/301276650 >> v2:172.21.15.176:3300/0 conn(0x7ffaac06c640 0x7ffaac15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 -- 172.21.15.67:0/301276650 >> 172.21.15.67:0/301276650 conn(0x7ffaac070db0 msgr2=0x7ffaac0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 -- 172.21.15.67:0/301276650 shutdown_connections 2024-04-25T15:53:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.766+0000 7ffab468e640 1 -- 172.21.15.67:0/301276650 wait complete. 2024-04-25T15:53:45.770 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:45.770+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:45.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.770+0000 7ffab468e640 1 Processor -- start 2024-04-25T15:53:45.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.770+0000 7ffab468e640 1 -- start start 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac06c640 0x7ffaac0c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaac074370 0x7ffaac150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaac074780 0x7ffaac1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac073890 con 0x7ffaac06c640 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac06e7b0 con 0x7ffaac074370 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac073b70 con 0x7ffaac074780 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac06c640 0x7ffaac0c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac06c640 0x7ffaac0c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43726/0 (socket says 172.21.15.67:43726) 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 -- 172.21.15.67:0/2942527344 learned_addr learned my addr 172.21.15.67:0/2942527344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab1c29640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaac074370 0x7ffaac150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaac074780 0x7ffaac1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaac074780 msgr2=0x7ffaac1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaac074780 0x7ffaac1592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaac074370 msgr2=0x7ffaac150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffaac074370 0x7ffaac150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac0c6f00 con 0x7ffaac06c640 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac06c640 0x7ffaac0c69c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d800 tx=0x7ffaa800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8018020 con 0x7ffaac06c640 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa80096a0 con 0x7ffaac06c640 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac1514b0 con 0x7ffaac06c640 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8009a90 con 0x7ffaac06c640 2024-04-25T15:53:45.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.774+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac0c5ee0 con 0x7ffaac06c640 2024-04-25T15:53:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ffaac06a480 con 0x7ffaac06c640 2024-04-25T15:53:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ffaa801b020 con 0x7ffaac06c640 2024-04-25T15:53:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffaa2ffd640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffa9006f400 0x7ffa90071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5944+0+0 (secure 0 0 0) 0x7ffaa808f850 con 0x7ffaac06c640 2024-04-25T15:53:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v89) ==== 54+0+16 (secure 0 0 0) 0x7ffaa8022020 con 0x7ffaac06c640 2024-04-25T15:53:45.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffab1c29640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffa9006f400 0x7ffa90071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ffaac15b790 0x7ffaac0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffaac1f3820 con 0x7ffaac15b790 2024-04-25T15:53:45.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.778+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ffaac15b790 0x7ffaac0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.782+0000 7ffab1c29640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffa9006f400 0x7ffa90071910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0ce1d0 tx=0x7ffa9c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.782+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ffaac15b790 0x7ffaac0c6120 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.782+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac1f3820 con 0x7ffaac15b790 2024-04-25T15:53:45.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:45.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.806+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac151c20 con 0x7ffaac15b790 2024-04-25T15:53:45.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.810+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac151c20 con 0x7ffaac15b790 2024-04-25T15:53:45.810 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.810+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ffaac15a360 0x7ffaac1f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.810+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ffaac0cb070 con 0x7ffaac15a360 2024-04-25T15:53:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.810+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ffaac15a360 0x7ffaac1f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.810+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ffaac15a360 0x7ffaac1f5de0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.810+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac0cb070 con 0x7ffaac15a360 2024-04-25T15:53:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.834+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac0ce2f0 con 0x7ffaac15a360 2024-04-25T15:53:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.834+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac0ce2f0 con 0x7ffaac15a360 2024-04-25T15:53:45.835 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.834+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ffaac1f6830 0x7ffaac063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.834+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ffaac1f6c40 con 0x7ffaac1f6830 2024-04-25T15:53:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.834+0000 7ffab1c29640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ffaac1f6830 0x7ffaac063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.834+0000 7ffab1c29640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ffaac1f6830 0x7ffaac063090 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.838+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac1f6c40 con 0x7ffaac1f6830 2024-04-25T15:53:45.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.858+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac0638a0 con 0x7ffaac1f6830 2024-04-25T15:53:45.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.858+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac0638a0 con 0x7ffaac1f6830 2024-04-25T15:53:45.860 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.858+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ffaac0c99e0 0x7ffaac0c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.858+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ffaac0ca330 con 0x7ffaac0c99e0 2024-04-25T15:53:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.862+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ffaac0c99e0 0x7ffaac0c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.862+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ffaac0c99e0 0x7ffaac0c9df0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.862+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac0ca330 con 0x7ffaac0c99e0 2024-04-25T15:53:45.871 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:45.870+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:45.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.882+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac0ca330 con 0x7ffaac0c99e0 2024-04-25T15:53:45.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.886+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac0ca330 con 0x7ffaac0c99e0 2024-04-25T15:53:45.887 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.886+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ffaac02cef0 0x7ffaac02d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.886+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ffaac02d840 con 0x7ffaac02cef0 2024-04-25T15:53:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.886+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ffaac02cef0 0x7ffaac02d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.886+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ffaac02cef0 0x7ffaac02d300 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.890+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac02d840 con 0x7ffaac02cef0 2024-04-25T15:53:45.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.910+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac02d840 con 0x7ffaac02cef0 2024-04-25T15:53:45.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.914+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac02d840 con 0x7ffaac02cef0 2024-04-25T15:53:45.916 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:45.917 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:45.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.914+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ffaac2071a0 0x7ffaac2095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.914+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ffaac20c390 con 0x7ffaac2071a0 2024-04-25T15:53:45.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.914+0000 7ffab1c29640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ffaac2071a0 0x7ffaac2095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.918+0000 7ffab1c29640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ffaac2071a0 0x7ffaac2095c0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.918+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac20c390 con 0x7ffaac2071a0 2024-04-25T15:53:45.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.938+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac209b00 con 0x7ffaac2071a0 2024-04-25T15:53:45.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.938+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac209b00 con 0x7ffaac2071a0 2024-04-25T15:53:45.940 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:45.941 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:45.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.938+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ffaac20a180 0x7ffaac20c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.938+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ffaac20f370 con 0x7ffaac20a180 2024-04-25T15:53:45.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.942+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ffaac20a180 0x7ffaac20c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.942+0000 7ffab2c2b640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ffaac20a180 0x7ffaac20c5a0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.942+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac20f370 con 0x7ffaac20a180 2024-04-25T15:53:45.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.962+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac20cae0 con 0x7ffaac20a180 2024-04-25T15:53:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.970+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac20cae0 con 0x7ffaac20a180 2024-04-25T15:53:45.969 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.970+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ffaac20d160 0x7ffaac20f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.970+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ffaac2124c0 con 0x7ffaac20d160 2024-04-25T15:53:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.970+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ffaac20d160 0x7ffaac20f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.970+0000 7ffab242a640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ffaac20d160 0x7ffaac20f580 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:45.971 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:45.970+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:45.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.970+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaac2124c0 con 0x7ffaac20d160 2024-04-25T15:53:45.973 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:45.973 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:45.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.990+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ffaac212490 con 0x7ffaac20d160 2024-04-25T15:53:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffaa2ffd640 1 -- 172.21.15.67:0/2942527344 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ffaac212490 con 0x7ffaac20d160 2024-04-25T15:53:45.994 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ffaac15b790 msgr2=0x7ffaac0c6120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ffaac15b790 0x7ffaac0c6120 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ffaac15a360 msgr2=0x7ffaac1f5de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ffaac15a360 0x7ffaac1f5de0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ffaac1f6830 msgr2=0x7ffaac063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7ffaac1f6830 0x7ffaac063090 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ffaac0c99e0 msgr2=0x7ffaac0c9df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7ffaac0c99e0 0x7ffaac0c9df0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ffaac02cef0 msgr2=0x7ffaac02d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7ffaac02cef0 0x7ffaac02d300 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ffaac2071a0 msgr2=0x7ffaac2095c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7ffaac2071a0 0x7ffaac2095c0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ffaac20a180 msgr2=0x7ffaac20c5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7ffaac20a180 0x7ffaac20c5a0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ffaac20d160 msgr2=0x7ffaac20f580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7ffaac20d160 0x7ffaac20f580 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab2c2b640 1 -- 172.21.15.67:0/2942527344 reap_dead start 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab2c2b640 1 -- 172.21.15.67:0/2942527344 reap_dead start 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab2c2b640 1 -- 172.21.15.67:0/2942527344 reap_dead start 2024-04-25T15:53:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab2c2b640 1 -- 172.21.15.67:0/2942527344 reap_dead start 2024-04-25T15:53:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab2c2b640 1 -- 172.21.15.67:0/2942527344 reap_dead start 2024-04-25T15:53:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffa9006f400 msgr2=0x7ffa90071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffa9006f400 0x7ffa90071910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0ce1d0 tx=0x7ffa9c009040 comp rx=0 tx=0).stop 2024-04-25T15:53:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac06c640 msgr2=0x7ffaac0c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac06c640 0x7ffaac0c69c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d800 tx=0x7ffaa800dcc0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 shutdown_connections 2024-04-25T15:53:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffa9006f400 0x7ffa90071910 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 --2- 172.21.15.67:0/2942527344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaac06c640 0x7ffaac0c69c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 >> 172.21.15.67:0/2942527344 conn(0x7ffaac070db0 msgr2=0x7ffaac1525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 shutdown_connections 2024-04-25T15:53:45.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:45.994+0000 7ffab468e640 1 -- 172.21.15.67:0/2942527344 wait complete. 2024-04-25T15:53:45.999 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:46.072 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:46.071+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:46.173 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:46.174+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:46.182 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:46.182 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:46.211 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:46.275 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:46.274+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:46.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:53:46.375 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:46.375+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:46.473 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:46.473 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:46.475 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:46.474+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:46.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57ca07640 1 Processor -- start 2024-04-25T15:53:46.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57ca07640 1 -- start start 2024-04-25T15:53:46.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57ca07640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5780736d0 0x7fd57806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57ca07640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5780caa60 con 0x7fd57806d460 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57ca07640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57815a5f0 con 0x7fd57806cb80 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57ca07640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd578068590 con 0x7fd5780736d0 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd575d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5780736d0 0x7fd57806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57659c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd57806d460 0x7fd5781547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35574/0 (socket says 172.21.15.67:35574) 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd575d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd5780736d0 0x7fd57806c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43728/0 (socket says 172.21.15.67:43728) 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57659c640 1 -- 172.21.15.67:0/1553988604 learned_addr learned my addr 172.21.15.67:0/1553988604 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57559a640 1 -- 172.21.15.67:0/1553988604 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146604971 0 0) 0x7fd5780caa60 con 0x7fd57806d460 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd57559a640 1 -- 172.21.15.67:0/1553988604 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd560003530 con 0x7fd57806d460 2024-04-25T15:53:46.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd575d9b640 1 -- 172.21.15.67:0/1553988604 >> v1:172.21.15.176:6789/0 conn(0x7fd57806cb80 legacy=0x7fd57815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd575d9b640 1 -- 172.21.15.67:0/1553988604 >> v1:172.21.15.67:6789/0 conn(0x7fd57806d460 legacy=0x7fd5781547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.506+0000 7fd575d9b640 1 -- 172.21.15.67:0/1553988604 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5780ca100 con 0x7fd5780736d0 2024-04-25T15:53:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd575d9b640 1 --2- 172.21.15.67:0/1553988604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5780736d0 0x7fd57806c640 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd578072690 tx=0x7fd56c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485a2ac50df47cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57559a640 1 -- 172.21.15.67:0/1553988604 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0b0070 con 0x7fd5780736d0 2024-04-25T15:53:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57559a640 1 -- 172.21.15.67:0/1553988604 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56c0ab600 con 0x7fd5780736d0 2024-04-25T15:53:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57559a640 1 -- 172.21.15.67:0/1553988604 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0ab9b0 con 0x7fd5780736d0 2024-04-25T15:53:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57ca07640 1 -- 172.21.15.67:0/1553988604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5780736d0 msgr2=0x7fd57806c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57ca07640 1 --2- 172.21.15.67:0/1553988604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5780736d0 0x7fd57806c640 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd578072690 tx=0x7fd56c01c400 comp rx=0 tx=0).stop 2024-04-25T15:53:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57ca07640 1 -- 172.21.15.67:0/1553988604 shutdown_connections 2024-04-25T15:53:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57ca07640 1 --2- 172.21.15.67:0/1553988604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5780736d0 0x7fd57806c640 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57ca07640 1 -- 172.21.15.67:0/1553988604 >> 172.21.15.67:0/1553988604 conn(0x7fd5780ce1c0 msgr2=0x7fd5780ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57ca07640 1 -- 172.21.15.67:0/1553988604 shutdown_connections 2024-04-25T15:53:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.510+0000 7fd57ca07640 1 -- 172.21.15.67:0/1553988604 wait complete. 2024-04-25T15:53:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 Processor -- start 2024-04-25T15:53:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 -- start start 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd57806cb80 0x7fd5780c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd57806d460 0x7fd5780c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5780736d0 0x7fd578148550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd578168920 con 0x7fd57806d460 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd5781687a0 con 0x7fd5780736d0 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57ca07640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd578168aa0 con 0x7fd57806cb80 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd57806d460 0x7fd5780c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd57806d460 0x7fd5780c56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43732/0 (socket says 172.21.15.67:43732) 2024-04-25T15:53:46.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 -- 172.21.15.67:0/647722434 learned_addr learned my addr 172.21.15.67:0/647722434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd57659c640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd57806cb80 0x7fd5780c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd576d9d640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5780736d0 0x7fd578148550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 -- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd57806cb80 msgr2=0x7fd5780c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd57806cb80 0x7fd5780c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 -- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5780736d0 msgr2=0x7fd578148550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5780736d0 0x7fd578148550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd575d9b640 1 -- 172.21.15.67:0/647722434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5780cd8f0 con 0x7fd57806d460 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.514+0000 7fd576d9d640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5780736d0 0x7fd578148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd575d9b640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd57806d460 0x7fd5780c56e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd56c01a030 tx=0x7fd56c0191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd566ffd640 1 -- 172.21.15.67:0/647722434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0b9030 con 0x7fd57806d460 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd566ffd640 1 -- 172.21.15.67:0/647722434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56c0ab400 con 0x7fd57806d460 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5780c5c20 con 0x7fd57806d460 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd566ffd640 1 -- 172.21.15.67:0/647722434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0b0040 con 0x7fd57806d460 2024-04-25T15:53:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd57814f8d0 con 0x7fd57806d460 2024-04-25T15:53:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd57806a480 con 0x7fd57806d460 2024-04-25T15:53:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd566ffd640 1 -- 172.21.15.67:0/647722434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd56c0bc070 con 0x7fd57806d460 2024-04-25T15:53:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd566ffd640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd55806f400 0x7fd558071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd566ffd640 1 -- 172.21.15.67:0/647722434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5944+0+0 (secure 0 0 0) 0x7fd56c129800 con 0x7fd57806d460 2024-04-25T15:53:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.518+0000 7fd57659c640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd55806f400 0x7fd558071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:46.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.522+0000 7fd57659c640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd55806f400 0x7fd558071910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd568000c10 tx=0x7fd568007c70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.526+0000 7fd566ffd640 1 -- 172.21.15.67:0/647722434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd56c015ff0 con 0x7fd57806d460 2024-04-25T15:53:46.599 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:46.599+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:46.677 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:46.675+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:46.743 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:46.743 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:46.773 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:46.777 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:46.778+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:46.778+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fd578150380 con 0x7fd57806d460 2024-04-25T15:53:46.878 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:46.878+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:46.979 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:46.978+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:47.018 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:47.018 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:47.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:47.080 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:47.079+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:47.182 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:47.179+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:47.239 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:47.239 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:47.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:47.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.310+0000 7fd566ffd640 1 -- 172.21.15.67:0/647722434 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v90) ==== 91+0+0 (secure 0 0 0) 0x7fd56c0f6700 con 0x7fd57806d460 2024-04-25T15:53:47.311 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:53:47.313 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd55806f400 msgr2=0x7fd558071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd55806f400 0x7fd558071910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd568000c10 tx=0x7fd568007c70 comp rx=0 tx=0).stop 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd57806d460 msgr2=0x7fd5780c56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd57806d460 0x7fd5780c56e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd56c01a030 tx=0x7fd56c0191a0 comp rx=0 tx=0).stop 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 shutdown_connections 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd55806f400 0x7fd558071910 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5780736d0 0x7fd578148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd57806d460 0x7fd5780c56e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 --2- 172.21.15.67:0/647722434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd57806cb80 0x7fd5780c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:47.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 >> 172.21.15.67:0/647722434 conn(0x7fd5780ce1c0 msgr2=0x7fd5780ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:47.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 shutdown_connections 2024-04-25T15:53:47.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:47.314+0000 7fd57ca07640 1 -- 172.21.15.67:0/647722434 wait complete. 2024-04-25T15:53:47.397 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:47.398+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:47.460 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:47.460 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:47.484 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:47.483+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:47.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:47.584 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:47.582+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:47.685 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:47.682+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:47.730 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:47.730 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:47.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:47.785 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:47.783+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:47.886 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:47.886+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:47.987 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:47.983+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:47.987 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:47.988 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:48.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:48.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:48.087 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:48.087+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:48.188 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:48.187+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.238+0000 7f3d42093640 1 Processor -- start 2024-04-25T15:53:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.238+0000 7f3d42093640 1 -- start start 2024-04-25T15:53:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d42093640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d3c160800 0x7f3d3c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d42093640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d42093640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3c06d8e0 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d42093640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d3c06dbc0 con 0x7f3d3c160800 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d42093640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c073c70 con 0x7f3d3c0c96e0 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d3c160800 0x7f3d3c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3b7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d3c160800 0x7f3d3c06d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43744/0 (socket says 172.21.15.67:43744) 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/776044893 learned_addr learned my addr 172.21.15.67:0/776044893 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d33fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d3c0c92d0 0x7f3d3c072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35578/0 (socket says 172.21.15.67:35578) 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/776044893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/776044893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 msgr2=0x7f3d3c167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/776044893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c167180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/776044893 >> v1:172.21.15.67:6789/0 conn(0x7f3d3c0c92d0 legacy=0x7f3d3c072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3affd640 1 -- 172.21.15.67:0/776044893 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3282606525 0 0) 0x7f3d3c06d8e0 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.242+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/776044893 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c073f50 con 0x7f3d3c160800 2024-04-25T15:53:48.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/776044893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c160800 0x7f3d3c06d030 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009870 tx=0x7f3d2c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7344be4cff4227bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d3affd640 1 -- 172.21.15.67:0/776044893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0b1070 con 0x7f3d3c160800 2024-04-25T15:53:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d3affd640 1 -- 172.21.15.67:0/776044893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c0ac620 con 0x7f3d3c160800 2024-04-25T15:53:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d3affd640 1 -- 172.21.15.67:0/776044893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0ac9f0 con 0x7f3d3c160800 2024-04-25T15:53:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 -- 172.21.15.67:0/776044893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c160800 msgr2=0x7f3d3c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 --2- 172.21.15.67:0/776044893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c160800 0x7f3d3c06d030 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009870 tx=0x7f3d2c01c4f0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 -- 172.21.15.67:0/776044893 shutdown_connections 2024-04-25T15:53:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 --2- 172.21.15.67:0/776044893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c167180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 --2- 172.21.15.67:0/776044893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c160800 0x7f3d3c06d030 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 -- 172.21.15.67:0/776044893 >> 172.21.15.67:0/776044893 conn(0x7f3d3c070db0 msgr2=0x7f3d3c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 -- 172.21.15.67:0/776044893 shutdown_connections 2024-04-25T15:53:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.246+0000 7f3d42093640 1 -- 172.21.15.67:0/776044893 wait complete. 2024-04-25T15:53:48.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 Processor -- start 2024-04-25T15:53:48.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 -- start start 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c0c92d0 0x7f3d3c15f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c15e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d3c160800 0x7f3d3c14a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c160c10 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c06d570 con 0x7f3d3c160800 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d42093640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c160d90 con 0x7f3d3c0c96e0 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c0c92d0 0x7f3d3c15f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c0c92d0 0x7f3d3c15f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43750/0 (socket says 172.21.15.67:43750) 2024-04-25T15:53:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/3939774749 learned_addr learned my addr 172.21.15.67:0/3939774749 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:48.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d3c160800 0x7f3d3c14a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c15e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 msgr2=0x7f3d3c15e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c15e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d3c160800 msgr2=0x7f3d3c14a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d3c160800 0x7f3d3c14a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d3b7fe640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c15eb70 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.250+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d3c0c96e0 0x7f3d3c15e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c0c92d0 0x7f3d3c15f780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c01a120 tx=0x7f3d2c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0b1040 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c004030 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c0c79d0 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0043e0 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c0c82d0 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.255 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:48.255 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3d20005280 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3d2c0b4070 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d38ff9640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d1c06f400 0x7f3d1c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.254+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5944+0+0 (secure 0 0 0) 0x7f3d2c129650 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.258+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v90) ==== 54+0+16 (secure 0 0 0) 0x7f3d2c0bb020 con 0x7f3d3c0c92d0 2024-04-25T15:53:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.258+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d1c06f400 0x7f3d1c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.258+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d1c06f400 0x7f3d1c071910 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c070000 tx=0x7f3d24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.258+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3d200018f0 0x7f3d20001d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.258+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d20002240 con 0x7f3d200018f0 2024-04-25T15:53:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.258+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3d200018f0 0x7f3d20001d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.262+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3d200018f0 0x7f3d20001d00 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.262+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d20002240 con 0x7f3d200018f0 2024-04-25T15:53:48.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.278+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d200029a0 con 0x7f3d200018f0 2024-04-25T15:53:48.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.282+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d200029a0 con 0x7f3d200018f0 2024-04-25T15:53:48.281 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.282+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3d20003460 0x7f3d20003870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.282+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3d20005040 con 0x7f3d20003460 2024-04-25T15:53:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.282+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3d20003460 0x7f3d20003870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.282+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3d20003460 0x7f3d20003870 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.282+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d20005040 con 0x7f3d20003460 2024-04-25T15:53:48.285 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:48.289 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:48.287+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:48.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.302+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d200040e0 con 0x7f3d20003460 2024-04-25T15:53:48.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.306+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d200040e0 con 0x7f3d20003460 2024-04-25T15:53:48.306 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:48.306 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:48.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.306+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3d20009c90 0x7f3d2000c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.306+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3d2000ee80 con 0x7f3d20009c90 2024-04-25T15:53:48.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.306+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3d20009c90 0x7f3d2000c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.306+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3d20009c90 0x7f3d2000c0b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.306+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d2000ee80 con 0x7f3d20009c90 2024-04-25T15:53:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.330+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d2000c980 con 0x7f3d20009c90 2024-04-25T15:53:48.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.330+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d2000c980 con 0x7f3d20009c90 2024-04-25T15:53:48.331 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:48.331 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:48.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.330+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3d2000cb30 0x7f3d2000cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.330+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3d2000d500 con 0x7f3d2000cb30 2024-04-25T15:53:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.330+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3d2000cb30 0x7f3d2000cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.330+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3d2000cb30 0x7f3d2000cf40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.334+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d2000d500 con 0x7f3d2000cb30 2024-04-25T15:53:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.350+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d2000da10 con 0x7f3d2000cb30 2024-04-25T15:53:48.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.354+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d2000da10 con 0x7f3d2000cb30 2024-04-25T15:53:48.354 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.354+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3d20011070 0x7f3d20013490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.354+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3d20016260 con 0x7f3d20011070 2024-04-25T15:53:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.354+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3d20011070 0x7f3d20013490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.354+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3d20011070 0x7f3d20013490 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.358+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d20016260 con 0x7f3d20011070 2024-04-25T15:53:48.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.382+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d200139d0 con 0x7f3d20011070 2024-04-25T15:53:48.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.386+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d200139d0 con 0x7f3d20011070 2024-04-25T15:53:48.387 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:48.387 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:48.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.386+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3d20014050 0x7f3d200144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.386+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3d200195f0 con 0x7f3d20014050 2024-04-25T15:53:48.388 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:48.387+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.390+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3d20014050 0x7f3d200144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.390+0000 7f3d33fff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3d20014050 0x7f3d200144a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.390+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d200195f0 con 0x7f3d20014050 2024-04-25T15:53:48.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.414+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d20014af0 con 0x7f3d20014050 2024-04-25T15:53:48.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.418+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d20014af0 con 0x7f3d20014050 2024-04-25T15:53:48.419 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:48.419 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:48.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.418+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3d20015210 0x7f3d20015660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.418+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f3d20015dd0 con 0x7f3d20015210 2024-04-25T15:53:48.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.418+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3d20015210 0x7f3d20015660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.418+0000 7f3d3bfff640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3d20015210 0x7f3d20015660 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.422+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d20015dd0 con 0x7f3d20015210 2024-04-25T15:53:48.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.442+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d20016010 con 0x7f3d20015210 2024-04-25T15:53:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.446+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d20016010 con 0x7f3d20015210 2024-04-25T15:53:48.447 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.446+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3d2001a990 0x7f3d2001cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.446+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3d2001fc00 con 0x7f3d2001a990 2024-04-25T15:53:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.446+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3d2001a990 0x7f3d2001cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:48.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.450+0000 7f3d3b7fe640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3d2001a990 0x7f3d2001cdb0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.450+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d2001fc00 con 0x7f3d2001a990 2024-04-25T15:53:48.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.470+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3d2001d370 con 0x7f3d2001a990 2024-04-25T15:53:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d38ff9640 1 -- 172.21.15.67:0/3939774749 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3d2001d370 con 0x7f3d2001a990 2024-04-25T15:53:48.476 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3d200018f0 msgr2=0x7f3d20001d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f3d200018f0 0x7f3d20001d00 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3d20003460 msgr2=0x7f3d20003870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f3d20003460 0x7f3d20003870 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3d20009c90 msgr2=0x7f3d2000c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3d20009c90 0x7f3d2000c0b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d3bfff640 1 -- 172.21.15.67:0/3939774749 reap_dead start 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3d2000cb30 msgr2=0x7f3d2000cf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3d2000cb30 0x7f3d2000cf40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3d20011070 msgr2=0x7f3d20013490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f3d20011070 0x7f3d20013490 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3d20014050 msgr2=0x7f3d200144a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.474+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f3d20014050 0x7f3d200144a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3d20015210 msgr2=0x7f3d20015660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f3d20015210 0x7f3d20015660 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3d2001a990 msgr2=0x7f3d2001cdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f3d2001a990 0x7f3d2001cdb0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d3bfff640 1 -- 172.21.15.67:0/3939774749 reap_dead start 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d1c06f400 msgr2=0x7f3d1c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d1c06f400 0x7f3d1c071910 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c070000 tx=0x7f3d24009040 comp rx=0 tx=0).stop 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c0c92d0 msgr2=0x7f3d3c15f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c0c92d0 0x7f3d3c15f780 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c01a120 tx=0x7f3d2c01ad30 comp rx=0 tx=0).stop 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 shutdown_connections 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d1c06f400 0x7f3d1c071910 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 --2- 172.21.15.67:0/3939774749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d3c0c92d0 0x7f3d3c15f780 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 >> 172.21.15.67:0/3939774749 conn(0x7f3d3c070db0 msgr2=0x7f3d3c151080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 shutdown_connections 2024-04-25T15:53:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:48.478+0000 7f3d42093640 1 -- 172.21.15.67:0/3939774749 wait complete. 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 186.240379711, 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.490 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:53:48.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:53:48.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:156 3.3e 3:7d47d675:::10000000002.00000098:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.928994+0000", 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 59.559120083000003, 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021932340000000002, 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.495 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 156 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.928994+0000", 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.928994+0000", 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.928997+0000", 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.933188+0000", 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041911789999999997 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.933251+0000", 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2334000000000003e-05 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.933275+0000", 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4405000000000001e-05 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.933426+0000", 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015126000000000001 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.933734+0000", 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030811499999999997 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:48.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.934041+0000", 2024-04-25T15:53:48.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00030668999999999998 2024-04-25T15:53:48.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950190+0000", 2024-04-25T15:53:48.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016148943999999998 2024-04-25T15:53:48.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950845+0000", 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000655084 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950882+0000", 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6668000000000002e-05 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.950926+0000", 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4181000000000003e-05 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.502 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:576 3.22 3:44d23391:::100000001f7.0000003c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.221297+0000", 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.266816556999999, 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024219094, 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 576 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.503 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.221297+0000", 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.221297+0000", 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.221300+0000", 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3280000000000002e-06 2024-04-25T15:53:48.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.223154+0000", 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001853658 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.223194+0000", 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0086000000000002e-05 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.223210+0000", 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5951e-05 2024-04-25T15:53:48.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.223299+0000", 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8857999999999995e-05 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op must wait for map", 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.223311+0000", 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2201999999999999e-05 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.228657+0000", 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053461840000000004 2024-04-25T15:53:48.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.228934+0000", 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00027652500000000001 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.229144+0000", 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021000000000000001 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.240798+0000", 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011653982 2024-04-25T15:53:48.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.245413+0000", 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0046150150000000001 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.245460+0000", 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7304e-05 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.245516+0000", 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6001000000000001e-05 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.508 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:644 3.22 3:477f9d5a:::100000001f7.00000080:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.698469+0000", 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.789644987999999, 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022179943000000001, 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 644 2024-04-25T15:53:48.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.698469+0000", 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.698469+0000", 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.698472+0000", 2024-04-25T15:53:48.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1250000000000001e-06 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.700264+0000", 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017921300000000001 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.700312+0000", 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7994999999999997e-05 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.700332+0000", 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0256999999999999e-05 2024-04-25T15:53:48.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.700419+0000", 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6403999999999995e-05 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.700566+0000", 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014753300000000001 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.700707+0000", 2024-04-25T15:53:48.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014052800000000001 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.708185+0000", 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074785110000000002 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.720510+0000", 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012324330999999999 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.720579+0000", 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9867e-05 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.720649+0000", 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9262000000000002e-05 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:676 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.558341380000002, 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053839007000000001, 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.514 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:53:48.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929776+0000", 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.743e-06 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931715+0000", 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019394130000000001 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931754+0000", 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8779000000000003e-05 2024-04-25T15:53:48.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931769+0000", 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5231999999999999e-05 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931867+0000", 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7485000000000006e-05 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932043+0000", 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017616100000000001 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932181+0000", 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013769100000000001 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.942677+0000", 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010495931 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983477+0000", 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040800621000000002 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983546+0000", 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9138000000000001e-05 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983611+0000", 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4813000000000004e-05 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:733 3.22 3:44586a4d:::100000001f7.000000d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:53:48.519 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.179875332000002, 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037465947999999999, 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 733 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308242+0000", 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.399e-06 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309826+0000", 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015840349999999999 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309863+0000", 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7122000000000001e-05 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309880+0000", 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6888000000000002e-05 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309986+0000", 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105979 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310169+0000", 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018349699999999999 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310353+0000", 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018415799999999999 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.342377+0000", 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032023169999999997 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345569+0000", 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031928680000000002 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345644+0000", 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4140999999999994e-05 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345704+0000", 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0690999999999999e-05 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:932 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.088886+0000", 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.399228031, 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028312902000000001, 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 932 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.088886+0000", 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.088886+0000", 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.088888+0000", 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4499999999999998e-06 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093536+0000", 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004647506 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093595+0000", 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9392999999999999e-05 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093617+0000", 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2062999999999999e-05 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093730+0000", 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011247399999999999 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093849+0000", 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011962499999999999 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.094001+0000", 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151404 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.106643+0000", 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01264216 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.117113+0000", 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010470405 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.117156+0000", 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2651000000000002e-05 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.117199+0000", 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2771000000000001e-05 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:962 3.22 3:470a77e8:::100000001f7.000000be:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.260688+0000", 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.227426055999999, 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022532144, 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.530 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 962 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.260688+0000", 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.260688+0000", 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.260690+0000", 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3889999999999999e-06 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.262606+0000", 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019161720000000001 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.262649+0000", 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3103999999999999e-05 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.262669+0000", 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9848e-05 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.262787+0000", 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011827799999999999 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.262888+0000", 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010107699999999999 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.263035+0000", 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014619200000000001 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.281793+0000", 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018758345999999999 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.283085+0000", 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012919610000000001 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.283155+0000", 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0370000000000006e-05 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.283220+0000", 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4406999999999995e-05 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1460 3.18 3:189ef23d:::100000001f8.000001b0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:48.535 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.396069+0000", 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.0920446410000002, 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020969261999999999, 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1460 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.396069+0000", 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.396069+0000", 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.396073+0000", 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.546e-06 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.397759+0000", 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016861860000000001 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.397801+0000", 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2491999999999999e-05 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.397821+0000", 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9921000000000001e-05 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.397920+0000", 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8723999999999999e-05 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.398105+0000", 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000184761 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.398279+0000", 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017434699999999999 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.407758+0000", 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0094790729999999993 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.416904+0000", 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0091454210000000008 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.416975+0000", 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1636000000000003e-05 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.417038+0000", 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3155000000000001e-05 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1515 3.18 3:1bff78ff:::100000001f8.000001e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.784293+0000", 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7038203049999998, 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025376673999999998, 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1515 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.784293+0000", 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.784293+0000", 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.784296+0000", 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8100000000000002e-06 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786418+0000", 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021213870000000002 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786473+0000", 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5640000000000003e-05 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786491+0000", 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.819e-05 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786669+0000", 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017772099999999999 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786881+0000", 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000212036 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.787039+0000", 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015808799999999999 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.801484+0000", 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014444477000000001 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.809535+0000", 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080517169999999999 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.809611+0000", 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5791000000000007e-05 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.809670+0000", 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8817000000000003e-05 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1543 3.32 3:4d482082:::100000001f8.00000203:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.947449+0000", 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.5406650050000001, 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026440832000000001, 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.546 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1543 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.947449+0000", 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.947449+0000", 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.947451+0000", 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3029999999999998e-06 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949063+0000", 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016119769999999999 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949101+0000", 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7978999999999997e-05 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949115+0000", 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4068e-05 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949187+0000", 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1627999999999999e-05 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949323+0000", 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013668299999999999 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:48.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949473+0000", 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014975000000000001 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973130+0000", 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023657114999999999 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973788+0000", 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065738899999999998 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973844+0000", 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6490999999999998e-05 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973890+0000", 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5448999999999998e-05 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1544 3.22 3:453283ef:::100000001f8.00000204:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.954353+0000", 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.5337602109999997, 2024-04-25T15:53:48.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025396769999999999, 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1544 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.954353+0000", 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.954353+0000", 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.954357+0000", 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1980000000000001e-06 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956165+0000", 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018087540000000001 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956202+0000", 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6248999999999998e-05 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956224+0000", 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2141999999999999e-05 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956316+0000", 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1724000000000005e-05 2024-04-25T15:53:48.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956484+0000", 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000168858 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956591+0000", 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106263 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973938+0000", 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017347122 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.979612+0000", 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056746360000000003 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.979682+0000", 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9142999999999999e-05 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.979750+0000", 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8681000000000002e-05 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.556 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1589 3.1a e88/76 3:590c0ff7:::100000001f8.00000231:head v 88'30, mlcod=88'28)", 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2412903310000001, 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032177728000000003, 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246826+0000", 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1900000000000002e-06 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045171680000000002 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251364+0000", 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1444999999999999e-05 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251375+0000", 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.115e-05 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251458+0000", 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3190999999999997e-05 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251485+0000", 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6661000000000001e-05 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.278940+0000", 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027455222000000001 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.279001+0000", 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0701e-05 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1595 3.18 3:1adf79c2:::100000001f8.00000237:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2059292939999997, 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038955741000000002, 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1595 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.561 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282187+0000", 2024-04-25T15:53:48.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1569999999999998e-06 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284089+0000", 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001902672 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284127+0000", 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7663000000000002e-05 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284141+0000", 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284236+0000", 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4895999999999994e-05 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284417+0000", 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018094500000000001 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284602+0000", 2024-04-25T15:53:48.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018422800000000001 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.293831+0000", 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092292650000000004 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321052+0000", 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027220978 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321098+0000", 2024-04-25T15:53:48.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6520000000000002e-05 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321140+0000", 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1850000000000001e-05 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1737 3.18 3:1bfb6017:::100000001f8.000002c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.167565+0000", 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.3205487140000001, 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020275571999999999, 2024-04-25T15:53:48.566 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1737 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.167565+0000", 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.167565+0000", 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.167569+0000", 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8060000000000001e-06 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.169547+0000", 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019786550000000002 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.169595+0000", 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7726999999999999e-05 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.169621+0000", 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5732e-05 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.169717+0000", 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6409999999999993e-05 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.169860+0000", 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014256499999999999 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.169967+0000", 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010720100000000001 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.180659+0000", 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010691600000000001 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.187708+0000", 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0070493029999999998 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.187783+0000", 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5066000000000003e-05 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.187841+0000", 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7506999999999997e-05 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1738 3.32 3:4f4b868d:::100000001f8.000002c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.173717+0000", 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.31439621, 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020601518999999999, 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1738 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.173717+0000", 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.173717+0000", 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.173720+0000", 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4389999999999999e-06 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.175545+0000", 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018253519999999999 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.175577+0000", 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1612000000000002e-05 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.175590+0000", 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.361e-05 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.175674+0000", 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3746999999999996e-05 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.175822+0000", 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014803200000000001 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.175987+0000", 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016459299999999999 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.191589+0000", 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015602130000000001 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.194217+0000", 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002627529 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.194272+0000", 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5906000000000003e-05 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.194319+0000", 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6569000000000001e-05 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1757 3.18 3:1b8a210a:::100000001f8.000002d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.288887+0000", 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.1992261810000002, 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022025704, 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.577 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1757 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.288887+0000", 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.288887+0000", 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.288890+0000", 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9670000000000001e-06 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.290238+0000", 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013479640000000001 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.290272+0000", 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4054000000000003e-05 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.290287+0000", 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.402e-05 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.290361+0000", 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4144e-05 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.290480+0000", 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119314 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.290656+0000", 2024-04-25T15:53:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017569800000000001 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.297605+0000", 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069495879999999996 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.310787+0000", 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013182093000000001 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.310858+0000", 2024-04-25T15:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0683000000000001e-05 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.310913+0000", 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5179000000000001e-05 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1814 3.18 3:1977fdc7:::100000001f8.00000312:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.636518+0000", 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.851595476, 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024442130999999999, 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.582 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1814 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.636518+0000", 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.636518+0000", 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.636521+0000", 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3010000000000001e-06 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638401+0000", 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018804379999999999 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638442+0000", 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1542999999999998e-05 2024-04-25T15:53:48.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638461+0000", 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8567e-05 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638559+0000", 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8355000000000003e-05 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638720+0000", 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016095200000000001 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638873+0000", 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015225899999999999 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.657264+0000", 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018391873999999999 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.660841+0000", 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035769170000000002 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.660910+0000", 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9021000000000002e-05 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.660960+0000", 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9904000000000002e-05 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1817 3.1a e89/76 3:5b497a17:::100000001f8.00000315:head v 89'33, mlcod=89'31)", 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.667493+0000", 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.8206209270000002, 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028489159, 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.587 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.667493+0000", 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.667493+0000", 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.667494+0000", 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4330000000000001e-06 2024-04-25T15:53:48.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680261+0000", 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012767152 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680284+0000", 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2846000000000001e-05 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680302+0000", 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7506000000000001e-05 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680403+0000", 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101007 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680432+0000", 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9417e-05 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695912+0000", 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015479655 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695982+0000", 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0142999999999996e-05 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1823 3.18 3:19a3da96:::100000001f8.0000031b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.697984+0000", 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 2.7901294050000001, 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023611703000000001, 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.591 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1823 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.697984+0000", 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.697984+0000", 2024-04-25T15:53:48.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.697986+0000", 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.165e-06 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.699519+0000", 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001532844 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.699564+0000", 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4688000000000003e-05 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.699582+0000", 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7768999999999999e-05 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.699673+0000", 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1354999999999995e-05 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.699856+0000", 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000182783 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.700037+0000", 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181163 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.706347+0000", 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0063099080000000004 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.721475+0000", 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015127784999999999 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:48.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.721546+0000", 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1509999999999998e-05 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.721596+0000", 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9733e-05 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.596 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:48.597 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:48.487+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:48.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:48.599 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:48.598+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:48.698 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:48.695+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:48.800 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:48.798+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:48.832 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:48.832 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:48.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.832 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2308 3.4 3:2343a40c:::100000001f8.00000500:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:48.832 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.012679487, 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012727466999999999, 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2308 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:48.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821088+0000", 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1599999999999998e-06 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:48.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823092+0000", 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020043909999999999 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823143+0000", 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0791999999999998e-05 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823162+0000", 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9332999999999999e-05 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:48.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823254+0000", 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2169000000000005e-05 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823637+0000", 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038265100000000002 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823849+0000", 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211879 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:48.837 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:48.837 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 1 2024-04-25T15:53:48.837 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:48.872 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:48.913 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:48.914+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:49.001 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:48.999+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:49.090 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:49.090 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:49.090 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:49.102 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:49.099+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:49.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:49.206 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:49.199+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:49.303 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:49.302+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:49.319 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:49.319 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:49.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:147 3.b 3:d1738051:::10000000002.0000008f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.506796+0000", 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.811453037, 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067075241999999993, 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.506796+0000", 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.506796+0000", 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.506799+0000", 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6170000000000001e-06 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.508301+0000", 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015025240000000001 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559741+0000", 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051439617999999999 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559757+0000", 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-05 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559865+0000", 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107383 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.560034+0000", 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016934299999999999 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.560164+0000", 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012956400000000001 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573028+0000", 2024-04-25T15:53:49.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01286471 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573792+0000", 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076319599999999995 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573833+0000", 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.138e-05 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573871+0000", 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8417000000000002e-05 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:148 3.9 3:91a8729a:::10000000002.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.760030968000002, 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.432712019, 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 148 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558221+0000", 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3929999999999998e-06 2024-04-25T15:53:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559704+0000", 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001483807 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975277+0000", 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.41557216400000002 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975295+0000", 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8408999999999999e-05 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975372+0000", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7235000000000002e-05 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975484+0000", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111499 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975582+0000", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8589000000000001e-05 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.980957+0000", 2024-04-25T15:53:49.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053747600000000001 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990855+0000", 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098981350000000006 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990894+0000", 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8436000000000003e-05 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990930+0000", 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6591999999999998e-05 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.329 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:157 3.2a 3:577cc1b8:::10000000002.00000099:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.973161+0000", 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.345088466, 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067779252999999998, 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 157 2024-04-25T15:53:49.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.973161+0000", 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.973161+0000", 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.973164+0000", 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.168e-06 2024-04-25T15:53:49.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975237+0000", 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020727599999999999 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027231+0000", 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051994686999999998 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027250+0000", 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8292e-05 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027337+0000", 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6977000000000004e-05 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027503+0000", 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016680300000000001 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027613+0000", 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110031 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.333 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034797+0000", 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071834669999999998 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.040858+0000", 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060610799999999999 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.040896+0000", 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8367e-05 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.040940+0000", 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3621000000000001e-05 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1009 3.3e 3:7fa01700:::100000001f7.000000ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.532897+0000", 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.785351765000001, 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046042244000000003, 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.335 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1009 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.532897+0000", 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.532897+0000", 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.532900+0000", 2024-04-25T15:53:49.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9849999999999998e-06 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534627+0000", 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263479999999999 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534658+0000", 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1600000000000002e-05 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534670+0000", 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.163e-05 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534798+0000", 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128355 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534901+0000", 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102429 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.535068+0000", 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016683600000000001 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.543609+0000", 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008541468 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.578815+0000", 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035206144000000002 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.578877+0000", 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1822000000000005e-05 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.578940+0000", 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2626999999999995e-05 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1473 3.0 3:008165f2:::100000001f8.000001bd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8260295019999999, 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.084007844999999998, 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.340 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1473 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492222+0000", 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6910000000000002e-06 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.494870+0000", 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026480620000000001 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557687+0000", 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.062816831000000004 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557707+0000", 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9262e-05 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557796+0000", 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9209000000000006e-05 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557984+0000", 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000188118 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:49.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.558165+0000", 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181292 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.565264+0000", 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0070987309999999996 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576094+0000", 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010829920999999999 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576165+0000", 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0951999999999994e-05 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576228+0000", 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2775999999999997e-05 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1481 3.2a 3:55ceaa15:::100000001f8.000001c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.555971+0000", 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.7622779279999996, 2024-04-25T15:53:49.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075920801999999996, 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1481 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.555971+0000", 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.555971+0000", 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.555974+0000", 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2850000000000001e-06 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557650+0000", 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001676716 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617464+0000", 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.059813893999999999 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617483+0000", 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8468999999999999e-05 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617590+0000", 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010698500000000001 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617779+0000", 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018936699999999999 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617928+0000", 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149188 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.627751+0000", 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098226839999999999 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.631788+0000", 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040376459999999998 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.631829+0000", 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0256999999999998e-05 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.631892+0000", 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3311000000000004e-05 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1490 3.0 3:008ec8c9:::100000001f8.000001ce:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.615629+0000", 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.7026206500000001, 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046517017000000001, 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1490 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.615629+0000", 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.615629+0000", 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.615631+0000", 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4109999999999999e-06 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617421+0000", 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017897900000000001 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.640726+0000", 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023305612999999999 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.640740+0000", 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3451999999999999e-05 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.640831+0000", 2024-04-25T15:53:49.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1593000000000002e-05 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.641019+0000", 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018716199999999999 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.641204+0000", 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185236 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.649450+0000", 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008246069 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.662008+0000", 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012558348 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.662082+0000", 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4192999999999999e-05 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.662146+0000", 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3150000000000004e-05 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1494 3.37 3:efe547ea:::100000001f8.000001d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6738464200000003, 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089208711999999996, 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.356 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1494 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644406+0000", 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2229999999999999e-06 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.646356+0000", 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019497990000000001 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718358+0000", 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072002624000000001 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718376+0000", 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7649999999999999e-05 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718458+0000", 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1984999999999995e-05 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718600+0000", 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000141922 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:49.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718731+0000", 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131488 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.724805+0000", 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006074032 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733498+0000", 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086928439999999999 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733549+0000", 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1137000000000002e-05 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733611+0000", 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2008000000000007e-05 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1504 3.2a 3:54fb97d1:::100000001f8.000001dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6015529839999996, 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121933077, 2024-04-25T15:53:49.361 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1504 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716699+0000", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4250000000000001e-06 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718317+0000", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016187440000000001 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.815989+0000", 2024-04-25T15:53:49.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.097671735999999995 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816008+0000", 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9131e-05 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816090+0000", 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1717000000000004e-05 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816244+0000", 2024-04-25T15:53:49.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015372199999999999 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816348+0000", 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010470499999999999 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.826448+0000", 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010099281999999999 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838528+0000", 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012080206 2024-04-25T15:53:49.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838583+0000", 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5105000000000002e-05 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838629+0000", 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6304000000000003e-05 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1521 3.21 3:84dc523d:::100000001f8.000001ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.366 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.822483+0000", 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4957660060000002, 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.060484519, 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1521 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.822483+0000", 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.367 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.822483+0000", 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.822486+0000", 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1580000000000001e-06 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.823900+0000", 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001413847 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.368 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865598+0000", 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041697565999999998 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865612+0000", 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4724e-05 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865689+0000", 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.369 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865824+0000", 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013481999999999999 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865930+0000", 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010693599999999999 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.872071+0000", 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061405640000000003 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.370 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.882880+0000", 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010809246999999999 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.882922+0000", 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2219999999999999e-05 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.882968+0000", 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5213999999999998e-05 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1528 3.3e 3:7f4f4afd:::100000001f8.000001f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.454079138, 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.133348521, 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1528 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.372 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864172+0000", 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.204e-06 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.373 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865554+0000", 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013821899999999999 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986006+0000", 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12045179 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986022+0000", 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999e-05 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986101+0000", 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9251000000000004e-05 2024-04-25T15:53:49.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986224+0000", 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012263899999999999 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986330+0000", 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106164 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.996058+0000", 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097274319999999994 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997353+0000", 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001294882 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997450+0000", 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7446000000000002e-05 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997519+0000", 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8523999999999996e-05 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1548 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.3346134310000002, 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096155170999999998, 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1548 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983639+0000", 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1750000000000001e-06 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.985960+0000", 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023213790000000002 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058434+0000", 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072473913000000001 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058454+0000", 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9329000000000001e-05 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058541+0000", 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7702999999999997e-05 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058708+0000", 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016712300000000001 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058864+0000", 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015600100000000001 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.065501+0000", 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066362299999999999 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079696+0000", 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014195228000000001 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079745+0000", 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8980000000000002e-05 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079791+0000", 2024-04-25T15:53:49.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6109999999999997e-05 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1561 3.2e 3:75253c31:::100000001f8.00000215:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.2502189010000002, 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099620287000000002, 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.382 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1561 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068033+0000", 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1240000000000001e-06 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.070277+0000", 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002243676 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143785+0000", 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073508283999999993 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143804+0000", 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8808e-05 2024-04-25T15:53:49.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143881+0000", 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6379999999999997e-05 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144063+0000", 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018198500000000001 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144168+0000", 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105253 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.154374+0000", 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010206374000000001 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167481+0000", 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013106405999999999 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167580+0000", 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9329999999999999e-05 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167651+0000", 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0667000000000006e-05 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1574 3.3e 3:7d4c752e:::100000001f8.00000222:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.141841+0000", 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1764085870000001, 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044150146000000001, 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.387 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1574 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.141841+0000", 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.141841+0000", 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.141843+0000", 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5500000000000001e-06 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143747+0000", 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019037559999999999 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.164024+0000", 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020276828 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.164038+0000", 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3814e-05 2024-04-25T15:53:49.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.164117+0000", 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9110000000000007e-05 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.164227+0000", 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110396 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.164373+0000", 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014598000000000001 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.183038+0000", 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018665210000000002 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.185915+0000", 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028766460000000001 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.185956+0000", 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1480000000000003e-05 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.185991+0000", 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4375999999999997e-05 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1649 3.37 3:ec163160:::100000001f8.0000026d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.631507+0000", 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.6867417439999999, 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043896939000000003, 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.392 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1649 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.631507+0000", 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.631507+0000", 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.631511+0000", 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.252e-06 2024-04-25T15:53:49.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.632735+0000", 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0012247880000000001 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.632785+0000", 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9614000000000003e-05 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.632799+0000", 2024-04-25T15:53:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4253000000000001e-05 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.632872+0000", 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2523999999999999e-05 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.633007+0000", 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013477899999999999 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.633113+0000", 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106808 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.639220+0000", 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061061420000000002 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.675323+0000", 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036102984999999997 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.675365+0000", 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2710000000000003e-05 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.675404+0000", 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9084000000000001e-05 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1688 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.865613+0000", 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4526364279999999, 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052534059000000001, 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.397 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1688 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.865613+0000", 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.865613+0000", 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.865615+0000", 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329999999999998e-06 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867053+0000", 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014382539999999999 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867094+0000", 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0695999999999998e-05 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867113+0000", 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9207000000000001e-05 2024-04-25T15:53:49.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867190+0000", 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6981e-05 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867339+0000", 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014954399999999999 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867473+0000", 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013331899999999999 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.873926+0000", 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064532010000000004 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.918006+0000", 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044080345999999999 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.918081+0000", 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5078000000000002e-05 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.918147+0000", 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5400000000000004e-05 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1704 3.4 3:230cd4d1:::100000001f8.000002a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.964986+0000", 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3532630140000004, 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044402101999999999, 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.402 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1704 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.964986+0000", 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.964986+0000", 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.964989+0000", 2024-04-25T15:53:49.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.061e-06 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966880+0000", 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001890907 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966907+0000", 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7237e-05 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966920+0000", 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2172e-05 2024-04-25T15:53:49.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966996+0000", 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6569999999999994e-05 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.967163+0000", 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000166983 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.967337+0000", 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017356399999999999 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.977633+0000", 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010296764 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.009257+0000", 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031623186999999997 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.009326+0000", 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9443999999999994e-05 2024-04-25T15:53:49.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.009388+0000", 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2212999999999995e-05 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1709 3.b 3:d0c4adc9:::100000001f8.000002a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.996354+0000", 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.321895628, 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044554143999999997, 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.407 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1709 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.996354+0000", 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.996354+0000", 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.996356+0000", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2309999999999999e-06 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998388+0000", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00203177 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998428+0000", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0862999999999998e-05 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998447+0000", 2024-04-25T15:53:49.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8783999999999998e-05 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998525+0000", 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8152000000000007e-05 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998701+0000", 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017595499999999999 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:49.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998821+0000", 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119335 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.028809+0000", 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029987949 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040800+0000", 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01199154 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040847+0000", 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7308999999999997e-05 2024-04-25T15:53:49.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040908+0000", 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0256e-05 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1710 3.b 3:d079ce42:::100000001f8.000002aa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.002097+0000", 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.316152389, 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043634797000000003, 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.412 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1710 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.002097+0000", 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.002097+0000", 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.002100+0000", 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.061e-06 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.003756+0000", 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001655959 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.003785+0000", 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9539e-05 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.003798+0000", 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.277e-05 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.003883+0000", 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4968000000000004e-05 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.004061+0000", 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000178159 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:49.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.004221+0000", 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016019999999999999 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.036633+0000", 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032411140999999997 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045627+0000", 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0089944599999999993 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045687+0000", 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0132e-05 2024-04-25T15:53:49.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045732+0000", 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4407999999999999e-05 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2308 3.4 3:2343a40c:::100000001f8.00000500:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.49716458899999999, 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.129707092, 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:49.417 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2308 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:49.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821088+0000", 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1599999999999998e-06 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823092+0000", 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020043909999999999 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823143+0000", 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0791999999999998e-05 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823162+0000", 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9332999999999999e-05 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823254+0000", 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2169000000000005e-05 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823637+0000", 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038265100000000002 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:49.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823849+0000", 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211879 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.834007+0000", 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010158377 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950654+0000", 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.116646229 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950721+0000", 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7020000000000005e-05 2024-04-25T15:53:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950792+0000", 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1091000000000001e-05 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:49.422 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:49.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:53:49.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:49.424 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:49.423+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:49.524 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:49.522+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:49.626 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:49.626+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.654+0000 7f22099f5640 1 Processor -- start 2024-04-25T15:53:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.654+0000 7f22099f5640 1 -- start start 2024-04-25T15:53:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f22099f5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22040caf60 0x7f220415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f22099f5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 0x7f220406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f22099f5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2204073bf0 0x7f220406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f22099f5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f22040cab60 con 0x7f22040caf60 2024-04-25T15:53:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f22099f5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2204159620 con 0x7f2204073bf0 2024-04-25T15:53:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f22099f5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f22040c77a0 con 0x7f220415d790 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22040caf60 0x7f220415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22040caf60 0x7f220415d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43764/0 (socket says 172.21.15.67:43764) 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 -- 172.21.15.67:0/3500320245 learned_addr learned my addr 172.21.15.67:0/3500320245 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f22037fe640 1 --2- 172.21.15.67:0/3500320245 >> v2:172.21.15.176:3300/0 conn(0x7f2204073bf0 0x7f220406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 -- 172.21.15.67:0/3500320245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 msgr2=0x7f220406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 --2- 172.21.15.67:0/3500320245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 0x7f220406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 -- 172.21.15.67:0/3500320245 >> v2:172.21.15.176:3300/0 conn(0x7f2204073bf0 msgr2=0x7f220406dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 --2- 172.21.15.67:0/3500320245 >> v2:172.21.15.176:3300/0 conn(0x7f2204073bf0 0x7f220406dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 -- 172.21.15.67:0/3500320245 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22040c7a60 con 0x7f22040caf60 2024-04-25T15:53:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.658+0000 7f2202ffd640 1 --2- 172.21.15.67:0/3500320245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f220415d250 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f220406fd00 tx=0x7f21f401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4eb4de734fde3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f2201ffb640 1 -- 172.21.15.67:0/3500320245 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f40b0070 con 0x7f22040caf60 2024-04-25T15:53:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f2201ffb640 1 -- 172.21.15.67:0/3500320245 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f40ab600 con 0x7f22040caf60 2024-04-25T15:53:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f2201ffb640 1 -- 172.21.15.67:0/3500320245 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f40ab9d0 con 0x7f22040caf60 2024-04-25T15:53:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 -- 172.21.15.67:0/3500320245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 msgr2=0x7f220415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 --2- 172.21.15.67:0/3500320245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f220415d250 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f220406fd00 tx=0x7f21f401c520 comp rx=0 tx=0).stop 2024-04-25T15:53:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 -- 172.21.15.67:0/3500320245 shutdown_connections 2024-04-25T15:53:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 --2- 172.21.15.67:0/3500320245 >> v2:172.21.15.176:3300/0 conn(0x7f2204073bf0 0x7f220406dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.663 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:49.663 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:49.663 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 --2- 172.21.15.67:0/3500320245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 0x7f220406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 --2- 172.21.15.67:0/3500320245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f220415d250 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 -- 172.21.15.67:0/3500320245 >> 172.21.15.67:0/3500320245 conn(0x7f2204070db0 msgr2=0x7f22040ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 -- 172.21.15.67:0/3500320245 shutdown_connections 2024-04-25T15:53:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 -- 172.21.15.67:0/3500320245 wait complete. 2024-04-25T15:53:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 Processor -- start 2024-04-25T15:53:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.662+0000 7f22099f5640 1 -- start start 2024-04-25T15:53:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2204073bf0 0x7f22040cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f22040cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 0x7f22040ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f22040733d0 con 0x7f22040caf60 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f220415a600 con 0x7f2204073bf0 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f22040cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f22040cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43780/0 (socket says 172.21.15.67:43780) 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 -- 172.21.15.67:0/2046486264 learned_addr learned my addr 172.21.15.67:0/2046486264 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2204073550 con 0x7f220415d790 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f2202ffd640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2204073bf0 0x7f22040cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22037fe640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 0x7f22040ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 -- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 msgr2=0x7f22040ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 0x7f22040ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 -- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2204073bf0 msgr2=0x7f22040cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2204073bf0 0x7f22040cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 -- 172.21.15.67:0/2046486264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22041f3980 con 0x7f22040caf60 2024-04-25T15:53:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22027fc640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f22040cc730 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f21f800b3f0 tx=0x7f21f800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f21ebfff640 1 -- 172.21.15.67:0/2046486264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8015020 con 0x7f22040caf60 2024-04-25T15:53:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f21ebfff640 1 -- 172.21.15.67:0/2046486264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f8011830 con 0x7f22040caf60 2024-04-25T15:53:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2204153c00 con 0x7f22040caf60 2024-04-25T15:53:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f21ebfff640 1 -- 172.21.15.67:0/2046486264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f8011c20 con 0x7f22040caf60 2024-04-25T15:53:49.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.666+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22041f3f20 con 0x7f22040caf60 2024-04-25T15:53:49.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.670+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e40052a0 con 0x7f22040caf60 2024-04-25T15:53:49.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.670+0000 7f21ebfff640 1 -- 172.21.15.67:0/2046486264 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f21f8018020 con 0x7f22040caf60 2024-04-25T15:53:49.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.670+0000 7f21ebfff640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f21f006f400 0x7f21f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.670+0000 7f21ebfff640 1 -- 172.21.15.67:0/2046486264 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5944+0+0 (secure 0 0 0) 0x7f21f808f0a0 con 0x7f22040caf60 2024-04-25T15:53:49.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.670+0000 7f2202ffd640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f21f006f400 0x7f21f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.670+0000 7f2202ffd640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f21f006f400 0x7f21f0071910 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f21f40ad000 tx=0x7f21f40b9040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:49.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.678+0000 7f21ebfff640 1 -- 172.21.15.67:0/2046486264 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f21f801f020 con 0x7f22040caf60 2024-04-25T15:53:49.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:49.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:49.688 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T15:53:49.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:53:49.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:49.741 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:49.742+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:49.828 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:49.827+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:49.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.882+0000 7ff2923f5640 1 Processor -- start 2024-04-25T15:53:49.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.882+0000 7ff2923f5640 1 -- start start 2024-04-25T15:53:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.882+0000 7ff2923f5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff28c160800 0x7ff28c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.882+0000 7ff2923f5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff28c06d8e0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff2923f5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff28c073c70 con 0x7ff28c0c96e0 2024-04-25T15:53:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff2923f5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff28c06dbc0 con 0x7ff28c160800 2024-04-25T15:53:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff28c160800 0x7ff28c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff28c0c92d0 0x7ff28c072dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35580/0 (socket says 172.21.15.67:35580) 2024-04-25T15:53:49.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28b7fe640 1 -- 172.21.15.67:0/3349862953 learned_addr learned my addr 172.21.15.67:0/3349862953 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28affd640 1 -- 172.21.15.67:0/3349862953 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155673205 0 0) 0x7ff28c06d8e0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28affd640 1 -- 172.21.15.67:0/3349862953 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff274003530 con 0x7ff28c0c92d0 2024-04-25T15:53:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28bfff640 1 -- 172.21.15.67:0/3349862953 >> v1:172.21.15.176:6789/0 conn(0x7ff28c0c96e0 legacy=0x7ff28c167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28bfff640 1 -- 172.21.15.67:0/3349862953 >> v1:172.21.15.67:6789/0 conn(0x7ff28c0c92d0 legacy=0x7ff28c072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:49.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.886+0000 7ff28bfff640 1 -- 172.21.15.67:0/3349862953 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c073f50 con 0x7ff28c160800 2024-04-25T15:53:49.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff28bfff640 1 --2- 172.21.15.67:0/3349862953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c160800 0x7ff28c06d030 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0097c0 tx=0x7ff27c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1d47731c092da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff28affd640 1 -- 172.21.15.67:0/3349862953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0b0070 con 0x7ff28c160800 2024-04-25T15:53:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff28affd640 1 -- 172.21.15.67:0/3349862953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff27c0ab600 con 0x7ff28c160800 2024-04-25T15:53:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff28affd640 1 -- 172.21.15.67:0/3349862953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0ab9b0 con 0x7ff28c160800 2024-04-25T15:53:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff2923f5640 1 -- 172.21.15.67:0/3349862953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c160800 msgr2=0x7ff28c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:49.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff2923f5640 1 --2- 172.21.15.67:0/3349862953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c160800 0x7ff28c06d030 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0097c0 tx=0x7ff27c01c3d0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff2923f5640 1 -- 172.21.15.67:0/3349862953 shutdown_connections 2024-04-25T15:53:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff2923f5640 1 --2- 172.21.15.67:0/3349862953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c160800 0x7ff28c06d030 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff2923f5640 1 -- 172.21.15.67:0/3349862953 >> 172.21.15.67:0/3349862953 conn(0x7ff28c070db0 msgr2=0x7ff28c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff2923f5640 1 -- 172.21.15.67:0/3349862953 shutdown_connections 2024-04-25T15:53:49.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.890+0000 7ff2923f5640 1 -- 172.21.15.67:0/3349862953 wait complete. 2024-04-25T15:53:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.894+0000 7ff2923f5640 1 Processor -- start 2024-04-25T15:53:49.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.894+0000 7ff2923f5640 1 -- start start 2024-04-25T15:53:49.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.894+0000 7ff2923f5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0c92d0 0x7ff28c0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff2923f5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c0c96e0 0x7ff28c1f9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0c92d0 0x7ff28c0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0c92d0 0x7ff28c0c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43800/0 (socket says 172.21.15.67:43800) 2024-04-25T15:53:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 -- 172.21.15.67:0/1811416605 learned_addr learned my addr 172.21.15.67:0/1811416605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff2923f5640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c152280 0x7ff28c1f9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c160c10 con 0x7ff28c0c92d0 2024-04-25T15:53:49.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c06d570 con 0x7ff28c0c96e0 2024-04-25T15:53:49.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff28c160d90 con 0x7ff28c152280 2024-04-25T15:53:49.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c0c96e0 0x7ff28c1f9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff290992640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c152280 0x7ff28c1f9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 -- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c152280 msgr2=0x7ff28c1f9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c152280 0x7ff28c1f9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 -- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c0c96e0 msgr2=0x7ff28c1f9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c0c96e0 0x7ff28c1f9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c0c70e0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c0c96e0 0x7ff28c1f9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:49.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff28bfff640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0c92d0 0x7ff28c0c6ba0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff27c000c10 tx=0x7ff27c01ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff288ff9640 1 -- 172.21.15.67:0/1811416605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0b0050 con 0x7ff28c0c92d0 2024-04-25T15:53:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff288ff9640 1 -- 172.21.15.67:0/1811416605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff27c0bc070 con 0x7ff28c0c92d0 2024-04-25T15:53:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff288ff9640 1 -- 172.21.15.67:0/1811416605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0b5810 con 0x7ff28c0c92d0 2024-04-25T15:53:49.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.898+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c1538e0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.902+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c153ea0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.902+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28c1540e0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.902+0000 7ff288ff9640 1 -- 172.21.15.67:0/1811416605 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff27c0b59b0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.902+0000 7ff288ff9640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff27806f430 0x7ff278071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:49.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.902+0000 7ff288ff9640 1 -- 172.21.15.67:0/1811416605 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5944+0+0 (secure 0 0 0) 0x7ff27c1298c0 con 0x7ff28c0c92d0 2024-04-25T15:53:49.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.902+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff27806f430 0x7ff278071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:49.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.906+0000 7ff28b7fe640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff27806f430 0x7ff278071940 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff28c14b540 tx=0x7ff280009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:49.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.910+0000 7ff288ff9640 1 -- 172.21.15.67:0/1811416605 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff27c0c6020 con 0x7ff28c0c92d0 2024-04-25T15:53:49.928 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:49.926+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:49.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:49.938+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f21e4005860 con 0x7f22040caf60 2024-04-25T15:53:49.946 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:49.946 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:49.947 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:49.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:50.057 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:50.058+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:50.130 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:50.127+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:50.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.146+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7ff28c06a480 con 0x7ff28c0c92d0 2024-04-25T15:53:50.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.150+0000 7ff288ff9640 1 -- 172.21.15.67:0/1811416605 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v90) ==== 81+0+35493 (secure 0 0 0) 0x7ff27c0c6020 con 0x7ff28c0c92d0 2024-04-25T15:53:50.153 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:53:50.153 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 90, 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:53:47.298293+0000", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:53:14.990625+0000", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5804032, 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "nodeep-scrub", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:53:50.154 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:53:50.155 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:53:50.156 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:50.157 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:50.158 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:50.159 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:53:50.160 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:50.161 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:50.162 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:50.163 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:50.164 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:53:50.165 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.75, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.75, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:50.166 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:53:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:53:50.168 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:53:50.169 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:53:50.170 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:53:50.171 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.4299999475479126, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.4299999475479126, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.6299999952316284, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.6299999952316284, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.172 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.175 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0.2953948974609375, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.178 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:53:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 73, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.183 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:53:50.184 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.186 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.187 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:53:50.189 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.190 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:53:50.191 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.192 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.194 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:53:50.195 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:50.197 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.198 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.199 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.200 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:53:50.201 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.204 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.205 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.206 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 85, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:53:50.207 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.208 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.210 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:53:50.211 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.213 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.214 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.215 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:53:50.216 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.18", 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: 4 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.27", 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: 7, 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: 6 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:53:50.218 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.219 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.3e", 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:53:50.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 0, 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:53:50.221 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:53:50.222 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.223 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff27806f430 msgr2=0x7ff278071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff27806f430 0x7ff278071940 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff28c14b540 tx=0x7ff280009040 comp rx=0 tx=0).stop 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0c92d0 msgr2=0x7ff28c0c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0c92d0 0x7ff28c0c6ba0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff27c000c10 tx=0x7ff27c01ac10 comp rx=0 tx=0).stop 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 shutdown_connections 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff27806f430 0x7ff278071940 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff28c152280 0x7ff28c1f9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28c0c96e0 0x7ff28c1f9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 --2- 172.21.15.67:0/1811416605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff28c0c92d0 0x7ff28c0c6ba0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 >> 172.21.15.67:0/1811416605 conn(0x7ff28c070db0 msgr2=0x7ff28c0c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:50.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 shutdown_connections 2024-04-25T15:53:50.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.154+0000 7ff2923f5640 1 -- 172.21.15.67:0/1811416605 wait complete. 2024-04-25T15:53:50.226 INFO:tasks.thrashosds.thrasher:j is {'epoch': 90, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:53:47.298293+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:53:14.990625+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4299999475479126, 'score_stable': 1.4299999475479126, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 0.2953948974609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 85, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [1, 4]}, {'pgid': '3.27', 'osds': [7, 6]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '3.3e', 'mappings': [{'from': 0, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:53:50.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:53:50.227 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:343 3.8 3:1115f3ca:::10000000002.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:53:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.196751867000003, 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.46089677899999998, 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 343 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006870+0000", 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9280000000000002e-06 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008363+0000", 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001493288 2024-04-25T15:53:50.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008398+0000", 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5067000000000002e-05 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008416+0000", 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7782e-05 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008522+0000", 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106327 2024-04-25T15:53:50.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008535+0000", 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2883e-05 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.387477+0000", 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.37894188200000001 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.392562+0000", 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050850870000000003 2024-04-25T15:53:50.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445264+0000", 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052702037 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445382+0000", 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118433 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445603+0000", 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022077500000000001 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.465250+0000", 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019646976 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467658+0000", 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024073850000000002 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467695+0000", 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7234999999999998e-05 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467763+0000", 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8694000000000003e-05 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:670 3.8 3:1323134e:::100000001f7.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.329681703999999, 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094982472999999998, 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 670 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873941+0000", 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0840000000000002e-06 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876488+0000", 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025470290000000001 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876647+0000", 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015912000000000001 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876664+0000", 2024-04-25T15:53:50.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7215e-05 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876782+0000", 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118082 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877222+0000", 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000439596 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877596+0000", 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037371700000000003 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.913106+0000", 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035509813000000001 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968792+0000", 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.055686739999999998 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968850+0000", 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8032000000000003e-05 2024-04-25T15:53:50.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968919+0000", 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9045e-05 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:666 3.15 e81/18 3:a92ac213:::100000001f7.00000096:head v 81'11, mlcod=81'9)", 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.328809467999999, 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046403201999999998, 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874809+0000", 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.874810+0000", 2024-04-25T15:53:50.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2500000000000001e-06 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882291+0000", 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074801800000000003 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882310+0000", 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9809999999999998e-05 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882330+0000", 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0026000000000001e-05 2024-04-25T15:53:50.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882471+0000", 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014052199999999999 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.882509+0000", 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7756999999999998e-05 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921136+0000", 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038627364999999997 2024-04-25T15:53:50.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921212+0000", 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6291999999999994e-05 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:671 3.38 3:1c0639bf:::100000001f7.0000009b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.319516669999999, 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073197039000000005, 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.243 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 671 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:53:50.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884106+0000", 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9709999999999998e-06 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886679+0000", 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025726770000000002 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886716+0000", 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.735e-05 2024-04-25T15:53:50.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886733+0000", 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6697999999999998e-05 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886843+0000", 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011074299999999999 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887031+0000", 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018804800000000001 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887215+0000", 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018384599999999999 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921326+0000", 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034110486000000002 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957180+0000", 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035854051999999997 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957227+0000", 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7290000000000003e-05 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957299+0000", 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1878000000000005e-05 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:732 3.15 e81/18 3:aadcba4a:::100000001f7.000000d8:head v 81'13, mlcod=81'11)", 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.895991681000002, 2024-04-25T15:53:50.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056638899999999999, 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307627+0000", 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.307628+0000", 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4529999999999999e-06 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.312965+0000", 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053367110000000001 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.312985+0000", 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9470000000000002e-05 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313001+0000", 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6840000000000001e-05 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313091+0000", 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9770000000000003e-05 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.313122+0000", 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1047000000000003e-05 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364214+0000", 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051091996000000001 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.364266+0000", 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1613000000000002e-05 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:736 3.25 3:a5356222:::100000001f7.000000dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.876919569999998, 2024-04-25T15:53:50.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041194891999999997, 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 736 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326699+0000", 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.326702+0000", 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5500000000000001e-06 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328063+0000", 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013610849999999999 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328097+0000", 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4638000000000003e-05 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328111+0000", 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3896999999999999e-05 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328183+0000", 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1859000000000004e-05 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328336+0000", 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015279700000000001 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.328465+0000", 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012961200000000001 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.346379+0000", 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017913097999999999 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367792+0000", 2024-04-25T15:53:50.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021413926 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367838+0000", 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5541999999999999e-05 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.367894+0000", 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5887999999999998e-05 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.257 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:959 3.33 3:cf81bd10:::100000001f7.000000bb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.959746853999999, 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042315856999999998, 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 959 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243872+0000", 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.243874+0000", 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.95e-06 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245749+0000", 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018754080000000001 2024-04-25T15:53:50.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245791+0000", 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2200000000000003e-05 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245807+0000", 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5379999999999998e-05 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245884+0000", 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7039999999999994e-05 2024-04-25T15:53:50.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.245972+0000", 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8459000000000001e-05 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.246076+0000", 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103861 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.252170+0000", 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060943439999999998 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286068+0000", 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033897570000000002 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286116+0000", 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8374999999999997e-05 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.286188+0000", 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.127e-05 2024-04-25T15:53:50.262 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1010 3.33 3:cf8599b8:::100000001f7.000000ee:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.662898198000001, 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043954280999999998, 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1010 2024-04-25T15:53:50.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540720+0000", 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.264 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.540723+0000", 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6029999999999999e-06 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542109+0000", 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013856610000000001 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542152+0000", 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3353999999999999e-05 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542167+0000", 2024-04-25T15:53:50.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4779e-05 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542259+0000", 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2472999999999994e-05 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542384+0000", 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012520200000000001 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:50.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.542571+0000", 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018691000000000001 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.551607+0000", 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090354900000000002 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584560+0000", 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032952759999999998 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584623+0000", 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3305999999999993e-05 2024-04-25T15:53:50.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584675+0000", 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1743000000000002e-05 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1011 3.7 3:e14869e0:::100000001f7.000000ef:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.657852034000001, 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038908115, 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.268 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1011 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545767+0000", 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.545769+0000", 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.587e-06 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547305+0000", 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015358590000000001 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547334+0000", 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8538e-05 2024-04-25T15:53:50.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547346+0000", 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2449000000000001e-05 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547459+0000", 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112759 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547581+0000", 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012228900000000001 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.547754+0000", 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017338 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.555402+0000", 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076472600000000003 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584550+0000", 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029148484999999998 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584616+0000", 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5528e-05 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.584675+0000", 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8981000000000003e-05 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1640 3.29 3:96b47918:::100000001f8.00000264:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.573553+0000", 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6300656279999997, 2024-04-25T15:53:50.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044379263000000002, 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1640 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.573553+0000", 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.573553+0000", 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.573556+0000", 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.993e-06 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.574886+0000", 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001329937 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.574932+0000", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6106999999999997e-05 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.574951+0000", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9440999999999999e-05 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.575033+0000", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1972999999999996e-05 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.575152+0000", 2024-04-25T15:53:50.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119029 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.575257+0000", 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010449 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.581333+0000", 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060757629999999997 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.617808+0000", 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036475661999999999 2024-04-25T15:53:50.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.617871+0000", 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3020000000000003e-05 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.617932+0000", 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0847999999999997e-05 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1689 3.8 3:13aaa185:::100000001f8.00000295:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:50.278 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.871337+0000", 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.3322817780000005, 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042391641000000001, 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1689 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.871337+0000", 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.871337+0000", 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.871340+0000", 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.106e-06 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.872773+0000", 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014333550000000001 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.872808+0000", 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5237000000000002e-05 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.872827+0000", 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8229000000000001e-05 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.872923+0000", 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6630000000000001e-05 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.873053+0000", 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012997900000000001 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.873160+0000", 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106629 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.881798+0000", 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086382239999999999 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.913645+0000", 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031846929000000003 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.913689+0000", 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3761000000000003e-05 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.913728+0000", 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9561999999999998e-05 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.283 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1733 3.7 3:e2edadef:::100000001f8.000002c1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.139013+0000", 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.0646051600000002, 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049403496999999998, 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1733 2024-04-25T15:53:50.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.139013+0000", 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.139013+0000", 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.139015+0000", 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5680000000000001e-06 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.140535+0000", 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00152005 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.140571+0000", 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5831000000000003e-05 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.140584+0000", 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3244999999999999e-05 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.140673+0000", 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8467999999999994e-05 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.140857+0000", 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018436200000000001 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.141033+0000", 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017637100000000001 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.151637+0000", 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010603668 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.188293+0000", 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036656053000000001 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.188362+0000", 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9040000000000003e-05 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.188417+0000", 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4840999999999999e-05 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1763 3.33 3:ccce5726:::100000001f8.000002df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.324114+0000", 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.879504764, 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050936052000000002, 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.289 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1763 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.324114+0000", 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.324114+0000", 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.324116+0000", 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5229999999999998e-06 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.326011+0000", 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001895021 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.326040+0000", 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8983999999999999e-05 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.326054+0000", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3171e-05 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.326146+0000", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2225000000000006e-05 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.326344+0000", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000198381 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.326524+0000", 2024-04-25T15:53:50.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018017400000000001 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.337817+0000", 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011292431 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.374947+0000", 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037130761999999998 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.374990+0000", 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2982000000000003e-05 2024-04-25T15:53:50.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.375050+0000", 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9398000000000003e-05 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1767 3.38 3:1fae2fe4:::100000001f8.000002e3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.344993+0000", 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8586250929999997, 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046239153999999998, 2024-04-25T15:53:50.294 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1767 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.344993+0000", 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.344993+0000", 2024-04-25T15:53:50.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.344997+0000", 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.765e-06 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.346790+0000", 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017927209999999999 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.346825+0000", 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4604999999999998e-05 2024-04-25T15:53:50.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.346842+0000", 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7588000000000002e-05 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.346934+0000", 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2244000000000007e-05 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.347064+0000", 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012999299999999999 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.347164+0000", 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9958999999999996e-05 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.353648+0000", 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006484114 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.391119+0000", 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037470261999999997 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.391171+0000", 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2153e-05 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.391233+0000", 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1749999999999997e-05 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1770 3.8 3:12b6887e:::100000001f8.000002e6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.359788+0000", 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8438307009999999, 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042666463000000002, 2024-04-25T15:53:50.299 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1770 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.359788+0000", 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.359788+0000", 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.359792+0000", 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9249999999999997e-06 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.361569+0000", 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00177709 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.361603+0000", 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4477999999999998e-05 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.361617+0000", 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3769e-05 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.361692+0000", 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4760999999999998e-05 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.361822+0000", 2024-04-25T15:53:50.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013024400000000001 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.361924+0000", 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010207 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.368393+0000", 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064683370000000002 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.402314+0000", 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033921933000000001 2024-04-25T15:53:50.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.402386+0000", 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1826e-05 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.402454+0000", 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8029999999999997e-05 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2313 3.1e 3:7a2a8d26:::100000001f8.00000505:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:53:50.304 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.338969332, 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091581081999999994, 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2313 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864653+0000", 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866695+0000", 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002042744 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866732+0000", 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7054000000000001e-05 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866765+0000", 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2141999999999998e-05 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866853+0000", 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7950999999999999e-05 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866986+0000", 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013319800000000001 2024-04-25T15:53:50.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.867189+0000", 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020333800000000001 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.883954+0000", 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016764729999999999 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956056+0000", 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072101922999999998 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956116+0000", 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9815000000000002e-05 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956230+0000", 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114817 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2314 3.29 3:9452ed8f:::100000001f8.00000506:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.871348+0000", 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.332270244, 2024-04-25T15:53:50.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.084828639999999997, 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2314 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.871348+0000", 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.871348+0000", 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.871370+0000", 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1458e-05 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873408+0000", 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020381259999999999 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873460+0000", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2216999999999998e-05 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873478+0000", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7481000000000001e-05 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873607+0000", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012908599999999999 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873821+0000", 2024-04-25T15:53:50.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000214322 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.874006+0000", 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018467099999999999 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.890192+0000", 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016186048000000001 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956045+0000", 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.065853729999999999 2024-04-25T15:53:50.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956112+0000", 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6701999999999999e-05 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956177+0000", 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799000000000001e-05 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2315 3.19 3:99bca703:::100000001f8.00000507:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:50.314 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.879908+0000", 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.323711018, 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.076440714000000007, 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2315 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.879908+0000", 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.879908+0000", 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.879910+0000", 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.097e-06 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881780+0000", 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001870402 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881836+0000", 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5985e-05 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881855+0000", 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8909000000000001e-05 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881937+0000", 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1948999999999998e-05 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.317 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.882115+0000", 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017834099999999999 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.882267+0000", 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151729 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.890850+0000", 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085832870000000002 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956251+0000", 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.065400729000000005 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956304+0000", 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3235000000000001e-05 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956348+0000", 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4051000000000002e-05 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2391 3.7 3:e256bd88:::100000001f8.00000553:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.400167+0000", 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.803452006, 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056655628999999999, 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2391 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.400167+0000", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.400167+0000", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.400169+0000", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.419e-06 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.401638+0000", 2024-04-25T15:53:50.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001468682 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.401671+0000", 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2895999999999998e-05 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.401683+0000", 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2924e-05 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.401766+0000", 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2467999999999997e-05 2024-04-25T15:53:50.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.401943+0000", 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017711699999999999 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.402080+0000", 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013661599999999999 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.417613+0000", 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015533234999999999 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.456678+0000", 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039065074999999998 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.456757+0000", 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9382999999999995e-05 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.456822+0000", 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4814000000000007e-05 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.324 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2393 3.22 e90/45 3:45fb7032:::100000001f8.00000555:head v 90'36, mlcod=90'34)", 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.414474+0000", 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.78914506399999995, 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050079559000000003, 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414474+0000", 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414474+0000", 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414475+0000", 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.404e-06 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.416416+0000", 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019409259999999999 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.416435+0000", 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9114999999999999e-05 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.416449+0000", 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3801000000000001e-05 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.416529+0000", 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9935e-05 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:50.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.416553+0000", 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4012999999999999e-05 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464449+0000", 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047896437 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464553+0000", 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103928 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:50.328 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:50.329 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:50.330 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:50.330+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:50.431 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:50.431+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:50.523 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:50.523 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2551 3.1d e90/76 3:b8839f0d:::100000001f8.000005f3:head v 90'59, mlcod=90'57)", 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.512667+0000", 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.010870698999999999, 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010922326, 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.512667+0000", 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.512667+0000", 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.512668+0000", 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.172e-06 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.515066+0000", 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023976190000000001 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:50.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.515086+0000", 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.995e-05 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.515100+0000", 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4208e-05 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.515181+0000", 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1144999999999996e-05 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:50.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.515218+0000", 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6922000000000003e-05 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 1 2024-04-25T15:53:50.527 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:50.532 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:50.530+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:50.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee584cf640 1 Processor -- start 2024-04-25T15:53:50.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee584cf640 1 -- start start 2024-04-25T15:53:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee584cf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee50074370 0x7fee50074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee584cf640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee500c92d0 con 0x7fee5006c640 2024-04-25T15:53:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee584cf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5006e210 con 0x7fee500c7fc0 2024-04-25T15:53:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee584cf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fee5006e4d0 con 0x7fee50074370 2024-04-25T15:53:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee55a6a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee50074370 0x7fee50074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee5626b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee5006c640 0x7fee5015b020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35582/0 (socket says 172.21.15.67:35582) 2024-04-25T15:53:50.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee55a6a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee50074370 0x7fee50074780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43814/0 (socket says 172.21.15.67:43814) 2024-04-25T15:53:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee5626b640 1 -- 172.21.15.67:0/3970446564 learned_addr learned my addr 172.21.15.67:0/3970446564 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee55a6a640 1 -- 172.21.15.67:0/3970446564 >> v1:172.21.15.176:6789/0 conn(0x7fee500c7fc0 legacy=0x7fee50164f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee55a6a640 1 -- 172.21.15.67:0/3970446564 >> v1:172.21.15.67:6789/0 conn(0x7fee5006c640 legacy=0x7fee5015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.534+0000 7fee55a6a640 1 -- 172.21.15.67:0/3970446564 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5006e7b0 con 0x7fee50074370 2024-04-25T15:53:50.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee55a6a640 1 --2- 172.21.15.67:0/3970446564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee50074370 0x7fee50074780 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fee400097c0 tx=0x7fee4001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8584207b3898ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee55269640 1 -- 172.21.15.67:0/3970446564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee400b5070 con 0x7fee50074370 2024-04-25T15:53:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee55269640 1 -- 172.21.15.67:0/3970446564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee400a8950 con 0x7fee50074370 2024-04-25T15:53:50.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee55269640 1 -- 172.21.15.67:0/3970446564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee400a8d20 con 0x7fee50074370 2024-04-25T15:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee584cf640 1 -- 172.21.15.67:0/3970446564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee50074370 msgr2=0x7fee50074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee584cf640 1 --2- 172.21.15.67:0/3970446564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee50074370 0x7fee50074780 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fee400097c0 tx=0x7fee4001ec50 comp rx=0 tx=0).stop 2024-04-25T15:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee584cf640 1 -- 172.21.15.67:0/3970446564 shutdown_connections 2024-04-25T15:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee584cf640 1 --2- 172.21.15.67:0/3970446564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee50074370 0x7fee50074780 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee584cf640 1 -- 172.21.15.67:0/3970446564 >> 172.21.15.67:0/3970446564 conn(0x7fee50070db0 msgr2=0x7fee500ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee584cf640 1 -- 172.21.15.67:0/3970446564 shutdown_connections 2024-04-25T15:53:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.538+0000 7fee584cf640 1 -- 172.21.15.67:0/3970446564 wait complete. 2024-04-25T15:53:50.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 Processor -- start 2024-04-25T15:53:50.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 -- start start 2024-04-25T15:53:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee5006c640 0x7fee50156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee50074370 0x7fee5015a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee500c7fc0 0x7fee5015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fee50073e50 con 0x7fee5006c640 2024-04-25T15:53:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fee50073b70 con 0x7fee500c7fc0 2024-04-25T15:53:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee584cf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee50073420 con 0x7fee50074370 2024-04-25T15:53:50.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee5626b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee5006c640 0x7fee50156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee5626b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee5006c640 0x7fee50156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43818/0 (socket says 172.21.15.67:43818) 2024-04-25T15:53:50.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.542+0000 7fee5626b640 1 -- 172.21.15.67:0/4113081418 learned_addr learned my addr 172.21.15.67:0/4113081418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:50.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee5626b640 1 -- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee50074370 msgr2=0x7fee5015a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee5626b640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee50074370 0x7fee5015a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee5626b640 1 -- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee500c7fc0 msgr2=0x7fee5015d880 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee5626b640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee500c7fc0 0x7fee5015d880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee5626b640 1 -- 172.21.15.67:0/4113081418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5015ace0 con 0x7fee5006c640 2024-04-25T15:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee5626b640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee5006c640 0x7fee50156d00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00c370 tx=0x7fee4c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee46ffd640 1 -- 172.21.15.67:0/4113081418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c017070 con 0x7fee5006c640 2024-04-25T15:53:50.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee46ffd640 1 -- 172.21.15.67:0/4113081418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee4c005cf0 con 0x7fee5006c640 2024-04-25T15:53:50.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee50159740 con 0x7fee5006c640 2024-04-25T15:53:50.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee46ffd640 1 -- 172.21.15.67:0/4113081418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c0072f0 con 0x7fee5006c640 2024-04-25T15:53:50.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.546+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee501599e0 con 0x7fee5006c640 2024-04-25T15:53:50.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.550+0000 7fee46ffd640 1 -- 172.21.15.67:0/4113081418 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fee4c01a070 con 0x7fee5006c640 2024-04-25T15:53:50.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.550+0000 7fee46ffd640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee3406f400 0x7fee34071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.550+0000 7fee46ffd640 1 -- 172.21.15.67:0/4113081418 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5944+0+0 (secure 0 0 0) 0x7fee4c093060 con 0x7fee5006c640 2024-04-25T15:53:50.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.550+0000 7fee55a6a640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee3406f400 0x7fee34071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.550+0000 7fee55a6a640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee3406f400 0x7fee34071910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fee400b2000 tx=0x7fee400aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.554+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee5006a480 con 0x7fee5006c640 2024-04-25T15:53:50.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:50.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.562+0000 7fee46ffd640 1 -- 172.21.15.67:0/4113081418 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fee4c05b2a0 con 0x7fee5006c640 2024-04-25T15:53:50.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:50.643 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:50.643+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:50.733 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:50.731+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.770+0000 7f61c6291640 1 Processor -- start 2024-04-25T15:53:50.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.770+0000 7f61c6291640 1 -- start start 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.770+0000 7f61c6291640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f61c0160800 0x7f61c006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c6291640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61c00c96e0 0x7f61c0167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c6291640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c006d8e0 con 0x7f61c00c92d0 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c6291640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f61c006dbc0 con 0x7f61c00c96e0 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c6291640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f61c0073c70 con 0x7f61c0160800 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c482e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61c00c96e0 0x7f61c0167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c482e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61c00c96e0 0x7f61c0167180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43828/0 (socket says 172.21.15.67:43828) 2024-04-25T15:53:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c482e640 1 -- 172.21.15.67:0/146422013 learned_addr learned my addr 172.21.15.67:0/146422013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61bffff640 1 --2- 172.21.15.67:0/146422013 >> v2:172.21.15.176:3300/0 conn(0x7f61c0160800 0x7f61c006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c482e640 1 -- 172.21.15.67:0/146422013 >> v2:172.21.15.176:3300/0 conn(0x7f61c0160800 msgr2=0x7f61c006d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c482e640 1 --2- 172.21.15.67:0/146422013 >> v2:172.21.15.176:3300/0 conn(0x7f61c0160800 0x7f61c006d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c482e640 1 -- 172.21.15.67:0/146422013 >> v1:172.21.15.67:6789/0 conn(0x7f61c00c92d0 legacy=0x7f61c0072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61c482e640 1 -- 172.21.15.67:0/146422013 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c0073f50 con 0x7f61c00c96e0 2024-04-25T15:53:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.774+0000 7f61bffff640 1 --2- 172.21.15.67:0/146422013 >> v2:172.21.15.176:3300/0 conn(0x7f61c0160800 0x7f61c006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61c482e640 1 --2- 172.21.15.67:0/146422013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c0167180 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f61c006dd40 tx=0x7f61b801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3547882e815de2e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61beffd640 1 -- 172.21.15.67:0/146422013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80a5710 con 0x7f61c00c96e0 2024-04-25T15:53:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61beffd640 1 -- 172.21.15.67:0/146422013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b80a58b0 con 0x7f61c00c96e0 2024-04-25T15:53:50.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61beffd640 1 -- 172.21.15.67:0/146422013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80a5c80 con 0x7f61c00c96e0 2024-04-25T15:53:50.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61c6291640 1 -- 172.21.15.67:0/146422013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 msgr2=0x7f61c0167180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61c6291640 1 --2- 172.21.15.67:0/146422013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c0167180 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f61c006dd40 tx=0x7f61b801c960 comp rx=0 tx=0).stop 2024-04-25T15:53:50.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61c6291640 1 -- 172.21.15.67:0/146422013 shutdown_connections 2024-04-25T15:53:50.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61c6291640 1 --2- 172.21.15.67:0/146422013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c0167180 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61c6291640 1 --2- 172.21.15.67:0/146422013 >> v2:172.21.15.176:3300/0 conn(0x7f61c0160800 0x7f61c006d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.778+0000 7f61c6291640 1 -- 172.21.15.67:0/146422013 >> 172.21.15.67:0/146422013 conn(0x7f61c0070db0 msgr2=0x7f61c00ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:50.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.782+0000 7f61c6291640 1 -- 172.21.15.67:0/146422013 shutdown_connections 2024-04-25T15:53:50.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.782+0000 7f61c6291640 1 -- 172.21.15.67:0/146422013 wait complete. 2024-04-25T15:53:50.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.782+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee500cb070 con 0x7fee3406f400 2024-04-25T15:53:50.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.782+0000 7f61c6291640 1 Processor -- start 2024-04-25T15:53:50.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.782+0000 7f61c6291640 1 -- start start 2024-04-25T15:53:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c6291640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61c00c92d0 0x7f61c01f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c6291640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c00c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c6291640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61c0160800 0x7f61c014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c00c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c00c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43838/0 (socket says 172.21.15.67:43838) 2024-04-25T15:53:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 -- 172.21.15.67:0/1665756708 learned_addr learned my addr 172.21.15.67:0/1665756708 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61c0160c10 con 0x7f61c00c96e0 2024-04-25T15:53:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f61c006d570 con 0x7f61c0160800 2024-04-25T15:53:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f61c0160d90 con 0x7f61c00c92d0 2024-04-25T15:53:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61c00c92d0 0x7f61c01f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c482e640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61c0160800 0x7f61c014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61c00c92d0 msgr2=0x7f61c01f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61c00c92d0 0x7f61c01f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61c0160800 msgr2=0x7f61c014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61c0160800 0x7f61c014ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c01577c0 con 0x7f61c00c96e0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bf7fe640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c00c7700 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f61b400c420 tx=0x7f61b400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4017070 con 0x7f61c00c96e0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b40052d0 con 0x7f61c00c96e0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c0159d00 con 0x7f61c00c96e0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.786+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4003840 con 0x7f61c00c96e0 2024-04-25T15:53:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.790+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c0159fa0 con 0x7f61c00c96e0 2024-04-25T15:53:50.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.790+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f61c006a480 con 0x7f61c00c96e0 2024-04-25T15:53:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.790+0000 7fee46ffd640 1 -- 172.21.15.67:0/4113081418 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254046 (secure 0 0 0) 0x7fee500cb070 con 0x7fee3406f400 2024-04-25T15:53:50.793 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:53:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:53:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.790+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f61b401a070 con 0x7f61c00c96e0 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.790+0000 7f61bcff9640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f61ac06f430 0x7f61ac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5944+0+0 (secure 0 0 0) 0x7f61b408eca0 con 0x7f61c00c96e0 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f61ac06f430 0x7f61ac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v90) ==== 54+0+16 (secure 0 0 0) 0x7f61b4021070 con 0x7f61c00c96e0 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f61c0150520 0x7f61c00ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61c00c8c50 con 0x7f61c0150520 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61c482e640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f61c0150520 0x7f61c00ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61c482e640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f61c0150520 0x7f61c00ce930 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f61ac06f430 0x7f61ac071940 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f61b00045f0 tx=0x7f61b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c00c8c50 con 0x7f61c0150520 2024-04-25T15:53:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee3406f400 msgr2=0x7fee34071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7fee584cf640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee3406f400 0x7fee34071910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fee400b2000 tx=0x7fee400aa070 comp rx=0 tx=0).stop 2024-04-25T15:53:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee5006c640 msgr2=0x7fee50156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.794+0000 7fee584cf640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee5006c640 0x7fee50156d00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00c370 tx=0x7fee4c00c830 comp rx=0 tx=0).stop 2024-04-25T15:53:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 shutdown_connections 2024-04-25T15:53:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fee3406f400 0x7fee34071910 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee500c7fc0 0x7fee5015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee50074370 0x7fee5015a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 --2- 172.21.15.67:0/4113081418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee5006c640 0x7fee50156d00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 >> 172.21.15.67:0/4113081418 conn(0x7fee50070db0 msgr2=0x7fee500ca360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 shutdown_connections 2024-04-25T15:53:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.798+0000 7fee584cf640 1 -- 172.21.15.67:0/4113081418 wait complete. 2024-04-25T15:53:50.808 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:50.808 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:50.808 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:50.808 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:50.808 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.818+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c0073920 con 0x7f61c0150520 2024-04-25T15:53:50.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.818+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c0073920 con 0x7f61c0150520 2024-04-25T15:53:50.819 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:50.820 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:50.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.818+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f61c01f66b0 0x7f61c01f9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.818+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f61c006c6a0 con 0x7f61c01f66b0 2024-04-25T15:53:50.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.818+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f61c01f66b0 0x7f61c01f9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.818+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f61c01f66b0 0x7f61c01f9710 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.822+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c006c6a0 con 0x7f61c01f66b0 2024-04-25T15:53:50.827 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-04-25T15:53:49.867541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4765495770,"num_objects":1237,"num_object_clones":0,"num_object_copies":2474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1237,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":3080,"num_write_kb":8849253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":557849859,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3019,"ondisk_log_size":3019,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8938876,"kb_used_data":8673064,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":746035844,"statfs":{"total":773094113280,"available":763940704256,"internally_reserved":0,"allocated":8881217536,"data_stored":8876456555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":113,"apply_latency_ms":113,"commit_latency_ns":113000000,"apply_latency_ns":113000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3691042875,"num_objects":880,"num_object_clones":0,"num_object_copies":1760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1421,"num_write_kb":4653120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003040"},"pg_stats":[{"pgid":"3.3f","version":"89'42","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325871+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:53:47.325871+0000","last_peered":"2024-04-25T15:53:47.325871+0000","last_clean":"2024-04-25T15:53:47.325871+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:53:47.325871+0000","last_undegraded":"2024-04-25T15:53:47.325871+0000","last_fullsized":"2024-04-25T15:53:47.325871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"90'35","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315705+0000","last_change":"2024-04-25T15:53:13.069367+0000","last_active":"2024-04-25T15:53:47.315705+0000","last_peered":"2024-04-25T15:53:47.315705+0000","last_clean":"2024-04-25T15:53:47.315705+0000","last_became_active":"2024-04-25T15:53:12.990829+0000","last_became_peered":"2024-04-25T15:53:12.990829+0000","last_unstale":"2024-04-25T15:53:47.315705+0000","last_undegraded":"2024-04-25T15:53:47.315705+0000","last_fullsized":"2024-04-25T15:53:47.315705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:46.276338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"90'36","reported_seq":144,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315586+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:53:47.315586+0000","last_peered":"2024-04-25T15:53:47.315586+0000","last_clean":"2024-04-25T15:53:47.315586+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:53:47.315586+0000","last_undegraded":"2024-04-25T15:53:47.315586+0000","last_fullsized":"2024-04-25T15:53:47.315586+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"90'34","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.731756+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:53:47.731756+0000","last_peered":"2024-04-25T15:53:47.731756+0000","last_clean":"2024-04-25T15:53:47.731756+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:53:47.731756+0000","last_undegraded":"2024-04-25T15:53:47.731756+0000","last_fullsized":"2024-04-25T15:53:47.731756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"90'42","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325140+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:53:47.325140+0000","last_peered":"2024-04-25T15:53:47.325140+0000","last_clean":"2024-04-25T15:53:47.325140+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:53:47.325140+0000","last_undegraded":"2024-04-25T15:53:47.325140+0000","last_fullsized":"2024-04-25T15:53:47.325140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":135168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'38","reported_seq":191,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316421+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:53:47.316421+0000","last_peered":"2024-04-25T15:53:47.316421+0000","last_clean":"2024-04-25T15:53:47.316421+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:53:47.316421+0000","last_undegraded":"2024-04-25T15:53:47.316421+0000","last_fullsized":"2024-04-25T15:53:47.316421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"90'51","reported_seq":202,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.574674+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:53:47.574674+0000","last_peered":"2024-04-25T15:53:47.574674+0000","last_clean":"2024-04-25T15:53:47.574674+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:53:47.574674+0000","last_undegraded":"2024-04-25T15:53:47.574674+0000","last_fullsized":"2024-04-25T15:53:47.574674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":176128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"90'50","reported_seq":203,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354741+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:53:47.354741+0000","last_peered":"2024-04-25T15:53:47.354741+0000","last_clean":"2024-04-25T15:53:47.354741+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:53:47.354741+0000","last_undegraded":"2024-04-25T15:53:47.354741+0000","last_fullsized":"2024-04-25T15:53:47.354741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":172032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"90'45","reported_seq":191,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.412791+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:53:47.412791+0000","last_peered":"2024-04-25T15:53:47.412791+0000","last_clean":"2024-04-25T15:53:47.412791+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:53:47.412791+0000","last_undegraded":"2024-04-25T15:53:47.412791+0000","last_fullsized":"2024-04-25T15:53:47.412791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"89'46","reported_seq":199,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355153+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:53:47.355153+0000","last_peered":"2024-04-25T15:53:47.355153+0000","last_clean":"2024-04-25T15:53:47.355153+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:53:47.355153+0000","last_undegraded":"2024-04-25T15:53:47.355153+0000","last_fullsized":"2024-04-25T15:53:47.355153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":151552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"90'45","reported_seq":198,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.744603+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:53:47.744603+0000","last_peered":"2024-04-25T15:53:47.744603+0000","last_clean":"2024-04-25T15:53:47.744603+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:53:47.744603+0000","last_undegraded":"2024-04-25T15:53:47.744603+0000","last_fullsized":"2024-04-25T15:53:47.744603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":155648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'46","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.548334+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:53:47.548334+0000","last_peered":"2024-04-25T15:53:47.548334+0000","last_clean":"2024-04-25T15:53:47.548334+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:53:47.548334+0000","last_undegraded":"2024-04-25T15:53:47.548334+0000","last_fullsized":"2024-04-25T15:53:47.548334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":143360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"90'36","reported_seq":189,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354947+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:53:47.354947+0000","last_peered":"2024-04-25T15:53:47.354947+0000","last_clean":"2024-04-25T15:53:47.354947+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:53:47.354947+0000","last_undegraded":"2024-04-25T15:53:47.354947+0000","last_fullsized":"2024-04-25T15:53:47.354947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"90'41","reported_seq":195,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.391663+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:53:47.391663+0000","last_peered":"2024-04-25T15:53:47.391663+0000","last_clean":"2024-04-25T15:53:47.391663+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:53:47.391663+0000","last_undegraded":"2024-04-25T15:53:47.391663+0000","last_fullsized":"2024-04-25T15:53:47.391663+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'37","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.518281+0000","last_change":"2024-04-25T15:53:17.148839+0000","last_active":"2024-04-25T15:53:47.518281+0000","last_peered":"2024-04-25T15:53:47.518281+0000","last_clean":"2024-04-25T15:53:47.518281+0000","last_became_active":"2024-04-25T15:53:16.064416+0000","last_became_peered":"2024-04-25T15:53:16.064416+0000","last_unstale":"2024-04-25T15:53:47.518281+0000","last_undegraded":"2024-04-25T15:53:47.518281+0000","last_fullsized":"2024-04-25T15:53:47.518281+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:58.798689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"90'52","reported_seq":205,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316248+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:53:47.316248+0000","last_peered":"2024-04-25T15:53:47.316248+0000","last_clean":"2024-04-25T15:53:47.316248+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:53:47.316248+0000","last_undegraded":"2024-04-25T15:53:47.316248+0000","last_fullsized":"2024-04-25T15:53:47.316248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":172032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"90'30","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316631+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:53:47.316631+0000","last_peered":"2024-04-25T15:53:47.316631+0000","last_clean":"2024-04-25T15:53:47.316631+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:53:47.316631+0000","last_undegraded":"2024-04-25T15:53:47.316631+0000","last_fullsized":"2024-04-25T15:53:47.316631+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"90'29","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315832+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:53:47.315832+0000","last_peered":"2024-04-25T15:53:47.315832+0000","last_clean":"2024-04-25T15:53:47.315832+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:53:47.315832+0000","last_undegraded":"2024-04-25T15:53:47.315832+0000","last_fullsized":"2024-04-25T15:53:47.315832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"90'46","reported_seq":197,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.479432+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:53:47.479432+0000","last_peered":"2024-04-25T15:53:47.479432+0000","last_clean":"2024-04-25T15:53:47.479432+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:53:47.479432+0000","last_undegraded":"2024-04-25T15:53:47.479432+0000","last_fullsized":"2024-04-25T15:53:47.479432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":163840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"89'38","reported_seq":207,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355124+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:53:47.355124+0000","last_peered":"2024-04-25T15:53:47.355124+0000","last_clean":"2024-04-25T15:53:47.355124+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:53:47.355124+0000","last_undegraded":"2024-04-25T15:53:47.355124+0000","last_fullsized":"2024-04-25T15:53:47.355124+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"90'33","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.362361+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:53:47.362361+0000","last_peered":"2024-04-25T15:53:47.362361+0000","last_clean":"2024-04-25T15:53:47.362361+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:53:47.362361+0000","last_undegraded":"2024-04-25T15:53:47.362361+0000","last_fullsized":"2024-04-25T15:53:47.362361+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"90'50","reported_seq":196,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.557989+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:53:47.557989+0000","last_peered":"2024-04-25T15:53:47.557989+0000","last_clean":"2024-04-25T15:53:47.557989+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:53:47.557989+0000","last_undegraded":"2024-04-25T15:53:47.557989+0000","last_fullsized":"2024-04-25T15:53:47.557989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":167936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"90'40","reported_seq":209,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.381596+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:53:47.381596+0000","last_peered":"2024-04-25T15:53:47.381596+0000","last_clean":"2024-04-25T15:53:47.381596+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:53:47.381596+0000","last_undegraded":"2024-04-25T15:53:47.381596+0000","last_fullsized":"2024-04-25T15:53:47.381596+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'57","reported_seq":210,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.606074+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:53:47.606074+0000","last_peered":"2024-04-25T15:53:47.606074+0000","last_clean":"2024-04-25T15:53:47.606074+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:53:47.606074+0000","last_undegraded":"2024-04-25T15:53:47.606074+0000","last_fullsized":"2024-04-25T15:53:47.606074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":196608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"89'31","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325590+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:53:47.325590+0000","last_peered":"2024-04-25T15:53:47.325590+0000","last_clean":"2024-04-25T15:53:47.325590+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:53:47.325590+0000","last_undegraded":"2024-04-25T15:53:47.325590+0000","last_fullsized":"2024-04-25T15:53:47.325590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"90'56","reported_seq":209,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355324+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:53:47.355324+0000","last_peered":"2024-04-25T15:53:47.355324+0000","last_clean":"2024-04-25T15:53:47.355324+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:53:47.355324+0000","last_undegraded":"2024-04-25T15:53:47.355324+0000","last_fullsized":"2024-04-25T15:53:47.355324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":176128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'39","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355520+0000","last_change":"2024-04-25T15:53:03.808515+0000","last_active":"2024-04-25T15:53:47.355520+0000","last_peered":"2024-04-25T15:53:47.355520+0000","last_clean":"2024-04-25T15:53:47.355520+0000","last_became_active":"2024-04-25T15:53:03.367253+0000","last_became_peered":"2024-04-25T15:53:03.367253+0000","last_unstale":"2024-04-25T15:53:47.355520+0000","last_undegraded":"2024-04-25T15:53:47.355520+0000","last_fullsized":"2024-04-25T15:53:47.355520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:05.644251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":135168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"90'42","reported_seq":230,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355148+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:53:47.355148+0000","last_peered":"2024-04-25T15:53:47.355148+0000","last_clean":"2024-04-25T15:53:47.355148+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:53:47.355148+0000","last_undegraded":"2024-04-25T15:53:47.355148+0000","last_fullsized":"2024-04-25T15:53:47.355148+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"90'37","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.843811+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:53:47.843811+0000","last_peered":"2024-04-25T15:53:47.843811+0000","last_clean":"2024-04-25T15:53:47.843811+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:53:47.843811+0000","last_undegraded":"2024-04-25T15:53:47.843811+0000","last_fullsized":"2024-04-25T15:53:47.843811+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"89'30","reported_seq":127,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.391313+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:53:47.391313+0000","last_peered":"2024-04-25T15:53:47.391313+0000","last_clean":"2024-04-25T15:53:47.391313+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:53:47.391313+0000","last_undegraded":"2024-04-25T15:53:47.391313+0000","last_fullsized":"2024-04-25T15:53:47.391313+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"90'42","reported_seq":204,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315428+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:53:47.315428+0000","last_peered":"2024-04-25T15:53:47.315428+0000","last_clean":"2024-04-25T15:53:47.315428+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:53:47.315428+0000","last_undegraded":"2024-04-25T15:53:47.315428+0000","last_fullsized":"2024-04-25T15:53:47.315428+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"89'48","reported_seq":201,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355926+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:53:47.355926+0000","last_peered":"2024-04-25T15:53:47.355926+0000","last_clean":"2024-04-25T15:53:47.355926+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:53:47.355926+0000","last_undegraded":"2024-04-25T15:53:47.355926+0000","last_fullsized":"2024-04-25T15:53:47.355926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"89'38","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315031+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:53:47.315031+0000","last_peered":"2024-04-25T15:53:47.315031+0000","last_clean":"2024-04-25T15:53:47.315031+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:53:47.315031+0000","last_undegraded":"2024-04-25T15:53:47.315031+0000","last_fullsized":"2024-04-25T15:53:47.315031+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":161,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354987+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:53:47.354987+0000","last_peered":"2024-04-25T15:53:47.354987+0000","last_clean":"2024-04-25T15:53:47.354987+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:53:47.354987+0000","last_undegraded":"2024-04-25T15:53:47.354987+0000","last_fullsized":"2024-04-25T15:53:47.354987+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"90'40","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315818+0000","last_change":"2024-04-25T15:53:16.510837+0000","last_active":"2024-04-25T15:53:47.315818+0000","last_peered":"2024-04-25T15:53:47.315818+0000","last_clean":"2024-04-25T15:53:47.315818+0000","last_became_active":"2024-04-25T15:53:16.041557+0000","last_became_peered":"2024-04-25T15:53:16.041557+0000","last_unstale":"2024-04-25T15:53:47.315818+0000","last_undegraded":"2024-04-25T15:53:47.315818+0000","last_fullsized":"2024-04-25T15:53:47.315818+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:06.600570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":188,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355604+0000","last_change":"2024-04-25T15:53:16.102712+0000","last_active":"2024-04-25T15:53:47.355604+0000","last_peered":"2024-04-25T15:53:47.355604+0000","last_clean":"2024-04-25T15:53:47.355604+0000","last_became_active":"2024-04-25T15:53:16.052729+0000","last_became_peered":"2024-04-25T15:53:16.052729+0000","last_unstale":"2024-04-25T15:53:47.355604+0000","last_undegraded":"2024-04-25T15:53:47.355604+0000","last_fullsized":"2024-04-25T15:53:47.355604+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:53.161289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":158,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355247+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:53:47.355247+0000","last_peered":"2024-04-25T15:53:47.355247+0000","last_clean":"2024-04-25T15:53:47.355247+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:53:47.355247+0000","last_undegraded":"2024-04-25T15:53:47.355247+0000","last_fullsized":"2024-04-25T15:53:47.355247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"89'45","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.391277+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:53:47.391277+0000","last_peered":"2024-04-25T15:53:47.391277+0000","last_clean":"2024-04-25T15:53:47.391277+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:53:47.391277+0000","last_undegraded":"2024-04-25T15:53:47.391277+0000","last_fullsized":"2024-04-25T15:53:47.391277+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":155648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":176,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316315+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:53:47.316315+0000","last_peered":"2024-04-25T15:53:47.316315+0000","last_clean":"2024-04-25T15:53:47.316315+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:53:47.316315+0000","last_undegraded":"2024-04-25T15:53:47.316315+0000","last_fullsized":"2024-04-25T15:53:47.316315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'38","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354653+0000","last_change":"2024-04-25T15:53:16.206656+0000","last_active":"2024-04-25T15:53:47.354653+0000","last_peered":"2024-04-25T15:53:47.354653+0000","last_clean":"2024-04-25T15:53:47.354653+0000","last_became_active":"2024-04-25T15:53:16.049908+0000","last_became_peered":"2024-04-25T15:53:16.049908+0000","last_unstale":"2024-04-25T15:53:47.354653+0000","last_undegraded":"2024-04-25T15:53:47.354653+0000","last_fullsized":"2024-04-25T15:53:47.354653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.217418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":161,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355157+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:53:47.355157+0000","last_peered":"2024-04-25T15:53:47.355157+0000","last_clean":"2024-04-25T15:53:47.355157+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:53:47.355157+0000","last_undegraded":"2024-04-25T15:53:47.355157+0000","last_fullsized":"2024-04-25T15:53:47.355157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"90'64","reported_seq":217,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.477292+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:53:47.477292+0000","last_peered":"2024-04-25T15:53:47.477292+0000","last_clean":"2024-04-25T15:53:47.477292+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:53:47.477292+0000","last_undegraded":"2024-04-25T15:53:47.477292+0000","last_fullsized":"2024-04-25T15:53:47.477292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":217088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316848+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:53:47.316848+0000","last_peered":"2024-04-25T15:53:47.316848+0000","last_clean":"2024-04-25T15:53:47.316848+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:53:47.316848+0000","last_undegraded":"2024-04-25T15:53:47.316848+0000","last_fullsized":"2024-04-25T15:53:47.316848+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"89'40","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316905+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:53:47.316905+0000","last_peered":"2024-04-25T15:53:47.316905+0000","last_clean":"2024-04-25T15:53:47.316905+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:53:47.316905+0000","last_undegraded":"2024-04-25T15:53:47.316905+0000","last_fullsized":"2024-04-25T15:53:47.316905+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"88'8","reported_seq":114,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316895+0000","last_change":"2024-04-25T15:53:03.435546+0000","last_active":"2024-04-25T15:53:47.316895+0000","last_peered":"2024-04-25T15:53:47.316895+0000","last_clean":"2024-04-25T15:53:47.316895+0000","last_became_active":"2024-04-25T15:53:03.387235+0000","last_became_peered":"2024-04-25T15:53:03.387235+0000","last_unstale":"2024-04-25T15:53:47.316895+0000","last_undegraded":"2024-04-25T15:53:47.316895+0000","last_fullsized":"2024-04-25T15:53:47.316895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:30.444213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1344,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'40","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.585743+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:53:47.585743+0000","last_peered":"2024-04-25T15:53:47.585743+0000","last_clean":"2024-04-25T15:53:47.585743+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:53:47.585743+0000","last_undegraded":"2024-04-25T15:53:47.585743+0000","last_fullsized":"2024-04-25T15:53:47.585743+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":135168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"89'78","reported_seq":233,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355339+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:53:47.355339+0000","last_peered":"2024-04-25T15:53:47.355339+0000","last_clean":"2024-04-25T15:53:47.355339+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:53:47.355339+0000","last_undegraded":"2024-04-25T15:53:47.355339+0000","last_fullsized":"2024-04-25T15:53:47.355339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168111,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"89'44","reported_seq":195,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325584+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:53:47.325584+0000","last_peered":"2024-04-25T15:53:47.325584+0000","last_clean":"2024-04-25T15:53:47.325584+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:53:47.325584+0000","last_undegraded":"2024-04-25T15:53:47.325584+0000","last_fullsized":"2024-04-25T15:53:47.325584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315894+0000","last_change":"2024-04-25T15:53:16.094342+0000","last_active":"2024-04-25T15:53:47.315894+0000","last_peered":"2024-04-25T15:53:47.315894+0000","last_clean":"2024-04-25T15:53:47.315894+0000","last_became_active":"2024-04-25T15:53:16.041484+0000","last_became_peered":"2024-04-25T15:53:16.041484+0000","last_unstale":"2024-04-25T15:53:47.315894+0000","last_undegraded":"2024-04-25T15:53:47.315894+0000","last_fullsized":"2024-04-25T15:53:47.315894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:42.531801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"89'45","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325027+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:53:47.325027+0000","last_peered":"2024-04-25T15:53:47.325027+0000","last_clean":"2024-04-25T15:53:47.325027+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:53:47.325027+0000","last_undegraded":"2024-04-25T15:53:47.325027+0000","last_fullsized":"2024-04-25T15:53:47.325027+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":143360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":160,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316370+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:53:47.316370+0000","last_peered":"2024-04-25T15:53:47.316370+0000","last_clean":"2024-04-25T15:53:47.316370+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:53:47.316370+0000","last_undegraded":"2024-04-25T15:53:47.316370+0000","last_fullsized":"2024-04-25T15:53:47.316370+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"90'40","reported_seq":211,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325201+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:53:47.325201+0000","last_peered":"2024-04-25T15:53:47.325201+0000","last_clean":"2024-04-25T15:53:47.325201+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:53:47.325201+0000","last_undegraded":"2024-04-25T15:53:47.325201+0000","last_fullsized":"2024-04-25T15:53:47.325201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":143360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":200,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354733+0000","last_change":"2024-04-25T15:53:16.115440+0000","last_active":"2024-04-25T15:53:47.354733+0000","last_peered":"2024-04-25T15:53:47.354733+0000","last_clean":"2024-04-25T15:53:47.354733+0000","last_became_active":"2024-04-25T15:53:16.053229+0000","last_became_peered":"2024-04-25T15:53:16.053229+0000","last_unstale":"2024-04-25T15:53:47.354733+0000","last_undegraded":"2024-04-25T15:53:47.354733+0000","last_fullsized":"2024-04-25T15:53:47.354733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:13.860370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3087,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'63","reported_seq":216,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354768+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:53:47.354768+0000","last_peered":"2024-04-25T15:53:47.354768+0000","last_clean":"2024-04-25T15:53:47.354768+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:53:47.354768+0000","last_undegraded":"2024-04-25T15:53:47.354768+0000","last_fullsized":"2024-04-25T15:53:47.354768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":240,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315265+0000","last_change":"2024-04-25T15:53:03.421455+0000","last_active":"2024-04-25T15:53:47.315265+0000","last_peered":"2024-04-25T15:53:47.315265+0000","last_clean":"2024-04-25T15:53:47.315265+0000","last_became_active":"2024-04-25T15:53:03.385909+0000","last_became_peered":"2024-04-25T15:53:03.385909+0000","last_unstale":"2024-04-25T15:53:47.315265+0000","last_undegraded":"2024-04-25T15:53:47.315265+0000","last_fullsized":"2024-04-25T15:53:47.315265+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:54.677309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"90'65","reported_seq":244,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325534+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:53:47.325534+0000","last_peered":"2024-04-25T15:53:47.325534+0000","last_clean":"2024-04-25T15:53:47.325534+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:53:47.325534+0000","last_undegraded":"2024-04-25T15:53:47.325534+0000","last_fullsized":"2024-04-25T15:53:47.325534+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315585+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:53:47.315585+0000","last_peered":"2024-04-25T15:53:47.315585+0000","last_clean":"2024-04-25T15:53:47.315585+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:53:47.315585+0000","last_undegraded":"2024-04-25T15:53:47.315585+0000","last_fullsized":"2024-04-25T15:53:47.315585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"90'38","reported_seq":205,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.518484+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:53:47.518484+0000","last_peered":"2024-04-25T15:53:47.518484+0000","last_clean":"2024-04-25T15:53:47.518484+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:53:47.518484+0000","last_undegraded":"2024-04-25T15:53:47.518484+0000","last_fullsized":"2024-04-25T15:53:47.518484+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315911+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:53:47.315911+0000","last_peered":"2024-04-25T15:53:47.315911+0000","last_clean":"2024-04-25T15:53:47.315911+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:53:47.315911+0000","last_undegraded":"2024-04-25T15:53:47.315911+0000","last_fullsized":"2024-04-25T15:53:47.315911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"90'31","reported_seq":202,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.518710+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:53:47.518710+0000","last_peered":"2024-04-25T15:53:47.518710+0000","last_clean":"2024-04-25T15:53:47.518710+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:53:47.518710+0000","last_undegraded":"2024-04-25T15:53:47.518710+0000","last_fullsized":"2024-04-25T15:53:47.518710+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315487+0000","last_change":"2024-04-25T15:53:16.098410+0000","last_active":"2024-04-25T15:53:47.315487+0000","last_peered":"2024-04-25T15:53:47.315487+0000","last_clean":"2024-04-25T15:53:47.315487+0000","last_became_active":"2024-04-25T15:53:16.053979+0000","last_became_peered":"2024-04-25T15:53:16.053979+0000","last_unstale":"2024-04-25T15:53:47.315487+0000","last_undegraded":"2024-04-25T15:53:47.315487+0000","last_fullsized":"2024-04-25T15:53:47.315487+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:24.859180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"89'38","reported_seq":191,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.320597+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:53:47.320597+0000","last_peered":"2024-04-25T15:53:47.320597+0000","last_clean":"2024-04-25T15:53:47.320597+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:53:47.320597+0000","last_undegraded":"2024-04-25T15:53:47.320597+0000","last_fullsized":"2024-04-25T15:53:47.320597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":160,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.317704+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:53:47.317704+0000","last_peered":"2024-04-25T15:53:47.317704+0000","last_clean":"2024-04-25T15:53:47.317704+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:53:47.317704+0000","last_undegraded":"2024-04-25T15:53:47.317704+0000","last_fullsized":"2024-04-25T15:53:47.317704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"90'45","reported_seq":196,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.647658+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:53:47.647658+0000","last_peered":"2024-04-25T15:53:47.647658+0000","last_clean":"2024-04-25T15:53:47.647658+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:53:47.647658+0000","last_undegraded":"2024-04-25T15:53:47.647658+0000","last_fullsized":"2024-04-25T15:53:47.647658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":155648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":249,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325048+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:53:47.325048+0000","last_peered":"2024-04-25T15:53:47.325048+0000","last_clean":"2024-04-25T15:53:47.325048+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:53:47.325048+0000","last_undegraded":"2024-04-25T15:53:47.325048+0000","last_fullsized":"2024-04-25T15:53:47.325048+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":161,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316268+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:53:47.316268+0000","last_peered":"2024-04-25T15:53:47.316268+0000","last_clean":"2024-04-25T15:53:47.316268+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:53:47.316268+0000","last_undegraded":"2024-04-25T15:53:47.316268+0000","last_fullsized":"2024-04-25T15:53:47.316268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'32","reported_seq":185,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.750812+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:53:47.750812+0000","last_peered":"2024-04-25T15:53:47.750812+0000","last_clean":"2024-04-25T15:53:47.750812+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:53:47.750812+0000","last_undegraded":"2024-04-25T15:53:47.750812+0000","last_fullsized":"2024-04-25T15:53:47.750812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.324853+0000","last_change":"2024-04-25T15:53:03.424512+0000","last_active":"2024-04-25T15:53:47.324853+0000","last_peered":"2024-04-25T15:53:47.324853+0000","last_clean":"2024-04-25T15:53:47.324853+0000","last_became_active":"2024-04-25T15:53:03.385988+0000","last_became_peered":"2024-04-25T15:53:03.385988+0000","last_unstale":"2024-04-25T15:53:47.324853+0000","last_undegraded":"2024-04-25T15:53:47.324853+0000","last_fullsized":"2024-04-25T15:53:47.324853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:55.224996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"90'48","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315649+0000","last_change":"2024-04-25T15:53:16.680938+0000","last_active":"2024-04-25T15:53:47.315649+0000","last_peered":"2024-04-25T15:53:47.315649+0000","last_clean":"2024-04-25T15:53:47.315649+0000","last_became_active":"2024-04-25T15:53:16.041368+0000","last_became_peered":"2024-04-25T15:53:16.041368+0000","last_unstale":"2024-04-25T15:53:47.315649+0000","last_undegraded":"2024-04-25T15:53:47.315649+0000","last_fullsized":"2024-04-25T15:53:47.315649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:49.681769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":151552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"88'9","reported_seq":165,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315488+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:53:47.315488+0000","last_peered":"2024-04-25T15:53:47.315488+0000","last_clean":"2024-04-25T15:53:47.315488+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:53:47.315488+0000","last_undegraded":"2024-04-25T15:53:47.315488+0000","last_fullsized":"2024-04-25T15:53:47.315488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"89'33","reported_seq":166,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315106+0000","last_change":"2024-04-25T15:53:03.530328+0000","last_active":"2024-04-25T15:53:47.315106+0000","last_peered":"2024-04-25T15:53:47.315106+0000","last_clean":"2024-04-25T15:53:47.315106+0000","last_became_active":"2024-04-25T15:53:03.382259+0000","last_became_peered":"2024-04-25T15:53:03.382259+0000","last_unstale":"2024-04-25T15:53:47.315106+0000","last_undegraded":"2024-04-25T15:53:47.315106+0000","last_fullsized":"2024-04-25T15:53:47.315106+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:43.788748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316532+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:53:47.316532+0000","last_peered":"2024-04-25T15:53:47.316532+0000","last_clean":"2024-04-25T15:53:47.316532+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:53:47.316532+0000","last_undegraded":"2024-04-25T15:53:47.316532+0000","last_fullsized":"2024-04-25T15:53:47.316532+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"90'34","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316740+0000","last_change":"2024-04-25T15:53:16.788986+0000","last_active":"2024-04-25T15:53:47.316740+0000","last_peered":"2024-04-25T15:53:47.316740+0000","last_clean":"2024-04-25T15:53:47.316740+0000","last_became_active":"2024-04-25T15:53:16.046426+0000","last_became_peered":"2024-04-25T15:53:16.046426+0000","last_unstale":"2024-04-25T15:53:47.316740+0000","last_undegraded":"2024-04-25T15:53:47.316740+0000","last_fullsized":"2024-04-25T15:53:47.316740+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:31:33.326656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.324993+0000","last_change":"2024-04-25T15:53:03.381919+0000","last_active":"2024-04-25T15:53:47.324993+0000","last_peered":"2024-04-25T15:53:47.324993+0000","last_clean":"2024-04-25T15:53:47.324993+0000","last_became_active":"2024-04-25T15:53:03.381463+0000","last_became_peered":"2024-04-25T15:53:03.381463+0000","last_unstale":"2024-04-25T15:53:47.324993+0000","last_undegraded":"2024-04-25T15:53:47.324993+0000","last_fullsized":"2024-04-25T15:53:47.324993+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:09.993891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"90'41","reported_seq":194,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.316817+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:53:47.316817+0000","last_peered":"2024-04-25T15:53:47.316817+0000","last_clean":"2024-04-25T15:53:47.316817+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:53:47.316817+0000","last_undegraded":"2024-04-25T15:53:47.316817+0000","last_fullsized":"2024-04-25T15:53:47.316817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":131072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":157,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325577+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:53:47.325577+0000","last_peered":"2024-04-25T15:53:47.325577+0000","last_clean":"2024-04-25T15:53:47.325577+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:53:47.325577+0000","last_undegraded":"2024-04-25T15:53:47.325577+0000","last_fullsized":"2024-04-25T15:53:47.325577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"90'32","reported_seq":151,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.327342+0000","last_change":"2024-04-25T15:53:03.480191+0000","last_active":"2024-04-25T15:53:47.327342+0000","last_peered":"2024-04-25T15:53:47.327342+0000","last_clean":"2024-04-25T15:53:47.327342+0000","last_became_active":"2024-04-25T15:53:03.380081+0000","last_became_peered":"2024-04-25T15:53:03.380081+0000","last_unstale":"2024-04-25T15:53:47.327342+0000","last_undegraded":"2024-04-25T15:53:47.327342+0000","last_fullsized":"2024-04-25T15:53:47.327342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:50.114177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"88'20","reported_seq":169,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355333+0000","last_change":"2024-04-25T15:53:16.083270+0000","last_active":"2024-04-25T15:53:47.355333+0000","last_peered":"2024-04-25T15:53:47.355333+0000","last_clean":"2024-04-25T15:53:47.355333+0000","last_became_active":"2024-04-25T15:53:16.045200+0000","last_became_peered":"2024-04-25T15:53:16.045200+0000","last_unstale":"2024-04-25T15:53:47.355333+0000","last_undegraded":"2024-04-25T15:53:47.355333+0000","last_fullsized":"2024-04-25T15:53:47.355333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:38.382894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"89'49","reported_seq":202,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355756+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:53:47.355756+0000","last_peered":"2024-04-25T15:53:47.355756+0000","last_clean":"2024-04-25T15:53:47.355756+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:53:47.355756+0000","last_undegraded":"2024-04-25T15:53:47.355756+0000","last_fullsized":"2024-04-25T15:53:47.355756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":162,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355347+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:53:47.355347+0000","last_peered":"2024-04-25T15:53:47.355347+0000","last_clean":"2024-04-25T15:53:47.355347+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:53:47.355347+0000","last_undegraded":"2024-04-25T15:53:47.355347+0000","last_fullsized":"2024-04-25T15:53:47.355347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"90'41","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315881+0000","last_change":"2024-04-25T15:53:16.863394+0000","last_active":"2024-04-25T15:53:47.315881+0000","last_peered":"2024-04-25T15:53:47.315881+0000","last_clean":"2024-04-25T15:53:47.315881+0000","last_became_active":"2024-04-25T15:53:16.048669+0000","last_became_peered":"2024-04-25T15:53:16.048669+0000","last_unstale":"2024-04-25T15:53:47.315881+0000","last_undegraded":"2024-04-25T15:53:47.315881+0000","last_fullsized":"2024-04-25T15:53:47.315881+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:53.132724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.324738+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:53:47.324738+0000","last_peered":"2024-04-25T15:53:47.324738+0000","last_clean":"2024-04-25T15:53:47.324738+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:53:47.324738+0000","last_undegraded":"2024-04-25T15:53:47.324738+0000","last_fullsized":"2024-04-25T15:53:47.324738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"90'51","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315227+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:53:47.315227+0000","last_peered":"2024-04-25T15:53:47.315227+0000","last_clean":"2024-04-25T15:53:47.315227+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:53:47.315227+0000","last_undegraded":"2024-04-25T15:53:47.315227+0000","last_fullsized":"2024-04-25T15:53:47.315227+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":155648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":127,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.391277+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:53:47.391277+0000","last_peered":"2024-04-25T15:53:47.391277+0000","last_clean":"2024-04-25T15:53:47.391277+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:53:47.391277+0000","last_undegraded":"2024-04-25T15:53:47.391277+0000","last_fullsized":"2024-04-25T15:53:47.391277+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"90'50","reported_seq":234,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354802+0000","last_change":"2024-04-25T15:53:03.761415+0000","last_active":"2024-04-25T15:53:47.354802+0000","last_peered":"2024-04-25T15:53:47.354802+0000","last_clean":"2024-04-25T15:53:47.354802+0000","last_became_active":"2024-04-25T15:53:03.384666+0000","last_became_peered":"2024-04-25T15:53:03.384666+0000","last_unstale":"2024-04-25T15:53:47.354802+0000","last_undegraded":"2024-04-25T15:53:47.354802+0000","last_fullsized":"2024-04-25T15:53:47.354802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:14.524576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":155648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":102,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354883+0000","last_change":"2024-04-25T15:53:03.409633+0000","last_active":"2024-04-25T15:53:47.354883+0000","last_peered":"2024-04-25T15:53:47.354883+0000","last_clean":"2024-04-25T15:53:47.354883+0000","last_became_active":"2024-04-25T15:53:03.384646+0000","last_became_peered":"2024-04-25T15:53:03.384646+0000","last_unstale":"2024-04-25T15:53:47.354883+0000","last_undegraded":"2024-04-25T15:53:47.354883+0000","last_fullsized":"2024-04-25T15:53:47.354883+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:43.442571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"89'42","reported_seq":188,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.315341+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:53:47.315341+0000","last_peered":"2024-04-25T15:53:47.315341+0000","last_clean":"2024-04-25T15:53:47.315341+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:53:47.315341+0000","last_undegraded":"2024-04-25T15:53:47.315341+0000","last_fullsized":"2024-04-25T15:53:47.315341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":135168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":180,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325282+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:53:47.325282+0000","last_peered":"2024-04-25T15:53:47.325282+0000","last_clean":"2024-04-25T15:53:47.325282+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:53:47.325282+0000","last_undegraded":"2024-04-25T15:53:47.325282+0000","last_fullsized":"2024-04-25T15:53:47.325282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"89'27","reported_seq":206,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.354721+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:53:47.354721+0000","last_peered":"2024-04-25T15:53:47.354721+0000","last_clean":"2024-04-25T15:53:47.354721+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:53:47.354721+0000","last_undegraded":"2024-04-25T15:53:47.354721+0000","last_fullsized":"2024-04-25T15:53:47.354721+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"90'45","reported_seq":224,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.533063+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:53:47.533063+0000","last_peered":"2024-04-25T15:53:47.533063+0000","last_clean":"2024-04-25T15:53:47.533063+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:53:47.533063+0000","last_undegraded":"2024-04-25T15:53:47.533063+0000","last_fullsized":"2024-04-25T15:53:47.533063+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":139264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":158,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325287+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:53:47.325287+0000","last_peered":"2024-04-25T15:53:47.325287+0000","last_clean":"2024-04-25T15:53:47.325287+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:53:47.325287+0000","last_undegraded":"2024-04-25T15:53:47.325287+0000","last_fullsized":"2024-04-25T15:53:47.325287+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"90'44","reported_seq":240,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.758727+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:53:47.758727+0000","last_peered":"2024-04-25T15:53:47.758727+0000","last_clean":"2024-04-25T15:53:47.758727+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:53:47.758727+0000","last_undegraded":"2024-04-25T15:53:47.758727+0000","last_fullsized":"2024-04-25T15:53:47.758727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":143360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":166,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.325265+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:53:47.325265+0000","last_peered":"2024-04-25T15:53:47.325265+0000","last_clean":"2024-04-25T15:53:47.325265+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:53:47.325265+0000","last_undegraded":"2024-04-25T15:53:47.325265+0000","last_fullsized":"2024-04-25T15:53:47.325265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"90'33","reported_seq":142,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.745831+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:53:47.745831+0000","last_peered":"2024-04-25T15:53:47.745831+0000","last_clean":"2024-04-25T15:53:47.745831+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:53:47.745831+0000","last_undegraded":"2024-04-25T15:53:47.745831+0000","last_fullsized":"2024-04-25T15:53:47.745831+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":157,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.355422+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:53:47.355422+0000","last_peered":"2024-04-25T15:53:47.355422+0000","last_clean":"2024-04-25T15:53:47.355422+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:53:47.355422+0000","last_undegraded":"2024-04-25T15:53:47.355422+0000","last_fullsized":"2024-04-25T15:53:47.355422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"90'33","reported_seq":186,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.703762+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:53:47.703762+0000","last_peered":"2024-04-25T15:53:47.703762+0000","last_clean":"2024-04-25T15:53:47.703762+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:53:47.703762+0000","last_undegraded":"2024-04-25T15:53:47.703762+0000","last_fullsized":"2024-04-25T15:53:47.703762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":131,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T15:53:47.326029+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:53:47.326029+0000","last_peered":"2024-04-25T15:53:47.326029+0000","last_clean":"2024-04-25T15:53:47.326029+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:53:47.326029+0000","last_undegraded":"2024-04-25T15:53:47.326029+0000","last_fullsized":"2024-04-25T15:53:47.326029+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":4764729344,"num_objects":1136,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2672,"num_write_kb":8847360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":557842432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9537847296,"data_stored":9537847296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2673,"ondisk_log_size":2673,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":176058,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":307,"num_write_kb":265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":7427,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":352116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1503300,"kb_used_data":1468024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92868540,"statfs":{"total":96636764160,"available":95097384960,"internally_reserved":0,"allocated":1503256576,"data_stored":1502653203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574895,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":970392,"kb_used_data":935116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93401448,"statfs":{"total":96636764160,"available":95643082752,"internally_reserved":0,"allocated":957558784,"data_stored":956970116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1617420,"kb_used_data":1582144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92754420,"statfs":{"total":96636764160,"available":94980526080,"internally_reserved":0,"allocated":1620115456,"data_stored":1619505179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1420772,"kb_used_data":1385500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92951068,"statfs":{"total":96636764160,"available":95181893632,"internally_reserved":0,"allocated":1418752000,"data_stored":1418169841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1052304,"kb_used_data":1021128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93319536,"statfs":{"total":96636764160,"available":95559204864,"internally_reserved":0,"allocated":1045635072,"data_stored":1045038141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937472,"kb_used_data":906292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93434368,"statfs":{"total":96636764160,"available":95676792832,"internally_reserved":0,"allocated":928043008,"data_stored":927442751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":343524,"kb_used_data":312344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94028316,"statfs":{"total":96636764160,"available":96284995584,"internally_reserved":0,"allocated":319840256,"data_stored":319267244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574893,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1093692,"kb_used_data":1062516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93278148,"statfs":{"total":96636764160,"available":95516823552,"internally_reserved":0,"allocated":1088016384,"data_stored":1087410080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":169648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":169342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1212153856,"data_stored":1212153856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":318767104,"data_stored":318767104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":935329792,"data_stored":935329792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1207959552,"data_stored":1207959552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1568669696,"data_stored":1568669696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1677721600,"data_stored":1677721600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1031798784,"data_stored":1031798784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1585446912,"data_stored":1585446912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:53:50.830 INFO:tasks.thrashosds.thrasher:Setting 3.3 to [5, 7, 0, 2] 2024-04-25T15:53:50.830 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.3', '5', '7', '0', '2'] 2024-04-25T15:53:50.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.3 5 7 0 2 2024-04-25T15:53:50.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:50.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.846+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c00cdf60 con 0x7f61c01f66b0 2024-04-25T15:53:50.848 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:50.846+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.850+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c00cdf60 con 0x7f61c01f66b0 2024-04-25T15:53:50.850 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.850+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f61c01fa160 0x7f61c0063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.850+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f61c01fa570 con 0x7f61c01fa160 2024-04-25T15:53:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.850+0000 7f61bf7fe640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f61c01fa160 0x7f61c0063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.850+0000 7f61bf7fe640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f61c01fa160 0x7f61c0063090 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.850+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c01fa570 con 0x7f61c01fa160 2024-04-25T15:53:50.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.870+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c00cd9a0 con 0x7f61c01fa160 2024-04-25T15:53:50.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.874+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c00cd9a0 con 0x7f61c01fa160 2024-04-25T15:53:50.876 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:50.876 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:50.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.874+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f61c0063ae0 0x7f61c015dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.874+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f61c00cd9a0 con 0x7f61c0063ae0 2024-04-25T15:53:50.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.874+0000 7f61c482e640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f61c0063ae0 0x7f61c015dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.874+0000 7f61c482e640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f61c0063ae0 0x7f61c015dbd0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.878+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c00cd9a0 con 0x7f61c0063ae0 2024-04-25T15:53:50.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.894+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c00cd9a0 con 0x7f61c0063ae0 2024-04-25T15:53:50.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.894+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c00cd9a0 con 0x7f61c0063ae0 2024-04-25T15:53:50.896 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:50.896 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:50.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.894+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f61c015e620 0x7f61c002c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.894+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f61c00cd6c0 con 0x7f61c015e620 2024-04-25T15:53:50.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.898+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f61c015e620 0x7f61c002c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.898+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f61c015e620 0x7f61c002c9e0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.898+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c00cd6c0 con 0x7f61c015e620 2024-04-25T15:53:50.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.922+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c00cd6c0 con 0x7f61c015e620 2024-04-25T15:53:50.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.922+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c00cd6c0 con 0x7f61c015e620 2024-04-25T15:53:50.925 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.926+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f61c002d430 0x7f61c002d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.926+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f61c002dd80 con 0x7f61c002d430 2024-04-25T15:53:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.926+0000 7f61bf7fe640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f61c002d430 0x7f61c002d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.926+0000 7f61bf7fe640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f61c002d430 0x7f61c002d840 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.926+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c002dd80 con 0x7f61c002d430 2024-04-25T15:53:50.935 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:50.934+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.942+0000 7f21ebfff640 1 -- 172.21.15.67:0/2046486264 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v91) ==== 101+0+0 (secure 0 0 0) 0x7f21f805bf30 con 0x7f22040caf60 2024-04-25T15:53:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f21f006f400 msgr2=0x7f21f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f21f006f400 0x7f21f0071910 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f21f40ad000 tx=0x7f21f40b9040 comp rx=0 tx=0).stop 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 msgr2=0x7f22040cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f22040cc730 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f21f800b3f0 tx=0x7f21f800b8b0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 shutdown_connections 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f21f006f400 0x7f21f0071910 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f220415d790 0x7f22040ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22040caf60 0x7f22040cc730 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 --2- 172.21.15.67:0/2046486264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2204073bf0 0x7f22040cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:50.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 >> 172.21.15.67:0/2046486264 conn(0x7f2204070db0 msgr2=0x7f22041523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:50.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 shutdown_connections 2024-04-25T15:53:50.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.946+0000 7f22099f5640 1 -- 172.21.15.67:0/2046486264 wait complete. 2024-04-25T15:53:50.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.950+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c002dd80 con 0x7f61c002d430 2024-04-25T15:53:50.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.954+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c002dd80 con 0x7f61c002d430 2024-04-25T15:53:50.954 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:50.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:50.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.954+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f61c020cfb0 0x7f61c020f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.954+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f61c00c7cc0 con 0x7f61c020cfb0 2024-04-25T15:53:50.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.954+0000 7f61c482e640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f61c020cfb0 0x7f61c020f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.954+0000 7f61c482e640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f61c020cfb0 0x7f61c020f3d0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.958+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c00c7cc0 con 0x7f61c020cfb0 2024-04-25T15:53:50.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.978+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c00008e0 con 0x7f61c020cfb0 2024-04-25T15:53:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.982+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c00008e0 con 0x7f61c020cfb0 2024-04-25T15:53:50.982 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.982+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f61c020fe20 0x7f61c0212240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.982+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f61c0215150 con 0x7f61c020fe20 2024-04-25T15:53:50.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.982+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f61c020fe20 0x7f61c0212240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:50.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.986+0000 7f61bffff640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f61c020fe20 0x7f61c0212240 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:50.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:50.986+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61c0215150 con 0x7f61c020fe20 2024-04-25T15:53:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.006+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f61c0212780 con 0x7f61c020fe20 2024-04-25T15:53:51.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61bcff9640 1 -- 172.21.15.67:0/1665756708 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f61c0212780 con 0x7f61c020fe20 2024-04-25T15:53:51.010 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:51.011 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:51.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f61c0150520 msgr2=0x7f61c00ce930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f61c0150520 0x7f61c00ce930 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f61c01f66b0 msgr2=0x7f61c01f9710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f61c01f66b0 0x7f61c01f9710 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f61c01fa160 msgr2=0x7f61c0063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f61c01fa160 0x7f61c0063090 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f61c0063ae0 msgr2=0x7f61c015dbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f61c0063ae0 0x7f61c015dbd0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f61c015e620 msgr2=0x7f61c002c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f61c015e620 0x7f61c002c9e0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c482e640 1 -- 172.21.15.67:0/1665756708 reap_dead start 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f61c002d430 msgr2=0x7f61c002d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f61c002d430 0x7f61c002d840 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c482e640 1 -- 172.21.15.67:0/1665756708 reap_dead start 2024-04-25T15:53:51.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f61c020cfb0 msgr2=0x7f61c020f3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f61c020cfb0 0x7f61c020f3d0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f61c020fe20 msgr2=0x7f61c0212240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f61c020fe20 0x7f61c0212240 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c482e640 1 -- 172.21.15.67:0/1665756708 reap_dead start 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f61ac06f430 msgr2=0x7f61ac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f61ac06f430 0x7f61ac071940 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f61b00045f0 tx=0x7f61b0008040 comp rx=0 tx=0).stop 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 msgr2=0x7f61c00c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c00c7700 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f61b400c420 tx=0x7f61b400c8e0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 shutdown_connections 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f61ac06f430 0x7f61ac071940 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 --2- 172.21.15.67:0/1665756708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61c00c96e0 0x7f61c00c7700 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 >> 172.21.15.67:0/1665756708 conn(0x7f61c0070db0 msgr2=0x7f61c0153b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 shutdown_connections 2024-04-25T15:53:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.010+0000 7f61c6291640 1 -- 172.21.15.67:0/1665756708 wait complete. 2024-04-25T15:53:51.036 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:51.034+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:51.051 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:51.052 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:51.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.054+0000 7f97bbb59640 1 Processor -- start 2024-04-25T15:53:51.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.054+0000 7f97bbb59640 1 -- start start 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97bbb59640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b4164590 0x7f97b41669b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97bbb59640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b406f220 con 0x7f97b40c9a00 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97bbb59640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b406f4e0 con 0x7f97b40c95f0 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97bbb59640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97b415a810 con 0x7f97b4164590 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97b90f4640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f97b40c9a00 0x7f97b4162e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35604/0 (socket says 172.21.15.67:35604) 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97b90f4640 1 -- 172.21.15.67:0/3704141229 learned_addr learned my addr 172.21.15.67:0/3704141229 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97ba0f6640 1 --2- 172.21.15.67:0/3704141229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b4164590 0x7f97b41669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97b88f3640 1 -- 172.21.15.67:0/3704141229 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453816874 0 0) 0x7f97b406f220 con 0x7f97b40c9a00 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97b88f3640 1 -- 172.21.15.67:0/3704141229 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97a0003530 con 0x7f97b40c9a00 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97b88f3640 1 -- 172.21.15.67:0/3704141229 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200024086 0 0) 0x7f97b406f4e0 con 0x7f97b40c95f0 2024-04-25T15:53:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97b88f3640 1 -- 172.21.15.67:0/3704141229 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97b406f220 con 0x7f97b40c95f0 2024-04-25T15:53:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97ba0f6640 1 -- 172.21.15.67:0/3704141229 >> v1:172.21.15.176:6789/0 conn(0x7f97b40c95f0 legacy=0x7f97b40cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97ba0f6640 1 -- 172.21.15.67:0/3704141229 >> v1:172.21.15.67:6789/0 conn(0x7f97b40c9a00 legacy=0x7f97b4162e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.058+0000 7f97ba0f6640 1 -- 172.21.15.67:0/3704141229 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b415aaf0 con 0x7f97b4164590 2024-04-25T15:53:51.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97ba0f6640 1 --2- 172.21.15.67:0/3704141229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b4164590 0x7f97b41669b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009dc0 tx=0x7f97ac01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96972babe30e09bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:51.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97b88f3640 1 -- 172.21.15.67:0/3704141229 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0b4070 con 0x7f97b4164590 2024-04-25T15:53:51.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97b88f3640 1 -- 172.21.15.67:0/3704141229 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac0af600 con 0x7f97b4164590 2024-04-25T15:53:51.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97b88f3640 1 -- 172.21.15.67:0/3704141229 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0af9d0 con 0x7f97b4164590 2024-04-25T15:53:51.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97bbb59640 1 -- 172.21.15.67:0/3704141229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b4164590 msgr2=0x7f97b41669b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3704141229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b4164590 0x7f97b41669b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97ac009dc0 tx=0x7f97ac01f170 comp rx=0 tx=0).stop 2024-04-25T15:53:51.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97bbb59640 1 -- 172.21.15.67:0/3704141229 shutdown_connections 2024-04-25T15:53:51.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3704141229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b4164590 0x7f97b41669b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97bbb59640 1 -- 172.21.15.67:0/3704141229 >> 172.21.15.67:0/3704141229 conn(0x7f97b40cebf0 msgr2=0x7f97b40cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:51.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97bbb59640 1 -- 172.21.15.67:0/3704141229 shutdown_connections 2024-04-25T15:53:51.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.062+0000 7f97bbb59640 1 -- 172.21.15.67:0/3704141229 wait complete. 2024-04-25T15:53:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 Processor -- start 2024-04-25T15:53:51.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 -- start start 2024-04-25T15:53:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97b40c95f0 0x7f97b415db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:51.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b40c9a00 0x7f97b4151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97b4164590 0x7f97b413d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4143b00 con 0x7f97b40c95f0 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f97b415a500 con 0x7f97b4164590 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97bbb59640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97b4143c80 con 0x7f97b40c9a00 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97b40c95f0 0x7f97b415db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97b40c95f0 0x7f97b415db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43850/0 (socket says 172.21.15.67:43850) 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 -- 172.21.15.67:0/3136123979 learned_addr learned my addr 172.21.15.67:0/3136123979 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 -- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b40c9a00 msgr2=0x7f97b4151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b40c9a00 0x7f97b4151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 -- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97b4164590 msgr2=0x7f97b413d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97b4164590 0x7f97b413d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.066+0000 7f97b98f5640 1 -- 172.21.15.67:0/3136123979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b415e0a0 con 0x7f97b40c95f0 2024-04-25T15:53:51.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97b98f5640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97b40c95f0 0x7f97b415db60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f97b000c430 tx=0x7f97b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:51.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97aa7fc640 1 -- 172.21.15.67:0/3136123979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0017070 con 0x7f97b40c95f0 2024-04-25T15:53:51.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97aa7fc640 1 -- 172.21.15.67:0/3136123979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b00052f0 con 0x7f97b40c95f0 2024-04-25T15:53:51.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b415cac0 con 0x7f97b40c95f0 2024-04-25T15:53:51.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b415d080 con 0x7f97b40c95f0 2024-04-25T15:53:51.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97aa7fc640 1 -- 172.21.15.67:0/3136123979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0003840 con 0x7f97b40c95f0 2024-04-25T15:53:51.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f979c0052a0 con 0x7f97b40c95f0 2024-04-25T15:53:51.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97aa7fc640 1 -- 172.21.15.67:0/3136123979 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f97b001a070 con 0x7f97b40c95f0 2024-04-25T15:53:51.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97aa7fc640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979806f430 0x7f9798071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:51.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97aa7fc640 1 -- 172.21.15.67:0/3136123979 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5944+0+0 (secure 0 0 0) 0x7f97b008e570 con 0x7f97b40c95f0 2024-04-25T15:53:51.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.070+0000 7f97b90f4640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979806f430 0x7f9798071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:51.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:51.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.074+0000 7f97b90f4640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979806f430 0x7f9798071940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4007940 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:51.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.078+0000 7f97aa7fc640 1 -- 172.21.15.67:0/3136123979 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f97b005b490 con 0x7f97b40c95f0 2024-04-25T15:53:51.137 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:51.135+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:51.237 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:51.238+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:51.246 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:51.246 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:51.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:51.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.342+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.3", "id": [5, 7, 0, 2]} v 0) -- 0x7f979c005860 con 0x7f97b40c95f0 2024-04-25T15:53:51.359 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:51.359+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:51.438 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:51.435+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:51.501 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:51.501 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:51.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:51.582 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:51.579+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:51.640 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:51.638+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:51.741 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:51.742+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:51.774 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:51.774 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:51.806 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:51.842 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:51.842+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:51.943 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:51.942+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:51.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.986+0000 7f97aa7fc640 1 -- 172.21.15.67:0/3136123979 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.3", "id": [5, 7, 0, 2]}]=0 set 3.3 pg_upmap_items mapping to [5->7,0->2] v92) ==== 146+0+0 (secure 0 0 0) 0x7f97b005ad90 con 0x7f97b40c95f0 2024-04-25T15:53:51.988 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.3 pg_upmap_items mapping to [5->7,0->2] 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979806f430 msgr2=0x7f9798071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979806f430 0x7f9798071940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a4007940 comp rx=0 tx=0).stop 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97b40c95f0 msgr2=0x7f97b415db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97b40c95f0 0x7f97b415db60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f97b000c430 tx=0x7f97b000c8f0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 shutdown_connections 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f979806f430 0x7f9798071940 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97b4164590 0x7f97b413d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97b40c9a00 0x7f97b4151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 --2- 172.21.15.67:0/3136123979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97b40c95f0 0x7f97b415db60 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 >> 172.21.15.67:0/3136123979 conn(0x7f97b40cebf0 msgr2=0x7f97b40734f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 shutdown_connections 2024-04-25T15:53:51.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:51.990+0000 7f97bbb59640 1 -- 172.21.15.67:0/3136123979 wait complete. 2024-04-25T15:53:52.038 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:52.038 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:52.044 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:52.043+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:52.065 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:52.145 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:52.146+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:52.245 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:52.243+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:52.282 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:52.282 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:52.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:52.355 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:52.355+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:52.447 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:52.443+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:52.547 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:52.546+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:52.548 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:52.548 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:52.578 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:52.667 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:52.667+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:52.749 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:52.747+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:52.822 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:52.823 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:52.848 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:52.849 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:52.850+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:52.954 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:52.951+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:52.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:53:53.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:53.084 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:53.084 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:53.089 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:53.090+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:53.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:53.151 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:53.151+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.182+0000 7f390ca10640 1 Processor -- start 2024-04-25T15:53:53.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.182+0000 7f390ca10640 1 -- start start 2024-04-25T15:53:53.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f390ca10640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3908074370 0x7f3908074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f390ca10640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39080c7fc0 0x7f39080c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3905d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3908074370 0x7f3908074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3905d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3908074370 0x7f3908074780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:53694/0 (socket says 172.21.15.67:53694) 2024-04-25T15:53:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3905d9b640 1 -- 172.21.15.67:0/3104862635 learned_addr learned my addr 172.21.15.67:0/3104862635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3906d9d640 1 --2- 172.21.15.67:0/3104862635 >> v2:172.21.15.67:3300/0 conn(0x7f39080c7fc0 0x7f39080c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f390ca10640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39080c92d0 con 0x7f390806c640 2024-04-25T15:53:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f390ca10640 1 -- 172.21.15.67:0/3104862635 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f390806e210 con 0x7f39080c7fc0 2024-04-25T15:53:53.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f390ca10640 1 -- 172.21.15.67:0/3104862635 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f390806e4d0 con 0x7f3908074370 2024-04-25T15:53:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3906d9d640 1 -- 172.21.15.67:0/3104862635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3908074370 msgr2=0x7f3908074780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3906d9d640 1 --2- 172.21.15.67:0/3104862635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3908074370 0x7f3908074780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f390559a640 1 -- 172.21.15.67:0/3104862635 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473713636 0 0) 0x7f39080c92d0 con 0x7f390806c640 2024-04-25T15:53:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3906d9d640 1 -- 172.21.15.67:0/3104862635 >> v1:172.21.15.176:6789/0 conn(0x7f390806c640 legacy=0x7f390815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.186+0000 7f3906d9d640 1 -- 172.21.15.67:0/3104862635 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390806e7b0 con 0x7f39080c7fc0 2024-04-25T15:53:53.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.190+0000 7f3906d9d640 1 --2- 172.21.15.67:0/3104862635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39080c7fc0 0x7f39080c83d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3900009d90 tx=0x7f390001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f07650900e1d58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.190+0000 7f390559a640 1 -- 172.21.15.67:0/3104862635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39000b4070 con 0x7f39080c7fc0 2024-04-25T15:53:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.190+0000 7f390559a640 1 -- 172.21.15.67:0/3104862635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39000af600 con 0x7f39080c7fc0 2024-04-25T15:53:53.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.190+0000 7f390559a640 1 -- 172.21.15.67:0/3104862635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39000af9d0 con 0x7f39080c7fc0 2024-04-25T15:53:53.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.194+0000 7f390ca10640 1 -- 172.21.15.67:0/3104862635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39080c7fc0 msgr2=0x7f39080c83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.194+0000 7f390ca10640 1 --2- 172.21.15.67:0/3104862635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39080c7fc0 0x7f39080c83d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3900009d90 tx=0x7f390001f150 comp rx=0 tx=0).stop 2024-04-25T15:53:53.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.194+0000 7f390ca10640 1 -- 172.21.15.67:0/3104862635 shutdown_connections 2024-04-25T15:53:53.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.194+0000 7f390ca10640 1 --2- 172.21.15.67:0/3104862635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39080c7fc0 0x7f39080c83d0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.194+0000 7f390ca10640 1 --2- 172.21.15.67:0/3104862635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3908074370 0x7f3908074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.194+0000 7f390ca10640 1 -- 172.21.15.67:0/3104862635 >> 172.21.15.67:0/3104862635 conn(0x7f3908070db0 msgr2=0x7f39080ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:53.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.198+0000 7f390ca10640 1 -- 172.21.15.67:0/3104862635 shutdown_connections 2024-04-25T15:53:53.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.198+0000 7f390ca10640 1 -- 172.21.15.67:0/3104862635 wait complete. 2024-04-25T15:53:53.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 Processor -- start 2024-04-25T15:53:53.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 -- start start 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390806c640 0x7f390815b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908074370 0x7f390814c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39080c7fc0 0x7f39081f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3908073e50 con 0x7f3908074370 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3908073b70 con 0x7f390806c640 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390ca10640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3908073420 con 0x7f39080c7fc0 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f3905d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908074370 0x7f390814c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f3905d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908074370 0x7f390814c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39968/0 (socket says 172.21.15.67:39968) 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f3905d9b640 1 -- 172.21.15.67:0/3721113921 learned_addr learned my addr 172.21.15.67:0/3721113921 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f390659c640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390806c640 0x7f390815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.202+0000 7f3906d9d640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39080c7fc0 0x7f39081f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f3906d9d640 1 -- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390806c640 msgr2=0x7f390815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f3906d9d640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390806c640 0x7f390815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f3906d9d640 1 -- 172.21.15.67:0/3721113921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908074370 msgr2=0x7f390814c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f3906d9d640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908074370 0x7f390814c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f3906d9d640 1 -- 172.21.15.67:0/3721113921 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390815be80 con 0x7f39080c7fc0 2024-04-25T15:53:53.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f390659c640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390806c640 0x7f390815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f3906d9d640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39080c7fc0 0x7f39081f4330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f39080c8e60 tx=0x7f390001bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f38f6ffd640 1 -- 172.21.15.67:0/3721113921 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39000b4040 con 0x7f39080c7fc0 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f38f6ffd640 1 -- 172.21.15.67:0/3721113921 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39000af400 con 0x7f39080c7fc0 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f38f6ffd640 1 -- 172.21.15.67:0/3721113921 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3900009450 con 0x7f39080c7fc0 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390814db10 con 0x7f39080c7fc0 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f390814f600 con 0x7f39080c7fc0 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f390806a480 con 0x7f39080c7fc0 2024-04-25T15:53:53.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f38f6ffd640 1 -- 172.21.15.67:0/3721113921 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f39000c3020 con 0x7f39080c7fc0 2024-04-25T15:53:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f38f6ffd640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f38e806f400 0x7f38e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.206+0000 7f38f6ffd640 1 -- 172.21.15.67:0/3721113921 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(93..93 src has 1..93) ==== 5973+0+0 (secure 0 0 0) 0x7f390012d280 con 0x7f39080c7fc0 2024-04-25T15:53:53.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.210+0000 7f390659c640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f38e806f400 0x7f38e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.210+0000 7f390659c640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f38e806f400 0x7f38e8071910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f38f8007fd0 tx=0x7f38f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.214+0000 7f38f6ffd640 1 -- 172.21.15.67:0/3721113921 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f39000ca020 con 0x7f39080c7fc0 2024-04-25T15:53:53.252 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:53.251+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae8dd23640 1 Processor -- start 2024-04-25T15:53:53.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae8dd23640 1 -- start start 2024-04-25T15:53:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae8dd23640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae88073bf0 0x7fae8806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae8dd23640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fae8806dc90 0x7fae88163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae8dd23640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae880c77a0 con 0x7fae880caf60 2024-04-25T15:53:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae8dd23640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fae880cab60 con 0x7fae88073bf0 2024-04-25T15:53:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae8dd23640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fae88159620 con 0x7fae8806dc90 2024-04-25T15:53:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae86ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae88073bf0 0x7fae8806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae86ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae88073bf0 0x7fae8806d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39974/0 (socket says 172.21.15.67:39974) 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae86ffd640 1 -- 172.21.15.67:0/1957354026 learned_addr learned my addr 172.21.15.67:0/1957354026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae87fff640 1 --2- 172.21.15.67:0/1957354026 >> v2:172.21.15.176:3300/0 conn(0x7fae8806dc90 0x7fae88163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae86ffd640 1 -- 172.21.15.67:0/1957354026 >> v2:172.21.15.176:3300/0 conn(0x7fae8806dc90 msgr2=0x7fae88163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae86ffd640 1 --2- 172.21.15.67:0/1957354026 >> v2:172.21.15.176:3300/0 conn(0x7fae8806dc90 0x7fae88163ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae86ffd640 1 -- 172.21.15.67:0/1957354026 >> v1:172.21.15.176:6789/0 conn(0x7fae880caf60 legacy=0x7fae8815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae86ffd640 1 -- 172.21.15.67:0/1957354026 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae880c7a60 con 0x7fae88073bf0 2024-04-25T15:53:53.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.254+0000 7fae87fff640 1 --2- 172.21.15.67:0/1957354026 >> v2:172.21.15.176:3300/0 conn(0x7fae8806dc90 0x7fae88163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae86ffd640 1 --2- 172.21.15.67:0/1957354026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae88073bf0 0x7fae8806d750 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fae880c61d0 tx=0x7fae7c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89103d6dee2363ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae867fc640 1 -- 172.21.15.67:0/1957354026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0b0070 con 0x7fae88073bf0 2024-04-25T15:53:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae867fc640 1 -- 172.21.15.67:0/1957354026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae7c0ab600 con 0x7fae88073bf0 2024-04-25T15:53:53.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae867fc640 1 -- 172.21.15.67:0/1957354026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c0ab9d0 con 0x7fae88073bf0 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 -- 172.21.15.67:0/1957354026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae88073bf0 msgr2=0x7fae8806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 --2- 172.21.15.67:0/1957354026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae88073bf0 0x7fae8806d750 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fae880c61d0 tx=0x7fae7c01c520 comp rx=0 tx=0).stop 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 -- 172.21.15.67:0/1957354026 shutdown_connections 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 --2- 172.21.15.67:0/1957354026 >> v2:172.21.15.176:3300/0 conn(0x7fae8806dc90 0x7fae88163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 --2- 172.21.15.67:0/1957354026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae88073bf0 0x7fae8806d750 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 -- 172.21.15.67:0/1957354026 >> 172.21.15.67:0/1957354026 conn(0x7fae88070db0 msgr2=0x7fae880ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 -- 172.21.15.67:0/1957354026 shutdown_connections 2024-04-25T15:53:53.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.258+0000 7fae8dd23640 1 -- 172.21.15.67:0/1957354026 wait complete. 2024-04-25T15:53:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 Processor -- start 2024-04-25T15:53:53.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 -- start start 2024-04-25T15:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8806dc90 0x7fae88141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae88073bf0 0x7fae88163a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae880caf60 0x7fae8813eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae8815a570 con 0x7fae880caf60 2024-04-25T15:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae8815a3f0 con 0x7fae8806dc90 2024-04-25T15:53:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae8dd23640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae8806c640 con 0x7fae88073bf0 2024-04-25T15:53:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.262+0000 7fae86ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae88073bf0 0x7fae88163a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae86ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae88073bf0 0x7fae88163a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:53716/0 (socket says 172.21.15.67:53716) 2024-04-25T15:53:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae86ffd640 1 -- 172.21.15.67:0/2049268060 learned_addr learned my addr 172.21.15.67:0/2049268060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae880caf60 0x7fae8813eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae88073bf0 msgr2=0x7fae88163a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae88073bf0 0x7fae88163a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8806dc90 msgr2=0x7fae88141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8806dc90 0x7fae88141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8806dc90 0x7fae88141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8813ce20 con 0x7fae880caf60 2024-04-25T15:53:53.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae8806dc90 0x7fae88141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae880caf60 0x7fae8813eca0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fae80007d10 tx=0x7fae8000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8001a070 con 0x7fae880caf60 2024-04-25T15:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae800039a0 con 0x7fae880caf60 2024-04-25T15:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae88163f70 con 0x7fae880caf60 2024-04-25T15:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80003d70 con 0x7fae880caf60 2024-04-25T15:53:53.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae880cd250 con 0x7fae880caf60 2024-04-25T15:53:53.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fae8001d070 con 0x7fae880caf60 2024-04-25T15:53:53.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.266+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fae8806a480 con 0x7fae880caf60 2024-04-25T15:53:53.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae6ffff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae7406f400 0x7fae74071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 5973+0+0 (secure 0 0 0) 0x7fae8008f4b0 con 0x7fae880caf60 2024-04-25T15:53:53.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v93) ==== 54+0+16 (secure 0 0 0) 0x7fae88163f70 con 0x7fae880caf60 2024-04-25T15:53:53.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae7406f400 0x7fae74071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fae8813f1e0 0x7fae8813d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae88157010 con 0x7fae8813f1e0 2024-04-25T15:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae86ffd640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fae8813f1e0 0x7fae8813d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae7406f400 0x7fae74071910 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae78001650 tx=0x7fae78009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae86ffd640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fae8813f1e0 0x7fae8813d890 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.270+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae88157010 con 0x7fae8813f1e0 2024-04-25T15:53:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.298+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae880c8a40 con 0x7fae8813f1e0 2024-04-25T15:53:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.302+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae880c8a40 con 0x7fae8813f1e0 2024-04-25T15:53:53.301 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:53.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.302+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fae880cba80 0x7fae881f9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.302+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fae880c8300 con 0x7fae880cba80 2024-04-25T15:53:53.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.302+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fae880cba80 0x7fae881f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.302+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fae880cba80 0x7fae881f9620 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.302+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae880c8300 con 0x7fae880cba80 2024-04-25T15:53:53.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.322+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae88159a30 con 0x7fae880cba80 2024-04-25T15:53:53.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.326+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae88159a30 con 0x7fae880cba80 2024-04-25T15:53:53.327 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.326+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fae881fa070 0x7fae88063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.326+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fae881fa480 con 0x7fae881fa070 2024-04-25T15:53:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.326+0000 7fae86ffd640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fae881fa070 0x7fae88063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.326+0000 7fae86ffd640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fae881fa070 0x7fae88063090 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.330+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae881fa480 con 0x7fae881fa070 2024-04-25T15:53:53.335 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:53.335 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:53.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.350+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae8813cff0 con 0x7fae881fa070 2024-04-25T15:53:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.350+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae8813cff0 con 0x7fae881fa070 2024-04-25T15:53:53.352 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.350+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fae880c8e90 0x7fae880c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.350+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fae8813d200 con 0x7fae880c8e90 2024-04-25T15:53:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.350+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fae880c8e90 0x7fae880c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.350+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fae880c8e90 0x7fae880c92a0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.354+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae8813d200 con 0x7fae880c8e90 2024-04-25T15:53:53.354 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:53.351+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:53.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:53.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.374+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae8813d200 con 0x7fae880c8e90 2024-04-25T15:53:53.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.378+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae8813d200 con 0x7fae880c8e90 2024-04-25T15:53:53.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:53.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.378+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fae880c9cf0 0x7fae8802c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.381 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:53.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.378+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fae8820c0b0 con 0x7fae880c9cf0 2024-04-25T15:53:53.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.382+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fae880c9cf0 0x7fae8802c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.382+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fae880c9cf0 0x7fae8802c9e0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.382+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae8820c0b0 con 0x7fae880c9cf0 2024-04-25T15:53:53.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.402+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae8802cf20 con 0x7fae880c9cf0 2024-04-25T15:53:53.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.406+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae8802cf20 con 0x7fae880c9cf0 2024-04-25T15:53:53.409 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.410+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fae8802d5a0 0x7fae8802d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.410+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fae8802def0 con 0x7fae8802d5a0 2024-04-25T15:53:53.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.410+0000 7fae86ffd640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fae8802d5a0 0x7fae8802d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.410+0000 7fae86ffd640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fae8802d5a0 0x7fae8802d9b0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.410+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae8802def0 con 0x7fae8802d5a0 2024-04-25T15:53:53.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.438+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae8802def0 con 0x7fae8802d5a0 2024-04-25T15:53:53.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.446+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae8802def0 con 0x7fae8802d5a0 2024-04-25T15:53:53.447 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:53.447 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.446+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fae8820cec0 0x7fae8820f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.446+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fae88212240 con 0x7fae8820cec0 2024-04-25T15:53:53.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.446+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fae8820cec0 0x7fae8820f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.450+0000 7fae87fff640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fae8820cec0 0x7fae8820f2e0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.450+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae88212240 con 0x7fae8820cec0 2024-04-25T15:53:53.454 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:53.454+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.474+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae88212210 con 0x7fae8820cec0 2024-04-25T15:53:53.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.474+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae88212210 con 0x7fae8820cec0 2024-04-25T15:53:53.476 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:53.477 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:53.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.474+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fae88211470 0x7fae882118e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.474+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fae88215660 con 0x7fae88211470 2024-04-25T15:53:53.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.478+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fae88211470 0x7fae882118e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.478+0000 7fae877fe640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fae88211470 0x7fae882118e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.482+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae88215660 con 0x7fae88211470 2024-04-25T15:53:53.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.490+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f39080c8620 con 0x7f39080c7fc0 2024-04-25T15:53:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.506+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fae882144e0 con 0x7fae88211470 2024-04-25T15:53:53.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.510+0000 7fae6ffff640 1 -- 172.21.15.67:0/2049268060 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fae882144e0 con 0x7fae88211470 2024-04-25T15:53:53.512 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.510+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fae8813f1e0 msgr2=0x7fae8813d890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fae8813f1e0 0x7fae8813d890 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fae880cba80 msgr2=0x7fae881f9620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fae880cba80 0x7fae881f9620 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fae881fa070 msgr2=0x7fae88063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fae881fa070 0x7fae88063090 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fae880c8e90 msgr2=0x7fae880c92a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fae880c8e90 0x7fae880c92a0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fae880c9cf0 msgr2=0x7fae8802c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fae880c9cf0 0x7fae8802c9e0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fae8802d5a0 msgr2=0x7fae8802d9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fae8802d5a0 0x7fae8802d9b0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fae8820cec0 msgr2=0x7fae8820f2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fae8820cec0 0x7fae8820f2e0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fae88211470 msgr2=0x7fae882118e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fae88211470 0x7fae882118e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 reap_dead start 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 reap_dead start 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 reap_dead start 2024-04-25T15:53:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 reap_dead start 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae87fff640 1 -- 172.21.15.67:0/2049268060 reap_dead start 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae7406f400 msgr2=0x7fae74071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae7406f400 0x7fae74071910 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae78001650 tx=0x7fae78009040 comp rx=0 tx=0).stop 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae880caf60 msgr2=0x7fae8813eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae880caf60 0x7fae8813eca0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fae80007d10 tx=0x7fae8000d4a0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 shutdown_connections 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fae7406f400 0x7fae74071910 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fae78001650 tx=0x7fae78009040 comp rx=0 tx=0).stop 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 --2- 172.21.15.67:0/2049268060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae880caf60 0x7fae8813eca0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 >> 172.21.15.67:0/2049268060 conn(0x7fae88070db0 msgr2=0x7fae880cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 shutdown_connections 2024-04-25T15:53:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:53.514+0000 7fae8dd23640 1 -- 172.21.15.67:0/2049268060 wait complete. 2024-04-25T15:53:53.555 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:53.555+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:53.578 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:53.578 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:53.578 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:53.578 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:53.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:53.656 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:53.655+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:53.757 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:53.758+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:53.857 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:53.857 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:53.857 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:53.857 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:53.857 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:53.860 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:53.858+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:53.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:53.959 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:53.955+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.030+0000 7f38f6ffd640 1 -- 172.21.15.67:0/3721113921 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v94) ==== 87+0+0 (secure 0 0 0) 0x7f3900018cb0 con 0x7f39080c7fc0 2024-04-25T15:53:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:53:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f38e806f400 msgr2=0x7f38e8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f38e806f400 0x7f38e8071910 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f38f8007fd0 tx=0x7f38f8009040 comp rx=0 tx=0).stop 2024-04-25T15:53:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39080c7fc0 msgr2=0x7f39081f4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39080c7fc0 0x7f39081f4330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f39080c8e60 tx=0x7f390001bef0 comp rx=0 tx=0).stop 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 shutdown_connections 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f38e806f400 0x7f38e8071910 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39080c7fc0 0x7f39081f4330 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3908074370 0x7f390814c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 --2- 172.21.15.67:0/3721113921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390806c640 0x7f390815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.034+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 >> 172.21.15.67:0/3721113921 conn(0x7f3908070db0 msgr2=0x7f39080ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.038+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 shutdown_connections 2024-04-25T15:53:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:54.038+0000 7f390ca10640 1 -- 172.21.15.67:0/3721113921 wait complete. 2024-04-25T15:53:54.059 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:54.059+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:668 3.1 e81/76 3:83ba1de1:::100000001f7.00000098:head v 81'14, mlcod=81'12)", 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.255843112000001, 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.045409913000000003, 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.129 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872468+0000", 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3659999999999999e-06 2024-04-25T15:53:54.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879062+0000", 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065941039999999999 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879116+0000", 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4372000000000001e-05 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879152+0000", 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6139000000000001e-05 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879318+0000", 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016595499999999999 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879365+0000", 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6897999999999997e-05 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917803+0000", 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038437773000000001 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.132 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917875+0000", 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2305999999999995e-05 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:637 3.20 e81/18 3:07263727:::100000001f7.00000079:head v 81'17, mlcod=81'15)", 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.246354620999998, 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040244958999999997, 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.133 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881957+0000", 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3890000000000001e-06 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.134 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888368+0000", 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006410682 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888395+0000", 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7477000000000001e-05 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888414+0000", 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9046e-05 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.135 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888512+0000", 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7876999999999998e-05 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888546+0000", 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3204000000000003e-05 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922131+0000", 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033585168999999998 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.136 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922199+0000", 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8114999999999994e-05 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:640 3.10 e81/18 3:092a6a2c:::100000001f7.0000007c:head v 81'13, mlcod=81'11)", 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.895692+0000", 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.232616068999999, 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027335905000000001, 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.137 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.895692+0000", 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.895692+0000", 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.895695+0000", 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1129999999999999e-06 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.138 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.900960+0000", 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052653109999999999 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901007+0000", 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6885000000000003e-05 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901033+0000", 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6160999999999999e-05 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.139 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901166+0000", 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013346000000000001 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901216+0000", 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9264999999999997e-05 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922970+0000", 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021753943000000001 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923028+0000", 2024-04-25T15:53:54.140 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8767000000000002e-05 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:762 3.3 e81/76 3:c0da3165:::100000001f7.000000f6:head v 81'12, mlcod=81'10)", 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.525079+0000", 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.603228981000001, 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024088555000000001, 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.525079+0000", 2024-04-25T15:53:54.141 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.525079+0000", 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.525080+0000", 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1600000000000002e-07 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.531160+0000", 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006080345 2024-04-25T15:53:54.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.531208+0000", 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8026999999999999e-05 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.531224+0000", 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5628999999999999e-05 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.531338+0000", 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011413000000000001 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.143 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.531395+0000", 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7030000000000001e-05 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.549124+0000", 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017728832 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.549168+0000", 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4146000000000001e-05 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.144 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1485 3.20 e88/18 3:04a26667:::100000001f8.000001c9:head v 88'41, mlcod=88'39)", 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.592564+0000", 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.535744294000001, 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025132409000000001, 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.592564+0000", 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.592564+0000", 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.592566+0000", 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7910000000000001e-06 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599273+0000", 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067066410000000002 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.146 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599297+0000", 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4178000000000001e-05 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599312+0000", 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4763000000000001e-05 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599396+0000", 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4790999999999995e-05 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599431+0000", 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4941000000000003e-05 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617638+0000", 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018206938999999998 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617697+0000", 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8365000000000001e-05 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.148 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1590 3.3d e88/40 3:bd018e47:::100000001f8.00000232:head v 88'32, mlcod=88'30)", 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.248469+0000", 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8798394149999993, 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025528071999999999, 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.248469+0000", 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.149 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.248469+0000", 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.248471+0000", 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3719999999999999e-06 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251313+0000", 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028412429999999998 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.150 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0558000000000001e-05 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251360+0000", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7172000000000001e-05 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251454+0000", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3931999999999994e-05 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251489+0000", 2024-04-25T15:53:54.151 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4224999999999998e-05 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.273899+0000", 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022410929999999999 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.273997+0000", 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7639999999999994e-05 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.152 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1598 3.1f e89/40 3:fa5184c2:::100000001f8.0000023a:head v 89'23, mlcod=88'21)", 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.311187+0000", 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8171212820000004, 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024934254999999999, 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.311187+0000", 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.311187+0000", 2024-04-25T15:53:54.153 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.311189+0000", 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.432e-06 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316338+0000", 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051491519999999997 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.154 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316360+0000", 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1914999999999999e-05 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316371+0000", 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1094000000000001e-05 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316433+0000", 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2242999999999993e-05 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.155 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316453+0000", 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0213000000000001e-05 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.336057+0000", 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019603438000000001 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.336121+0000", 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4768000000000001e-05 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.156 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1709 3.b e89/21 3:d0c4adc9:::100000001f8.000002a9:head v 89'35, mlcod=89'33)", 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.129378634, 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041562524000000003, 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.157 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998932+0000", 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7570000000000001e-06 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.158 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000395+0000", 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014629389999999999 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000414+0000", 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9908999999999999e-05 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000432+0000", 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249999999999999e-05 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.159 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000518+0000", 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6501000000000004e-05 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000548+0000", 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9777e-05 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040424+0000", 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039875586999999997 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.160 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040492+0000", 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8804000000000001e-05 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1710 3.b e89/21 3:d079ce42:::100000001f8.000002aa:head v 89'36, mlcod=89'33)", 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1229499169999997, 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040049727, 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.161 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005360+0000", 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9290000000000001e-06 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006358+0000", 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099724699999999993 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006377+0000", 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9656e-05 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006392+0000", 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4207e-05 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006477+0000", 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4989999999999998e-05 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006507+0000", 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0786e-05 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045329+0000", 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038821829000000002 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045408+0000", 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9083000000000002e-05 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1711 3.12 e89/40 3:4b7bcb17:::100000001f8.000002ab:head v 89'37, mlcod=89'35)", 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.013771+0000", 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1145372600000005, 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031890573999999998, 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.013771+0000", 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.013771+0000", 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.013772+0000", 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8599999999999996e-07 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017771+0000", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039990889999999999 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017791+0000", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9976999999999999e-05 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017805+0000", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3413e-05 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017869+0000", 2024-04-25T15:53:54.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4417000000000003e-05 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017894+0000", 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.512e-05 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045623+0000", 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027728778999999999 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045662+0000", 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8792999999999999e-05 2024-04-25T15:53:54.168 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1712 3.30 e89/18 3:0f3afaad:::100000001f8.000002ac:head v 89'47, mlcod=89'45)", 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.020649+0000", 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1076596199999997, 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025202367999999999, 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.020649+0000", 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.020649+0000", 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.020650+0000", 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4559999999999999e-06 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025465+0000", 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004815059 2024-04-25T15:53:54.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025486+0000", 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0706000000000001e-05 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025500+0000", 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4141000000000001e-05 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025565+0000", 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5024000000000006e-05 2024-04-25T15:53:54.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025586+0000", 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0930000000000001e-05 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045786+0000", 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020199994999999998 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045851+0000", 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5056999999999997e-05 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1802 3.23 e89/40 3:c6704eb5:::100000001f8.00000306:head v 89'33, mlcod=88'31)", 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.558838+0000", 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.5694701569999996, 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02500527, 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.558838+0000", 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.558838+0000", 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.558839+0000", 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4700000000000001e-07 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560492+0000", 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001652854 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560516+0000", 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4742000000000001e-05 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560534+0000", 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7898e-05 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560626+0000", 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1414000000000003e-05 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560658+0000", 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1853999999999997e-05 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.583765+0000", 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023107544000000001 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.583844+0000", 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8417000000000005e-05 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1811 3.10 e89/18 3:0b5e6101:::100000001f8.0000030f:head v 89'58, mlcod=89'56)", 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.668830+0000", 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.4594787500000006, 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026867552999999999, 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.668830+0000", 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.668830+0000", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.668832+0000", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9589999999999998e-06 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677622+0000", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087903660000000008 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677650+0000", 2024-04-25T15:53:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8167e-05 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677670+0000", 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9643000000000001e-05 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677754+0000", 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4019999999999999e-05 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677778+0000", 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4042000000000001e-05 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695608+0000", 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017830141000000001 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695697+0000", 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9215000000000006e-05 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2273 3.39 e90/18 3:9fcbcc01:::100000001f8.000004dd:head v 90'54, mlcod=90'52)", 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.302800672, 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035128116000000001, 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:53:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825510+0000", 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837142+0000", 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011632136 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837175+0000", 2024-04-25T15:53:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3166000000000001e-05 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837197+0000", 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1495e-05 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837293+0000", 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6353000000000004e-05 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837329+0000", 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5901000000000001e-05 2024-04-25T15:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860257+0000", 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022927804999999999 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860636+0000", 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000379327 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2275 3.12 e90/40 3:492b2588:::100000001f8.000004df:head v 90'47, mlcod=89'45)", 2024-04-25T15:53:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.837247+0000", 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.2910619639999998, 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024204458000000002, 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837247+0000", 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837247+0000", 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837248+0000", 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1289999999999999e-06 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.843461+0000", 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062132539999999997 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.843482+0000", 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0860999999999999e-05 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.843499+0000", 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7677999999999999e-05 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.843594+0000", 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4909999999999998e-05 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.843664+0000", 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9336000000000002e-05 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.861408+0000", 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017744770999999999 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.861451+0000", 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2518999999999997e-05 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2392 3.32 e90/40 3:4cdbb504:::100000001f8.00000554:head v 90'45, mlcod=90'43)", 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7189859109999999, 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044535299, 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409325+0000", 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2220000000000001e-06 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410846+0000", 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001521454 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410866+0000", 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0129999999999999e-05 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410882+0000", 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5452e-05 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410962+0000", 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9833999999999995e-05 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410991+0000", 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.968e-05 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453793+0000", 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042801621999999998 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453858+0000", 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4905000000000003e-05 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.192 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2394 3.6 e90/76 3:63a5c66e:::100000001f8.00000556:head v 90'46, mlcod=90'44)", 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7042907879999998, 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038940880999999997, 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424019+0000", 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6160000000000001e-06 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428140+0000", 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041208529999999998 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428169+0000", 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8824999999999999e-05 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428190+0000", 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0894999999999999e-05 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428283+0000", 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3277999999999997e-05 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428318+0000", 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5191000000000002e-05 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462890+0000", 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034571283000000001 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462959+0000", 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000001e-05 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2396 3.3d e90/40 3:bc4ba9bc:::100000001f8.00000558:head v 90'44, mlcod=90'42)", 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.6944855670000001, 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033872159999999998, 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433825+0000", 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.951e-06 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435239+0000", 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014141729999999999 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435260+0000", 2024-04-25T15:53:54.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0775e-05 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435278+0000", 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8317000000000001e-05 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435363+0000", 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4441e-05 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435395+0000", 2024-04-25T15:53:54.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1939e-05 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467549+0000", 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032154061999999997 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467695+0000", 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014650199999999999 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2556 3.17 e90/40 3:e9827f3e:::100000001f8.000005f8:head v 90'46, mlcod=90'44)", 2024-04-25T15:53:54.200 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:50.546366+0000", 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.5819425549999999, 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025043425000000001, 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.546366+0000", 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.546366+0000", 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.546369+0000", 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2109999999999998e-06 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552050+0000", 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056808589999999999 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552109+0000", 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8536000000000003e-05 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552129+0000", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0752000000000001e-05 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552217+0000", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.81e-05 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552243+0000", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5568999999999999e-05 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.571338+0000", 2024-04-25T15:53:54.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019095035999999999 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.571409+0000", 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1361999999999999e-05 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3052 3.30 e93/18 3:0fd3b3ba:::100000001f8.000007e8:head v 93'59, mlcod=91'57)", 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:53.799314+0000", 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.32899407000000003, 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024425879000000001, 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.204 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.799314+0000", 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.799314+0000", 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.799316+0000", 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2699999999999999e-06 2024-04-25T15:53:54.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.804197+0000", 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048812869999999998 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.804223+0000", 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6049000000000001e-05 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.804242+0000", 2024-04-25T15:53:54.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8810000000000001e-05 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.804376+0000", 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000133724 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.804403+0000", 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7756999999999999e-05 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.823649+0000", 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019245409000000002 2024-04-25T15:53:54.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:53.823740+0000", 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1572999999999999e-05 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.208 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:54.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:54.210 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:54.210+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:54.312 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:54.307+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:54.412 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:54.411+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3145 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.013722059, 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013769823, 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.447 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3145 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:53:54.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434689+0000", 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.005e-06 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436062+0000", 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001373682 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436096+0000", 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3797000000000002e-05 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436107+0000", 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436177+0000", 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0069999999999998e-05 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436346+0000", 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169168 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436510+0000", 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163807 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.443699+0000", 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007189394 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 1 2024-04-25T15:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:54.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:53:54.521 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:54.522+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:54.613 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:54.611+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:54.655 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:54.655 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:54.655 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:54.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:53:54.714 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:54.711+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:54.815 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:54.811+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:54.910 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:54.910 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 192.66223251, 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:53:54.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:676 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.980194179000001, 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053839007000000001, 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.916 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929776+0000", 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.743e-06 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931715+0000", 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019394130000000001 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931754+0000", 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8779000000000003e-05 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931769+0000", 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5231999999999999e-05 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931867+0000", 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7485000000000006e-05 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932043+0000", 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017616100000000001 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932181+0000", 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013769100000000001 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.942677+0000", 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010495931 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983477+0000", 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040800621000000002 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983546+0000", 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9138000000000001e-05 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983611+0000", 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4813000000000004e-05 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:733 3.22 3:44586a4d:::100000001f7.000000d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.601728131000002, 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037465947999999999, 2024-04-25T15:53:54.921 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 733 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308242+0000", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.399e-06 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309826+0000", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015840349999999999 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309863+0000", 2024-04-25T15:53:54.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7122000000000001e-05 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309880+0000", 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6888000000000002e-05 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309986+0000", 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105979 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310169+0000", 2024-04-25T15:53:54.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018349699999999999 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310353+0000", 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018415799999999999 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.342377+0000", 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032023169999999997 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345569+0000", 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031928680000000002 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345644+0000", 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4140999999999994e-05 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345704+0000", 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0690999999999999e-05 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:932 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.088886+0000", 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.82108083, 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028312902000000001, 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 932 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.088886+0000", 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.088886+0000", 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.088888+0000", 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4499999999999998e-06 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093536+0000", 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004647506 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093595+0000", 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9392999999999999e-05 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093617+0000", 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2062999999999999e-05 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093730+0000", 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011247399999999999 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.093849+0000", 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011962499999999999 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.094001+0000", 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151404 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.106643+0000", 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01264216 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.117113+0000", 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010470405 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.117156+0000", 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2651000000000002e-05 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.117199+0000", 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2771000000000001e-05 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1515 3.18 3:1bff78ff:::100000001f8.000001e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.784293+0000", 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.125673104000001, 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025376673999999998, 2024-04-25T15:53:54.932 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1515 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.784293+0000", 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.784293+0000", 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.784296+0000", 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8100000000000002e-06 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786418+0000", 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021213870000000002 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786473+0000", 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5640000000000003e-05 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786491+0000", 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.819e-05 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786669+0000", 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017772099999999999 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.786881+0000", 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000212036 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.787039+0000", 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015808799999999999 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.801484+0000", 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014444477000000001 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.809535+0000", 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0080517169999999999 2024-04-25T15:53:54.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.809611+0000", 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5791000000000007e-05 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.809670+0000", 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8817000000000003e-05 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.937 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1543 3.32 3:4d482082:::100000001f8.00000203:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.947449+0000", 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.962517804000001, 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026440832000000001, 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1543 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.947449+0000", 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.947449+0000", 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.947451+0000", 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3029999999999998e-06 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949063+0000", 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016119769999999999 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949101+0000", 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7978999999999997e-05 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949115+0000", 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4068e-05 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949187+0000", 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1627999999999999e-05 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949323+0000", 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013668299999999999 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.949473+0000", 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014975000000000001 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973130+0000", 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023657114999999999 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973788+0000", 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00065738899999999998 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973844+0000", 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6490999999999998e-05 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973890+0000", 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5448999999999998e-05 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1544 3.22 3:453283ef:::100000001f8.00000204:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.954353+0000", 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.95561301, 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025396769999999999, 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1544 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.954353+0000", 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.954353+0000", 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.954357+0000", 2024-04-25T15:53:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1980000000000001e-06 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956165+0000", 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018087540000000001 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956202+0000", 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6248999999999998e-05 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956224+0000", 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2141999999999999e-05 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956316+0000", 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1724000000000005e-05 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956484+0000", 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000168858 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.956591+0000", 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106263 2024-04-25T15:53:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.973938+0000", 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017347122 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.979612+0000", 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056746360000000003 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.979682+0000", 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9142999999999999e-05 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.979750+0000", 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8681000000000002e-05 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1589 3.1a e88/76 3:590c0ff7:::100000001f8.00000231:head v 88'30, mlcod=88'28)", 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.66314313, 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032177728000000003, 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246826+0000", 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1900000000000002e-06 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045171680000000002 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251364+0000", 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1444999999999999e-05 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251375+0000", 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.115e-05 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251458+0000", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3190999999999997e-05 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251485+0000", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6661000000000001e-05 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.278940+0000", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027455222000000001 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.279001+0000", 2024-04-25T15:53:54.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0701e-05 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1595 3.18 3:1adf79c2:::100000001f8.00000237:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.627782093, 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038955741000000002, 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.952 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1595 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282187+0000", 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1569999999999998e-06 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284089+0000", 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001902672 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284127+0000", 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7663000000000002e-05 2024-04-25T15:53:54.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284141+0000", 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284236+0000", 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4895999999999994e-05 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284417+0000", 2024-04-25T15:53:54.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018094500000000001 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284602+0000", 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018422800000000001 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.293831+0000", 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092292650000000004 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321052+0000", 2024-04-25T15:53:54.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027220978 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321098+0000", 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6520000000000002e-05 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321140+0000", 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1850000000000001e-05 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1814 3.18 3:1977fdc7:::100000001f8.00000312:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.636518+0000", 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.2734482749999998, 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024442130999999999, 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1814 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.636518+0000", 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.636518+0000", 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.636521+0000", 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3010000000000001e-06 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638401+0000", 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018804379999999999 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638442+0000", 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1542999999999998e-05 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638461+0000", 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8567e-05 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638559+0000", 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8355000000000003e-05 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638720+0000", 2024-04-25T15:53:54.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016095200000000001 2024-04-25T15:53:54.962 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:54.959+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.638873+0000", 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015225899999999999 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.657264+0000", 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018391873999999999 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.660841+0000", 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0035769170000000002 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.660910+0000", 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9021000000000002e-05 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.660960+0000", 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9904000000000002e-05 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1817 3.1a e89/76 3:5b497a17:::100000001f8.00000315:head v 89'33, mlcod=89'31)", 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.667493+0000", 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.2424737260000001, 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028489159, 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.965 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.667493+0000", 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.667493+0000", 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.667494+0000", 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4330000000000001e-06 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680261+0000", 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012767152 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680284+0000", 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2846000000000001e-05 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680302+0000", 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7506000000000001e-05 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680403+0000", 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101007 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.680432+0000", 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9417e-05 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695912+0000", 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015479655 2024-04-25T15:53:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695982+0000", 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0142999999999996e-05 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2263 3.18 3:1b2b5a9e:::100000001f8.000004d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.550992+0000", 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.3589745210000004, 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02958715, 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.969 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2263 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.550992+0000", 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.550992+0000", 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.550995+0000", 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8150000000000002e-06 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.552761+0000", 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017658839999999999 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.552980+0000", 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021939299999999999 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.552994+0000", 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4416999999999999e-05 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.553100+0000", 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106028 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.553258+0000", 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157159 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.553400+0000", 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142323 2024-04-25T15:53:54.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.559455+0000", 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006054785 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.580489+0000", 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021034177000000001 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.580531+0000", 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1777999999999999e-05 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.580579+0000", 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8390999999999998e-05 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2389 3.18 3:1bdc9869:::100000001f8.00000551:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5207308910000004, 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030905229999999999, 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.974 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2389 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389238+0000", 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7099999999999999e-06 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391140+0000", 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001901359 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391174+0000", 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4165000000000002e-05 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391187+0000", 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3188e-05 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391277+0000", 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0267999999999997e-05 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391455+0000", 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017728700000000001 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391618+0000", 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163308 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.419720+0000", 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028102374999999999 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420001+0000", 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028060900000000002 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420083+0000", 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2010999999999998e-05 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420141+0000", 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7949999999999999e-05 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2392 3.32 3:4cdbb504:::100000001f8.00000554:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5040321329999999, 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048266518000000001, 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2392 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405937+0000", 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6759999999999999e-06 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408696+0000", 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027593589999999999 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408735+0000", 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8516000000000002e-05 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408749+0000", 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4117000000000001e-05 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408844+0000", 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4891999999999999e-05 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409024+0000", 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018000199999999999 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409200+0000", 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017591800000000001 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.427538+0000", 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018337728000000001 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454069+0000", 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026531451000000001 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454141+0000", 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2316000000000004e-05 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454201+0000", 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9543000000000002e-05 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2393 3.22 3:45fb7032:::100000001f8.00000555:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:53:54.985 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4976395619999998, 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052582106000000003, 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2393 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412330+0000", 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3119999999999998e-06 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413928+0000", 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597521 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413971+0000", 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3086e-05 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413988+0000", 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7336000000000001e-05 2024-04-25T15:53:54.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414091+0000", 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103099 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414245+0000", 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015350300000000001 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414359+0000", 2024-04-25T15:53:54.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011410500000000001 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.446661+0000", 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032302571000000002 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464781+0000", 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018119283999999999 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464856+0000", 2024-04-25T15:53:54.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5153000000000004e-05 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464909+0000", 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3136000000000001e-05 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2395 3.18 3:183ba26c:::100000001f8.00000557:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4852394540000002, 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029801055999999999, 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2395 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424735+0000", 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1780000000000003e-06 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426303+0000", 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015679170000000001 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.993 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426349+0000", 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6343e-05 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426370+0000", 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329000000000001e-05 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426458+0000", 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.831e-05 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426609+0000", 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150629 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426735+0000", 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012625900000000001 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.441627+0000", 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014891824999999999 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454417+0000", 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012789994000000001 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454476+0000", 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8752000000000003e-05 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454528+0000", 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2519999999999999e-05 2024-04-25T15:53:54.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2420 3.18 3:1bcc4fe4:::100000001f8.00000570:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.600704+0000", 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.309262146, 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026186556, 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:54.997 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2420 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.600704+0000", 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.600704+0000", 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.600707+0000", 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.757e-06 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.602494+0000", 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017868230000000001 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.602541+0000", 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6681999999999998e-05 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:54.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.602560+0000", 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9690999999999998e-05 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.602665+0000", 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010431 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.602884+0000", 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000219263 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.603073+0000", 2024-04-25T15:53:55.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018875900000000001 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.614119+0000", 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011046721000000001 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.626803+0000", 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012683665 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.626854+0000", 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1025e-05 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.626891+0000", 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6860000000000003e-05 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2564 3.18 3:1b311046:::100000001f8.00000600:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:50.585869+0000", 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3240977689999998, 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024985052000000001, 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.002 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2564 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.585869+0000", 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.585869+0000", 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.585872+0000", 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3459999999999998e-06 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.587657+0000", 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017852580000000001 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.587692+0000", 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4684000000000002e-05 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.587710+0000", 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7904999999999999e-05 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.587814+0000", 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010426500000000001 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.588007+0000", 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193174 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:55.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.588193+0000", 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018565699999999999 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.596164+0000", 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079706140000000009 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.610742+0000", 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014578778000000001 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.610816+0000", 2024-04-25T15:53:55.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3761000000000001e-05 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.610854+0000", 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7610000000000001e-05 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3095 3.18 3:1a5a6080:::100000001f8.00000813:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.098603+0000", 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.81136345099999996, 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027550497, 2024-04-25T15:53:55.007 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3095 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.098603+0000", 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.098603+0000", 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.098606+0000", 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0800000000000002e-06 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.100847+0000", 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022405900000000002 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.100887+0000", 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0683999999999999e-05 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.100903+0000", 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5201999999999999e-05 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.100978+0000", 2024-04-25T15:53:55.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5547e-05 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.101134+0000", 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015624399999999999 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.101299+0000", 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016483599999999999 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.117009+0000", 2024-04-25T15:53:55.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015709473000000002 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.126028+0000", 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090196889999999991 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.126096+0000", 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7434000000000005e-05 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.126154+0000", 2024-04-25T15:53:55.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7717999999999999e-05 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3145 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.47528071999999999, 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030601595999999998, 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3145 2024-04-25T15:53:55.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434689+0000", 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.005e-06 2024-04-25T15:53:55.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436062+0000", 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001373682 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436096+0000", 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3797000000000002e-05 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436107+0000", 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436177+0000", 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0069999999999998e-05 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436346+0000", 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169168 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436510+0000", 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163807 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.443699+0000", 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007189394 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465141+0000", 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021441386999999999 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465223+0000", 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1920000000000002e-05 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465287+0000", 2024-04-25T15:53:55.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4692999999999999e-05 2024-04-25T15:53:55.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.018 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.018 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.018 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:55.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:53:55.065 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:55.066+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:55.164 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:55.163+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3263 3.3b e94/18 3:deccab11:::100000001f8.000008bb:head v 94'59, mlcod=93'57)", 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:55.199402+0000", 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.016196067000000001, 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016223508000000001, 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.199402+0000", 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.199402+0000", 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.199403+0000", 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.269e-06 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.204484+0000", 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005081258 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.204523+0000", 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9032000000000002e-05 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.204546+0000", 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3286e-05 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.204661+0000", 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114752 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.204701+0000", 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9891000000000001e-05 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.217 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:53:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 1 2024-04-25T15:53:55.218 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:55.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:53:55.264 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:55.263+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:55.365 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:55.366+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:55.466 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:55.466+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:55.476 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:55.476 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:55.476 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:55.476 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:55.476 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:55.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:53:55.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:53:55.567 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:55.567+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:55.669 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:55.667+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 Processor -- start 2024-04-25T15:53:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 -- start start 2024-04-25T15:53:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 0x7f56100ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56100c92d0 0x7f56100c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5610160d90 0x7f56101611a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f561006d050 con 0x7f56100c92d0 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5610074720 con 0x7f5610160d90 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.742+0000 7f5615f23640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f56100c9c20 con 0x7f56100733f0 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56100c92d0 0x7f56100c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56100c92d0 0x7f56100c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40002/0 (socket says 172.21.15.67:40002) 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 -- 172.21.15.67:0/1444722156 learned_addr learned my addr 172.21.15.67:0/1444722156 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 -- 172.21.15.67:0/1444722156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 msgr2=0x7f56100ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 --2- 172.21.15.67:0/1444722156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 0x7f56100ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 -- 172.21.15.67:0/1444722156 >> v2:172.21.15.176:3300/0 conn(0x7f5610160d90 msgr2=0x7f56101611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 --2- 172.21.15.67:0/1444722156 >> v2:172.21.15.176:3300/0 conn(0x7f5610160d90 0x7f56101611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 -- 172.21.15.67:0/1444722156 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5610074990 con 0x7f56100c92d0 2024-04-25T15:53:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.746+0000 7f560effd640 1 --2- 172.21.15.67:0/1444722156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f56100c96e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5610073c60 tx=0x7f560001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c1b277e855f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f560e7fc640 1 -- 172.21.15.67:0/1444722156 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56000ab5f0 con 0x7f56100c92d0 2024-04-25T15:53:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f560e7fc640 1 -- 172.21.15.67:0/1444722156 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56000b0070 con 0x7f56100c92d0 2024-04-25T15:53:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f560e7fc640 1 -- 172.21.15.67:0/1444722156 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56000ab9b0 con 0x7f56100c92d0 2024-04-25T15:53:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f5615f23640 1 -- 172.21.15.67:0/1444722156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 msgr2=0x7f56100c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f5615f23640 1 --2- 172.21.15.67:0/1444722156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f56100c96e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f5610073c60 tx=0x7f560001c540 comp rx=0 tx=0).stop 2024-04-25T15:53:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f5615f23640 1 -- 172.21.15.67:0/1444722156 shutdown_connections 2024-04-25T15:53:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f5615f23640 1 --2- 172.21.15.67:0/1444722156 >> v2:172.21.15.176:3300/0 conn(0x7f5610160d90 0x7f56101611a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f5615f23640 1 --2- 172.21.15.67:0/1444722156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f56100c96e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f5615f23640 1 --2- 172.21.15.67:0/1444722156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 0x7f56100ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.750+0000 7f5615f23640 1 -- 172.21.15.67:0/1444722156 >> 172.21.15.67:0/1444722156 conn(0x7f5610070db0 msgr2=0x7f56100ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:55.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.754+0000 7f5615f23640 1 -- 172.21.15.67:0/1444722156 shutdown_connections 2024-04-25T15:53:55.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.758+0000 7f5615f23640 1 -- 172.21.15.67:0/1444722156 wait complete. 2024-04-25T15:53:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.758+0000 7f5615f23640 1 Processor -- start 2024-04-25T15:53:55.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.758+0000 7f5615f23640 1 -- start start 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:147 3.b 3:d1738051:::10000000002.0000008f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.506796+0000", 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.253492344999998, 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067075241999999993, 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.761 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 147 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.506796+0000", 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.506796+0000", 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.506799+0000", 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6170000000000001e-06 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.508301+0000", 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015025240000000001 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559741+0000", 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051439617999999999 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559757+0000", 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.649e-05 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559865+0000", 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000107383 2024-04-25T15:53:55.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.560034+0000", 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016934299999999999 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.560164+0000", 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012956400000000001 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573028+0000", 2024-04-25T15:53:55.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01286471 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573792+0000", 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00076319599999999995 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573833+0000", 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.138e-05 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.573871+0000", 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8417000000000002e-05 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:148 3.9 3:91a8729a:::10000000002.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 67.202070276000001, 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.432712019, 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 148 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558221+0000", 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3929999999999998e-06 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559704+0000", 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001483807 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975277+0000", 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.41557216400000002 2024-04-25T15:53:55.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975295+0000", 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8408999999999999e-05 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975372+0000", 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7235000000000002e-05 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975484+0000", 2024-04-25T15:53:55.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111499 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975582+0000", 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8589000000000001e-05 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.980957+0000", 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053747600000000001 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990855+0000", 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098981350000000006 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990894+0000", 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8436000000000003e-05 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990930+0000", 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6591999999999998e-05 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:157 3.2a 3:577cc1b8:::10000000002.00000099:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.973161+0000", 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 66.787127773999998, 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.067779252999999998, 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 157 2024-04-25T15:53:55.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.973161+0000", 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.973161+0000", 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.973164+0000", 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.168e-06 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975237+0000", 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020727599999999999 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027231+0000", 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051994686999999998 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027250+0000", 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8292e-05 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027337+0000", 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6977000000000004e-05 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027503+0000", 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016680300000000001 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.027613+0000", 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110031 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.034797+0000", 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071834669999999998 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.040858+0000", 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0060610799999999999 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.040896+0000", 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8367e-05 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:49.040940+0000", 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3621000000000001e-05 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1009 3.3e 3:7fa01700:::100000001f7.000000ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:28.532897+0000", 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 27.227391073, 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046042244000000003, 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1009 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.532897+0000", 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.532897+0000", 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.532900+0000", 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9849999999999998e-06 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534627+0000", 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017263479999999999 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534658+0000", 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1600000000000002e-05 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534670+0000", 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.163e-05 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534798+0000", 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000128355 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.534901+0000", 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102429 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.535068+0000", 2024-04-25T15:53:55.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016683600000000001 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.543609+0000", 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008541468 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.578815+0000", 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035206144000000002 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.578877+0000", 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1822000000000005e-05 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:28.578940+0000", 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2626999999999995e-05 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1473 3.0 3:008165f2:::100000001f8.000001bd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.268068810000001, 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.084007844999999998, 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.784 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1473 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492222+0000", 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6910000000000002e-06 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.494870+0000", 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026480620000000001 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557687+0000", 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.062816831000000004 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557707+0000", 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9262e-05 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557796+0000", 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9209000000000006e-05 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557984+0000", 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000188118 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.558165+0000", 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181292 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.565264+0000", 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0070987309999999996 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576094+0000", 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010829920999999999 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576165+0000", 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0951999999999994e-05 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576228+0000", 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2775999999999997e-05 2024-04-25T15:53:55.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1481 3.2a 3:55ceaa15:::100000001f8.000001c5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.555971+0000", 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.204317236, 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075920801999999996, 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.790 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1481 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.555971+0000", 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.555971+0000", 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.555974+0000", 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2850000000000001e-06 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557650+0000", 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001676716 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617464+0000", 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.059813893999999999 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617483+0000", 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8468999999999999e-05 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617590+0000", 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010698500000000001 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617779+0000", 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018936699999999999 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:55.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617928+0000", 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000149188 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.627751+0000", 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098226839999999999 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.631788+0000", 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0040376459999999998 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.631829+0000", 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0256999999999998e-05 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.631892+0000", 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3311000000000004e-05 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.795 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1490 3.0 3:008ec8c9:::100000001f8.000001ce:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.615629+0000", 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.144659958, 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046517017000000001, 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1490 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.615629+0000", 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.615629+0000", 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.615631+0000", 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4109999999999999e-06 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617421+0000", 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017897900000000001 2024-04-25T15:53:55.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.640726+0000", 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023305612999999999 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.640740+0000", 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3451999999999999e-05 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.640831+0000", 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1593000000000002e-05 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.641019+0000", 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018716199999999999 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.641204+0000", 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000185236 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.649450+0000", 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008246069 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.662008+0000", 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012558348 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.662082+0000", 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4192999999999999e-05 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.662146+0000", 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3150000000000004e-05 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1494 3.37 3:efe547ea:::100000001f8.000001d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.115885728, 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089208711999999996, 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1494 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644406+0000", 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2229999999999999e-06 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.646356+0000", 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019497990000000001 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718358+0000", 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072002624000000001 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718376+0000", 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7649999999999999e-05 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718458+0000", 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1984999999999995e-05 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718600+0000", 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000141922 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718731+0000", 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131488 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.724805+0000", 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006074032 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733498+0000", 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086928439999999999 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733549+0000", 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1137000000000002e-05 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733611+0000", 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2008000000000007e-05 2024-04-25T15:53:55.805 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1504 3.2a 3:54fb97d1:::100000001f8.000001dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 12.043592292, 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121933077, 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.806 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1504 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716699+0000", 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4250000000000001e-06 2024-04-25T15:53:55.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718317+0000", 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016187440000000001 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.815989+0000", 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.097671735999999995 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816008+0000", 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9131e-05 2024-04-25T15:53:55.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816090+0000", 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1717000000000004e-05 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816244+0000", 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015372199999999999 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816348+0000", 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010470499999999999 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.826448+0000", 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010099281999999999 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838528+0000", 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012080206 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838583+0000", 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5105000000000002e-05 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838629+0000", 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6304000000000003e-05 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1521 3.21 3:84dc523d:::100000001f8.000001ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.822483+0000", 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.937805314, 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.060484519, 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.811 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1521 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.822483+0000", 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.822483+0000", 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.822486+0000", 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1580000000000001e-06 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.823900+0000", 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001413847 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865598+0000", 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041697565999999998 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865612+0000", 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4724e-05 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865689+0000", 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6223000000000005e-05 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865824+0000", 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013481999999999999 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865930+0000", 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010693599999999999 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.872071+0000", 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0061405640000000003 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.882880+0000", 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010809246999999999 2024-04-25T15:53:55.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.882922+0000", 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2219999999999999e-05 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.882968+0000", 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5213999999999998e-05 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1528 3.3e 3:7f4f4afd:::100000001f8.000001f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.896118445999999, 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.133348521, 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1528 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864172+0000", 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.204e-06 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865554+0000", 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013821899999999999 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986006+0000", 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12045179 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986022+0000", 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999e-05 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986101+0000", 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9251000000000004e-05 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986224+0000", 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012263899999999999 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986330+0000", 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106164 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.996058+0000", 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097274319999999994 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997353+0000", 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001294882 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997450+0000", 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7446000000000002e-05 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997519+0000", 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8523999999999996e-05 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1548 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.776652738999999, 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096155170999999998, 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.822 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1548 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:53:55.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983639+0000", 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1750000000000001e-06 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.985960+0000", 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023213790000000002 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058434+0000", 2024-04-25T15:53:55.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072473913000000001 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058454+0000", 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9329000000000001e-05 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058541+0000", 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7702999999999997e-05 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058708+0000", 2024-04-25T15:53:55.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016712300000000001 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058864+0000", 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015600100000000001 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.065501+0000", 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066362299999999999 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079696+0000", 2024-04-25T15:53:55.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014195228000000001 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079745+0000", 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8980000000000002e-05 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079791+0000", 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6109999999999997e-05 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1561 3.2e 3:75253c31:::100000001f8.00000215:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.692258209, 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099620287000000002, 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1561 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068033+0000", 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1240000000000001e-06 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.070277+0000", 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002243676 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143785+0000", 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073508283999999993 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143804+0000", 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8808e-05 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143881+0000", 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6379999999999997e-05 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144063+0000", 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018198500000000001 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144168+0000", 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105253 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.154374+0000", 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010206374000000001 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167481+0000", 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013106405999999999 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167580+0000", 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9329999999999999e-05 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167651+0000", 2024-04-25T15:53:55.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0667000000000006e-05 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1688 3.2e 3:764e45bb:::100000001f8.00000294:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.865613+0000", 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.894675736, 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052534059000000001, 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.833 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1688 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.865613+0000", 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.865613+0000", 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.865615+0000", 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329999999999998e-06 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867053+0000", 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014382539999999999 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867094+0000", 2024-04-25T15:53:55.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0695999999999998e-05 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867113+0000", 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9207000000000001e-05 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867190+0000", 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6981e-05 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867339+0000", 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014954399999999999 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.867473+0000", 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013331899999999999 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.873926+0000", 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0064532010000000004 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.918006+0000", 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044080345999999999 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.918081+0000", 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5078000000000002e-05 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.918147+0000", 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5400000000000004e-05 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1704 3.4 3:230cd4d1:::100000001f8.000002a4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.964986+0000", 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.795302322, 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044402101999999999, 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1704 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.964986+0000", 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.964986+0000", 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.964989+0000", 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.061e-06 2024-04-25T15:53:55.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966880+0000", 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001890907 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966907+0000", 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7237e-05 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966920+0000", 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2172e-05 2024-04-25T15:53:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.966996+0000", 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6569999999999994e-05 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.967163+0000", 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000166983 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:55.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.967337+0000", 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017356399999999999 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.977633+0000", 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010296764 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.009257+0000", 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031623186999999997 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.009326+0000", 2024-04-25T15:53:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9443999999999994e-05 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.009388+0000", 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2212999999999995e-05 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1709 3.b 3:d0c4adc9:::100000001f8.000002a9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.996354+0000", 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.763934936, 2024-04-25T15:53:55.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044554143999999997, 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1709 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.996354+0000", 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.996354+0000", 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.996356+0000", 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2309999999999999e-06 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998388+0000", 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00203177 2024-04-25T15:53:55.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998428+0000", 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0862999999999998e-05 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998447+0000", 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8783999999999998e-05 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998525+0000", 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8152000000000007e-05 2024-04-25T15:53:55.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998701+0000", 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017595499999999999 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998821+0000", 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119335 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.028809+0000", 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029987949 2024-04-25T15:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040800+0000", 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.01199154 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040847+0000", 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7308999999999997e-05 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040908+0000", 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0256e-05 2024-04-25T15:53:55.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2308 3.4 3:2343a40c:::100000001f8.00000500:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.9392038969999996, 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.129707092, 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.850 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2308 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821088+0000", 2024-04-25T15:53:55.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1599999999999998e-06 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823092+0000", 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020043909999999999 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823143+0000", 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0791999999999998e-05 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823162+0000", 2024-04-25T15:53:55.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9332999999999999e-05 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823254+0000", 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2169000000000005e-05 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823637+0000", 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038265100000000002 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:55.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823849+0000", 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211879 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.834007+0000", 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010158377 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950654+0000", 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.116646229 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950721+0000", 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7020000000000005e-05 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950792+0000", 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1091000000000001e-05 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2489 3.2e 3:740912a7:::100000001f8.000005b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:50.107521+0000", 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6527679370000001, 2024-04-25T15:53:55.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.057692686999999999, 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2489 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.107521+0000", 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.107521+0000", 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.107523+0000", 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.503e-06 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.109053+0000", 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015295040000000001 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.109101+0000", 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8806e-05 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.109121+0000", 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9550000000000001e-05 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.109213+0000", 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2095999999999994e-05 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.109364+0000", 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150789 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.109530+0000", 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016629399999999999 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.122029+0000", 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012498763 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.165101+0000", 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043071810000000002 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.165170+0000", 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9616000000000006e-05 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.165213+0000", 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2956e-05 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3128 3.37 3:ec781aeb:::100000001f8.00000834:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.328074+0000", 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.43221441, 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.055269110000000003, 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3128 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.328074+0000", 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.328074+0000", 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.328077+0000", 2024-04-25T15:53:55.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2949999999999998e-06 2024-04-25T15:53:55.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.329570+0000", 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001492288 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.329620+0000", 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0658999999999997e-05 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.329640+0000", 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9686000000000001e-05 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.329734+0000", 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4097000000000004e-05 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.329862+0000", 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012805499999999999 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.329992+0000", 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012997599999999999 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.340097+0000", 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010105012 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.383165+0000", 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043068290000000002 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.383281+0000", 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115402 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.383343+0000", 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2349999999999998e-05 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3326 3.4 3:23741ef8:::100000001f8.000008fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:55.555237+0000", 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.205051391, 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048104751000000001, 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:55.869 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3326 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.555237+0000", 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:55.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.555237+0000", 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.555241+0000", 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.557e-06 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.556902+0000", 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016615779999999999 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:55.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.556941+0000", 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8336e-05 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.556957+0000", 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6614999999999999e-05 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.557034+0000", 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6569000000000005e-05 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.557166+0000", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000132731 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.557270+0000", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103902 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.564380+0000", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0071100640000000001 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.603198+0000", 2024-04-25T15:53:55.873 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038817849000000001 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.603277+0000", 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8535000000000006e-05 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.603342+0000", 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5015e-05 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:55.874 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:55.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.758+0000 7f5615f23640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 0x7f5610153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.758+0000 7f5615f23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f561015eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.758+0000 7f5615f23640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5610160d90 0x7f561014e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f5615f23640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5610161c70 con 0x7f56100c92d0 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f5615f23640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5610161af0 con 0x7f5610160d90 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f5615f23640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5610164b10 con 0x7f56100733f0 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f561015eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f561015eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40008/0 (socket says 172.21.15.67:40008) 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 -- 172.21.15.67:0/1884300523 learned_addr learned my addr 172.21.15.67:0/1884300523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 0x7f5610153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560ffff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5610160d90 0x7f561014e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 msgr2=0x7f5610153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56100733f0 0x7f5610153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5610160d90 msgr2=0x7f561014e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5610160d90 0x7f561014e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561015f2e0 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f560effd640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f561015eda0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f560001abf0 tx=0x7f560001ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56000be5f0 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56000be7b0 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56000beba0 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56101fec20 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.762+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5610154880 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f5610154ac0 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f56000b43f0 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f55f7fff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f55fc06f430 0x7f55fc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5973+0+0 (secure 0 0 0) 0x7f5600129340 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v94) ==== 54+0+16 (secure 0 0 0) 0x7f5600016130 con 0x7f56100c92d0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5610152bb0 0x7f561015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f55fc06f430 0x7f55fc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f561006a480 con 0x7f5610152bb0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f560ffff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5610152bb0 0x7f561015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.766+0000 7f560ffff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5610152bb0 0x7f561015cc60 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.770+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f55fc06f430 0x7f55fc071940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f56040098f0 tx=0x7f5604006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.770+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f561006a480 con 0x7f5610152bb0 2024-04-25T15:53:55.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:53:55.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.790+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f56100cf080 con 0x7f5610152bb0 2024-04-25T15:53:55.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.794+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f56100cf080 con 0x7f5610152bb0 2024-04-25T15:53:55.882 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.794+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f56101fdf00 0x7f56101fe310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.794+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f5610074120 con 0x7f56101fdf00 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.794+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f56101fdf00 0x7f56101fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.794+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f56101fdf00 0x7f56101fe310 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.794+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5610074120 con 0x7f56101fdf00 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.814+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5610073920 con 0x7f56101fdf00 2024-04-25T15:53:55.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.818+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5610073920 con 0x7f56101fdf00 2024-04-25T15:53:55.884 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:55.884 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.818+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f56101facd0 0x7f56101fb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.818+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f56101fb620 con 0x7f56101facd0 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.818+0000 7f560effd640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f56101facd0 0x7f56101fb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.818+0000 7f560effd640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f56101facd0 0x7f56101fb0e0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.822+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f56101fb620 con 0x7f56101facd0 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.846+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5610063090 con 0x7f56101facd0 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.846+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5610063090 con 0x7f56101facd0 2024-04-25T15:53:55.885 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.846+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5610063b10 0x7f561015b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.846+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f5610063f20 con 0x7f5610063b10 2024-04-25T15:53:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.846+0000 7f560ffff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5610063b10 0x7f561015b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.846+0000 7f560ffff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5610063b10 0x7f561015b8b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.850+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5610063f20 con 0x7f5610063b10 2024-04-25T15:53:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.866+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f561015c0c0 con 0x7f5610063b10 2024-04-25T15:53:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.870+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f561015c0c0 con 0x7f5610063b10 2024-04-25T15:53:55.887 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.870+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f561002c9e0 0x7f561002cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.870+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f5610063f20 con 0x7f561002c9e0 2024-04-25T15:53:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.870+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f561002c9e0 0x7f561002cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.870+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f561002c9e0 0x7f561002cdf0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.870+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5610063f20 con 0x7f561002c9e0 2024-04-25T15:53:55.888 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:55.878+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:55.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.890+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5610063f20 con 0x7f561002c9e0 2024-04-25T15:53:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.894+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5610063f20 con 0x7f561002c9e0 2024-04-25T15:53:55.894 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:55.894 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:55.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.894+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f561002d840 0x7f56102110c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.894+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f561002ddc0 con 0x7f561002d840 2024-04-25T15:53:55.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.894+0000 7f560effd640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f561002d840 0x7f56102110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.894+0000 7f560effd640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f561002d840 0x7f56102110c0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.898+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f561002ddc0 con 0x7f561002d840 2024-04-25T15:53:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.922+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f561002dd10 con 0x7f561002d840 2024-04-25T15:53:55.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.926+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f561002dd10 con 0x7f561002d840 2024-04-25T15:53:55.927 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:55.927 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.926+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5610211b10 0x7f5610213f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.926+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f5610216e00 con 0x7f5610211b10 2024-04-25T15:53:55.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.926+0000 7f560ffff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5610211b10 0x7f5610213f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.930+0000 7f560ffff640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5610211b10 0x7f5610213f30 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.930+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5610216e00 con 0x7f5610211b10 2024-04-25T15:53:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.954+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f56100008e0 con 0x7f5610211b10 2024-04-25T15:53:55.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.962+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f56100008e0 con 0x7f5610211b10 2024-04-25T15:53:55.961 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.962+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5610214980 0x7f5610214d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.962+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f5610216ab0 con 0x7f5610214980 2024-04-25T15:53:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.962+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5610214980 0x7f5610214d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:55.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.966+0000 7f560f7fe640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5610214980 0x7f5610214d90 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:55.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.966+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5610216ab0 con 0x7f5610214980 2024-04-25T15:53:55.978 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:55.978+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:55.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.986+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f5610216ab0 con 0x7f5610214980 2024-04-25T15:53:55.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f55f7fff640 1 -- 172.21.15.67:0/1884300523 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f5610216ab0 con 0x7f5610214980 2024-04-25T15:53:55.992 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5610152bb0 msgr2=0x7f561015cc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f5610152bb0 0x7f561015cc60 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f56101fdf00 msgr2=0x7f56101fe310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f56101fdf00 0x7f56101fe310 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f56101facd0 msgr2=0x7f56101fb0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f56101facd0 0x7f56101fb0e0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.990+0000 7f560ffff640 1 -- 172.21.15.67:0/1884300523 reap_dead start 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5610063b10 msgr2=0x7f561015b8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f5610063b10 0x7f561015b8b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f561002c9e0 msgr2=0x7f561002cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f561002c9e0 0x7f561002cdf0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f561002d840 msgr2=0x7f56102110c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f561002d840 0x7f56102110c0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5610211b10 msgr2=0x7f5610213f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f5610211b10 0x7f5610213f30 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5610214980 msgr2=0x7f5610214d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f5610214980 0x7f5610214d90 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f560ffff640 1 -- 172.21.15.67:0/1884300523 reap_dead start 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f55fc06f430 msgr2=0x7f55fc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f55fc06f430 0x7f55fc071940 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f56040098f0 tx=0x7f5604006f90 comp rx=0 tx=0).stop 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 msgr2=0x7f561015eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f561015eda0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f560001abf0 tx=0x7f560001ad80 comp rx=0 tx=0).stop 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 shutdown_connections 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f55fc06f430 0x7f55fc071940 secure :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f56040098f0 tx=0x7f5604006f90 comp rx=0 tx=0).stop 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 --2- 172.21.15.67:0/1884300523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56100c92d0 0x7f561015eda0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 >> 172.21.15.67:0/1884300523 conn(0x7f5610070db0 msgr2=0x7f561015df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 shutdown_connections 2024-04-25T15:53:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:55.994+0000 7f5615f23640 1 -- 172.21.15.67:0/1884300523 wait complete. 2024-04-25T15:53:56.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:53:56.095 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:56.095 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:56.096 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:56.094+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:56.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:53:56.180 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:56.179+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.266+0000 7fcc97e1f640 1 Processor -- start 2024-04-25T15:53:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.266+0000 7fcc97e1f640 1 -- start start 2024-04-25T15:53:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc97e1f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900c49f0 0x7fcc900c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc97e1f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc9006fa10 con 0x7fcc90161620 2024-04-25T15:53:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc963bc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc90161620 0x7fcc90163a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41380/0 (socket says 172.21.15.67:41380) 2024-04-25T15:53:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc97e1f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc900ca8f0 con 0x7fcc90156330 2024-04-25T15:53:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc963bc640 1 -- 172.21.15.67:0/2966622096 learned_addr learned my addr 172.21.15.67:0/2966622096 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2966622096 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcc900ca6c0 con 0x7fcc900c49f0 2024-04-25T15:53:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900c49f0 0x7fcc900c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 347634709 0 0) 0x7fcc9006fa10 con 0x7fcc90161620 2024-04-25T15:53:56.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc7c003530 con 0x7fcc90161620 2024-04-25T15:53:56.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 814161053 0 0) 0x7fcc7c003530 con 0x7fcc90161620 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900c49f0 msgr2=0x7fcc900c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900c49f0 0x7fcc900c4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 >> v1:172.21.15.176:6789/0 conn(0x7fcc90156330 legacy=0x7fcc9006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc900c3aa0 con 0x7fcc90161620 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fcc88001a00 con 0x7fcc90161620 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 >> v1:172.21.15.67:6789/0 conn(0x7fcc90161620 legacy=0x7fcc90163a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc7c0039f0 0x7fcc7c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c004810 0x7fcc7c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7c005160 0x7fcc7c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c00c050 con 0x7fcc7c004810 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c00d2e0 con 0x7fcc7c005160 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7c00c250 con 0x7fcc7c0039f0 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc95bbb640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc7c0039f0 0x7fcc7c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c004810 0x7fcc7c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc963bc640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7c005160 0x7fcc7c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc953ba640 1 -- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc7c0039f0 msgr2=0x7fcc7c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc7c0039f0 0x7fcc7c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc953ba640 1 -- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7c005160 msgr2=0x7fcc7c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc7c005160 0x7fcc7c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.270+0000 7fcc953ba640 1 -- 172.21.15.67:0/2966622096 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc900c3aa0 con 0x7fcc7c004810 2024-04-25T15:53:56.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.274+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c004810 0x7fcc7c004c20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fcc900ca0e0 tx=0x7fcc80020d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205bd4cf58721553 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:56.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.274+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc800b6070 con 0x7fcc7c004810 2024-04-25T15:53:56.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.274+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc800a9940 con 0x7fcc7c004810 2024-04-25T15:53:56.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.274+0000 7fcc94bb9640 1 -- 172.21.15.67:0/2966622096 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc80009650 con 0x7fcc7c004810 2024-04-25T15:53:56.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.278+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2966622096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c004810 msgr2=0x7fcc7c004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:56.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.278+0000 7fcc97e1f640 1 --2- 172.21.15.67:0/2966622096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc7c004810 0x7fcc7c004c20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fcc900ca0e0 tx=0x7fcc80020d40 comp rx=0 tx=0).stop 2024-04-25T15:53:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.282+0000 7fcc963bc640 1 -- 172.21.15.67:0/2966622096 reap_dead start 2024-04-25T15:53:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.282+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2966622096 shutdown_connections 2024-04-25T15:53:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.282+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2966622096 >> 172.21.15.67:0/2966622096 conn(0x7fcc900ce1c0 msgr2=0x7fcc900ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.282+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2966622096 shutdown_connections 2024-04-25T15:53:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.282+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2966622096 wait complete. 2024-04-25T15:53:56.281 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:56.282+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.282+0000 7fcc97e1f640 1 Processor -- start 2024-04-25T15:53:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.282+0000 7fcc97e1f640 1 -- start start 2024-04-25T15:53:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900c7950 0x7fcc9014e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900b1160 0x7fcc900b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc900c5000 0x7fcc900c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc900c4060 con 0x7fcc900c7950 2024-04-25T15:53:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcc900c43d0 con 0x7fcc900c5000 2024-04-25T15:53:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcc90156740 con 0x7fcc900b1160 2024-04-25T15:53:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900c7950 0x7fcc9014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900c7950 0x7fcc9014e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40034/0 (socket says 172.21.15.67:40034) 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 -- 172.21.15.67:0/2936310775 learned_addr learned my addr 172.21.15.67:0/2936310775 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900b1160 0x7fcc900b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc963bc640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc900c5000 0x7fcc900c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 -- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900b1160 msgr2=0x7fcc900b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900b1160 0x7fcc900b1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 -- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc900c5000 msgr2=0x7fcc900c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc900c5000 0x7fcc900c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 -- 172.21.15.67:0/2936310775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc900cdc50 con 0x7fcc900c7950 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc95bbb640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900c7950 0x7fcc9014e730 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00f7f0 tx=0x7fcc8c00fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc867fc640 1 -- 172.21.15.67:0/2936310775 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c0050d0 con 0x7fcc900c7950 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc867fc640 1 -- 172.21.15.67:0/2936310775 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc8c018070 con 0x7fcc900c7950 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc901f1e30 con 0x7fcc900c7950 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc867fc640 1 -- 172.21.15.67:0/2936310775 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c00d730 con 0x7fcc900c7950 2024-04-25T15:53:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc901f23f0 con 0x7fcc900c7950 2024-04-25T15:53:56.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.286+0000 7fcc867fc640 1 -- 172.21.15.67:0/2936310775 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcc8c00d8d0 con 0x7fcc900c7950 2024-04-25T15:53:56.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.290+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc9006a480 con 0x7fcc900c7950 2024-04-25T15:53:56.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.290+0000 7fcc867fc640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcc7406f430 0x7fcc74071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:56.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.290+0000 7fcc867fc640 1 -- 172.21.15.67:0/2936310775 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5973+0+0 (secure 0 0 0) 0x7fcc8c08efb0 con 0x7fcc900c7950 2024-04-25T15:53:56.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.290+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcc7406f430 0x7fcc74071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:56.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.290+0000 7fcc953ba640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcc7406f430 0x7fcc74071940 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcc800b3000 tx=0x7fcc800046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:56.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.298+0000 7fcc867fc640 1 -- 172.21.15.67:0/2936310775 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcc900cdc50 con 0x7fcc900c7950 2024-04-25T15:53:56.362 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:56.362 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:56.382 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:56.382+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:56.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:53:56.483 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:56.483+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:56.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:56.534+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fcc900c3650 con 0x7fcc900c7950 2024-04-25T15:53:56.584 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:56.583+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:56.618 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:56.618 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:56.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:53:56.687 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:56.687+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:56.785 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:56.783+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:56.841 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:56.841 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:56.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:53:56.911 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:56.911+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:56.986 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:56.983+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:57.026 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:53:57.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:53:57.026 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-04-25T15:53:57.026 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:53:57.026 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:53:57.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-04-25T15:53:57.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.042+0000 7fcc867fc640 1 -- 172.21.15.67:0/2936310775 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v95) ==== 97+0+0 (secure 0 0 0) 0x7fcc8c05beb0 con 0x7fcc900c7950 2024-04-25T15:53:57.043 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:53:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcc7406f430 msgr2=0x7fcc74071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcc7406f430 0x7fcc74071940 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcc800b3000 tx=0x7fcc800046d0 comp rx=0 tx=0).stop 2024-04-25T15:53:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900c7950 msgr2=0x7fcc9014e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900c7950 0x7fcc9014e730 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00f7f0 tx=0x7fcc8c00fbf0 comp rx=0 tx=0).stop 2024-04-25T15:53:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 shutdown_connections 2024-04-25T15:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcc7406f430 0x7fcc74071940 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc900c5000 0x7fcc900c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc900b1160 0x7fcc900b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 --2- 172.21.15.67:0/2936310775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc900c7950 0x7fcc9014e730 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 >> 172.21.15.67:0/2936310775 conn(0x7fcc900ce1c0 msgr2=0x7fcc90154f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 shutdown_connections 2024-04-25T15:53:57.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:57.046+0000 7fcc97e1f640 1 -- 172.21.15.67:0/2936310775 wait complete. 2024-04-25T15:53:57.087 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:57.087+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:57.100 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:57.100 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:57.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:53:57.188 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:57.186+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:57.225 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:57.225 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "" 2024-04-25T15:53:57.225 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:57.289 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:57.287+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:57.335 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:53:57.335 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 initial writers = 620545.94 kB/sec 2024-04-25T15:53:57.335 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 initial writers = 620525.60 kB/sec 2024-04-25T15:53:57.335 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 620545.94 kB/sec 2024-04-25T15:53:57.335 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 620545.94 kB/sec 2024-04-25T15:53:57.336 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 620545.94 kB/sec 2024-04-25T15:53:57.336 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-04-25T15:53:57.350 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:57.350 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:57.367 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:53:57.390 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:57.390+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:57.491 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:57.491+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:57.562 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:57.562 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:57.579 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:53:57.592 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:57.590+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:57.693 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:53:57.691+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:53:57.771 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:57.772 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:57.789 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:53:57.794 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:57.794+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:57.895 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:57.894+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:57.984 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:57.984 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:57.996 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:57.995+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:58.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:53:58.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:53:58.097 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:58.098+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9dd7c1640 1 Processor -- start 2024-04-25T15:53:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9dd7c1640 1 -- start start 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9dd7c1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb9d80caf60 0x7fb9d815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9dd7c1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d815d790 0x7fb9d806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9dd7c1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9d80c77a0 con 0x7fb9d8073bf0 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9dd7c1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d80cab60 con 0x7fb9d80caf60 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9dd7c1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d8159620 con 0x7fb9d815d790 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9d77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb9d8073bf0 0x7fb9d8072bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41394/0 (socket says 172.21.15.67:41394) 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9d77fe640 1 -- 172.21.15.67:0/3901455865 learned_addr learned my addr 172.21.15.67:0/3901455865 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/3901455865 >> v2:172.21.15.176:3300/0 conn(0x7fb9d80caf60 0x7fb9d815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/3901455865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d815d790 0x7fb9d806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.190+0000 7fb9d5ffb640 1 -- 172.21.15.67:0/3901455865 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957744248 0 0) 0x7fb9d80c77a0 con 0x7fb9d8073bf0 2024-04-25T15:53:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d5ffb640 1 -- 172.21.15.67:0/3901455865 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9c0003530 con 0x7fb9d8073bf0 2024-04-25T15:53:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d67fc640 1 -- 172.21.15.67:0/3901455865 >> v2:172.21.15.176:3300/0 conn(0x7fb9d80caf60 msgr2=0x7fb9d815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:53:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/3901455865 >> v2:172.21.15.176:3300/0 conn(0x7fb9d80caf60 0x7fb9d815d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d67fc640 1 -- 172.21.15.67:0/3901455865 >> v1:172.21.15.67:6789/0 conn(0x7fb9d8073bf0 legacy=0x7fb9d8072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d67fc640 1 -- 172.21.15.67:0/3901455865 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d80c7a60 con 0x7fb9d815d790 2024-04-25T15:53:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/3901455865 >> v2:172.21.15.176:3300/0 conn(0x7fb9d80caf60 0x7fb9d815d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:53:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/3901455865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d815d790 0x7fb9d806d750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8007b80 tx=0x7fb9c8020b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ec41dfda00abb2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d5ffb640 1 -- 172.21.15.67:0/3901455865 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8005c80 con 0x7fb9d815d790 2024-04-25T15:53:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d5ffb640 1 -- 172.21.15.67:0/3901455865 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c8005e20 con 0x7fb9d815d790 2024-04-25T15:53:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.194+0000 7fb9d5ffb640 1 -- 172.21.15.67:0/3901455865 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c80a9a40 con 0x7fb9d815d790 2024-04-25T15:53:58.197 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:53:58.197 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/3901455865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d815d790 msgr2=0x7fb9d806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/3901455865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d815d790 0x7fb9d806d750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8007b80 tx=0x7fb9c8020b50 comp rx=0 tx=0).stop 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/3901455865 shutdown_connections 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/3901455865 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d815d790 0x7fb9d806d750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/3901455865 >> v2:172.21.15.176:3300/0 conn(0x7fb9d80caf60 0x7fb9d815d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/3901455865 >> 172.21.15.67:0/3901455865 conn(0x7fb9d8070db0 msgr2=0x7fb9d80ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/3901455865 shutdown_connections 2024-04-25T15:53:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/3901455865 wait complete. 2024-04-25T15:53:58.199 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:58.195+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 Processor -- start 2024-04-25T15:53:58.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- start start 2024-04-25T15:53:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d8073bf0 0x7fb9d81f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9d80caf60 0x7fb9d815e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9d815d790 0x7fb9d815a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d806c640 con 0x7fb9d815d790 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d815a600 con 0x7fb9d80caf60 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9dd7c1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d806c7c0 con 0x7fb9d8073bf0 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9d77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9d815d790 0x7fb9d815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9d77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9d815d790 0x7fb9d815a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40046/0 (socket says 172.21.15.67:40046) 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9d77fe640 1 -- 172.21.15.67:0/2988902994 learned_addr learned my addr 172.21.15.67:0/2988902994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9d80caf60 0x7fb9d815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.198+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d8073bf0 0x7fb9d81f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d77fe640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d8073bf0 msgr2=0x7fb9d81f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d77fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9d8073bf0 0x7fb9d81f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d77fe640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9d80caf60 msgr2=0x7fb9d815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d77fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9d80caf60 0x7fb9d815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d77fe640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d81f2be0 con 0x7fb9d815d790 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d77fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9d815d790 0x7fb9d815a950 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000ca70 tx=0x7fb9d000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0017070 con 0x7fb9d815d790 2024-04-25T15:53:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d000abd0 con 0x7fb9d815d790 2024-04-25T15:53:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d81f3180 con 0x7fb9d815d790 2024-04-25T15:53:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0012400 con 0x7fb9d815d790 2024-04-25T15:53:58.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d815dba0 con 0x7fb9d815d790 2024-04-25T15:53:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb9d81f2ed0 con 0x7fb9d815d790 2024-04-25T15:53:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb9d001a070 con 0x7fb9d815d790 2024-04-25T15:53:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9bf7fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb9c406f400 0x7fb9c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5973+0+0 (secure 0 0 0) 0x7fb9d008efc0 con 0x7fb9d815d790 2024-04-25T15:53:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v95) ==== 54+0+16 (secure 0 0 0) 0x7fb9d0091640 con 0x7fb9d815d790 2024-04-25T15:53:58.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb9c406f400 0x7fb9c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb9d8156a90 0x7fb9d8157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9d815e2f0 con 0x7fb9d8156a90 2024-04-25T15:53:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb9d8156a90 0x7fb9d8157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.202+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb9d8156a90 0x7fb9d8157490 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.206+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb9c406f400 0x7fb9c4071910 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009a50 tx=0x7fb9cc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.206+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9c8005b00 con 0x7fb9d8156a90 2024-04-25T15:53:58.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:53:58.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.218+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d806a480 con 0x7fb9d8156a90 2024-04-25T15:53:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.222+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d806a480 con 0x7fb9d8156a90 2024-04-25T15:53:58.222 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:53:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:53:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.222+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb9d81ec8b0 0x7fb9d81f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.222+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb9d80c5930 con 0x7fb9d81ec8b0 2024-04-25T15:53:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.222+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb9d81ec8b0 0x7fb9d81f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.222+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb9d81ec8b0 0x7fb9d81f5b10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.222+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9d80c5930 con 0x7fb9d81ec8b0 2024-04-25T15:53:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.234+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d8157010 con 0x7fb9d81ec8b0 2024-04-25T15:53:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.238+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d8157010 con 0x7fb9d81ec8b0 2024-04-25T15:53:58.238 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:53:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:53:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.238+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb9d81f6560 0x7fb9d8063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.238+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb9d81f6970 con 0x7fb9d81f6560 2024-04-25T15:53:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.238+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb9d81f6560 0x7fb9d8063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.238+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb9d81f6560 0x7fb9d8063090 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.238+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9d81f6970 con 0x7fb9d81f6560 2024-04-25T15:53:58.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.250+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d80c8a40 con 0x7fb9d81f6560 2024-04-25T15:53:58.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.254+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d80c8a40 con 0x7fb9d81f6560 2024-04-25T15:53:58.254 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:53:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:53:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.254+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb9d8153fe0 0x7fb9d81543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.254+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb9d80cad40 con 0x7fb9d8153fe0 2024-04-25T15:53:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.254+0000 7fb9d77fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb9d8153fe0 0x7fb9d81543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.254+0000 7fb9d77fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb9d8153fe0 0x7fb9d81543f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.254+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9d80cad40 con 0x7fb9d8153fe0 2024-04-25T15:53:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.266+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d80cad40 con 0x7fb9d8153fe0 2024-04-25T15:53:58.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.270+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d80cad40 con 0x7fb9d8153fe0 2024-04-25T15:53:58.270 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:53:58.271 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:53:58.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.270+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb9d802c9e0 0x7fb9d802cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.270+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb9d80cad40 con 0x7fb9d802c9e0 2024-04-25T15:53:58.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.270+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb9d802c9e0 0x7fb9d802cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.270+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb9d802c9e0 0x7fb9d802cdf0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.270+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9d80cad40 con 0x7fb9d802c9e0 2024-04-25T15:53:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.286+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d80cad40 con 0x7fb9d802c9e0 2024-04-25T15:53:58.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.290+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d80cad40 con 0x7fb9d802c9e0 2024-04-25T15:53:58.289 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:53:58.290 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:53:58.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.290+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb9d802d840 0x7fb9d8208ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.290+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb9d80c52f0 con 0x7fb9d802d840 2024-04-25T15:53:58.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.290+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb9d802d840 0x7fb9d8208ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.290+0000 7fb9d67fc640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb9d802d840 0x7fb9d8208ef0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.294+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9c80c2b40 con 0x7fb9d802d840 2024-04-25T15:53:58.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.306+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d8159be0 con 0x7fb9d802d840 2024-04-25T15:53:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.310+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d8159be0 con 0x7fb9d802d840 2024-04-25T15:53:58.311 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:53:58.311 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:53:58.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.310+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb9d802dc50 0x7fb9d820b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.310+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb9d820e720 con 0x7fb9d802dc50 2024-04-25T15:53:58.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.310+0000 7fb9d77fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb9d802dc50 0x7fb9d820b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.312 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:58.311+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:58.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.310+0000 7fb9d77fe640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb9d802dc50 0x7fb9d820b950 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.314+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9d820e720 con 0x7fb9d802dc50 2024-04-25T15:53:58.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.326+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d820be90 con 0x7fb9d802dc50 2024-04-25T15:53:58.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.330+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d820be90 con 0x7fb9d802dc50 2024-04-25T15:53:58.329 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:53:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:53:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.330+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb9d820c5b0 0x7fb9d820e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.330+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb9d8211930 con 0x7fb9d820c5b0 2024-04-25T15:53:58.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.330+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb9d820c5b0 0x7fb9d820e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.330+0000 7fb9d6ffd640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb9d820c5b0 0x7fb9d820e9d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.330+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9d8211930 con 0x7fb9d820c5b0 2024-04-25T15:53:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.342+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fb9d8000960 con 0x7fb9d820c5b0 2024-04-25T15:53:58.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9bf7fe640 1 -- 172.21.15.67:0/2988902994 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb9d8000960 con 0x7fb9d820c5b0 2024-04-25T15:53:58.348 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb9d8156a90 msgr2=0x7fb9d8157490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb9d8156a90 0x7fb9d8157490 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb9d81ec8b0 msgr2=0x7fb9d81f5b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb9d81ec8b0 0x7fb9d81f5b10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb9d81f6560 msgr2=0x7fb9d8063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb9d81f6560 0x7fb9d8063090 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb9d8153fe0 msgr2=0x7fb9d81543f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9d77fe640 1 -- 172.21.15.67:0/2988902994 reap_dead start 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb9d8153fe0 0x7fb9d81543f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb9d802c9e0 msgr2=0x7fb9d802cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb9d802c9e0 0x7fb9d802cdf0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb9d802d840 msgr2=0x7fb9d8208ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb9d802d840 0x7fb9d8208ef0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb9d802dc50 msgr2=0x7fb9d820b950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.346+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb9d802dc50 0x7fb9d820b950 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb9d820c5b0 msgr2=0x7fb9d820e9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb9d820c5b0 0x7fb9d820e9d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9d77fe640 1 -- 172.21.15.67:0/2988902994 reap_dead start 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb9c406f400 msgr2=0x7fb9c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb9c406f400 0x7fb9c4071910 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009a50 tx=0x7fb9cc008040 comp rx=0 tx=0).stop 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9d815d790 msgr2=0x7fb9d815a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9d815d790 0x7fb9d815a950 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000ca70 tx=0x7fb9d000cf30 comp rx=0 tx=0).stop 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 shutdown_connections 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb9c406f400 0x7fb9c4071910 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 --2- 172.21.15.67:0/2988902994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9d815d790 0x7fb9d815a950 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 >> 172.21.15.67:0/2988902994 conn(0x7fb9d8070db0 msgr2=0x7fb9d80c8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 shutdown_connections 2024-04-25T15:53:58.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:58.350+0000 7fb9dd7c1640 1 -- 172.21.15.67:0/2988902994 wait complete. 2024-04-25T15:53:58.399 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:58.395+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:58.416 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:58.416 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:58.417 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:58.417 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:58.417 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:58.434 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:53:58.499 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:58.498+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:58.601 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:58.598+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:58.623 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:58.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.962200103999997, 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.26243838600000002, 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:58.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:53:58.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920373+0000", 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039448372000000002 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921649+0000", 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012757210000000001 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921724+0000", 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5549000000000005e-05 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921802+0000", 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8150000000000002e-05 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:647 3.a 3:524e89bd:::100000001f7.00000083:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.727414146999998, 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083393058000000006, 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.629 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 647 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894150+0000", 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.894152+0000", 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.968e-06 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.902694+0000", 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085420719999999995 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903482+0000", 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078797999999999997 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903499+0000", 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7082e-05 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903593+0000", 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4452999999999998e-05 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903774+0000", 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180803 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.903953+0000", 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179106 2024-04-25T15:53:58.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.965763+0000", 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061810057000000002 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977390+0000", 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01162697 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977474+0000", 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3812000000000003e-05 2024-04-25T15:53:58.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.977543+0000", 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8755000000000002e-05 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:354 3.10 3:0ba0dc78:::10000000002.000000fd:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.022247313999998, 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56691576200000005, 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.634 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599318+0000", 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599326+0000", 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9459999999999998e-06 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161759+0000", 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56243287799999997 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161783+0000", 2024-04-25T15:53:58.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4057e-05 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161899+0000", 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011615799999999999 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162007+0000", 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107823 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162110+0000", 2024-04-25T15:53:58.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010235500000000001 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.163904+0000", 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017941470000000001 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166138+0000", 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022346190000000002 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166186+0000", 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7902999999999999e-05 2024-04-25T15:53:58.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166232+0000", 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6220999999999997e-05 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1799 3.20 3:06ec13d8:::100000001f8.00000303:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:45.542374+0000", 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.079189911, 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.091822033999999997, 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.639 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1799 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.542374+0000", 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.542374+0000", 2024-04-25T15:53:58.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.542376+0000", 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.73e-06 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.547641+0000", 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052651379999999999 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.620544+0000", 2024-04-25T15:53:58.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072903067000000002 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.620564+0000", 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0475999999999998e-05 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.620668+0000", 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103362 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.620862+0000", 2024-04-25T15:53:58.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019379400000000001 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.621018+0000", 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015654500000000001 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.634016+0000", 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012998024 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.634091+0000", 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4617999999999996e-05 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.634137+0000", 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5988000000000001e-05 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.634196+0000", 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9292000000000001e-05 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.644 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1811 3.10 3:0b5e6101:::100000001f8.0000030f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:45.615236+0000", 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.006327697, 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.081304886000000007, 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1811 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.615236+0000", 2024-04-25T15:53:58.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.615236+0000", 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.615239+0000", 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8439999999999998e-06 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.620497+0000", 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052581310000000001 2024-04-25T15:53:58.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.666876+0000", 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046378939000000001 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.666898+0000", 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1328e-05 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.667026+0000", 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128703 2024-04-25T15:53:58.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.667285+0000", 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025901500000000003 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.667532+0000", 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024653400000000002 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.694979+0000", 2024-04-25T15:53:58.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027446735999999999 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.696417+0000", 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014388109999999999 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.696472+0000", 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4398999999999999e-05 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.696541+0000", 2024-04-25T15:53:58.649 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9445999999999999e-05 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2304 3.36 3:6f9fe141:::100000001f8.000004fc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.8310024140000003, 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15859584900000001, 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2304 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790566+0000", 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5369999999999999e-06 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802415+0000", 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011848747999999999 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802510+0000", 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5036000000000003e-05 2024-04-25T15:53:58.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802550+0000", 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9807e-05 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802712+0000", 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162619 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.803017+0000", 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000304433 2024-04-25T15:53:58.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.803308+0000", 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029095100000000001 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.835450+0000", 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032142776999999997 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949058+0000", 2024-04-25T15:53:58.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.113607898 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949100+0000", 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1213e-05 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949157+0000", 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.783e-05 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2307 3.35 3:ad417e89:::100000001f8.000004ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.8104327340000008, 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.138515105, 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2307 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811135+0000", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1629999999999998e-06 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824210+0000", 2024-04-25T15:53:58.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013074564 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824274+0000", 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4131e-05 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824302+0000", 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8167e-05 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824464+0000", 2024-04-25T15:53:58.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000161605 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824749+0000", 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028465599999999999 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.825010+0000", 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026149799999999998 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844599+0000", 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019588795999999999 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949581+0000", 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10498215399999999 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949617+0000", 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6498000000000002e-05 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949646+0000", 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8873e-05 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.660 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2309 3.2c 3:354b7d43:::100000001f8.00000501:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7945757990000004, 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.127916427, 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2309 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.661 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826993+0000", 2024-04-25T15:53:58.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.746e-06 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839270+0000", 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012277221 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839339+0000", 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8709999999999998e-05 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839369+0000", 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0590999999999999e-05 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839557+0000", 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018798099999999999 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839975+0000", 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041773500000000002 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.840214+0000", 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023835300000000001 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.925866+0000", 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.085652939999999997 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954707+0000", 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028840472999999998 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954819+0000", 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011174 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954905+0000", 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5937000000000001e-05 2024-04-25T15:53:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2311 3.35 3:aef469dc:::100000001f8.00000503:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.849872+0000", 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7716924380000005, 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10447076, 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2311 2024-04-25T15:53:58.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.849872+0000", 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.849872+0000", 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.849875+0000", 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.551e-06 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.857887+0000", 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080120380000000008 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.857941+0000", 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4242e-05 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.857963+0000", 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1472999999999999e-05 2024-04-25T15:53:58.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858081+0000", 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118423 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858307+0000", 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022578400000000001 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858487+0000", 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179488 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.952277+0000", 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.093790084999999995 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954127+0000", 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001850637 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954222+0000", 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4806e-05 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954342+0000", 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120233 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2283 3.27 e90/85 3:e56ee505:::100000001f8.000004e7:head v 90'34, mlcod=90'32)", 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.852664+0000", 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7689004399999995, 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10103540900000001, 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.671 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852664+0000", 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852664+0000", 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852665+0000", 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3120000000000001e-06 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858302+0000", 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056368649999999996 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858322+0000", 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9936999999999999e-05 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858339+0000", 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6911000000000001e-05 2024-04-25T15:53:58.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858429+0000", 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0550999999999994e-05 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858463+0000", 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3664000000000002e-05 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953607+0000", 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095144571999999997 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953699+0000", 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1596999999999997e-05 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2313 3.1e e90/18 3:7a2a8d26:::100000001f8.00000505:head v 90'37, mlcod=90'35)", 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.866368+0000", 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7551958140000004, 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.088323592000000006, 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.675 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.866368+0000", 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.866368+0000", 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.866370+0000", 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1670000000000002e-06 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.873930+0000", 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075593049999999997 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.873966+0000", 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5992999999999999e-05 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.873990+0000", 2024-04-25T15:53:58.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4754000000000001e-05 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.874108+0000", 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000117576 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.874146+0000", 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7775999999999999e-05 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954493+0000", 2024-04-25T15:53:58.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080346994000000005 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954692+0000", 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199027 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2297 3.3f e90/18 3:fe369af4:::100000001f8.000004f5:head v 90'43, mlcod=89'41)", 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.879099+0000", 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.7424653970000001, 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074689271000000002, 2024-04-25T15:53:58.679 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.879099+0000", 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.879099+0000", 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.879101+0000", 2024-04-25T15:53:58.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0959999999999999e-06 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.882773+0000", 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036721039999999998 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.882803+0000", 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9879999999999999e-05 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.882822+0000", 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9718e-05 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.882928+0000", 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105858 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.882962+0000", 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3254999999999999e-05 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953731+0000", 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.070769958999999993 2024-04-25T15:53:58.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953788+0000", 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6400999999999998e-05 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2447 3.26 3:6778e6f4:::100000001f8.0000058b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:49.797131+0000", 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.8244333749999999, 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.111032146, 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.683 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2447 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.797131+0000", 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.797131+0000", 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.797146+0000", 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5187999999999999e-05 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.802084+0000", 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049379699999999999 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.888315+0000", 2024-04-25T15:53:58.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.086230875999999998 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.888340+0000", 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4922999999999998e-05 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.888515+0000", 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017587899999999999 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.888872+0000", 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035645700000000002 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.889214+0000", 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000342312 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.898678+0000", 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0094637660000000002 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.907787+0000", 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0091093279999999999 2024-04-25T15:53:58.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.907962+0000", 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017493200000000001 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.908163+0000", 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020051499999999999 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.688 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2463 3.2c 3:37e979db:::100000001f8.0000059b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:49.914260+0000", 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.7073044769999992, 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.091490401999999998, 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2463 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.914260+0000", 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.914260+0000", 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.914262+0000", 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.757e-06 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.920007+0000", 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0057447660000000001 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.968485+0000", 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048477971000000002 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.968508+0000", 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2921e-05 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.968642+0000", 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013407 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.968929+0000", 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028705700000000001 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:49.969161+0000", 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000231852 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.003183+0000", 2024-04-25T15:53:58.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.034021968 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.005560+0000", 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023775720000000001 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.005667+0000", 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106579 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.005750+0000", 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2888999999999999e-05 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2480 3.35 3:afdde0ae:::100000001f8.000005ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.5830942320000005, 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14409117900000001, 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2480 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038475+0000", 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0000000000000004e-06 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.046847+0000", 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083724569999999998 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131712+0000", 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084864593000000002 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131735+0000", 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3448e-05 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131860+0000", 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124258 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.132129+0000", 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026964399999999998 2024-04-25T15:53:58.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.132397+0000", 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026743300000000003 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.167840+0000", 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035443241 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182375+0000", 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014535002 2024-04-25T15:53:58.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182462+0000", 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6887999999999999e-05 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182561+0000", 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9215000000000005e-05 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.699 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2492 3.35 3:af04461b:::100000001f8.000005b8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.127568+0000", 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.4939958069999992, 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095578910000000003, 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2492 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.127568+0000", 2024-04-25T15:53:58.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.127568+0000", 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.127572+0000", 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9430000000000002e-06 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131677+0000", 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041047219999999999 2024-04-25T15:53:58.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.163165+0000", 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031488034999999998 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.163201+0000", 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5899000000000003e-05 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.163295+0000", 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4254999999999998e-05 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.163488+0000", 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019294900000000001 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.163717+0000", 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022890399999999999 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182595+0000", 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018878176999999999 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.223017+0000", 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040421788 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.223080+0000", 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3262000000000005e-05 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.223147+0000", 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6976000000000004e-05 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2499 3.26 3:65b3bea5:::100000001f8.000005bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.4508488059999998, 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.32409411900000001, 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.705 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2499 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170718+0000", 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9699999999999999e-06 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.178145+0000", 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074272360000000003 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457402+0000", 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.27925624300000002 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.707 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457419+0000", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7387000000000001e-05 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457521+0000", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101615 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457720+0000", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019950299999999999 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457907+0000", 2024-04-25T15:53:58.708 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000186973 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.478350+0000", 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020442846000000001 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494652+0000", 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016301573999999999 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494737+0000", 2024-04-25T15:53:58.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5216000000000005e-05 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494809+0000", 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2556000000000001e-05 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2553 3.36 3:6f867a4d:::100000001f8.000005f5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.516323+0000", 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.1052410760000004, 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.077533049000000007, 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.710 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2553 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.516323+0000", 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.516323+0000", 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.516326+0000", 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8590000000000001e-06 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.521806+0000", 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054799330000000002 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.558710+0000", 2024-04-25T15:53:58.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036904571999999997 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.558726+0000", 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5441e-05 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.558809+0000", 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3676000000000003e-05 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.558972+0000", 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016288500000000001 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.559125+0000", 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015259299999999999 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.587658+0000", 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028532747000000001 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.593711+0000", 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060534300000000003 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.593787+0000", 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5523000000000002e-05 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.593856+0000", 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9389999999999998e-05 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3312 3.35 3:af24358c:::100000001f8.000008ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.152647612, 2024-04-25T15:53:58.715 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14714041, 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3312 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.716 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468918+0000", 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9929999999999998e-06 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.473248+0000", 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043301030000000001 2024-04-25T15:53:58.717 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585462+0000", 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112213177 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585475+0000", 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3438000000000001e-05 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585550+0000", 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4975000000000006e-05 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.718 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585692+0000", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142022 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585805+0000", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112736 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.601250+0000", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015444866999999999 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.615882+0000", 2024-04-25T15:53:58.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014632116000000001 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.615956+0000", 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3769000000000005e-05 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.616057+0000", 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101214 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3331 3.2c 3:373546b8:::100000001f8.000008ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:58.720 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:55.586952+0000", 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.0346124290000001, 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074674041999999996, 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3331 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.586952+0000", 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.586952+0000", 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.586954+0000", 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6649999999999999e-06 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.592610+0000", 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056557489999999998 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.632278+0000", 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039668271999999997 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.632297+0000", 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8357000000000001e-05 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.632402+0000", 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105523 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.632548+0000", 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0001459 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.632673+0000", 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124609 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.661127+0000", 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028454232999999999 2024-04-25T15:53:58.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.661434+0000", 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030750299999999999 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.661541+0000", 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010675600000000001 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.661626+0000", 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4474999999999993e-05 2024-04-25T15:53:58.725 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:58.726 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:58.726 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:58.727 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:53:58.728 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:58.726+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:58.828 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:58.827+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:58.882 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:58.882 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:58.882 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:53:58.883 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:58.900 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:53:58.928 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:58.930+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:53:59.029 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:53:59.030+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:53:59.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:53:59.098 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:53:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:53:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:53:59.098 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:59.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:53:59.137 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:53:59.138+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:53:59.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:53:59.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:53:59.243 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:59.243+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:59.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d7f0f640 1 Processor -- start 2024-04-25T15:53:59.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d7f0f640 1 -- start start 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d7f0f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f82d0163ed0 0x7f82d01662f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d7f0f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d0072c50 con 0x7f82d0074650 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d7f0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d0072f30 con 0x7f82d00723a0 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d7f0f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f82d015a860 con 0x7f82d0163ed0 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d5cab640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82d0074650 0x7f82d015db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41410/0 (socket says 172.21.15.67:41410) 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d5cab640 1 -- 172.21.15.67:0/1326185527 learned_addr learned my addr 172.21.15.67:0/1326185527 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d64ac640 1 --2- 172.21.15.67:0/1326185527 >> v2:172.21.15.176:3300/0 conn(0x7f82d0163ed0 0x7f82d01662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d4ca9640 1 -- 172.21.15.67:0/1326185527 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054519690 0 0) 0x7f82d0072c50 con 0x7f82d0074650 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d4ca9640 1 -- 172.21.15.67:0/1326185527 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82bc003530 con 0x7f82d0074650 2024-04-25T15:53:59.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d64ac640 1 -- 172.21.15.67:0/1326185527 >> v1:172.21.15.176:6789/0 conn(0x7f82d00723a0 legacy=0x7f82d00c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:59.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d64ac640 1 -- 172.21.15.67:0/1326185527 >> v1:172.21.15.67:6789/0 conn(0x7f82d0074650 legacy=0x7f82d015db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:59.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.254+0000 7f82d64ac640 1 -- 172.21.15.67:0/1326185527 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d015ab40 con 0x7f82d0163ed0 2024-04-25T15:53:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d64ac640 1 --2- 172.21.15.67:0/1326185527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 0x7f82d01662f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009d90 tx=0x7f82c801f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=702d10777c75a45a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:59.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d4ca9640 1 -- 172.21.15.67:0/1326185527 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c80b4070 con 0x7f82d0163ed0 2024-04-25T15:53:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d4ca9640 1 -- 172.21.15.67:0/1326185527 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c80af600 con 0x7f82d0163ed0 2024-04-25T15:53:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d4ca9640 1 -- 172.21.15.67:0/1326185527 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c80af9d0 con 0x7f82d0163ed0 2024-04-25T15:53:59.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d7f0f640 1 -- 172.21.15.67:0/1326185527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 msgr2=0x7f82d01662f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/1326185527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 0x7f82d01662f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009d90 tx=0x7f82c801f150 comp rx=0 tx=0).stop 2024-04-25T15:53:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d7f0f640 1 -- 172.21.15.67:0/1326185527 shutdown_connections 2024-04-25T15:53:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/1326185527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 0x7f82d01662f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d7f0f640 1 -- 172.21.15.67:0/1326185527 >> 172.21.15.67:0/1326185527 conn(0x7f82d006fe70 msgr2=0x7f82d00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d7f0f640 1 -- 172.21.15.67:0/1326185527 shutdown_connections 2024-04-25T15:53:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.258+0000 7f82d7f0f640 1 -- 172.21.15.67:0/1326185527 wait complete. 2024-04-25T15:53:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 Processor -- start 2024-04-25T15:53:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 -- start start 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82d00723a0 0x7f82d01f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0074650 0x7f82d015d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 0x7f82d0145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f82d01548f0 con 0x7f82d0074650 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f82d015a550 con 0x7f82d0163ed0 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f82d0154a70 con 0x7f82d00723a0 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0074650 0x7f82d015d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0074650 0x7f82d015d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40058/0 (socket says 172.21.15.67:40058) 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 -- 172.21.15.67:0/2330331373 learned_addr learned my addr 172.21.15.67:0/2330331373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d64ac640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 0x7f82d0145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d5cab640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82d00723a0 0x7f82d01f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 -- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82d00723a0 msgr2=0x7f82d01f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:59.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82d00723a0 0x7f82d01f3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 -- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 msgr2=0x7f82d0145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 0x7f82d0145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 -- 172.21.15.67:0/2330331373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d01f3b50 con 0x7f82d0074650 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d54aa640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0074650 0x7f82d015d1f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f82c000c340 tx=0x7f82c000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82c67fc640 1 -- 172.21.15.67:0/2330331373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c00052d0 con 0x7f82d0074650 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82c67fc640 1 -- 172.21.15.67:0/2330331373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c0005470 con 0x7f82d0074650 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d015d730 con 0x7f82d0074650 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82c67fc640 1 -- 172.21.15.67:0/2330331373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0007260 con 0x7f82d0074650 2024-04-25T15:53:59.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d01f2850 con 0x7f82d0074650 2024-04-25T15:53:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.262+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d01f22e0 con 0x7f82d0074650 2024-04-25T15:53:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.266+0000 7f82c67fc640 1 -- 172.21.15.67:0/2330331373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f82c0007400 con 0x7f82d0074650 2024-04-25T15:53:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.266+0000 7f82c67fc640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82b406f430 0x7f82b4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.266+0000 7f82c67fc640 1 -- 172.21.15.67:0/2330331373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5973+0+0 (secure 0 0 0) 0x7f82c008e6c0 con 0x7f82d0074650 2024-04-25T15:53:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.266+0000 7f82d5cab640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82b406f430 0x7f82b4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.266+0000 7f82d5cab640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82b406f430 0x7f82b4071940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f82cc0011f0 tx=0x7f82cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:59.270 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 220.16998279500001, 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:53:59.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:53:59.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:53:59.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 66.050742314000004, 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:53:59.276 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:53:59.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:53:59.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 65.956108262000001, 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.685430310000001, 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.287 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:53:59.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:53:59.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:666 3.15 3:a92ac213:::100000001f7.00000096:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.403098944, 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10217435, 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.293 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 666 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865348+0000", 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.865350+0000", 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9360000000000002e-06 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871962+0000", 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006612349 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872932+0000", 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00097010200000000003 2024-04-25T15:53:59.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.872950+0000", 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7193999999999998e-05 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873133+0000", 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183265 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873494+0000", 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00036074300000000001 2024-04-25T15:53:59.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.873747+0000", 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025347100000000001 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920781+0000", 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.047033951999999997 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967388+0000", 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.046606748000000003 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967458+0000", 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0339999999999994e-05 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967522+0000", 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4250000000000003e-05 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:667 3.2d 3:b6615a40:::100000001f7.00000097:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.396469031999999, 2024-04-25T15:53:59.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095661866999999998, 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 667 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871978+0000", 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.871981+0000", 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0759999999999999e-06 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877165+0000", 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051845709999999998 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878247+0000", 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001081448 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878262+0000", 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5196e-05 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885690+0000", 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074279159999999997 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886266+0000", 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057628900000000001 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886495+0000", 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022871600000000001 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.910525+0000", 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024029452 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967541+0000", 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.057016940000000002 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967590+0000", 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8585999999999999e-05 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967640+0000", 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9676999999999999e-05 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:670 3.8 e81/68 3:1323134e:::100000001f7.0000009a:head v 81'17, mlcod=80'15)", 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.391326790999997, 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090558897999999999, 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877120+0000", 2024-04-25T15:53:59.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877122+0000", 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.883982+0000", 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068602339999999998 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884003+0000", 2024-04-25T15:53:59.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0418999999999999e-05 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884024+0000", 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1225000000000001e-05 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884154+0000", 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012977099999999999 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884218+0000", 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3830000000000004e-05 2024-04-25T15:53:59.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967623+0000", 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083405807999999998 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967679+0000", 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5559000000000001e-05 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:669 3.f 3:f0675d7f:::100000001f7.00000099:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:59.307 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.391271185000001, 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090729281999999994, 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 669 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877176+0000", 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.877178+0000", 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.88e-06 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884938+0000", 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0077602569999999996 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884975+0000", 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7157999999999999e-05 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.884991+0000", 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5637999999999998e-05 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885075+0000", 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4696000000000004e-05 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885226+0000", 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015028799999999999 2024-04-25T15:53:59.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.885331+0000", 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010580400000000001 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967662+0000", 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.082330667999999996 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967792+0000", 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130259 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967861+0000", 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9002000000000001e-05 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967905+0000", 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3631999999999998e-05 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2266 3.11 3:89528692:::100000001f8.000004d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.697263594000001, 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.38717112199999998, 2024-04-25T15:53:59.312 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2266 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571186+0000", 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4569999999999999e-06 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784888+0000", 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.213702742 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785181+0000", 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029301099999999998 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785201+0000", 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9683000000000001e-05 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785333+0000", 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013224600000000001 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785779+0000", 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044548299999999999 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.786106+0000", 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032678999999999998 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870593+0000", 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084487524999999994 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958234+0000", 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087640933000000004 2024-04-25T15:53:59.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958283+0000", 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8473999999999997e-05 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958354+0000", 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1778000000000003e-05 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2269 3.d 3:b24099bb:::100000001f8.000004d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.317 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.784900+0000", 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.483547337999999, 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.120695039, 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2269 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784900+0000", 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784900+0000", 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784902+0000", 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.17e-06 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795914+0000", 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011012399000000001 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.796532+0000", 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061838699999999997 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.796568+0000", 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5135999999999997e-05 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.796733+0000", 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016577900000000001 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.797066+0000", 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033217099999999997 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.797359+0000", 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029385799999999998 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.848518+0000", 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051158207999999997 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905450+0000", 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056931926000000001 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905526+0000", 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6137999999999995e-05 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905595+0000", 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8867000000000003e-05 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2272 3.13 3:ca3e88be:::100000001f8.000004dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.472514946, 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13173032000000001, 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2272 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795936+0000", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.754e-06 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805167+0000", 2024-04-25T15:53:59.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092308890000000008 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805557+0000", 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039059400000000002 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805588+0000", 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0865999999999999e-05 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805759+0000", 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017085599999999999 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806172+0000", 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000413416 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806552+0000", 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000379417 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905757+0000", 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099205023000000003 2024-04-25T15:53:59.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927568+0000", 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021810800000000002 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927616+0000", 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927662+0000", 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6535000000000001e-05 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.327 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2273 3.39 3:9fcbcc01:::100000001f8.000004dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.463262831, 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15983650499999999, 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2273 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805188+0000", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816847+0000", 2024-04-25T15:53:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011659119000000001 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817576+0000", 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00072975000000000004 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817601+0000", 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4450000000000001e-05 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817741+0000", 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014058500000000001 2024-04-25T15:53:59.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818074+0000", 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033291000000000001 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818385+0000", 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031034999999999998 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870618+0000", 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052233743999999999 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964814+0000", 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094195216999999998 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964945+0000", 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013086799999999999 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965021+0000", 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6032000000000007e-05 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2275 3.12 3:492b2588:::100000001f8.000004df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.451587452, 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14794684399999999, 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2275 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816863+0000", 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4149999999999999e-06 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829311+0000", 2024-04-25T15:53:59.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012448222 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829569+0000", 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025766800000000001 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829598+0000", 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.968e-05 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829786+0000", 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187746 2024-04-25T15:53:59.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830129+0000", 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034284099999999998 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830479+0000", 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034961500000000001 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867146+0000", 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036667733000000001 2024-04-25T15:53:59.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964688+0000", 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097541577000000004 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964748+0000", 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9626000000000001e-05 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964806+0000", 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8721000000000002e-05 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.337 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2278 3.2b 3:d79b9024:::100000001f8.000004e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.439122193999999, 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135647779, 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2278 2024-04-25T15:53:59.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829328+0000", 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7309999999999999e-06 2024-04-25T15:53:59.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843821+0000", 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014493012 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844787+0000", 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096513199999999995 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844876+0000", 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9147999999999994e-05 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844972+0000", 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6566999999999999e-05 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845195+0000", 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222699 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845422+0000", 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022700699999999999 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870319+0000", 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024897319000000001 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964892+0000", 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094572233000000006 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964938+0000", 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6646e-05 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964972+0000", 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4285000000000001e-05 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2305 3.2f e90/51 3:f6648653:::100000001f8.000004fd:head v 90'33, mlcod=90'30)", 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.431188404, 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12780160600000001, 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.343 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837261+0000", 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1679999999999998e-06 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839587+0000", 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00232607 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839620+0000", 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3503000000000001e-05 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839642+0000", 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1302e-05 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839754+0000", 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112454 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839798+0000", 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4240999999999999e-05 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965003+0000", 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12520515600000001 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965060+0000", 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6712000000000002e-05 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2283 3.27 3:e56ee505:::100000001f8.000004e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.843837+0000", 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.424609665, 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.121476668, 2024-04-25T15:53:59.347 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2283 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843837+0000", 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843837+0000", 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843842+0000", 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0329999999999999e-06 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851910+0000", 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080682299999999992 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852141+0000", 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023074400000000001 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852162+0000", 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0652000000000001e-05 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852248+0000", 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6112000000000005e-05 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852390+0000", 2024-04-25T15:53:59.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000141731 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852509+0000", 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011903100000000001 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953894+0000", 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.101384997 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965175+0000", 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011280874999999999 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965255+0000", 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0238e-05 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965314+0000", 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9024999999999998e-05 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2292 3.12 3:4a3a3c79:::100000001f8.000004f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.851920+0000", 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.416526878999999, 2024-04-25T15:53:59.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11395511899999999, 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2292 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851920+0000", 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851920+0000", 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851923+0000", 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3299999999999999e-06 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860067+0000", 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081432929999999994 2024-04-25T15:53:59.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860474+0000", 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000407694 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860489+0000", 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4568e-05 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867223+0000", 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067344479999999996 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867532+0000", 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000308203 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867816+0000", 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028490399999999999 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.923967+0000", 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056150243000000002 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965776+0000", 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041809499 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965829+0000", 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3062000000000001e-05 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965875+0000", 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5874999999999997e-05 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.357 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2294 3.11 3:8bba9e6f:::100000001f8.000004f2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.860077+0000", 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.408370337000001, 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10564612499999999, 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2294 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.358 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860077+0000", 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860077+0000", 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860079+0000", 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2069999999999998e-06 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870074+0000", 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0099951290000000002 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870612+0000", 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00053807900000000001 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870629+0000", 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7000999999999999e-05 2024-04-25T15:53:59.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870708+0000", 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9095000000000001e-05 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870900+0000", 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019231700000000001 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.871064+0000", 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016398899999999999 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958152+0000", 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087087585999999995 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965569+0000", 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074167779999999997 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965655+0000", 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6252e-05 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965723+0000", 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7692000000000002e-05 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2297 3.3f 3:fe369af4:::100000001f8.000004f5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.870085+0000", 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.398361817, 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095408468999999996, 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2297 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870085+0000", 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870085+0000", 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870088+0000", 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0460000000000001e-06 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.878097+0000", 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080088260000000001 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.878425+0000", 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000328568 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.878446+0000", 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0330999999999999e-05 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.878548+0000", 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010238699999999999 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.878740+0000", 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000191561 2024-04-25T15:53:59.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.878917+0000", 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017712399999999999 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953985+0000", 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.075068088000000005 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965374+0000", 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011388973 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965442+0000", 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7729000000000002e-05 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965494+0000", 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1836000000000003e-05 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.368 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3237 3.2b 3:d61b6bf4:::100000001f8.000008a1:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:54.989799+0000", 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.27864763, 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094009341999999996, 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3237 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:54.989799+0000", 2024-04-25T15:53:59.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:54.989799+0000", 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:54.989802+0000", 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.615e-06 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:54.994304+0000", 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045025610000000004 2024-04-25T15:53:59.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.068484+0000", 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074179139000000005 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.068505+0000", 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1809000000000001e-05 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.068614+0000", 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108701 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.068808+0000", 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019392899999999999 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.068988+0000", 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018035600000000001 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.075843+0000", 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068549429999999996 2024-04-25T15:53:59.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.083717+0000", 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0078737549999999996 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.083765+0000", 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7929999999999997e-05 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.083809+0000", 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3603999999999998e-05 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.373 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:53:59.374 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:53:59.374 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:53:59.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.270+0000 7f82c67fc640 1 -- 172.21.15.67:0/2330331373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f82c005aef0 con 0x7f82d0074650 2024-04-25T15:53:59.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:53:59.376 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:59.374+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:59.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:53:59.442+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f82d006a480 con 0x7f82d0074650 2024-04-25T15:53:59.476 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:59.474+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:59.527 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:59.527 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:59.527 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:53:59.527 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:59.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:53:59.577 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:53:59.575+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:53:59.677 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:53:59.675+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:53:59.737 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:59.738 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:53:59.738 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:53:59.738 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:53:59.738 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:59.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:53:59.801 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:53:59.802+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:53:59.879 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:53:59.879+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:668 3.1 e81/76 3:83ba1de1:::100000001f7.00000098:head v 81'14, mlcod=81'12)", 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.034954222000003, 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.045409913000000003, 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.909 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.910 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872468+0000", 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3659999999999999e-06 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879062+0000", 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065941039999999999 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879116+0000", 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4372000000000001e-05 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.911 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879152+0000", 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6139000000000001e-05 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879318+0000", 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016595499999999999 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879365+0000", 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6897999999999997e-05 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.912 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917803+0000", 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038437773000000001 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917875+0000", 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2305999999999995e-05 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:637 3.20 e81/18 3:07263727:::100000001f7.00000079:head v 81'17, mlcod=81'15)", 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:53:59.913 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.025465730999997, 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040244958999999997, 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.914 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881957+0000", 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3890000000000001e-06 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888368+0000", 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006410682 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888395+0000", 2024-04-25T15:53:59.915 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7477000000000001e-05 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888414+0000", 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9046e-05 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888512+0000", 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7876999999999998e-05 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.916 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888546+0000", 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3204000000000003e-05 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922131+0000", 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033585168999999998 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922199+0000", 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8114999999999994e-05 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.917 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:640 3.10 e81/18 3:092a6a2c:::100000001f7.0000007c:head v 81'13, mlcod=81'11)", 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.895692+0000", 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.011727178999998, 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027335905000000001, 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.895692+0000", 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.895692+0000", 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.895695+0000", 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1129999999999999e-06 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.900960+0000", 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0052653109999999999 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901007+0000", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6885000000000003e-05 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901033+0000", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6160999999999999e-05 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901166+0000", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013346000000000001 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.901216+0000", 2024-04-25T15:53:59.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9264999999999997e-05 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922970+0000", 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021753943000000001 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.923028+0000", 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8767000000000002e-05 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1485 3.20 e88/18 3:04a26667:::100000001f8.000001c9:head v 88'41, mlcod=88'39)", 2024-04-25T15:53:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.592564+0000", 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.314855403999999, 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025132409000000001, 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.592564+0000", 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.592564+0000", 2024-04-25T15:53:59.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.592566+0000", 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7910000000000001e-06 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599273+0000", 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067066410000000002 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599297+0000", 2024-04-25T15:53:59.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4178000000000001e-05 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599312+0000", 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4763000000000001e-05 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599396+0000", 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4790999999999995e-05 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.599431+0000", 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4941000000000003e-05 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617638+0000", 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018206938999999998 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.617697+0000", 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8365000000000001e-05 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1590 3.3d e88/40 3:bd018e47:::100000001f8.00000232:head v 88'32, mlcod=88'30)", 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.248469+0000", 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.658950525, 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025528071999999999, 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.248469+0000", 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.248469+0000", 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.248471+0000", 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3719999999999999e-06 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251313+0000", 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0028412429999999998 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0558000000000001e-05 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251360+0000", 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7172000000000001e-05 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251454+0000", 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3931999999999994e-05 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251489+0000", 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4224999999999998e-05 2024-04-25T15:53:59.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.273899+0000", 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022410929999999999 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.273997+0000", 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7639999999999994e-05 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1598 3.1f e89/40 3:fa5184c2:::100000001f8.0000023a:head v 89'23, mlcod=88'21)", 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.311187+0000", 2024-04-25T15:53:59.929 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.596232391999999, 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024934254999999999, 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.311187+0000", 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.311187+0000", 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.311189+0000", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.432e-06 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316338+0000", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0051491519999999997 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316360+0000", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1914999999999999e-05 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316371+0000", 2024-04-25T15:53:59.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.1094000000000001e-05 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316433+0000", 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2242999999999993e-05 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.316453+0000", 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0213000000000001e-05 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.336057+0000", 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019603438000000001 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.336121+0000", 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4768000000000001e-05 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1709 3.b e89/21 3:d0c4adc9:::100000001f8.000002a9:head v 89'35, mlcod=89'33)", 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.908489744000001, 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041562524000000003, 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998932+0000", 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7570000000000001e-06 2024-04-25T15:53:59.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000395+0000", 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014629389999999999 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000414+0000", 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9908999999999999e-05 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000432+0000", 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249999999999999e-05 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000518+0000", 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6501000000000004e-05 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000548+0000", 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9777e-05 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040424+0000", 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039875586999999997 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040492+0000", 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8804000000000001e-05 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1710 3.b e89/21 3:d079ce42:::100000001f8.000002aa:head v 89'36, mlcod=89'33)", 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.902061027, 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040049727, 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005360+0000", 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9290000000000001e-06 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.938 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006358+0000", 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099724699999999993 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006377+0000", 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9656e-05 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006392+0000", 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4207e-05 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006477+0000", 2024-04-25T15:53:59.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4989999999999998e-05 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006507+0000", 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0786e-05 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045329+0000", 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038821829000000002 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045408+0000", 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9083000000000002e-05 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.940 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1711 3.12 e89/40 3:4b7bcb17:::100000001f8.000002ab:head v 89'37, mlcod=89'35)", 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.013771+0000", 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.893648369999999, 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031890573999999998, 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.013771+0000", 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.013771+0000", 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.013772+0000", 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8599999999999996e-07 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017771+0000", 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039990889999999999 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017791+0000", 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9976999999999999e-05 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017805+0000", 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3413e-05 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017869+0000", 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4417000000000003e-05 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.017894+0000", 2024-04-25T15:53:59.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.512e-05 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045623+0000", 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027728778999999999 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045662+0000", 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8792999999999999e-05 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1712 3.30 e89/18 3:0f3afaad:::100000001f8.000002ac:head v 89'47, mlcod=89'45)", 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.020649+0000", 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.88677073, 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025202367999999999, 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.020649+0000", 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.020649+0000", 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.020650+0000", 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4559999999999999e-06 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025465+0000", 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.004815059 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025486+0000", 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0706000000000001e-05 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025500+0000", 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4141000000000001e-05 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025565+0000", 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5024000000000006e-05 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.025586+0000", 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0930000000000001e-05 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045786+0000", 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020199994999999998 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045851+0000", 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5056999999999997e-05 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.948 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1802 3.23 e89/40 3:c6704eb5:::100000001f8.00000306:head v 89'33, mlcod=88'31)", 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.558838+0000", 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.348581267, 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02500527, 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.558838+0000", 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.558838+0000", 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.558839+0000", 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4700000000000001e-07 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560492+0000", 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001652854 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560516+0000", 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4742000000000001e-05 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560534+0000", 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7898e-05 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560626+0000", 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1414000000000003e-05 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.560658+0000", 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1853999999999997e-05 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.583765+0000", 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.023107544000000001 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.583844+0000", 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8417000000000005e-05 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1811 3.10 e89/18 3:0b5e6101:::100000001f8.0000030f:head v 89'58, mlcod=89'56)", 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.668830+0000", 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.238589859999999, 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026867552999999999, 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.668830+0000", 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.668830+0000", 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.668832+0000", 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9589999999999998e-06 2024-04-25T15:53:59.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677622+0000", 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087903660000000008 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677650+0000", 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8167e-05 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677670+0000", 2024-04-25T15:53:59.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9643000000000001e-05 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677754+0000", 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4019999999999999e-05 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.677778+0000", 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4042000000000001e-05 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695608+0000", 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017830141000000001 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.695697+0000", 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9215000000000006e-05 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2273 3.39 e90/18 3:9fcbcc01:::100000001f8.000004dd:head v 90'54, mlcod=90'52)", 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.081911782000001, 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035128116000000001, 2024-04-25T15:53:59.957 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825510+0000", 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837142+0000", 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011632136 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837175+0000", 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3166000000000001e-05 2024-04-25T15:53:59.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837197+0000", 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1495e-05 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837293+0000", 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6353000000000004e-05 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837329+0000", 2024-04-25T15:53:59.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5901000000000001e-05 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860257+0000", 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022927804999999999 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860636+0000", 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000379327 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2392 3.32 e90/40 3:4cdbb504:::100000001f8.00000554:head v 90'45, mlcod=90'43)", 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.498097021, 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044535299, 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409325+0000", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2220000000000001e-06 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410846+0000", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001521454 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410866+0000", 2024-04-25T15:53:59.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0129999999999999e-05 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410882+0000", 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5452e-05 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410962+0000", 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9833999999999995e-05 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410991+0000", 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.968e-05 2024-04-25T15:53:59.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453793+0000", 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042801621999999998 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453858+0000", 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4905000000000003e-05 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.965 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2394 3.6 e90/76 3:63a5c66e:::100000001f8.00000556:head v 90'46, mlcod=90'44)", 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.483401898, 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038940880999999997, 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424019+0000", 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6160000000000001e-06 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428140+0000", 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041208529999999998 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428169+0000", 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8824999999999999e-05 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428190+0000", 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0894999999999999e-05 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428283+0000", 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3277999999999997e-05 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.968 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428318+0000", 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5191000000000002e-05 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462890+0000", 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034571283000000001 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462959+0000", 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000001e-05 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2396 3.3d e90/40 3:bc4ba9bc:::100000001f8.00000558:head v 90'44, mlcod=90'42)", 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.473596677, 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033872159999999998, 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433825+0000", 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.951e-06 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435239+0000", 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014141729999999999 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435260+0000", 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0775e-05 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435278+0000", 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8317000000000001e-05 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435363+0000", 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4441e-05 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435395+0000", 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1939e-05 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467549+0000", 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032154061999999997 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467695+0000", 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014650199999999999 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.973 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2556 3.17 e90/40 3:e9827f3e:::100000001f8.000005f8:head v 90'46, mlcod=90'44)", 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:50.546366+0000", 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.361053665, 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025043425000000001, 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.546366+0000", 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.546366+0000", 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.546369+0000", 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2109999999999998e-06 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552050+0000", 2024-04-25T15:53:59.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056808589999999999 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552109+0000", 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8536000000000003e-05 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552129+0000", 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0752000000000001e-05 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552217+0000", 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.81e-05 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.552243+0000", 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5568999999999999e-05 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.571338+0000", 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019095035999999999 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:50.571409+0000", 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1361999999999999e-05 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3111 3.23 e94/40 3:c54285fe:::100000001f8.00000823:head v 94'50, mlcod=93'48)", 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.211252+0000", 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6961677609999999, 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026616490999999999, 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.211252+0000", 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.211252+0000", 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.211254+0000", 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8589999999999999e-06 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.212881+0000", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00162707 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.212912+0000", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1219e-05 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.212934+0000", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1641000000000002e-05 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.213029+0000", 2024-04-25T15:53:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5526000000000007e-05 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.213051+0000", 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1993000000000001e-05 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.237802+0000", 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024750846 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.237868+0000", 2024-04-25T15:53:59.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6336999999999999e-05 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3345 3.17 e94/40 3:e83df825:::100000001f8.0000090d:head v 94'58, mlcod=93'56)", 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:55.712016+0000", 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.1954030959999997, 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025603068, 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.712016+0000", 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.712016+0000", 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.712019+0000", 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.503e-06 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.719494+0000", 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0074755100000000003 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.719514+0000", 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9976999999999999e-05 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.719532+0000", 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.734e-05 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.719617+0000", 2024-04-25T15:53:59.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5419000000000003e-05 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.719649+0000", 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1596e-05 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.737527+0000", 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017877965999999999 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.737620+0000", 2024-04-25T15:53:59.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2757000000000007e-05 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3347 3.2b e94/40 3:d46a96dc:::100000001f8.0000090f:head v 94'50, mlcod=94'48)", 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:55.712840+0000", 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.1945798270000001, 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025109026, 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.712840+0000", 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.712840+0000", 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.712841+0000", 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.55e-06 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:53:59.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.721603+0000", 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087614070000000006 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.721622+0000", 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9485000000000001e-05 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.721639+0000", 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6554000000000001e-05 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.721723+0000", 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4460000000000001e-05 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.721745+0000", 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1376999999999999e-05 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.737908+0000", 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016163712 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:53:59.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:55.737949+0000", 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0481000000000001e-05 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:53:59.990 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:53:59.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:53:59.992 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:53:59.990+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:00.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.074+0000 7f82c67fc640 1 -- 172.21.15.67:0/2330331373 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v96) ==== 91+0+0 (secure 0 0 0) 0x7f82c005f1f0 con 0x7f82d0074650 2024-04-25T15:54:00.074 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82b406f430 msgr2=0x7f82b4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82b406f430 0x7f82b4071940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f82cc0011f0 tx=0x7f82cc009040 comp rx=0 tx=0).stop 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0074650 msgr2=0x7f82d015d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0074650 0x7f82d015d1f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f82c000c340 tx=0x7f82c000c800 comp rx=0 tx=0).stop 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 shutdown_connections 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82b406f430 0x7f82b4071940 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82d0163ed0 0x7f82d0145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d0074650 0x7f82d015d1f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 --2- 172.21.15.67:0/2330331373 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82d00723a0 0x7f82d01f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 >> 172.21.15.67:0/2330331373 conn(0x7f82d006fe70 msgr2=0x7f82d0159560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 shutdown_connections 2024-04-25T15:54:00.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.078+0000 7f82d7f0f640 1 -- 172.21.15.67:0/2330331373 wait complete. 2024-04-25T15:54:00.092 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:00.091+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:00.193 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:00.194+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:00.204 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:00.204 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:00.204 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:00.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:54:00.309 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:00.310+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:00.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:54:00.395 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:00.394+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:00.415 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:00.415 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:00.415 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:00.415 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:00.415 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:00.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:54:00.495 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:00.494+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.506+0000 7f07fe9ca640 1 Processor -- start 2024-04-25T15:54:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.506+0000 7f07fe9ca640 1 -- start start 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fe9ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 0x7f07f806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fe9ca640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f07f815fe00 0x7f07f8160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fe9ca640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f806d780 con 0x7f07f80b5660 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fe9ca640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f07f815d510 con 0x7f07f815fe00 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fe9ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07f8160be0 con 0x7f07f806ca60 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f77fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07f80b5660 0x7f07f80b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41416/0 (socket says 172.21.15.67:41416) 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f77fe640 1 -- 172.21.15.67:0/2378629553 learned_addr learned my addr 172.21.15.67:0/2378629553 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 0x7f07f806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2378629553 >> v2:172.21.15.176:3300/0 conn(0x7f07f815fe00 0x7f07f8160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089009287 0 0) 0x7f07f806d780 con 0x7f07f80b5660 2024-04-25T15:54:00.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07e0003530 con 0x7f07f80b5660 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 945613293 0 0) 0x7f07e0003530 con 0x7f07f80b5660 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 msgr2=0x7f07f806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 0x7f07f806ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 >> v2:172.21.15.176:3300/0 conn(0x7f07f815fe00 msgr2=0x7f07f8160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 --2- 172.21.15.67:0/2378629553 >> v2:172.21.15.176:3300/0 conn(0x7f07f815fe00 0x7f07f8160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f815cbe0 con 0x7f07f80b5660 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f07ec0016a0 con 0x7f07f80b5660 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 >> v1:172.21.15.67:6789/0 conn(0x7f07f80b5660 legacy=0x7f07f80b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07e00039c0 0x7f07e0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 0x7f07e0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e0004d50 0x7f07e0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07f815d510 con 0x7f07e0004d50 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f07f8160be0 con 0x7f07e00039c0 2024-04-25T15:54:00.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07e000ba50 con 0x7f07f806ca60 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e0004d50 0x7f07e0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07e00039c0 0x7f07e0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 0x7f07e0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fcf67640 1 -- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 msgr2=0x7f07e0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f806ca60 0x7f07e0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fcf67640 1 -- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07e00039c0 msgr2=0x7f07e0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07e00039c0 0x7f07e0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.510+0000 7f07fcf67640 1 -- 172.21.15.67:0/2378629553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e0003530 con 0x7f07e0004d50 2024-04-25T15:54:00.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.514+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e0004d50 0x7f07e0005160 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009f90 tx=0x7f07f001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70604713c06ca5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.514+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f00b2070 con 0x7f07e0004d50 2024-04-25T15:54:00.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.514+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f00a5c90 con 0x7f07e0004d50 2024-04-25T15:54:00.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.514+0000 7f07f6ffd640 1 -- 172.21.15.67:0/2378629553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0009c10 con 0x7f07e0004d50 2024-04-25T15:54:00.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2378629553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e0004d50 msgr2=0x7f07e0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2378629553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07e0004d50 0x7f07e0005160 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009f90 tx=0x7f07f001cc80 comp rx=0 tx=0).stop 2024-04-25T15:54:00.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fcf67640 1 -- 172.21.15.67:0/2378629553 reap_dead start 2024-04-25T15:54:00.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2378629553 shutdown_connections 2024-04-25T15:54:00.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2378629553 >> 172.21.15.67:0/2378629553 conn(0x7f07f806e420 msgr2=0x7f07f806e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:00.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2378629553 shutdown_connections 2024-04-25T15:54:00.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2378629553 wait complete. 2024-04-25T15:54:00.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.518+0000 7f07fe9ca640 1 Processor -- start 2024-04-25T15:54:00.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 -- start start 2024-04-25T15:54:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f81544a0 0x7f07f8153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07f81e90e0 0x7f07f81f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f81e94f0 0x7f07f8152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07f815c890 con 0x7f07f81e94f0 2024-04-25T15:54:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f07f815d1a0 con 0x7f07f81e90e0 2024-04-25T15:54:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07f80b6e90 con 0x7f07f81544a0 2024-04-25T15:54:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f81e94f0 0x7f07f8152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f81e94f0 0x7f07f8152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40070/0 (socket says 172.21.15.67:40070) 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 -- 172.21.15.67:0/2622377427 learned_addr learned my addr 172.21.15.67:0/2622377427 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f81544a0 0x7f07f8153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07f81e90e0 0x7f07f81f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f81544a0 msgr2=0x7f07f8153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07f81544a0 0x7f07f8153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07f81e90e0 msgr2=0x7f07f81f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07f81e90e0 0x7f07f81f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f815f250 con 0x7f07f81e94f0 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07f81e90e0 0x7f07f81f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f81e94f0 0x7f07f8152590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009f90 tx=0x7f07f0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f00bd070 con 0x7f07f81e94f0 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f00a5a90 con 0x7f07f81e94f0 2024-04-25T15:54:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f00b2040 con 0x7f07f81e94f0 2024-04-25T15:54:00.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f815f530 con 0x7f07f81e94f0 2024-04-25T15:54:00.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.522+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f815faf0 con 0x7f07f81e94f0 2024-04-25T15:54:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f07f806a480 con 0x7f07f81e94f0 2024-04-25T15:54:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f07f00c0070 con 0x7f07f81e94f0 2024-04-25T15:54:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f4ff9640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07e406f400 0x7f07e4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5973+0+0 (secure 0 0 0) 0x7f07f0129370 con 0x7f07f81e94f0 2024-04-25T15:54:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07e406f400 0x7f07e4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v96) ==== 54+0+16 (secure 0 0 0) 0x7f07f0016990 con 0x7f07f81e94f0 2024-04-25T15:54:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f07f8152ad0 0x7f07f8075df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07f80cd380 con 0x7f07f8152ad0 2024-04-25T15:54:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f07f8152ad0 0x7f07f8075df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07e406f400 0x7f07e4071910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f07f81ed5e0 tx=0x7f07e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.526+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f07f8152ad0 0x7f07f8075df0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.530+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f80cd380 con 0x7f07f8152ad0 2024-04-25T15:54:00.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.542+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f8076e20 con 0x7f07f8152ad0 2024-04-25T15:54:00.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.546+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f8076e20 con 0x7f07f8152ad0 2024-04-25T15:54:00.546 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:00.546 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.546+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f07f81f6720 0x7f07f81f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.546+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f07f81f7070 con 0x7f07f81f6720 2024-04-25T15:54:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.546+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f07f81f6720 0x7f07f81f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.546+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f07f81f6720 0x7f07f81f6b30 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.546+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f81f7070 con 0x7f07f81f6720 2024-04-25T15:54:00.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.562+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f806c740 con 0x7f07f81f6720 2024-04-25T15:54:00.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.562+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f806c740 con 0x7f07f81f6720 2024-04-25T15:54:00.563 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.562+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f07f80635a0 0x7f07f80639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.562+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f07f8063ef0 con 0x7f07f80635a0 2024-04-25T15:54:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.562+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f07f80635a0 0x7f07f80639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.562+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f07f80635a0 0x7f07f80639b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.566+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f8063ef0 con 0x7f07f80635a0 2024-04-25T15:54:00.573 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:00.573 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:00.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.578+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f81f3a50 con 0x7f07f80635a0 2024-04-25T15:54:00.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.582+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f81f3a50 con 0x7f07f80635a0 2024-04-25T15:54:00.581 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.582+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f07f815afd0 0x7f07f802c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.582+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f07f815b3e0 con 0x7f07f815afd0 2024-04-25T15:54:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.582+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f07f815afd0 0x7f07f802c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.582+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f07f815afd0 0x7f07f802c9e0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.582+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f815b3e0 con 0x7f07f815afd0 2024-04-25T15:54:00.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:54:00.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.594+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f815b3e0 con 0x7f07f815afd0 2024-04-25T15:54:00.596 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:00.594+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:00.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.598+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f815b3e0 con 0x7f07f815afd0 2024-04-25T15:54:00.598 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:00.599 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:00.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.598+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f07f802d430 0x7f07f802d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.598+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f07f815b3e0 con 0x7f07f802d430 2024-04-25T15:54:00.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.598+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f07f802d430 0x7f07f802d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.598+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f07f802d430 0x7f07f802d840 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.602+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f815b3e0 con 0x7f07f802d430 2024-04-25T15:54:00.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.614+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f815b3e0 con 0x7f07f802d430 2024-04-25T15:54:00.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.618+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f815b3e0 con 0x7f07f802d430 2024-04-25T15:54:00.620 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:00.620 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:00.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.618+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f07f8207aa0 0x7f07f8209ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.618+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f07f802dd80 con 0x7f07f8207aa0 2024-04-25T15:54:00.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.618+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f07f8207aa0 0x7f07f8209ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.622+0000 7f07f77fe640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f07f8207aa0 0x7f07f8209ec0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.622+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f802dd80 con 0x7f07f8207aa0 2024-04-25T15:54:00.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.638+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f802dd80 con 0x7f07f8207aa0 2024-04-25T15:54:00.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.642+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f802dd80 con 0x7f07f8207aa0 2024-04-25T15:54:00.642 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:00.642 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:00.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.642+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07f820a910 0x7f07f820cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.642+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f07f820fb00 con 0x7f07f820a910 2024-04-25T15:54:00.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.642+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07f820a910 0x7f07f820cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.642+0000 7f07fcf67640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07f820a910 0x7f07f820cd30 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.646+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f820fb00 con 0x7f07f820a910 2024-04-25T15:54:00.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.658+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f820d270 con 0x7f07f820a910 2024-04-25T15:54:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.662+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f820d270 con 0x7f07f820a910 2024-04-25T15:54:00.664 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.662+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f07f820d990 0x7f07f820de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.666+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f07f8212ed0 con 0x7f07f820d990 2024-04-25T15:54:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.666+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f07f820d990 0x7f07f820de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:00.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.666+0000 7f07f7fff640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f07f820d990 0x7f07f820de00 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:00.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.666+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07f8212ed0 con 0x7f07f820d990 2024-04-25T15:54:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.682+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f07f8211d50 con 0x7f07f820d990 2024-04-25T15:54:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07f4ff9640 1 -- 172.21.15.67:0/2622377427 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f07f8211d50 con 0x7f07f820d990 2024-04-25T15:54:00.687 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f07f8152ad0 msgr2=0x7f07f8075df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f07f8152ad0 0x7f07f8075df0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f07f81f6720 msgr2=0x7f07f81f6b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f07f81f6720 0x7f07f81f6b30 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f07f80635a0 msgr2=0x7f07f80639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f07f80635a0 0x7f07f80639b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f07f815afd0 msgr2=0x7f07f802c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f07f815afd0 0x7f07f802c9e0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f07f802d430 msgr2=0x7f07f802d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f07f802d430 0x7f07f802d840 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fcf67640 1 -- 172.21.15.67:0/2622377427 reap_dead start 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f07f8207aa0 msgr2=0x7f07f8209ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f07f8207aa0 0x7f07f8209ec0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fcf67640 1 -- 172.21.15.67:0/2622377427 reap_dead start 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07f820a910 msgr2=0x7f07f820cd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f07f820a910 0x7f07f820cd30 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f07f820d990 msgr2=0x7f07f820de00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f07f820d990 0x7f07f820de00 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fcf67640 1 -- 172.21.15.67:0/2622377427 reap_dead start 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07e406f400 msgr2=0x7f07e4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07e406f400 0x7f07e4071910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f07f81ed5e0 tx=0x7f07e8008040 comp rx=0 tx=0).stop 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f81e94f0 msgr2=0x7f07f8152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f81e94f0 0x7f07f8152590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009f90 tx=0x7f07f0009750 comp rx=0 tx=0).stop 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 shutdown_connections 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f07e406f400 0x7f07e4071910 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 --2- 172.21.15.67:0/2622377427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f81e94f0 0x7f07f8152590 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 >> 172.21.15.67:0/2622377427 conn(0x7f07f806e420 msgr2=0x7f07f80b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 shutdown_connections 2024-04-25T15:54:00.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:00.686+0000 7f07fe9ca640 1 -- 172.21.15.67:0/2622377427 wait complete. 2024-04-25T15:54:00.698 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:00.694+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:00.734 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:00.734 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:00.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:54:00.837 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:00.838+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:00.899 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:00.898+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:00.940 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:00.940 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:00.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:54:01.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:01.042+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:01.101 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:01.098+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:01.148 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:01.148 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:01.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:54:01.209 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:01.210+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:01.303 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:01.302+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:01.364 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:01.365 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:01.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:54:01.425 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:01.426+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:01.505 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:01.502+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:01.579 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:01.579 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:01.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:54:01.605 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:01.602+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:01.706 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:01.706+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:01.793 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:01.793 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:01.806 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:01.802+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:01.823 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:54:01.908 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:01.906+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:02.008 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:02.006+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:02.019 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:02.019 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:02.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:54:02.103 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:54:02.143 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:02.142+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:02.210 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:02.210+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:02.260 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:54:02.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:54:02.261 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-04-25T15:54:02.261 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:54:02.261 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:54:02.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-04-25T15:54:02.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.298+0000 7fd136571640 1 Processor -- start 2024-04-25T15:54:02.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.298+0000 7fd136571640 1 -- start start 2024-04-25T15:54:02.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd136571640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1300caf60 0x7fd13015d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd136571640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd13015d790 0x7fd13006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd136571640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1300c77a0 con 0x7fd130073bf0 2024-04-25T15:54:02.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd136571640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd1300cab60 con 0x7fd13015d790 2024-04-25T15:54:02.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd136571640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd130159620 con 0x7fd1300caf60 2024-04-25T15:54:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12f7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd13015d790 0x7fd13006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12f7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd13015d790 0x7fd13006d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:50200/0 (socket says 172.21.15.67:50200) 2024-04-25T15:54:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12f7fe640 1 -- 172.21.15.67:0/1703162106 learned_addr learned my addr 172.21.15.67:0/1703162106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd134b0e640 1 --1- 172.21.15.67:0/1703162106 >> v1:172.21.15.176:6789/0 conn(0x7fd130073bf0 0x7fd130072bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:52312/0 (socket says 172.21.15.67:52312) 2024-04-25T15:54:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12ffff640 1 --2- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1300caf60 0x7fd13015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12f7fe640 1 -- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1300caf60 msgr2=0x7fd13015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:02.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12f7fe640 1 --2- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1300caf60 0x7fd13015d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12f7fe640 1 -- 172.21.15.67:0/1703162106 >> v1:172.21.15.176:6789/0 conn(0x7fd130073bf0 legacy=0x7fd130072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:02.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12effd640 1 -- 172.21.15.67:0/1703162106 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2917522601 0 0) 0x7fd1300c77a0 con 0x7fd130073bf0 2024-04-25T15:54:02.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.302+0000 7fd12f7fe640 1 -- 172.21.15.67:0/1703162106 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1300c7a60 con 0x7fd13015d790 2024-04-25T15:54:02.304 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:02.304 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:02.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd12f7fe640 1 --2- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13015d790 0x7fd13006d750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd12400bb10 tx=0x7fd12401ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae890e1815d71b28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd12effd640 1 -- 172.21.15.67:0/1703162106 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1240b4070 con 0x7fd13015d790 2024-04-25T15:54:02.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd12effd640 1 -- 172.21.15.67:0/1703162106 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1240a7b00 con 0x7fd13015d790 2024-04-25T15:54:02.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd12effd640 1 -- 172.21.15.67:0/1703162106 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1240af440 con 0x7fd13015d790 2024-04-25T15:54:02.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 -- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13015d790 msgr2=0x7fd13006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:02.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 --2- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13015d790 0x7fd13006d750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd12400bb10 tx=0x7fd12401ec30 comp rx=0 tx=0).stop 2024-04-25T15:54:02.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 -- 172.21.15.67:0/1703162106 shutdown_connections 2024-04-25T15:54:02.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 --2- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd13015d790 0x7fd13006d750 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 --2- 172.21.15.67:0/1703162106 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1300caf60 0x7fd13015d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 -- 172.21.15.67:0/1703162106 >> 172.21.15.67:0/1703162106 conn(0x7fd130070db0 msgr2=0x7fd1300ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 -- 172.21.15.67:0/1703162106 shutdown_connections 2024-04-25T15:54:02.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.306+0000 7fd136571640 1 -- 172.21.15.67:0/1703162106 wait complete. 2024-04-25T15:54:02.310 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:02.310+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 Processor -- start 2024-04-25T15:54:02.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 -- start start 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130073bf0 0x7fd1301f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1300caf60 0x7fd1300c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd13015d790 0x7fd13014ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd13006c7c0 con 0x7fd130073bf0 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd13006c640 con 0x7fd1300caf60 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd136571640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd13006c940 con 0x7fd13015d790 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd12ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130073bf0 0x7fd1301f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd12ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130073bf0 0x7fd1301f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40078/0 (socket says 172.21.15.67:40078) 2024-04-25T15:54:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.310+0000 7fd12ffff640 1 -- 172.21.15.67:0/2897688731 learned_addr learned my addr 172.21.15.67:0/2897688731 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd134b0e640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd13015d790 0x7fd13014ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12ffff640 1 -- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd13015d790 msgr2=0x7fd13014ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12ffff640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd13015d790 0x7fd13014ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12ffff640 1 -- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1300caf60 msgr2=0x7fd1300c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12ffff640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1300caf60 0x7fd1300c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12ffff640 1 -- 172.21.15.67:0/2897688731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1301f80a0 con 0x7fd130073bf0 2024-04-25T15:54:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12ffff640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130073bf0 0x7fd1301f7b60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd12000e870 tx=0x7fd12000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12cff9640 1 -- 172.21.15.67:0/2897688731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd120017070 con 0x7fd130073bf0 2024-04-25T15:54:02.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1301551f0 con 0x7fd130073bf0 2024-04-25T15:54:02.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd130156b60 con 0x7fd130073bf0 2024-04-25T15:54:02.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12cff9640 1 -- 172.21.15.67:0/2897688731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd12000c8e0 con 0x7fd130073bf0 2024-04-25T15:54:02.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12cff9640 1 -- 172.21.15.67:0/2897688731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12000cc90 con 0x7fd130073bf0 2024-04-25T15:54:02.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1100052a0 con 0x7fd130073bf0 2024-04-25T15:54:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12cff9640 1 -- 172.21.15.67:0/2897688731 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd12001a070 con 0x7fd130073bf0 2024-04-25T15:54:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12cff9640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd11c06f430 0x7fd11c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.314+0000 7fd12cff9640 1 -- 172.21.15.67:0/2897688731 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5973+0+0 (secure 0 0 0) 0x7fd12008f370 con 0x7fd130073bf0 2024-04-25T15:54:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.318+0000 7fd12f7fe640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd11c06f430 0x7fd11c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.318+0000 7fd12f7fe640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd11c06f430 0x7fd11c071940 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd1240b1000 tx=0x7fd124008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.326+0000 7fd12cff9640 1 -- 172.21.15.67:0/2897688731 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd12005bc20 con 0x7fd130073bf0 2024-04-25T15:54:02.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:54:02.411 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:02.410+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:02.463 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:02.463 INFO:teuthology.orchestra.run.smithi176.stdout: "success": "" 2024-04-25T15:54:02.463 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:02.512 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:02.510+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:02.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.590+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fd110005860 con 0x7fd130073bf0 2024-04-25T15:54:02.600 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:02.601 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:02.613 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:02.610+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:02.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:54:02.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:54:02.753 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:02.754+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:02.814 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:02.814+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:02.875 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:02.876 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:02.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:54:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.910+0000 7f8c4ed33640 1 Processor -- start 2024-04-25T15:54:02.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.910+0000 7f8c4ed33640 1 -- start start 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c4ed33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c48073bf0 0x7f8c4806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c47fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c48073bf0 0x7f8c4806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c47fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c48073bf0 0x7f8c4806d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59852/0 (socket says 172.21.15.67:59852) 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c4cacf640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f8c480caf60 0x7f8c4815d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:51818/0 (socket says 172.21.15.67:51818) 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c4ed33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4806dc90 0x7f8c48163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c47fff640 1 -- 172.21.15.67:0/2444032862 learned_addr learned my addr 172.21.15.67:0/2444032862 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c480c77a0 con 0x7f8c480caf60 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f8c480cab60 con 0x7f8c48073bf0 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c48159620 con 0x7f8c4806dc90 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c47fff640 1 -- 172.21.15.67:0/2444032862 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4806dc90 msgr2=0x7f8c48163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c47fff640 1 --2- 172.21.15.67:0/2444032862 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4806dc90 0x7f8c48163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c47fff640 1 -- 172.21.15.67:0/2444032862 >> v1:172.21.15.176:6789/0 conn(0x7f8c480caf60 legacy=0x7f8c4815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:02.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.914+0000 7f8c47fff640 1 -- 172.21.15.67:0/2444032862 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c480c7a60 con 0x7f8c48073bf0 2024-04-25T15:54:02.917 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c47fff640 1 --2- 172.21.15.67:0/2444032862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c4806d750 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8c480cf8d0 tx=0x7f8c3401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bf48cc398cc864 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c477fe640 1 -- 172.21.15.67:0/2444032862 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c340b0070 con 0x7f8c48073bf0 2024-04-25T15:54:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c477fe640 1 -- 172.21.15.67:0/2444032862 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c340ab600 con 0x7f8c48073bf0 2024-04-25T15:54:02.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c477fe640 1 -- 172.21.15.67:0/2444032862 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c340ab9d0 con 0x7f8c48073bf0 2024-04-25T15:54:02.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 msgr2=0x7f8c4806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:02.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/2444032862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c4806d750 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8c480cf8d0 tx=0x7f8c3401c4e0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 shutdown_connections 2024-04-25T15:54:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/2444032862 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4806dc90 0x7f8c48163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/2444032862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c4806d750 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 >> 172.21.15.67:0/2444032862 conn(0x7f8c48070db0 msgr2=0x7f8c480ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 shutdown_connections 2024-04-25T15:54:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.918+0000 7f8c4ed33640 1 -- 172.21.15.67:0/2444032862 wait complete. 2024-04-25T15:54:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 Processor -- start 2024-04-25T15:54:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 -- start start 2024-04-25T15:54:02.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4806dc90 0x7f8c481f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c48160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c480caf60 0x7f8c480ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4815a570 con 0x7f8c48073bf0 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4815a3f0 con 0x7f8c4806dc90 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c48166c20 con 0x7f8c480caf60 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c48160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c48160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59860/0 (socket says 172.21.15.67:59860) 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 -- 172.21.15.67:0/1364830103 learned_addr learned my addr 172.21.15.67:0/1364830103 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4806dc90 0x7f8c481f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4d2d0640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c480caf60 0x7f8c480ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c480caf60 msgr2=0x7f8c480ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c480caf60 0x7f8c480ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4806dc90 msgr2=0x7f8c481f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4806dc90 0x7f8c481f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c481f46b0 con 0x7f8c48073bf0 2024-04-25T15:54:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c47fff640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c48160d60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8c3401cd80 tx=0x7f8c3401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c340b0040 con 0x7f8c48073bf0 2024-04-25T15:54:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c340bc070 con 0x7f8c48073bf0 2024-04-25T15:54:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c481629e0 con 0x7f8c48073bf0 2024-04-25T15:54:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c340b6840 con 0x7f8c48073bf0 2024-04-25T15:54:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.922+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c481f4c00 con 0x7f8c48073bf0 2024-04-25T15:54:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8c4806a480 con 0x7f8c48073bf0 2024-04-25T15:54:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8c340b69e0 con 0x7f8c48073bf0 2024-04-25T15:54:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c457fa640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c2c06f450 0x7f8c2c071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5973+0+0 (secure 0 0 0) 0x7f8c341295e0 con 0x7f8c48073bf0 2024-04-25T15:54:02.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v96) ==== 54+0+16 (secure 0 0 0) 0x7f8c340c6020 con 0x7f8c48073bf0 2024-04-25T15:54:02.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c48160480 0x7f8c4815bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c481614b0 con 0x7f8c48160480 2024-04-25T15:54:02.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.926+0000 7f8c4d2d0640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c48160480 0x7f8c4815bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.930+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c2c06f450 0x7f8c2c071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.930+0000 7f8c4d2d0640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c48160480 0x7f8c4815bcb0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.930+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c2c06f450 0x7f8c2c071960 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00b830 tx=0x7f8c3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.930+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c481614b0 con 0x7f8c48160480 2024-04-25T15:54:02.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.958+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c481f49b0 con 0x7f8c48160480 2024-04-25T15:54:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.958+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c481f49b0 con 0x7f8c48160480 2024-04-25T15:54:02.960 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.962+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c481507a0 0x7f8c4806c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.962+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8c48150bb0 con 0x7f8c481507a0 2024-04-25T15:54:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.962+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c481507a0 0x7f8c4806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.962+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c481507a0 0x7f8c4806c640 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.962+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c48150bb0 con 0x7f8c481507a0 2024-04-25T15:54:02.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.982+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c480c8a40 con 0x7f8c481507a0 2024-04-25T15:54:02.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.986+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c480c8a40 con 0x7f8c481507a0 2024-04-25T15:54:02.986 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:02.987 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:02.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.986+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c4806d090 0x7f8c48063090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:02.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.986+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8c480cad40 con 0x7f8c4806d090 2024-04-25T15:54:02.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.986+0000 7f8c47fff640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c4806d090 0x7f8c48063090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:02.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.986+0000 7f8c47fff640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c4806d090 0x7f8c48063090 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:02.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:02.990+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c480cad40 con 0x7f8c4806d090 2024-04-25T15:54:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.006+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c48159bd0 con 0x7f8c4806d090 2024-04-25T15:54:03.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.006+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c48159bd0 con 0x7f8c4806d090 2024-04-25T15:54:03.009 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.010+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c48063ae0 0x7f8c48150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.010+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8c481598f0 con 0x7f8c48063ae0 2024-04-25T15:54:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.010+0000 7f8c4d2d0640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c48063ae0 0x7f8c48150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.010+0000 7f8c4d2d0640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c48063ae0 0x7f8c48150fd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.010+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c481598f0 con 0x7f8c48063ae0 2024-04-25T15:54:03.017 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:03.014+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:03.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.030+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c4815d510 con 0x7f8c48063ae0 2024-04-25T15:54:03.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.034+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c4815d510 con 0x7f8c48063ae0 2024-04-25T15:54:03.035 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.034+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c48151a20 0x7f8c4802c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.034+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8c48151f70 con 0x7f8c48151a20 2024-04-25T15:54:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.034+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c48151a20 0x7f8c4802c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.034+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c48151a20 0x7f8c4802c9e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.038+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c48151f70 con 0x7f8c48151a20 2024-04-25T15:54:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.062+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c48151f70 con 0x7f8c48151a20 2024-04-25T15:54:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.062+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c48151f70 con 0x7f8c48151a20 2024-04-25T15:54:03.064 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.066+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c4802d430 0x7f8c4802d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.066+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8c4802dd80 con 0x7f8c4802d430 2024-04-25T15:54:03.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.066+0000 7f8c47fff640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c4802d430 0x7f8c4802d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:03.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.070+0000 7f8c47fff640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c4802d430 0x7f8c4802d840 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.070+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c4802dd80 con 0x7f8c4802d430 2024-04-25T15:54:03.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.098+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c4802dd80 con 0x7f8c4802d430 2024-04-25T15:54:03.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.098+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c4802dd80 con 0x7f8c4802d430 2024-04-25T15:54:03.101 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:03.101 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:03.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.102+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c4820af80 0x7f8c4820d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:03.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.102+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8c482102d0 con 0x7f8c4820af80 2024-04-25T15:54:03.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.102+0000 7f8c4d2d0640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c4820af80 0x7f8c4820d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:03.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.102+0000 7fd12cff9640 1 -- 172.21.15.67:0/2897688731 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v97) ==== 101+0+0 (secure 0 0 0) 0x7fd12005ff20 con 0x7fd130073bf0 2024-04-25T15:54:03.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.102+0000 7f8c4d2d0640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c4820af80 0x7f8c4820d3a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:03.103 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:54:03.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.102+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c482102d0 con 0x7f8c4820af80 2024-04-25T15:54:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd11c06f430 msgr2=0x7fd11c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd11c06f430 0x7fd11c071940 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd1240b1000 tx=0x7fd124008040 comp rx=0 tx=0).stop 2024-04-25T15:54:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130073bf0 msgr2=0x7fd1301f7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130073bf0 0x7fd1301f7b60 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd12000e870 tx=0x7fd12000ec70 comp rx=0 tx=0).stop 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 shutdown_connections 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd11c06f430 0x7fd11c071940 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd13015d790 0x7fd13014ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1300caf60 0x7fd1300c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 --2- 172.21.15.67:0/2897688731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd130073bf0 0x7fd1301f7b60 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 >> 172.21.15.67:0/2897688731 conn(0x7fd130070db0 msgr2=0x7fd130158920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 shutdown_connections 2024-04-25T15:54:03.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.106+0000 7fd136571640 1 -- 172.21.15.67:0/2897688731 wait complete. 2024-04-25T15:54:03.118 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:03.114+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:03.119 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:03.119 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:03.119 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.130+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c4820d8e0 con 0x7f8c4820af80 2024-04-25T15:54:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.134+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c4820d8e0 con 0x7f8c4820af80 2024-04-25T15:54:03.137 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:03.137 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.138+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c4820e000 0x7f8c4820e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.138+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8c482136a0 con 0x7f8c4820e000 2024-04-25T15:54:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.138+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c4820e000 0x7f8c4820e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.138+0000 7f8c4cacf640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c4820e000 0x7f8c4820e410 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.138+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c482136a0 con 0x7f8c4820e000 2024-04-25T15:54:03.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:54:03.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.162+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8c48212520 con 0x7f8c4820e000 2024-04-25T15:54:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c457fa640 1 -- 172.21.15.67:0/1364830103 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8c48212520 con 0x7f8c4820e000 2024-04-25T15:54:03.167 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c48160480 msgr2=0x7f8c4815bcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f8c48160480 0x7f8c4815bcb0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c481507a0 msgr2=0x7f8c4806c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f8c481507a0 0x7f8c4806c640 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c4806d090 msgr2=0x7f8c48063090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f8c4806d090 0x7f8c48063090 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c48063ae0 msgr2=0x7f8c48150fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f8c48063ae0 0x7f8c48150fd0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c48151a20 msgr2=0x7f8c4802c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f8c48151a20 0x7f8c4802c9e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c4802d430 msgr2=0x7f8c4802d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4d2d0640 1 -- 172.21.15.67:0/1364830103 reap_dead start 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f8c4802d430 0x7f8c4802d840 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4d2d0640 1 -- 172.21.15.67:0/1364830103 reap_dead start 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4d2d0640 1 -- 172.21.15.67:0/1364830103 reap_dead start 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c4820af80 msgr2=0x7f8c4820d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c4820af80 0x7f8c4820d3a0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c4820e000 msgr2=0x7f8c4820e410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c4820e000 0x7f8c4820e410 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c2c06f450 msgr2=0x7f8c2c071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c2c06f450 0x7f8c2c071960 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00b830 tx=0x7f8c3c008040 comp rx=0 tx=0).stop 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 msgr2=0x7f8c48160d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c48160d60 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8c3401cd80 tx=0x7f8c3401ad20 comp rx=0 tx=0).stop 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 shutdown_connections 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8c4820af80 0x7f8c4820d3a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f8c4820e000 0x7f8c4820e410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8c2c06f450 0x7f8c2c071960 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 --2- 172.21.15.67:0/1364830103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c48073bf0 0x7f8c48160d60 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 >> 172.21.15.67:0/1364830103 conn(0x7f8c48070db0 msgr2=0x7f8c4815b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 shutdown_connections 2024-04-25T15:54:03.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:03.166+0000 7f8c4ed33640 1 -- 172.21.15.67:0/1364830103 wait complete. 2024-04-25T15:54:03.245 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:03.246+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:03.320 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:03.318+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 60.161648139999997, 2024-04-25T15:54:03.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:54:03.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:54:03.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:54:03.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:54:03.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:671 3.38 e81/18 3:1c0639bf:::100000001f7.0000009b:head v 81'8, mlcod=81'6)", 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.494028063000002, 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069441491999999994, 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.389 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886647+0000", 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.886668+0000", 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0876000000000001e-05 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893242+0000", 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0065744849999999997 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893267+0000", 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5321e-05 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893283+0000", 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5454000000000001e-05 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893362+0000", 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8770000000000006e-05 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.893382+0000", 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0585999999999999e-05 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.955934+0000", 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062552041000000003 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.956088+0000", 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015395900000000001 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:673 3.28 3:153e0bf4:::100000001f7.0000009d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.475738774, 2024-04-25T15:54:03.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063190614000000006, 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 673 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904936+0000", 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.904939+0000", 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8739999999999999e-06 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911908+0000", 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0069695449999999997 2024-04-25T15:54:03.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911944+0000", 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5973000000000003e-05 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.911960+0000", 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5319999999999999e-05 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912057+0000", 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7427000000000001e-05 2024-04-25T15:54:03.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912221+0000", 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016433600000000001 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.912375+0000", 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015359299999999999 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.941806+0000", 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029430523 2024-04-25T15:54:03.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.967997+0000", 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026191589000000001 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968063+0000", 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6321000000000004e-05 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.968127+0000", 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3113000000000004e-05 2024-04-25T15:54:03.398 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1699 3.2f 3:f7a50725:::100000001f8.0000029f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:44.934543+0000", 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.446131969, 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067346198999999995, 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.399 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1699 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.934543+0000", 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.934543+0000", 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.934547+0000", 2024-04-25T15:54:03.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2270000000000004e-06 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939709+0000", 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051621649999999998 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939765+0000", 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5924999999999997e-05 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939789+0000", 2024-04-25T15:54:03.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4383e-05 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.939913+0000", 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000123738 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.940118+0000", 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020446799999999999 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:44.940324+0000", 2024-04-25T15:54:03.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000206582 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.000408+0000", 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.060084151000000002 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.001776+0000", 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013679359999999999 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.001824+0000", 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7366e-05 2024-04-25T15:54:03.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.001889+0000", 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5258000000000004e-05 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2301 3.2f 3:f7556ebc:::100000001f8.000004f9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.773283+0000", 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.607392209, 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.066818095999999993, 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.404 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2301 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.773283+0000", 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.773283+0000", 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.773286+0000", 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5839999999999999e-06 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.778778+0000", 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054920710000000003 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.778837+0000", 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8473000000000001e-05 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.778862+0000", 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5249000000000002e-05 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.779074+0000", 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021172899999999999 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.779338+0000", 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026401400000000002 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.779586+0000", 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000248478 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.796756+0000", 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017169897999999999 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839913+0000", 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.043156844 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839998+0000", 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4728999999999995e-05 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.840101+0000", 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103027 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.409 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2305 3.2f 3:f6648653:::100000001f8.000004fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.577884503, 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16263440000000001, 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2305 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.410 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802795+0000", 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7530000000000001e-06 2024-04-25T15:54:03.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815251+0000", 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012456312000000001 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815675+0000", 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042352399999999999 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815700+0000", 2024-04-25T15:54:03.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5225e-05 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815859+0000", 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015856099999999999 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816239+0000", 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038042099999999998 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816444+0000", 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020472099999999999 2024-04-25T15:54:03.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.906577+0000", 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090133536 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965298+0000", 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058721044999999999 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965364+0000", 2024-04-25T15:54:03.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6013000000000006e-05 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965425+0000", 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0288999999999998e-05 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2304 3.36 e90/18 3:6f9fe141:::100000001f8.000004fc:head v 90'48, mlcod=89'46)", 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.576558240000001, 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14476808899999999, 2024-04-25T15:54:03.415 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804121+0000", 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3950000000000003e-06 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818809+0000", 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014687881999999999 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818838+0000", 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9689e-05 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818859+0000", 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0353999999999999e-05 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.819110+0000", 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025134700000000002 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.819181+0000", 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1094999999999996e-05 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.948829+0000", 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12964726300000001 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.948885+0000", 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6063999999999997e-05 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2306 3.5 3:a0445fe6:::100000001f8.000004fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.419 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.565412114000001, 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.1339195, 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2306 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815265+0000", 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2759999999999999e-06 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830149+0000", 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014884298000000001 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830753+0000", 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060401599999999997 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830772+0000", 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9017000000000001e-05 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.831046+0000", 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027419800000000002 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.832603+0000", 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015568450000000001 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833182+0000", 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057851500000000004 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.859487+0000", 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026305354 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949024+0000", 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.089536491999999995 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949121+0000", 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7589000000000004e-05 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949182+0000", 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0900000000000003e-05 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2308 3.4 e90/68 3:2343a40c:::100000001f8.00000500:head v 90'35, mlcod=90'33)", 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.557477677, 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12610180200000001, 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.425 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.426 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:03.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:03.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823200+0000", 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4590000000000001e-06 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833167+0000", 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0099672500000000004 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833193+0000", 2024-04-25T15:54:03.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6231999999999999e-05 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833214+0000", 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0942e-05 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833316+0000", 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101772 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833360+0000", 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3835999999999998e-05 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949250+0000", 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11589087000000001 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949299+0000", 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8440999999999999e-05 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2307 3.35 e90/18 3:ad417e89:::100000001f8.000004ff:head v 90'49, mlcod=90'47)", 2024-04-25T15:54:03.430 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.828366+0000", 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.552308999999999, 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.121096469, 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.828366+0000", 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.828366+0000", 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.828368+0000", 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2000000000000001e-06 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833667+0000", 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052990789999999999 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833694+0000", 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6800000000000001e-05 2024-04-25T15:54:03.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833715+0000", 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379e-05 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.841285+0000", 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075698600000000003 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.841333+0000", 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8004999999999999e-05 2024-04-25T15:54:03.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949416+0000", 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10808335099999999 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949462+0000", 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5794999999999998e-05 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2312 3.5 3:a05781df:::100000001f8.00000504:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.434 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.853782+0000", 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.526892745, 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099756845999999996, 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2312 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.853782+0000", 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.853782+0000", 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.853785+0000", 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2949999999999998e-06 2024-04-25T15:54:03.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.863044+0000", 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092590620000000002 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.863075+0000", 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1084999999999998e-05 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.863106+0000", 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0374000000000001e-05 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.863218+0000", 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111718 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.863428+0000", 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000210235 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.863622+0000", 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019466199999999999 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.889073+0000", 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025450383 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953394+0000", 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.064321342000000004 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953490+0000", 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5849999999999999e-05 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953539+0000", 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.884e-05 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2311 3.35 e90/18 3:aef469dc:::100000001f8.00000503:head v 90'50, mlcod=90'47)", 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.858653+0000", 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.52202192, 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094324564999999999, 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858653+0000", 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858653+0000", 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.858655+0000", 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.176e-06 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.861447+0000", 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027917490000000001 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.861474+0000", 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7368999999999998e-05 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.861493+0000", 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8601e-05 2024-04-25T15:54:03.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.861599+0000", 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106585 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.861642+0000", 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3103999999999999e-05 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.952925+0000", 2024-04-25T15:54:03.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.091282199999999994 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.952977+0000", 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2781000000000002e-05 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2316 3.1d 3:b9f5e24a:::100000001f8.00000508:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.886073+0000", 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.494601712, 2024-04-25T15:54:03.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069394102999999999, 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2316 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.886073+0000", 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.886073+0000", 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.886078+0000", 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3399999999999997e-06 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.890926+0000", 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048477720000000002 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.890960+0000", 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4144999999999999e-05 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.890979+0000", 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8451999999999999e-05 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.891047+0000", 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7768999999999994e-05 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.891173+0000", 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012687399999999999 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.891285+0000", 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111146 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953550+0000", 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062265274000000002 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.955344+0000", 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001794552 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.955416+0000", 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1840000000000003e-05 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.955467+0000", 2024-04-25T15:54:03.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0939000000000001e-05 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2495 3.1c 3:394277a5:::100000001f8.000005bb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.147076+0000", 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.233598327999999, 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084955764000000003, 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.450 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2495 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.147076+0000", 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.147076+0000", 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.147079+0000", 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9040000000000001e-06 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.152587+0000", 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055073930000000002 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.152626+0000", 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8964999999999997e-05 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.152644+0000", 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7870999999999999e-05 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.152761+0000", 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011728899999999999 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.152913+0000", 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015229299999999999 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.153055+0000", 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142206 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.188875+0000", 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035819465000000002 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.231945+0000", 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.043070566999999997 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.231990+0000", 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5105000000000003e-05 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.232032+0000", 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1705999999999997e-05 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2498 3.17 3:eaf2c8b6:::100000001f8.000005be:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.455 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.165100+0000", 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.215574948, 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.066554200999999993, 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2498 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.165100+0000", 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.165100+0000", 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.165104+0000", 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1200000000000004e-06 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.169916+0000", 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048118229999999998 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.169954+0000", 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7815999999999998e-05 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.169967+0000", 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3645000000000001e-05 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170086+0000", 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118792 2024-04-25T15:54:03.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170332+0000", 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024649300000000003 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170571+0000", 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023895200000000001 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.189203+0000", 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018631407999999999 2024-04-25T15:54:03.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.231548+0000", 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.042344840000000002 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.231606+0000", 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7825999999999998e-05 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.231654+0000", 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8486000000000003e-05 2024-04-25T15:54:03.460 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2513 3.3c 3:3c46fcc2:::100000001f8.000005cd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.256485+0000", 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.124189448999999, 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.085953117999999995, 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2513 2024-04-25T15:54:03.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.256485+0000", 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.256485+0000", 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.256487+0000", 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1220000000000002e-06 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.261328+0000", 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048404499999999996 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.329127+0000", 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067799553999999998 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.329142+0000", 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4718999999999999e-05 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.329250+0000", 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010785099999999999 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.329430+0000", 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017990200000000001 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:03.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.329600+0000", 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017052699999999999 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.342148+0000", 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012547310000000001 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.342340+0000", 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000192332 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.342386+0000", 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5578000000000003e-05 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.342438+0000", 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2772999999999998e-05 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2530 3.5 3:a39a885e:::100000001f8.000005de:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.361538+0000", 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.019136402000001, 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.077346114999999993, 2024-04-25T15:54:03.466 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2530 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.361538+0000", 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.361538+0000", 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.361541+0000", 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2450000000000001e-06 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.365791+0000", 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042506439999999996 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.416752+0000", 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.050960473999999999 2024-04-25T15:54:03.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.416770+0000", 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8003999999999999e-05 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.416886+0000", 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116259 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.417138+0000", 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025251499999999998 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.417776+0000", 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063756299999999995 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.429765+0000", 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011988989 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.438734+0000", 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089687629999999994 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.438821+0000", 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7504999999999997e-05 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.438884+0000", 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3153999999999999e-05 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2567 3.1c 3:3be4d2bf:::100000001f8.00000603:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.610105+0000", 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 12.770570044999999, 2024-04-25T15:54:03.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.077380852, 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2567 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.610105+0000", 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.610105+0000", 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.610106+0000", 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.341e-06 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.619120+0000", 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0090141460000000007 2024-04-25T15:54:03.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.619156+0000", 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6066000000000003e-05 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.619171+0000", 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4428e-05 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.619250+0000", 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9719e-05 2024-04-25T15:54:03.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.619389+0000", 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013856299999999999 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.619505+0000", 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116278 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.627429+0000", 2024-04-25T15:54:03.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079237279999999997 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.687373+0000", 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.059944075999999999 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.687438+0000", 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5157e-05 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.687486+0000", 2024-04-25T15:54:03.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7349999999999999e-05 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3289 3.2f 3:f5008011:::100000001f8.000008d5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:55.325580+0000", 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.0550951269999995, 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.075942233999999997, 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.477 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3289 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.325580+0000", 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.325580+0000", 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.325583+0000", 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5209999999999998e-06 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.329921+0000", 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043378540000000004 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.329981+0000", 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0266999999999997e-05 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.330002+0000", 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0948e-05 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.330111+0000", 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010891999999999999 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.330292+0000", 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018104899999999999 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:03.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.330437+0000", 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014440899999999999 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.361650+0000", 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.031213040000000001 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.401385+0000", 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039734876000000002 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.401447+0000", 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2553999999999998e-05 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.401522+0000", 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4795999999999993e-05 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3299 3.1c 3:3abe956f:::100000001f8.000008df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:55.382833+0000", 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 7.9978417049999999, 2024-04-25T15:54:03.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063082895999999999, 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3299 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.382833+0000", 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.382833+0000", 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.382837+0000", 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5420000000000001e-06 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.389287+0000", 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064502500000000003 2024-04-25T15:54:03.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.389330+0000", 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3263000000000002e-05 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.389346+0000", 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5823000000000001e-05 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.389495+0000", 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014889300000000001 2024-04-25T15:54:03.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.389710+0000", 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021477399999999999 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.389845+0000", 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013556000000000001 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.416259+0000", 2024-04-25T15:54:03.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026413849999999999 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.445774+0000", 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.029514827 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.445856+0000", 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2176999999999996e-05 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.445916+0000", 2024-04-25T15:54:03.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9936999999999999e-05 2024-04-25T15:54:03.488 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.488 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.488 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.488 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:03.488 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:03.488 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:03.489 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:54:03.490 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:03.489+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:03.590 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:03.586+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:03.690 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:03.686+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:03.711 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:03.711 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:03.711 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:03.711 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:03.739 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:54:03.792 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:03.789+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:03.896 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:03.890+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:03.991 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:03.991 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:03.991 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:03.991 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:03.991 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:03.994 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:03.993+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:04.028 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:54:04.115 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:04.114+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:04.195 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:04.194+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:04.266 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:04.266 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 40.605622154000002, 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.26243838600000002, 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:54:04.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:54:04.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:54:04.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920373+0000", 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039448372000000002 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921649+0000", 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012757210000000001 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921724+0000", 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5549000000000005e-05 2024-04-25T15:54:04.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921802+0000", 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8150000000000002e-05 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:354 3.10 3:0ba0dc78:::10000000002.000000fd:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.665669364000003, 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56691576200000005, 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599318+0000", 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599326+0000", 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9459999999999998e-06 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161759+0000", 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56243287799999997 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161783+0000", 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4057e-05 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161899+0000", 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011615799999999999 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162007+0000", 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107823 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162110+0000", 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010235500000000001 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.163904+0000", 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017941470000000001 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166138+0000", 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022346190000000002 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166186+0000", 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7902999999999999e-05 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166232+0000", 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6220999999999997e-05 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2304 3.36 3:6f9fe141:::100000001f8.000004fc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.474424464, 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15859584900000001, 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2304 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:54:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790566+0000", 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5369999999999999e-06 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802415+0000", 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011848747999999999 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802510+0000", 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5036000000000003e-05 2024-04-25T15:54:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802550+0000", 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9807e-05 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802712+0000", 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162619 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.803017+0000", 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000304433 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.803308+0000", 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029095100000000001 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.835450+0000", 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032142776999999997 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949058+0000", 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.113607898 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949100+0000", 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1213e-05 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949157+0000", 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.783e-05 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2307 3.35 3:ad417e89:::100000001f8.000004ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.453854784000001, 2024-04-25T15:54:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.138515105, 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2307 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811135+0000", 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1629999999999998e-06 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824210+0000", 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013074564 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824274+0000", 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4131e-05 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824302+0000", 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8167e-05 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824464+0000", 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000161605 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824749+0000", 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028465599999999999 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.825010+0000", 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026149799999999998 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844599+0000", 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019588795999999999 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949581+0000", 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10498215399999999 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949617+0000", 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6498000000000002e-05 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949646+0000", 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8873e-05 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2309 3.2c 3:354b7d43:::100000001f8.00000501:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 15.437997849, 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.127916427, 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2309 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826993+0000", 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.746e-06 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839270+0000", 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012277221 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839339+0000", 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8709999999999998e-05 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839369+0000", 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0590999999999999e-05 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839557+0000", 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018798099999999999 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839975+0000", 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041773500000000002 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.840214+0000", 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023835300000000001 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.925866+0000", 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.085652939999999997 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954707+0000", 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028840472999999998 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954819+0000", 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011174 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954905+0000", 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5937000000000001e-05 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2480 3.35 3:afdde0ae:::100000001f8.000005ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.226516282, 2024-04-25T15:54:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14409117900000001, 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2480 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038475+0000", 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0000000000000004e-06 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.046847+0000", 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083724569999999998 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131712+0000", 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084864593000000002 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131735+0000", 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3448e-05 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131860+0000", 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124258 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.132129+0000", 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026964399999999998 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.132397+0000", 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026743300000000003 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.167840+0000", 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035443241 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182375+0000", 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014535002 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182462+0000", 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6887999999999999e-05 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182561+0000", 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9215000000000005e-05 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2499 3.26 3:65b3bea5:::100000001f8.000005bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.094270856, 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.32409411900000001, 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2499 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170718+0000", 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9699999999999999e-06 2024-04-25T15:54:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.178145+0000", 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074272360000000003 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457402+0000", 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.27925624300000002 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457419+0000", 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7387000000000001e-05 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457521+0000", 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101615 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457720+0000", 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019950299999999999 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457907+0000", 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000186973 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.478350+0000", 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020442846000000001 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494652+0000", 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016301573999999999 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494737+0000", 2024-04-25T15:54:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5216000000000005e-05 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494809+0000", 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2556000000000001e-05 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3312 3.35 3:af24358c:::100000001f8.000008ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.7960696620000007, 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14714041, 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3312 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468918+0000", 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9929999999999998e-06 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.473248+0000", 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043301030000000001 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585462+0000", 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112213177 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585475+0000", 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3438000000000001e-05 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585550+0000", 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4975000000000006e-05 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585692+0000", 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142022 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585805+0000", 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112736 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.601250+0000", 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015444866999999999 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.615882+0000", 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014632116000000001 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.615956+0000", 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3769000000000005e-05 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.616057+0000", 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101214 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3915 3.35 3:ac1713fb:::100000001f8.00000147:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.493809+0000", 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.77117715799999997, 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17255479300000001, 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3915 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493809+0000", 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493809+0000", 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493812+0000", 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3440000000000001e-06 2024-04-25T15:54:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500643+0000", 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068309570000000003 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500700+0000", 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7003999999999998e-05 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500724+0000", 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3328000000000001e-05 2024-04-25T15:54:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500816+0000", 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2429999999999994e-05 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500912+0000", 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6422000000000006e-05 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501075+0000", 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016272500000000001 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653383+0000", 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15230775299999999 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666228+0000", 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012845132 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666300+0000", 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2504999999999998e-05 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666364+0000", 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3193000000000003e-05 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3919 3.2c 3:37bce3fe:::100000001f8.0000014b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.504543+0000", 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.76044332000000003, 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16566847200000001, 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3919 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504543+0000", 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504543+0000", 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504546+0000", 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0960000000000001e-06 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512225+0000", 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076787909999999999 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514415+0000", 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021900980000000001 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514429+0000", 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3848e-05 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514537+0000", 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108316 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514672+0000", 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013469099999999999 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515838+0000", 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001166411 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666543+0000", 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15070541400000001 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670106+0000", 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0035621989999999998 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670163+0000", 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7689000000000003e-05 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670211+0000", 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7919e-05 2024-04-25T15:54:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3917 3.27 e97/85 3:e652e8b8:::100000001f8.00000149:head v 97'58, mlcod=95'56)", 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.510092+0000", 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.75489362800000004, 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15662477599999999, 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.510092+0000", 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.510092+0000", 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.510094+0000", 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.455e-06 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514379+0000", 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042855510000000003 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514408+0000", 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8903000000000001e-05 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514429+0000", 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0230000000000001e-05 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514575+0000", 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146596 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514612+0000", 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7092000000000003e-05 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666650+0000", 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15203760599999999 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666717+0000", 2024-04-25T15:54:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7342999999999995e-05 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3920 3.26 3:6422fb70:::100000001f8.0000014c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.512238+0000", 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.75274776200000004, 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15489787399999999, 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3920 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512238+0000", 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512238+0000", 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512241+0000", 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8849999999999999e-06 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520461+0000", 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082202270000000001 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520514+0000", 2024-04-25T15:54:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2318999999999998e-05 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520535+0000", 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1488000000000002e-05 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520660+0000", 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012459999999999999 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520822+0000", 2024-04-25T15:54:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000161862 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.521056+0000", 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000234384 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666410+0000", 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14535368400000001 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667015+0000", 2024-04-25T15:54:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060488299999999996 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667082+0000", 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7322000000000004e-05 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667136+0000", 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4219999999999999e-05 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3923 3.35 3:ac352473:::100000001f8.0000014f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523946+0000", 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.74104043500000005, 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14618500700000001, 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3923 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523946+0000", 2024-04-25T15:54:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523946+0000", 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523949+0000", 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9299999999999999e-06 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531931+0000", 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007982701 2024-04-25T15:54:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531977+0000", 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5793e-05 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531998+0000", 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1097000000000001e-05 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532113+0000", 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114997 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532237+0000", 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012414699999999999 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532445+0000", 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020746499999999999 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669235+0000", 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.136789986 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670026+0000", 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00079101099999999999 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670086+0000", 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0399999999999998e-05 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670131+0000", 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4480000000000001e-05 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3928 3.10 3:0a3c763a:::100000001f8.00000154:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.546912+0000", 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.71807429300000003, 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122483755, 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3928 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546912+0000", 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546912+0000", 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546916+0000", 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2239999999999997e-06 2024-04-25T15:54:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.552858+0000", 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059422939999999999 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.552911+0000", 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3139000000000001e-05 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.552934+0000", 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2367000000000001e-05 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553076+0000", 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142454 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553195+0000", 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118437 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553308+0000", 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011311699999999999 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.580586+0000", 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027278046 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669260+0000", 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.088673972000000004 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669345+0000", 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5142999999999994e-05 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669396+0000", 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0562000000000001e-05 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3966 3.2c 3:36f7a988:::100000001f8.0000017a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.793699+0000", 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.471286859, 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13581676300000001, 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3966 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.793699+0000", 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.793699+0000", 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.793703+0000", 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6770000000000001e-06 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801135+0000", 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007432564 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801198+0000", 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.232e-05 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801213+0000", 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5705e-05 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801321+0000", 2024-04-25T15:54:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010709000000000001 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801404+0000", 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3844000000000006e-05 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801525+0000", 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121007 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.852854+0000", 2024-04-25T15:54:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051328560000000002 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929426+0000", 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.076572232000000004 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929470+0000", 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3918000000000002e-05 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929516+0000", 2024-04-25T15:54:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5846000000000001e-05 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3972 3.a 3:506f1d9f:::100000001f8.00000180:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.821706+0000", 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.44328035900000001, 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15946833799999999, 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3972 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821706+0000", 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821706+0000", 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821708+0000", 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7480000000000002e-06 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829901+0000", 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081928009999999996 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830176+0000", 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000275006 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830207+0000", 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1019999999999998e-05 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830305+0000", 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8097999999999995e-05 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830484+0000", 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017865099999999999 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830670+0000", 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018631799999999999 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971559+0000", 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14088915499999999 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981057+0000", 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0094979709999999992 2024-04-25T15:54:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981123+0000", 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5902999999999996e-05 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981174+0000", 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0667000000000001e-05 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3971 3.3f e97/18 3:febcb5ea:::100000001f8.0000017f:head v 97'71, mlcod=97'68)", 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.827561+0000", 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.43742541200000001, 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14429470999999999, 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.827561+0000", 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.827561+0000", 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.827562+0000", 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.849e-06 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.828970+0000", 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001407438 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829000+0000", 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0320000000000001e-05 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829020+0000", 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9624999999999999e-05 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829144+0000", 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012414199999999999 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829169+0000", 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4834999999999999e-05 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971813+0000", 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14264386100000001 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971855+0000", 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2639999999999998e-05 2024-04-25T15:54:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3974 3.2c 3:35c2bab0:::100000001f8.00000182:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.829920+0000", 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.43506621499999998, 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14917461800000001, 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3974 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829920+0000", 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829920+0000", 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829922+0000", 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3879999999999998e-06 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835844+0000", 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059220439999999996 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836262+0000", 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041762199999999998 2024-04-25T15:54:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836280+0000", 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8255999999999999e-05 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836402+0000", 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012145400000000001 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836576+0000", 2024-04-25T15:54:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000174184 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836798+0000", 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022228499999999999 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971962+0000", 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135163916 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978979+0000", 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070174729999999998 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979046+0000", 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6931e-05 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979094+0000", 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8065000000000001e-05 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3975 3.20 3:06072bd3:::100000001f8.00000183:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.835855+0000", 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.42913116200000001, 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135919927, 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3975 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835855+0000", 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835855+0000", 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835857+0000", 2024-04-25T15:54:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3980000000000002e-06 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841292+0000", 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054348349999999998 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841593+0000", 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030063000000000003 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841605+0000", 2024-04-25T15:54:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2245e-05 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841688+0000", 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3443999999999996e-05 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841801+0000", 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011285600000000001 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841952+0000", 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150717 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.856301+0000", 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014349259 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971673+0000", 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.115372129 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971720+0000", 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6700999999999999e-05 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971775+0000", 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4713000000000003e-05 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3976 3.2c 3:35666940:::100000001f8.00000184:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.841300+0000", 2024-04-25T15:54:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.42368571399999999, 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13985472299999999, 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3976 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841300+0000", 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841300+0000", 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841302+0000", 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6190000000000001e-06 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845593+0000", 2024-04-25T15:54:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042905900000000004 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845639+0000", 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6851999999999998e-05 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845664+0000", 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4301e-05 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845780+0000", 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011649799999999999 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845927+0000", 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014731 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846114+0000", 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018610400000000001 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972147+0000", 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12603325100000001 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981025+0000", 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088782259999999995 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981104+0000", 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8702000000000006e-05 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981155+0000", 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1270000000000002e-05 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:04.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:54:04.376 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:04.374+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:04.475 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:04.473+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:04.577 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:04.574+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:04.607 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:04.608 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:04.608 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:04.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:54:04.688 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:04.686+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:04.779 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:04.777+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:04.869 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:04.869 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:04.869 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:04.869 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:04.869 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:04.880 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:04.877+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:04.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:54:04.980 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:04.978+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:05.082 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:05.081+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:05.112 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:05.112 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:05.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:05.112 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:05.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 226.01219027799999, 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:54:05.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:54:05.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:54:05.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:252 3.2b 3:d64bac37:::10000000002.000000f8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.892949797, 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112378492, 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 252 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217705+0000", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.217709+0000", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1300000000000003e-06 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.223626+0000", 2024-04-25T15:54:05.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059177639999999998 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318358+0000", 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094731446999999996 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318383+0000", 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4992999999999999e-05 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318504+0000", 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121586 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318702+0000", 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019752800000000001 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318884+0000", 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018153200000000001 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.327804+0000", 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089205980000000001 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.329909+0000", 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021050359999999998 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330014+0000", 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010501300000000001 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.330083+0000", 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8864999999999999e-05 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 71.798315744999996, 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.124 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:05.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:54:05.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:54:05.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:05.129 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 41.527637792999997, 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:05.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:54:05.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:54:05.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.134 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2266 3.11 3:89528692:::100000001f8.000004d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.539471077000002, 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.38717112199999998, 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2266 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.135 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571186+0000", 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4569999999999999e-06 2024-04-25T15:54:05.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784888+0000", 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.213702742 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785181+0000", 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029301099999999998 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785201+0000", 2024-04-25T15:54:05.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9683000000000001e-05 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785333+0000", 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013224600000000001 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785779+0000", 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044548299999999999 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.786106+0000", 2024-04-25T15:54:05.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032678999999999998 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870593+0000", 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084487524999999994 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958234+0000", 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087640933000000004 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958283+0000", 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8473999999999997e-05 2024-04-25T15:54:05.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958354+0000", 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1778000000000003e-05 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2269 3.d 3:b24099bb:::100000001f8.000004d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.784900+0000", 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.325754821, 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.120695039, 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.140 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2269 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784900+0000", 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784900+0000", 2024-04-25T15:54:05.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784902+0000", 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.17e-06 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795914+0000", 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011012399000000001 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.796532+0000", 2024-04-25T15:54:05.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061838699999999997 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.796568+0000", 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5135999999999997e-05 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.796733+0000", 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016577900000000001 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.797066+0000", 2024-04-25T15:54:05.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033217099999999997 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.797359+0000", 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029385799999999998 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.848518+0000", 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051158207999999997 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905450+0000", 2024-04-25T15:54:05.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056931926000000001 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905526+0000", 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6137999999999995e-05 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905595+0000", 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8867000000000003e-05 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.145 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2272 3.13 3:ca3e88be:::100000001f8.000004dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.314722429, 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13173032000000001, 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2272 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795936+0000", 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.754e-06 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805167+0000", 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092308890000000008 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805557+0000", 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039059400000000002 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805588+0000", 2024-04-25T15:54:05.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0865999999999999e-05 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805759+0000", 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017085599999999999 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806172+0000", 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000413416 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806552+0000", 2024-04-25T15:54:05.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000379417 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905757+0000", 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099205023000000003 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927568+0000", 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021810800000000002 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927616+0000", 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927662+0000", 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6535000000000001e-05 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2273 3.39 3:9fcbcc01:::100000001f8.000004dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.305470314000001, 2024-04-25T15:54:05.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15983650499999999, 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2273 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805188+0000", 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816847+0000", 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011659119000000001 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817576+0000", 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00072975000000000004 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817601+0000", 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4450000000000001e-05 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817741+0000", 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014058500000000001 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818074+0000", 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033291000000000001 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818385+0000", 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031034999999999998 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870618+0000", 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052233743999999999 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964814+0000", 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094195216999999998 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964945+0000", 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013086799999999999 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965021+0000", 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6032000000000007e-05 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2275 3.12 3:492b2588:::100000001f8.000004df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.293794935000001, 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14794684399999999, 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2275 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816863+0000", 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4149999999999999e-06 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829311+0000", 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012448222 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829569+0000", 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025766800000000001 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829598+0000", 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.968e-05 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829786+0000", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187746 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830129+0000", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034284099999999998 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830479+0000", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034961500000000001 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867146+0000", 2024-04-25T15:54:05.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036667733000000001 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964688+0000", 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097541577000000004 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964748+0000", 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9626000000000001e-05 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964806+0000", 2024-04-25T15:54:05.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8721000000000002e-05 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2278 3.2b 3:d79b9024:::100000001f8.000004e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.281329676999999, 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135647779, 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.162 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2278 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829328+0000", 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7309999999999999e-06 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843821+0000", 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014493012 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844787+0000", 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096513199999999995 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844876+0000", 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9147999999999994e-05 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844972+0000", 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6566999999999999e-05 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845195+0000", 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222699 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845422+0000", 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022700699999999999 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870319+0000", 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024897319000000001 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964892+0000", 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094572233000000006 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964938+0000", 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6646e-05 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964972+0000", 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4285000000000001e-05 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2305 3.2f e90/51 3:f6648653:::100000001f8.000004fd:head v 90'33, mlcod=90'30)", 2024-04-25T15:54:05.167 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.273395887, 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12780160600000001, 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837261+0000", 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1679999999999998e-06 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839587+0000", 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00232607 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839620+0000", 2024-04-25T15:54:05.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3503000000000001e-05 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839642+0000", 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1302e-05 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839754+0000", 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112454 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839798+0000", 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4240999999999999e-05 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965003+0000", 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12520515600000001 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965060+0000", 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6712000000000002e-05 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2283 3.27 3:e56ee505:::100000001f8.000004e7:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.843837+0000", 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.266817148000001, 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.121476668, 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2283 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843837+0000", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843837+0000", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843842+0000", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0329999999999999e-06 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851910+0000", 2024-04-25T15:54:05.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080682299999999992 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852141+0000", 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023074400000000001 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852162+0000", 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0652000000000001e-05 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852248+0000", 2024-04-25T15:54:05.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6112000000000005e-05 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852390+0000", 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000141731 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.852509+0000", 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011903100000000001 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.953894+0000", 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.101384997 2024-04-25T15:54:05.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965175+0000", 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011280874999999999 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965255+0000", 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0238e-05 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965314+0000", 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9024999999999998e-05 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.176 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2292 3.12 3:4a3a3c79:::100000001f8.000004f0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.851920+0000", 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 16.258734361999998, 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11395511899999999, 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2292 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.177 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851920+0000", 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851920+0000", 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.851923+0000", 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3299999999999999e-06 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860067+0000", 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081432929999999994 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860474+0000", 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000407694 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.860489+0000", 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4568e-05 2024-04-25T15:54:05.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867223+0000", 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067344479999999996 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867532+0000", 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000308203 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867816+0000", 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028490399999999999 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.923967+0000", 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056150243000000002 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965776+0000", 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.041809499 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965829+0000", 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3062000000000001e-05 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965875+0000", 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5874999999999997e-05 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3917 3.27 3:e652e8b8:::100000001f8.00000149:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.6125853829999999, 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16896387500000001, 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.182 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3917 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498071+0000", 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4779999999999998e-06 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509296+0000", 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011224433000000001 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509572+0000", 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027576500000000003 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509591+0000", 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9466e-05 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509677+0000", 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5607999999999998e-05 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509773+0000", 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6136000000000002e-05 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509942+0000", 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016951999999999999 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.610111+0000", 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.100168231 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666922+0000", 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056811077000000001 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666991+0000", 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9758000000000005e-05 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667033+0000", 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1403000000000003e-05 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3918 3.11 3:8b260724:::100000001f8.0000014a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.60134917, 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15748047100000001, 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3918 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509307+0000", 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.12e-06 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515132+0000", 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058250860000000002 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515164+0000", 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1563000000000003e-05 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515179+0000", 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5102e-05 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515262+0000", 2024-04-25T15:54:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2917999999999995e-05 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515360+0000", 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7532000000000001e-05 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515512+0000", 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152442 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654409+0000", 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13889727499999999 2024-04-25T15:54:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666644+0000", 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012234584999999999 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666730+0000", 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5793999999999999e-05 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666786+0000", 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6054000000000002e-05 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3922 3.d 3:b3f0d00e:::100000001f8.0000014e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.591987249, 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.150212927, 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3922 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518670+0000", 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1039999999999998e-06 2024-04-25T15:54:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529034+0000", 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010364237 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529073+0000", 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8407e-05 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529092+0000", 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9411000000000001e-05 2024-04-25T15:54:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529203+0000", 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011113 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529330+0000", 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012611499999999999 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529517+0000", 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187639 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667032+0000", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13751481300000001 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668779+0000", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017469740000000001 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668835+0000", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6251000000000001e-05 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668880+0000", 2024-04-25T15:54:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4845999999999997e-05 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3920 3.26 e97/18 3:6422fb70:::100000001f8.0000014c:head v 97'93, mlcod=96'91)", 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.586964101, 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14317563999999999, 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523692+0000", 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8050000000000001e-06 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525897+0000", 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022045290000000002 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525923+0000", 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6225e-05 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525941+0000", 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8304e-05 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526045+0000", 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104158 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526076+0000", 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0454e-05 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666788+0000", 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140712106 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666866+0000", 2024-04-25T15:54:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8059000000000006e-05 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3926 3.d 3:b20dc027:::100000001f8.00000152:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.572477554, 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131821096, 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3926 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538180+0000", 2024-04-25T15:54:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4690000000000002e-06 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542940+0000", 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047600250000000002 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542988+0000", 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7633999999999998e-05 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543006+0000", 2024-04-25T15:54:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8114999999999999e-05 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543103+0000", 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6723000000000002e-05 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543283+0000", 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018030100000000001 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543476+0000", 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019295200000000001 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668734+0000", 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12525763200000001 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669901+0000", 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011670890000000001 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669955+0000", 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4373999999999998e-05 2024-04-25T15:54:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669998+0000", 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3971 3.3f 3:febcb5ea:::100000001f8.0000017f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2929537980000001, 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15452237999999999, 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3971 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817703+0000", 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822518+0000", 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048147939999999998 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822563+0000", 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4515000000000003e-05 2024-04-25T15:54:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822577+0000", 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.432e-05 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822664+0000", 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7440000000000003e-05 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822786+0000", 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012110800000000001 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822960+0000", 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017468900000000001 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945290+0000", 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122329907 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972057+0000", 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026767102000000001 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972139+0000", 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2070000000000005e-05 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972223+0000", 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3565000000000004e-05 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3970 3.16 e97/18 3:694e1180:::100000001f8.0000017e:head v 97'101, mlcod=96'99)", 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.818899+0000", 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2917557239999999, 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12624318300000001, 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818899+0000", 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818899+0000", 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818900+0000", 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7740000000000001e-06 2024-04-25T15:54:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821578+0000", 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026776360000000002 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821603+0000", 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4553999999999999e-05 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821625+0000", 2024-04-25T15:54:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2476e-05 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821737+0000", 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011185599999999999 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821777+0000", 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0058000000000002e-05 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945064+0000", 2024-04-25T15:54:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12328665699999999 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945142+0000", 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8171999999999996e-05 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:05.216 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:05.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:54:05.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:54:05.218 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:05.217+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:05.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:54:05.319 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:05.317+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:05.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.413+0000 7f117dbba640 1 Processor -- start 2024-04-25T15:54:05.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.413+0000 7f117dbba640 1 -- start start 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f117dbba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f117806c640 0x7f117815b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f117dbba640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1178074370 0x7f1178074780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f117dbba640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11780c92d0 con 0x7f11780c7fc0 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f117dbba640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f117815b560 con 0x7f1178074370 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f117dbba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f117806e210 con 0x7f117806c640 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f11777fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f117806c640 0x7f117815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f1177fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f11780c7fc0 0x7f117815dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:51824/0 (socket says 172.21.15.67:51824) 2024-04-25T15:54:05.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f1177fff640 1 -- 172.21.15.67:0/3414566321 learned_addr learned my addr 172.21.15.67:0/3414566321 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:05.421 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:05.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f1176ffd640 1 --2- 172.21.15.67:0/3414566321 >> v2:172.21.15.176:3300/0 conn(0x7f1178074370 0x7f1178074780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f11767fc640 1 -- 172.21.15.67:0/3414566321 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 137456446 0 0) 0x7f11780c92d0 con 0x7f11780c7fc0 2024-04-25T15:54:05.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.417+0000 7f11767fc640 1 -- 172.21.15.67:0/3414566321 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1164003530 con 0x7f11780c7fc0 2024-04-25T15:54:05.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.421+0000 7f11777fe640 1 -- 172.21.15.67:0/3414566321 >> v2:172.21.15.176:3300/0 conn(0x7f1178074370 msgr2=0x7f1178074780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:05.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.421+0000 7f11777fe640 1 --2- 172.21.15.67:0/3414566321 >> v2:172.21.15.176:3300/0 conn(0x7f1178074370 0x7f1178074780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.421+0000 7f11777fe640 1 -- 172.21.15.67:0/3414566321 >> v1:172.21.15.176:6789/0 conn(0x7f11780c7fc0 legacy=0x7f117815dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.421+0000 7f11777fe640 1 -- 172.21.15.67:0/3414566321 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117806e4d0 con 0x7f117806c640 2024-04-25T15:54:05.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.421+0000 7f1176ffd640 1 --2- 172.21.15.67:0/3414566321 >> v2:172.21.15.176:3300/0 conn(0x7f1178074370 0x7f1178074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:05.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.421+0000 7f11777fe640 1 --2- 172.21.15.67:0/3414566321 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f117806c640 0x7f117815b020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1168009d90 tx=0x7f116801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6a20c873744c4ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f11767fc640 1 -- 172.21.15.67:0/3414566321 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11680b4070 con 0x7f117806c640 2024-04-25T15:54:05.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f11767fc640 1 -- 172.21.15.67:0/3414566321 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11680af600 con 0x7f117806c640 2024-04-25T15:54:05.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f11767fc640 1 -- 172.21.15.67:0/3414566321 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11680af9d0 con 0x7f117806c640 2024-04-25T15:54:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 -- 172.21.15.67:0/3414566321 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f117806c640 msgr2=0x7f117815b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 --2- 172.21.15.67:0/3414566321 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f117806c640 0x7f117815b020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1168009d90 tx=0x7f116801f150 comp rx=0 tx=0).stop 2024-04-25T15:54:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 -- 172.21.15.67:0/3414566321 shutdown_connections 2024-04-25T15:54:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 --2- 172.21.15.67:0/3414566321 >> v2:172.21.15.176:3300/0 conn(0x7f1178074370 0x7f1178074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 --2- 172.21.15.67:0/3414566321 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f117806c640 0x7f117815b020 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 -- 172.21.15.67:0/3414566321 >> 172.21.15.67:0/3414566321 conn(0x7f1178070db0 msgr2=0x7f11780ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 -- 172.21.15.67:0/3414566321 shutdown_connections 2024-04-25T15:54:05.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.425+0000 7f117dbba640 1 -- 172.21.15.67:0/3414566321 wait complete. 2024-04-25T15:54:05.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.429+0000 7f117dbba640 1 Processor -- start 2024-04-25T15:54:05.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.429+0000 7f117dbba640 1 -- start start 2024-04-25T15:54:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178074370 0x7f11780c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11780c7fc0 0x7f1178155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11781474c0 0x7f11781567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1178073b70 con 0x7f1178074370 2024-04-25T15:54:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1178073890 con 0x7f11781474c0 2024-04-25T15:54:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1178073e50 con 0x7f11780c7fc0 2024-04-25T15:54:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178074370 0x7f11780c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178074370 0x7f11780c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59864/0 (socket says 172.21.15.67:59864) 2024-04-25T15:54:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 -- 172.21.15.67:0/1817366024 learned_addr learned my addr 172.21.15.67:0/1817366024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f1177fff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11781474c0 0x7f11781567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f1176ffd640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11780c7fc0 0x7f1178155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11780c7fc0 msgr2=0x7f1178155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11780c7fc0 0x7f1178155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11781474c0 msgr2=0x7f11781567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11781474c0 0x7f11781567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11780c5ee0 con 0x7f1178074370 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178074370 0x7f11780c74c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f116801bd60 tx=0x7f116801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11680c0020 con 0x7f1178074370 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11680ba470 con 0x7f1178074370 2024-04-25T15:54:05.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11680b4040 con 0x7f1178074370 2024-04-25T15:54:05.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1178142200 con 0x7f1178074370 2024-04-25T15:54:05.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.433+0000 7f117dbba640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11781470a0 con 0x7f1178074370 2024-04-25T15:54:05.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864cba3640 1 Processor -- start 2024-04-25T15:54:05.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864cba3640 1 -- start start 2024-04-25T15:54:05.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f11680c3070 con 0x7f1178074370 2024-04-25T15:54:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864cba3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 0x7f86480c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864cba3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f86480c6140 0x7f86480c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864cba3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86480c7340 con 0x7f8648159520 2024-04-25T15:54:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864cba3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f86480c67c0 con 0x7f86480c6140 2024-04-25T15:54:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864cba3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86480c6aa0 con 0x7f86480c4760 2024-04-25T15:54:05.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f117dbba640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f117806a480 con 0x7f1178074370 2024-04-25T15:54:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f8648159520 0x7f8648073bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:51836/0 (socket says 172.21.15.67:51836) 2024-04-25T15:54:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864659c640 1 -- 172.21.15.67:0/1135004951 learned_addr learned my addr 172.21.15.67:0/1135004951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f116ffff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f115806f430 0x7f1158071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5973+0+0 (secure 0 0 0) 0x7f116812cd60 con 0x7f1178074370 2024-04-25T15:54:05.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v97) ==== 54+0+16 (secure 0 0 0) 0x7f1168018cb0 con 0x7f1178074370 2024-04-25T15:54:05.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235907480 0 0) 0x7f86480c7340 con 0x7f8648159520 2024-04-25T15:54:05.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f8646d9d640 1 --2- 172.21.15.67:0/1135004951 >> v2:172.21.15.67:3300/0 conn(0x7f86480c6140 0x7f86480c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8630003530 con 0x7f8648159520 2024-04-25T15:54:05.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f8645d9b640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 0x7f86480c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f1176ffd640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f115806f430 0x7f1158071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1150001660 0x7f1150003b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3032154421 0 0) 0x7f8630003530 con 0x7f8648159520 2024-04-25T15:54:05.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 msgr2=0x7f86480c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:05.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f864559a640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 0x7f86480c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 >> v2:172.21.15.67:3300/0 conn(0x7f86480c6140 msgr2=0x7f86480c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:05.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 --2- 172.21.15.67:0/1135004951 >> v2:172.21.15.67:3300/0 conn(0x7f86480c6140 0x7f86480c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1150006c10 con 0x7f1150001660 2024-04-25T15:54:05.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.437+0000 7f1177fff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1150001660 0x7f1150003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86480c6d80 con 0x7f8648159520 2024-04-25T15:54:05.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f863c001680 con 0x7f8648159520 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 >> v1:172.21.15.176:6789/0 conn(0x7f8648159520 legacy=0x7f8648073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86300039c0 0x7f8630003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 0x7f8630004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8630004370 0x7f8630004dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86480c67c0 con 0x7f8630004370 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86480c6aa0 con 0x7f86300039c0 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8630009be0 con 0x7f86480c4760 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8645d9b640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 0x7f8630004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f1176ffd640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f115806f430 0x7f1158071940 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1178070460 tx=0x7f1160009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8646d9d640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8630004370 0x7f8630004dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864659c640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86300039c0 0x7f8630003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f1177fff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1150001660 0x7f1150003b70 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8645d9b640 1 -- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86300039c0 msgr2=0x7f8630003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8645d9b640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86300039c0 0x7f8630003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8645d9b640 1 -- 172.21.15.67:0/1135004951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8630004370 msgr2=0x7f8630004dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8645d9b640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8630004370 0x7f8630004dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8645d9b640 1 -- 172.21.15.67:0/1135004951 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8630003530 con 0x7f86480c4760 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f8646d9d640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8630004370 0x7f8630004dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f864659c640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86300039c0 0x7f8630003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:05.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.441+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1150006c10 con 0x7f1150001660 2024-04-25T15:54:05.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.445+0000 7f8645d9b640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 0x7f8630004810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8638009ae0 tx=0x7f863801cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cfc5a1c83110e4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.445+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86380b2070 con 0x7f86480c4760 2024-04-25T15:54:05.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.445+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86380a5590 con 0x7f86480c4760 2024-04-25T15:54:05.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.445+0000 7f864559a640 1 -- 172.21.15.67:0/1135004951 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86380096b0 con 0x7f86480c4760 2024-04-25T15:54:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.449+0000 7f864cba3640 1 -- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 msgr2=0x7f8630004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.449+0000 7f864cba3640 1 --2- 172.21.15.67:0/1135004951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c4760 0x7f8630004810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8638009ae0 tx=0x7f863801cab0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.449+0000 7f8646d9d640 1 -- 172.21.15.67:0/1135004951 reap_dead start 2024-04-25T15:54:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.449+0000 7f864cba3640 1 -- 172.21.15.67:0/1135004951 shutdown_connections 2024-04-25T15:54:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.449+0000 7f864cba3640 1 -- 172.21.15.67:0/1135004951 >> 172.21.15.67:0/1135004951 conn(0x7f864806fe70 msgr2=0x7f86480cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:05.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.449+0000 7f864cba3640 1 -- 172.21.15.67:0/1135004951 shutdown_connections 2024-04-25T15:54:05.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.449+0000 7f864cba3640 1 -- 172.21.15.67:0/1135004951 wait complete. 2024-04-25T15:54:05.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 Processor -- start 2024-04-25T15:54:05.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 -- start start 2024-04-25T15:54:05.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86480c4760 0x7f864815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c7e70 0x7f86481453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86480c8280 0x7f86481458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86480c5c00 con 0x7f86480c4760 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86480736b0 con 0x7f86480c8280 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864cba3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86480c5d80 con 0x7f86480c7e70 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86480c4760 0x7f864815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86480c4760 0x7f864815c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59890/0 (socket says 172.21.15.67:59890) 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.453+0000 7f864659c640 1 -- 172.21.15.67:0/2442246573 learned_addr learned my addr 172.21.15.67:0/2442246573 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f8645d9b640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c7e70 0x7f86481453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f8646d9d640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86480c8280 0x7f86481458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864659c640 1 -- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c7e70 msgr2=0x7f86481453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864659c640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c7e70 0x7f86481453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864659c640 1 -- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86480c8280 msgr2=0x7f86481458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864659c640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86480c8280 0x7f86481458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864659c640 1 -- 172.21.15.67:0/2442246573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864815cc10 con 0x7f86480c4760 2024-04-25T15:54:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864659c640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86480c4760 0x7f864815c6d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f863c00e7e0 tx=0x7f863c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f8636ffd640 1 -- 172.21.15.67:0/2442246573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c00ee90 con 0x7f86480c4760 2024-04-25T15:54:05.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8648150160 con 0x7f86480c4760 2024-04-25T15:54:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f8636ffd640 1 -- 172.21.15.67:0/2442246573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f863c018070 con 0x7f86480c4760 2024-04-25T15:54:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f8636ffd640 1 -- 172.21.15.67:0/2442246573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c00cad0 con 0x7f86480c4760 2024-04-25T15:54:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8648150d00 con 0x7f86480c4760 2024-04-25T15:54:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.457+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864806a480 con 0x7f86480c4760 2024-04-25T15:54:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.461+0000 7f8636ffd640 1 -- 172.21.15.67:0/2442246573 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f863c00cc70 con 0x7f86480c4760 2024-04-25T15:54:05.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.461+0000 7f8636ffd640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f862806f5d0 0x7f8628071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.461+0000 7f8636ffd640 1 -- 172.21.15.67:0/2442246573 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5973+0+0 (secure 0 0 0) 0x7f863c08fc50 con 0x7f86480c4760 2024-04-25T15:54:05.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.465+0000 7f8636ffd640 1 -- 172.21.15.67:0/2442246573 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f863c05c3e0 con 0x7f86480c4760 2024-04-25T15:54:05.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.465+0000 7f8645d9b640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f862806f5d0 0x7f8628071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.465+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1150005cc0 con 0x7f1150001660 2024-04-25T15:54:05.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.465+0000 7f8645d9b640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f862806f5d0 0x7f8628071ae0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f86380af000 tx=0x7f86380046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.469+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1150005cc0 con 0x7f1150001660 2024-04-25T15:54:05.472 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.469+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11500041f0 0x7f1150004600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.469+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1150004c80 con 0x7f11500041f0 2024-04-25T15:54:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.473+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11500041f0 0x7f1150004600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.473+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11500041f0 0x7f1150004600 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.473+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1150004c80 con 0x7f11500041f0 2024-04-25T15:54:05.479 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:05.480 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:05.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.497+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f11500051f0 con 0x7f11500041f0 2024-04-25T15:54:05.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.497+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f11500051f0 con 0x7f11500041f0 2024-04-25T15:54:05.501 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.501+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f11500093d0 0x7f115000b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.501+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f115000e770 con 0x7f11500093d0 2024-04-25T15:54:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.501+0000 7f1176ffd640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f11500093d0 0x7f115000b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.501+0000 7f1176ffd640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f11500093d0 0x7f115000b7f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.501+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f115000e770 con 0x7f11500093d0 2024-04-25T15:54:05.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:54:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.525+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f115000c110 con 0x7f11500093d0 2024-04-25T15:54:05.528 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:05.529+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:05.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.529+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f115000c110 con 0x7f11500093d0 2024-04-25T15:54:05.532 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:05.532 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.529+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f115000c2c0 0x7f115000c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.529+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f115000cc90 con 0x7f115000c2c0 2024-04-25T15:54:05.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.533+0000 7f1177fff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f115000c2c0 0x7f115000c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.533+0000 7f1177fff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f115000c2c0 0x7f115000c6d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.533+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f115000cc90 con 0x7f115000c2c0 2024-04-25T15:54:05.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.561+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f115000d1a0 con 0x7f115000c2c0 2024-04-25T15:54:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.565+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f115000d1a0 con 0x7f115000c2c0 2024-04-25T15:54:05.565 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.565+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f11500109c0 0x7f1150012de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.565+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1150015bb0 con 0x7f11500109c0 2024-04-25T15:54:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.565+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f11500109c0 0x7f1150012de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.565+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f11500109c0 0x7f1150012de0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.565+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1150015bb0 con 0x7f11500109c0 2024-04-25T15:54:05.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.585+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1150013570 con 0x7f11500109c0 2024-04-25T15:54:05.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.589+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1150013570 con 0x7f11500109c0 2024-04-25T15:54:05.591 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:05.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.589+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1150013830 0x7f1150015d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.589+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1150018da0 con 0x7f1150013830 2024-04-25T15:54:05.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.589+0000 7f1176ffd640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1150013830 0x7f1150015d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.593+0000 7f1176ffd640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1150013830 0x7f1150015d00 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.593+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1150018da0 con 0x7f1150013830 2024-04-25T15:54:05.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.613+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1150017ac0 con 0x7f1150013830 2024-04-25T15:54:05.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.617+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1150017ac0 con 0x7f1150013830 2024-04-25T15:54:05.618 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:05.619 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:05.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.617+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f11500181e0 0x7f11500185f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.617+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f115001c1f0 con 0x7f11500181e0 2024-04-25T15:54:05.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.617+0000 7f1177fff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f11500181e0 0x7f11500185f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.617+0000 7f1177fff640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f11500181e0 0x7f11500185f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.621 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:05.618+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:05.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.621+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f115001c1f0 con 0x7f11500181e0 2024-04-25T15:54:05.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.641+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f115001b070 con 0x7f11500181e0 2024-04-25T15:54:05.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.645+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f115001b070 con 0x7f11500181e0 2024-04-25T15:54:05.648 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:05.649 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:05.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.649+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f115001b6f0 0x7f115001bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:05.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.649+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f115001f610 con 0x7f115001b6f0 2024-04-25T15:54:05.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.649+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f115001b6f0 0x7f115001bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:05.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.649+0000 7f11777fe640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f115001b6f0 0x7f115001bb20 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:05.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.649+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f115001f610 con 0x7f115001b6f0 2024-04-25T15:54:05.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.673+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1150016380 con 0x7f115001b6f0 2024-04-25T15:54:05.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.677+0000 7f116ffff640 1 -- 172.21.15.67:0/1817366024 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1150016380 con 0x7f115001b6f0 2024-04-25T15:54:05.681 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1150001660 msgr2=0x7f1150003b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1150001660 0x7f1150003b70 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11500041f0 msgr2=0x7f1150004600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f11500041f0 0x7f1150004600 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f11500093d0 msgr2=0x7f115000b7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f11500093d0 0x7f115000b7f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f1177fff640 1 -- 172.21.15.67:0/1817366024 reap_dead start 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f115000c2c0 msgr2=0x7f115000c6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f115000c2c0 0x7f115000c6d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f11500109c0 msgr2=0x7f1150012de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f11500109c0 0x7f1150012de0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1150013830 msgr2=0x7f1150015d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1150013830 0x7f1150015d00 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f11500181e0 msgr2=0x7f11500185f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f11500181e0 0x7f11500185f0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f115001b6f0 msgr2=0x7f115001bb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f115001b6f0 0x7f115001bb20 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f1177fff640 1 -- 172.21.15.67:0/1817366024 reap_dead start 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f115806f430 msgr2=0x7f1158071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f115806f430 0x7f1158071940 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1178070460 tx=0x7f1160009190 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178074370 msgr2=0x7f11780c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178074370 0x7f11780c74c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f116801bd60 tx=0x7f116801bef0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 shutdown_connections 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f115806f430 0x7f1158071940 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 --2- 172.21.15.67:0/1817366024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178074370 0x7f11780c74c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 >> 172.21.15.67:0/1817366024 conn(0x7f1178070db0 msgr2=0x7f1178153440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:05.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 shutdown_connections 2024-04-25T15:54:05.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.681+0000 7f116dffb640 1 -- 172.21.15.67:0/1817366024 wait complete. 2024-04-25T15:54:05.722 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:05.718+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:05.768 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:05.768 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:05.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:05.785+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f86480c9bc0 con 0x7f86480c4760 2024-04-25T15:54:05.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:54:05.823 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:05.818+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:05.923 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:05.922+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:06.023 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:06.023 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:06.024 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:06.022+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:06.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:54:06.124 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:06.122+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:06.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.197+0000 7f8636ffd640 1 -- 172.21.15.67:0/2442246573 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v98) ==== 87+0+0 (secure 0 0 0) 0x7f863c0606e0 con 0x7f86480c4760 2024-04-25T15:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f862806f5d0 msgr2=0x7f8628071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:06.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f862806f5d0 0x7f8628071ae0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f86380af000 tx=0x7f86380046d0 comp rx=0 tx=0).stop 2024-04-25T15:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86480c4760 msgr2=0x7f864815c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86480c4760 0x7f864815c6d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f863c00e7e0 tx=0x7f863c00eca0 comp rx=0 tx=0).stop 2024-04-25T15:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 shutdown_connections 2024-04-25T15:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f862806f5d0 0x7f8628071ae0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86480c8280 0x7f86481458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:06.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86480c7e70 0x7f86481453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 --2- 172.21.15.67:0/2442246573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86480c4760 0x7f864815c6d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 >> 172.21.15.67:0/2442246573 conn(0x7f864806fe70 msgr2=0x7f86480c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 shutdown_connections 2024-04-25T15:54:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:06.201+0000 7f864cba3640 1 -- 172.21.15.67:0/2442246573 wait complete. 2024-04-25T15:54:06.225 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:06.222+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:06.279 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:06.279 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:06.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:54:06.357 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:06.357+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:06.427 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:06.425+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:06.526 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:06.526 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:06.527 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:06.525+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:06.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:54:06.628 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:06.626+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:06.729 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:06.726+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:06.823 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:06.823 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:06.830 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:06.829+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:06.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:54:06.931 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:06.930+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:07.032 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:07.030+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:07.107 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:07.107 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:07.133 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:07.133+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:07.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:54:07.234 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:07.233+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:07.334 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:07.330+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:07.407 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:07.408 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:07.435 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:07.434+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:07.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:54:07.504 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:54:07.504 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:54:07.504 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-25T15:54:07.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:54:07.569 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:07.569+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:07.637 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:07.637+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:07.692 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:07.692 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:07.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:54:07.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:54:07.738 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:07.734+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc2539a1640 1 Processor -- start 2024-04-25T15:54:07.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc2539a1640 1 -- start start 2024-04-25T15:54:07.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc2539a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c156330 0x7fc24c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc2539a1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc24c06fa10 con 0x7fc24c06cb80 2024-04-25T15:54:07.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc2539a1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc24c0ca8f0 con 0x7fc24c0c5100 2024-04-25T15:54:07.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc2539a1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c0ca6c0 con 0x7fc24c156330 2024-04-25T15:54:07.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc25173d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c156330 0x7fc24c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc25173d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c156330 0x7fc24c06c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:38722/0 (socket says 172.21.15.67:38722) 2024-04-25T15:54:07.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.741+0000 7fc25173d640 1 -- 172.21.15.67:0/3154398303 learned_addr learned my addr 172.21.15.67:0/3154398303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:07.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.745+0000 7fc25173d640 1 -- 172.21.15.67:0/3154398303 >> v1:172.21.15.176:6789/0 conn(0x7fc24c0c5100 legacy=0x7fc24c163630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:07.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.745+0000 7fc25173d640 1 -- 172.21.15.67:0/3154398303 >> v1:172.21.15.67:6789/0 conn(0x7fc24c06cb80 legacy=0x7fc24c0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:07.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.745+0000 7fc25173d640 1 -- 172.21.15.67:0/3154398303 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c0c3aa0 con 0x7fc24c156330 2024-04-25T15:54:07.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.745+0000 7fc25173d640 1 --2- 172.21.15.67:0/3154398303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c156330 0x7fc24c06c640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2480098a0 tx=0x7fc24801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9594b596d5eb27a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.745+0000 7fc243fff640 1 -- 172.21.15.67:0/3154398303 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2480b5070 con 0x7fc24c156330 2024-04-25T15:54:07.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.745+0000 7fc243fff640 1 -- 172.21.15.67:0/3154398303 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2480a8a30 con 0x7fc24c156330 2024-04-25T15:54:07.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.745+0000 7fc243fff640 1 -- 172.21.15.67:0/3154398303 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2480a8e00 con 0x7fc24c156330 2024-04-25T15:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.749+0000 7fc2539a1640 1 -- 172.21.15.67:0/3154398303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c156330 msgr2=0x7fc24c06c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.749+0000 7fc2539a1640 1 --2- 172.21.15.67:0/3154398303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c156330 0x7fc24c06c640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2480098a0 tx=0x7fc24801ed30 comp rx=0 tx=0).stop 2024-04-25T15:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.749+0000 7fc2539a1640 1 -- 172.21.15.67:0/3154398303 shutdown_connections 2024-04-25T15:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.749+0000 7fc2539a1640 1 --2- 172.21.15.67:0/3154398303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c156330 0x7fc24c06c640 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.749+0000 7fc2539a1640 1 -- 172.21.15.67:0/3154398303 >> 172.21.15.67:0/3154398303 conn(0x7fc24c0ce1c0 msgr2=0x7fc24c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.749+0000 7fc2539a1640 1 -- 172.21.15.67:0/3154398303 shutdown_connections 2024-04-25T15:54:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.749+0000 7fc2539a1640 1 -- 172.21.15.67:0/3154398303 wait complete. 2024-04-25T15:54:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 Processor -- start 2024-04-25T15:54:07.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 -- start start 2024-04-25T15:54:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c06cb80 0x7fc24c1ef680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc24c0c5100 0x7fc24c0c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc24c156330 0x7fc24c0b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c0c43d0 con 0x7fc24c0c5100 2024-04-25T15:54:07.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c0c4060 con 0x7fc24c156330 2024-04-25T15:54:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc2539a1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c156740 con 0x7fc24c06cb80 2024-04-25T15:54:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc250f3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc24c0c5100 0x7fc24c0c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc250f3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc24c0c5100 0x7fc24c0c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59902/0 (socket says 172.21.15.67:59902) 2024-04-25T15:54:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc250f3c640 1 -- 172.21.15.67:0/4252223551 learned_addr learned my addr 172.21.15.67:0/4252223551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc25173d640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c06cb80 0x7fc24c1ef680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.753+0000 7fc251f3e640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc24c156330 0x7fc24c0b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc25173d640 1 -- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc24c156330 msgr2=0x7fc24c0b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc25173d640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc24c156330 0x7fc24c0b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc25173d640 1 -- 172.21.15.67:0/4252223551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc24c0c5100 msgr2=0x7fc24c0c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc25173d640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc24c0c5100 0x7fc24c0c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc25173d640 1 -- 172.21.15.67:0/4252223551 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c1efbc0 con 0x7fc24c06cb80 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc250f3c640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc24c0c5100 0x7fc24c0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc25173d640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c06cb80 0x7fc24c1ef680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2480af000 tx=0x7fc24801bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc241ffb640 1 -- 172.21.15.67:0/4252223551 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2480c0070 con 0x7fc24c06cb80 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc241ffb640 1 -- 172.21.15.67:0/4252223551 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2480b08f0 con 0x7fc24c06cb80 2024-04-25T15:54:07.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c0c7b90 con 0x7fc24c06cb80 2024-04-25T15:54:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc241ffb640 1 -- 172.21.15.67:0/4252223551 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2480b5040 con 0x7fc24c06cb80 2024-04-25T15:54:07.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c0c6160 con 0x7fc24c06cb80 2024-04-25T15:54:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.757+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc24c0c4690 con 0x7fc24c06cb80 2024-04-25T15:54:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.761+0000 7fc241ffb640 1 -- 172.21.15.67:0/4252223551 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc2480c3070 con 0x7fc24c06cb80 2024-04-25T15:54:07.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.761+0000 7fc241ffb640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc23006f430 0x7fc230071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.761+0000 7fc241ffb640 1 -- 172.21.15.67:0/4252223551 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(98..98 src has 1..98) ==== 5973+0+0 (secure 0 0 0) 0x7fc24812c1d0 con 0x7fc24c06cb80 2024-04-25T15:54:07.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.761+0000 7fc250f3c640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc23006f430 0x7fc230071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.761+0000 7fc250f3c640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc23006f430 0x7fc230071940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0b4650 tx=0x7fc23c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.769+0000 7fc241ffb640 1 -- 172.21.15.67:0/4252223551 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc24c1efbc0 con 0x7fc24c06cb80 2024-04-25T15:54:07.839 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:07.837+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:07.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f15e13640 1 Processor -- start 2024-04-25T15:54:07.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f15e13640 1 -- start start 2024-04-25T15:54:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f15e13640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f1006d460 0x7f6f100736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f15e13640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f100caa60 con 0x7f6f1015aaf0 2024-04-25T15:54:07.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6f1015aaf0 0x7f6f1015af00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36812/0 (socket says 172.21.15.67:36812) 2024-04-25T15:54:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0ffff640 1 -- 172.21.15.67:0/323338585 learned_addr learned my addr 172.21.15.67:0/323338585 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f15e13640 1 -- 172.21.15.67:0/323338585 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f1015a5f0 con 0x7f6f101547e0 2024-04-25T15:54:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f15e13640 1 -- 172.21.15.67:0/323338585 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6f10068590 con 0x7f6f1006d460 2024-04-25T15:54:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3706626879 0 0) 0x7f6f100caa60 con 0x7f6f1015aaf0 2024-04-25T15:54:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ef8003530 con 0x7f6f1015aaf0 2024-04-25T15:54:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f1006d460 0x7f6f100736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3352850430 0 0) 0x7f6ef8003530 con 0x7f6f1015aaf0 2024-04-25T15:54:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f1006d460 msgr2=0x7f6f100736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f1006d460 0x7f6f100736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 >> v1:172.21.15.176:6789/0 conn(0x7f6f101547e0 legacy=0x7f6f1006c640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f100ca100 con 0x7f6f1015aaf0 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f6f08001a00 con 0x7f6f1015aaf0 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 >> v1:172.21.15.67:6789/0 conn(0x7f6f1015aaf0 legacy=0x7f6f1015af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef80039f0 0x7f6ef8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef8004810 0x7f6ef8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f1006d460 0x7f6ef8005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f10068590 con 0x7f6ef80039f0 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef800bb90 con 0x7f6ef8004810 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef800bd90 con 0x7f6f1006d460 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.925+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef80039f0 0x7f6ef8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.929+0000 7f6f0f7fe640 1 -- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f1006d460 msgr2=0x7f6ef8005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.929+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f1006d460 0x7f6ef8005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.929+0000 7f6f0f7fe640 1 -- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef8004810 msgr2=0x7f6ef8004c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:54:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.929+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef8004810 0x7f6ef8004c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.929+0000 7f6f0f7fe640 1 -- 172.21.15.67:0/323338585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef8003530 con 0x7f6ef80039f0 2024-04-25T15:54:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.929+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef80039f0 0x7f6ef8003e40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000ba00 tx=0x7f6f0001c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41666d7e51dbce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.929+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000b600 con 0x7f6ef80039f0 2024-04-25T15:54:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f000b1070 con 0x7f6ef80039f0 2024-04-25T15:54:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f0e7fc640 1 -- 172.21.15.67:0/323338585 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f000ac830 con 0x7f6ef80039f0 2024-04-25T15:54:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 -- 172.21.15.67:0/323338585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef80039f0 msgr2=0x7f6ef8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 --2- 172.21.15.67:0/323338585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ef80039f0 0x7f6ef8003e40 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000ba00 tx=0x7f6f0001c670 comp rx=0 tx=0).stop 2024-04-25T15:54:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f0ffff640 1 -- 172.21.15.67:0/323338585 reap_dead start 2024-04-25T15:54:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 -- 172.21.15.67:0/323338585 shutdown_connections 2024-04-25T15:54:07.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 -- 172.21.15.67:0/323338585 >> 172.21.15.67:0/323338585 conn(0x7f6f100ce1c0 msgr2=0x7f6f100ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:07.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 -- 172.21.15.67:0/323338585 shutdown_connections 2024-04-25T15:54:07.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 -- 172.21.15.67:0/323338585 wait complete. 2024-04-25T15:54:07.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 Processor -- start 2024-04-25T15:54:07.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.933+0000 7f6f15e13640 1 -- start start 2024-04-25T15:54:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef80039f0 0x7f6f10155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f101f6590 0x7f6f100b2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f100c4cc0 0x7f6f100c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f10154d70 con 0x7f6f101f6590 2024-04-25T15:54:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6f10154bf0 con 0x7f6ef80039f0 2024-04-25T15:54:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1006cd50 con 0x7f6f100c4cc0 2024-04-25T15:54:07.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f101f6590 0x7f6f100b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f101f6590 0x7f6f100b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59922/0 (socket says 172.21.15.67:59922) 2024-04-25T15:54:07.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0ffff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f100c4cc0 0x7f6f100c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0f7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef80039f0 0x7f6f10155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 -- 172.21.15.67:0/3465341508 learned_addr learned my addr 172.21.15.67:0/3465341508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:07.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f100c4cc0 msgr2=0x7f6f100c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:07.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f100c4cc0 0x7f6f100c3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef80039f0 msgr2=0x7f6f10155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef80039f0 0x7f6f10155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f100b2dd0 con 0x7f6f101f6590 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ef80039f0 0x7f6f10155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0ffff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f100c4cc0 0x7f6f100c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f0effd640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f101f6590 0x7f6f100b2890 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b3f0 tx=0x7f6f0400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04015020 con 0x7f6f101f6590 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f100b1780 con 0x7f6f101f6590 2024-04-25T15:54:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f100b1d40 con 0x7f6f101f6590 2024-04-25T15:54:07.941 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:07.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f04011830 con 0x7f6f101f6590 2024-04-25T15:54:07.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.937+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04011c20 con 0x7f6f101f6590 2024-04-25T15:54:07.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6f1006a480 con 0x7f6f101f6590 2024-04-25T15:54:07.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6f04018020 con 0x7f6f101f6590 2024-04-25T15:54:07.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6ef7fff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6efc06f400 0x7f6efc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5973+0+0 (secure 0 0 0) 0x7f6f0408f340 con 0x7f6f101f6590 2024-04-25T15:54:07.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v98) ==== 54+0+16 (secure 0 0 0) 0x7f6f0401f020 con 0x7f6f101f6590 2024-04-25T15:54:07.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6efc06f400 0x7f6efc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6f10165580 0x7f6f10165990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f100b24e0 con 0x7f6f10165580 2024-04-25T15:54:07.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.941+0000 7f6f0ffff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6f10165580 0x7f6f10165990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.945+0000 7f6f0ffff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6f10165580 0x7f6f10165990 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.945+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6efc06f400 0x7f6efc071910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b0000 tx=0x7f6f000046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.945+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f100b24e0 con 0x7f6f10165580 2024-04-25T15:54:07.965 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:07.965 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:07.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.965+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f1015b1e0 con 0x7f6f10165580 2024-04-25T15:54:07.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.969+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f1015b1e0 con 0x7f6f10165580 2024-04-25T15:54:07.970 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.969+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 0x7f6f101f9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.969+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6f1014d530 con 0x7f6f1014d120 2024-04-25T15:54:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.969+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 0x7f6f101f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.969+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 0x7f6f101f9080 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.969+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f1014d530 con 0x7f6f1014d120 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f101f95c0 con 0x7f6f1014d120 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6f0f7fe640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 msgr2=0x7f6f101f9080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 22 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6f0f7fe640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 msgr2=0x7f6f101f9080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 0x7f6f101f9080 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 0x7f6f101f9080 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 msgr2=0x7f6f101f9080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6ef7fff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6f1014d120 0x7f6f101f9080 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6ef7fff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6efc0794b0 0x7f6efc07b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f1014d530 con 0x7f6efc0794b0 2024-04-25T15:54:07.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=99}) -- 0x7f6efc07be30 con 0x7f6f101f6590 2024-04-25T15:54:07.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6efc0794b0 0x7f6efc07b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.993+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6efc0794b0 0x7f6efc07b8d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:54:07.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.997+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f1014d530 con 0x7f6efc0794b0 2024-04-25T15:54:07.997 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.997+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f6f10063090 0x7f6f100634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.997+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6f101f9ce0 con 0x7f6f10063090 2024-04-25T15:54:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.997+0000 7f6f0effd640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f6f10063090 0x7f6f100634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:07.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.997+0000 7f6f0effd640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f6f10063090 0x7f6f100634a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:07.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:07.997+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f101f9ce0 con 0x7f6f10063090 2024-04-25T15:54:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.017+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fc24c06a480 con 0x7fc24c06cb80 2024-04-25T15:54:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.017+0000 7fc241ffb640 1 -- 172.21.15.67:0/4252223551 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v98) ==== 81+0+35666 (secure 0 0 0) 0x7fc248018890 con 0x7fc24c06cb80 2024-04-25T15:54:08.021 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:08.021 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:08.021 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 98, 2024-04-25T15:54:08.021 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:54:08.021 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:54:08.021 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:54:06.178294+0000", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:53:14.990625+0000", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5801984, 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:54:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:54:08.023 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:54:08.024 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:54:08.025 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:54:08.026 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:54:08.027 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:54:08.028 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:54:08.029 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:54:08.030 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:54:08.031 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:54:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:54:08.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.75, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.75, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:54:08.034 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:54:08.035 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:54:08.036 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:54:08.037 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:54:08.038 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:54:08.039 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.4299999475479126, 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.4299999475479126, 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.6299999952316284, 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.6299999952316284, 2024-04-25T15:54:08.040 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.041 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.042 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.046 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0.2953948974609375, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.047 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.048 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.049 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.051 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 73, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.053 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.054 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.055 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:54:08.056 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:54:08.057 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:54:08.058 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.060 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.061 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 92, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.063 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:54:08.064 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:54:08.065 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.066 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.067 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.069 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:54:08.071 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.072 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:54:08.074 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.075 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:54:08.076 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 85, 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:54:08.078 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.079 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.080 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:54:08.081 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.082 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:54:08.083 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:54:08.084 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:54:08.086 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.18", 2024-04-25T15:54:08.087 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: 4 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.27", 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: 7, 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: 6 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.3", 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:08.088 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 0, 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:54:08.089 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.090 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.3e", 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.091 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 0, 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:54:08.092 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.093 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc23006f430 msgr2=0x7fc230071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc23006f430 0x7fc230071940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0b4650 tx=0x7fc23c009040 comp rx=0 tx=0).stop 2024-04-25T15:54:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c06cb80 msgr2=0x7fc24c1ef680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c06cb80 0x7fc24c1ef680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2480af000 tx=0x7fc24801bad0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f10063cb0 con 0x7f6f10063090 2024-04-25T15:54:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 shutdown_connections 2024-04-25T15:54:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc23006f430 0x7fc230071940 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc24c156330 0x7fc24c0b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc24c0c5100 0x7fc24c0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 --2- 172.21.15.67:0/4252223551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc24c06cb80 0x7fc24c1ef680 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.025+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 >> 172.21.15.67:0/4252223551 conn(0x7fc24c0ce1c0 msgr2=0x7fc24c154510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 shutdown_connections 2024-04-25T15:54:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f10063cb0 con 0x7f6f10063090 2024-04-25T15:54:08.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7fc2539a1640 1 -- 172.21.15.67:0/4252223551 wait complete. 2024-04-25T15:54:08.096 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6f100cd320 0x7f6f100cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6f100cdc70 con 0x7f6f100cd320 2024-04-25T15:54:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7f6f0ffff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6f100cd320 0x7f6f100cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7f6f0ffff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6f100cd320 0x7f6f100cd730 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.029+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f100cdc70 con 0x7f6f100cd320 2024-04-25T15:54:08.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.049+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f1002cc30 con 0x7f6f100cd320 2024-04-25T15:54:08.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.053+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f1002cc30 con 0x7f6f100cd320 2024-04-25T15:54:08.098 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:08.099 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:08.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.053+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f6f1002d2b0 0x7f6f1002d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.053+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6f100cdc70 con 0x7f6f1002d2b0 2024-04-25T15:54:08.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.053+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f6f1002d2b0 0x7f6f1002d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.057+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f6f1002d2b0 0x7f6f1002d6c0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.057+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f100cdc70 con 0x7f6f1002d2b0 2024-04-25T15:54:08.101 INFO:tasks.thrashosds.thrasher:j is {'epoch': 98, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:54:06.178294+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:53:14.990625+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4299999475479126, 'score_stable': 1.4299999475479126, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 0.2953948974609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 85, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [1, 4]}, {'pgid': '3.27', 'osds': [7, 6]}], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '3.3e', 'mappings': [{'from': 0, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:54:08.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:54:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.085+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f100cdc70 con 0x7f6f1002d2b0 2024-04-25T15:54:08.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.089+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f100cdc70 con 0x7f6f1002d2b0 2024-04-25T15:54:08.102 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.089+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6f1002dc00 0x7f6f1020a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.089+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6f100cdc70 con 0x7f6f1002dc00 2024-04-25T15:54:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.089+0000 7f6f0effd640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6f1002dc00 0x7f6f1020a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.089+0000 7f6f0effd640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6f1002dc00 0x7f6f1020a910 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.093+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f100cdc70 con 0x7f6f1002dc00 2024-04-25T15:54:08.103 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:08.090+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:08.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.121+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f100cdc70 con 0x7f6f1002dc00 2024-04-25T15:54:08.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.125+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f100cdc70 con 0x7f6f1002dc00 2024-04-25T15:54:08.126 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:08.127 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:08.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.125+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f6f1020b360 0x7f6f1020d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.125+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6f10210550 con 0x7f6f1020b360 2024-04-25T15:54:08.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.125+0000 7f6f0ffff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f6f1020b360 0x7f6f1020d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.125+0000 7f6f0ffff640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f6f1020b360 0x7f6f1020d780 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.129+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f10210550 con 0x7f6f1020b360 2024-04-25T15:54:08.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.153+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f1020dcc0 con 0x7f6f1020b360 2024-04-25T15:54:08.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.157+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f1020dcc0 con 0x7f6f1020b360 2024-04-25T15:54:08.158 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.157+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6f1020e340 0x7f6f10210760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.157+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6f10213630 con 0x7f6f1020e340 2024-04-25T15:54:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.157+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6f1020e340 0x7f6f10210760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.161+0000 7f6f0f7fe640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6f1020e340 0x7f6f10210760 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.161+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f10213630 con 0x7f6f1020e340 2024-04-25T15:54:08.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.193+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6f10210ca0 con 0x7f6f1020e340 2024-04-25T15:54:08.197 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:08.194+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:08.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6ef7fff640 1 -- 172.21.15.67:0/3465341508 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6f10210ca0 con 0x7f6f1020e340 2024-04-25T15:54:08.199 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6f10165580 msgr2=0x7f6f10165990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f6f10165580 0x7f6f10165990 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6efc0794b0 msgr2=0x7f6efc07b8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f6efc0794b0 0x7f6efc07b8d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f6f10063090 msgr2=0x7f6f100634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f6f10063090 0x7f6f100634a0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6f100cd320 msgr2=0x7f6f100cd730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f6f100cd320 0x7f6f100cd730 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f6f1002d2b0 msgr2=0x7f6f1002d6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f6f1002d2b0 0x7f6f1002d6c0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6f1002dc00 msgr2=0x7f6f1020a910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f6f1002dc00 0x7f6f1020a910 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f6f1020b360 msgr2=0x7f6f1020d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f6f1020b360 0x7f6f1020d780 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6f1020e340 msgr2=0x7f6f10210760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f6f1020e340 0x7f6f10210760 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f0ffff640 1 -- 172.21.15.67:0/3465341508 reap_dead start 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f0ffff640 1 -- 172.21.15.67:0/3465341508 reap_dead start 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f0ffff640 1 -- 172.21.15.67:0/3465341508 reap_dead start 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f0ffff640 1 -- 172.21.15.67:0/3465341508 reap_dead start 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f0ffff640 1 -- 172.21.15.67:0/3465341508 reap_dead start 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6efc06f400 msgr2=0x7f6efc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6efc06f400 0x7f6efc071910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b0000 tx=0x7f6f000046d0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f101f6590 msgr2=0x7f6f100b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f101f6590 0x7f6f100b2890 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b3f0 tx=0x7f6f0400b8b0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 shutdown_connections 2024-04-25T15:54:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6efc06f400 0x7f6efc071910 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 --2- 172.21.15.67:0/3465341508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f101f6590 0x7f6f100b2890 secure :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b3f0 tx=0x7f6f0400b8b0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.197+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 >> 172.21.15.67:0/3465341508 conn(0x7f6f100ce1c0 msgr2=0x7f6f100ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.201+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 shutdown_connections 2024-04-25T15:54:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.205+0000 7f6f15e13640 1 -- 172.21.15.67:0/3465341508 wait complete. 2024-04-25T15:54:08.231 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:08.231 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:08.232 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:08.232 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:08.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:54:08.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:54:08.317 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:08.317+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:08.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.329+0000 7f66bde84640 1 Processor -- start 2024-04-25T15:54:08.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.329+0000 7f66bde84640 1 -- start start 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66bde84640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66b80c4010 0x7f66b80725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66bde84640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f66b815a0b0 0x7f66b80746a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66bde84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b80c3bc0 con 0x7f66b8153a00 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66bde84640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f66b8153e10 con 0x7f66b80c4010 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66bde84640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f66b8068c40 con 0x7f66b815a0b0 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66b6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66b80c4010 0x7f66b80725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66b6ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66b80c4010 0x7f66b80725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59932/0 (socket says 172.21.15.67:59932) 2024-04-25T15:54:08.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66b6ffd640 1 -- 172.21.15.67:0/3026414232 learned_addr learned my addr 172.21.15.67:0/3026414232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:08.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66b6ffd640 1 -- 172.21.15.67:0/3026414232 >> v2:172.21.15.176:3300/0 conn(0x7f66b815a0b0 msgr2=0x7f66b80746a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:08.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66b6ffd640 1 --2- 172.21.15.67:0/3026414232 >> v2:172.21.15.176:3300/0 conn(0x7f66b815a0b0 0x7f66b80746a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66b6ffd640 1 -- 172.21.15.67:0/3026414232 >> v1:172.21.15.176:6789/0 conn(0x7f66b8153a00 legacy=0x7f66b8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:08.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.333+0000 7f66b6ffd640 1 -- 172.21.15.67:0/3026414232 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b80c53a0 con 0x7f66b80c4010 2024-04-25T15:54:08.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.337+0000 7f66b6ffd640 1 --2- 172.21.15.67:0/3026414232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80c4010 0x7f66b80725c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f66a8006fd0 tx=0x7f66a801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e991b585175b7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.337+0000 7f66b67fc640 1 -- 172.21.15.67:0/3026414232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80b0070 con 0x7f66b80c4010 2024-04-25T15:54:08.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.337+0000 7f66b67fc640 1 -- 172.21.15.67:0/3026414232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a80ab600 con 0x7f66b80c4010 2024-04-25T15:54:08.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66b67fc640 1 -- 172.21.15.67:0/3026414232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80ab9d0 con 0x7f66b80c4010 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 -- 172.21.15.67:0/3026414232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80c4010 msgr2=0x7f66b80725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 --2- 172.21.15.67:0/3026414232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80c4010 0x7f66b80725c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f66a8006fd0 tx=0x7f66a801c4c0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 -- 172.21.15.67:0/3026414232 shutdown_connections 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 --2- 172.21.15.67:0/3026414232 >> v2:172.21.15.176:3300/0 conn(0x7f66b815a0b0 0x7f66b80746a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 --2- 172.21.15.67:0/3026414232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b80c4010 0x7f66b80725c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 -- 172.21.15.67:0/3026414232 >> 172.21.15.67:0/3026414232 conn(0x7f66b80ce1c0 msgr2=0x7f66b80ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 -- 172.21.15.67:0/3026414232 shutdown_connections 2024-04-25T15:54:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.345+0000 7f66bde84640 1 -- 172.21.15.67:0/3026414232 wait complete. 2024-04-25T15:54:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 Processor -- start 2024-04-25T15:54:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 -- start start 2024-04-25T15:54:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66b80c4010 0x7f66b80b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66b8153a00 0x7f66b80c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b815a0b0 0x7f66b814abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66b80686c0 con 0x7f66b815a0b0 2024-04-25T15:54:08.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f66b80c4420 con 0x7f66b8153a00 2024-04-25T15:54:08.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f66b8074be0 con 0x7f66b80c4010 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b815a0b0 0x7f66b814abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b815a0b0 0x7f66b814abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59940/0 (socket says 172.21.15.67:59940) 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 -- 172.21.15.67:0/1238575769 learned_addr learned my addr 172.21.15.67:0/1238575769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b6ffd640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66b8153a00 0x7f66b80c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b77fe640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66b80c4010 0x7f66b80b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 -- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66b80c4010 msgr2=0x7f66b80b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66b80c4010 0x7f66b80b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 -- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66b8153a00 msgr2=0x7f66b80c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66b8153a00 0x7f66b80c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 -- 172.21.15.67:0/1238575769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b8168190 con 0x7f66b815a0b0 2024-04-25T15:54:08.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b6ffd640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66b8153a00 0x7f66b80c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:08.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66b7fff640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b815a0b0 0x7f66b814abf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f66b000b9e0 tx=0x7f66b000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f669ffff640 1 -- 172.21.15.67:0/1238575769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0019070 con 0x7f66b815a0b0 2024-04-25T15:54:08.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f669ffff640 1 -- 172.21.15.67:0/1238575769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b000cb20 con 0x7f66b815a0b0 2024-04-25T15:54:08.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f669ffff640 1 -- 172.21.15.67:0/1238575769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0014440 con 0x7f66b815a0b0 2024-04-25T15:54:08.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.349+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b8168470 con 0x7f66b815a0b0 2024-04-25T15:54:08.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.353+0000 7f669ffff640 1 -- 172.21.15.67:0/1238575769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f66b001c070 con 0x7f66b815a0b0 2024-04-25T15:54:08.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.353+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b8166bb0 con 0x7f66b815a0b0 2024-04-25T15:54:08.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.353+0000 7f669ffff640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66a406f450 0x7f66a4071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.353+0000 7f669ffff640 1 -- 172.21.15.67:0/1238575769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5973+0+0 (secure 0 0 0) 0x7f66b0093170 con 0x7f66b815a0b0 2024-04-25T15:54:08.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.353+0000 7f66b77fe640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66a406f450 0x7f66a4071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.353+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b806a480 con 0x7f66b815a0b0 2024-04-25T15:54:08.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.357+0000 7f66b77fe640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66a406f450 0x7f66a4071960 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00a790 tx=0x7f66ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.361+0000 7f669ffff640 1 -- 172.21.15.67:0/1238575769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f66b005bce0 con 0x7f66b815a0b0 2024-04-25T15:54:08.398 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:08.394+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.453+0000 7fe2d9a63640 1 Processor -- start 2024-04-25T15:54:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.453+0000 7fe2d9a63640 1 -- start start 2024-04-25T15:54:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d9a63640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe2d415aaf0 0x7fe2d415af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d9a63640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d40caa60 con 0x7fe2d406d460 2024-04-25T15:54:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d9a63640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d415a5f0 con 0x7fe2d40736d0 2024-04-25T15:54:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d9a63640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d4068590 con 0x7fe2d415aaf0 2024-04-25T15:54:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe2d406d460 0x7fe2d41547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36818/0 (socket says 172.21.15.67:36818) 2024-04-25T15:54:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d37fe640 1 -- 172.21.15.67:0/2593246961 learned_addr learned my addr 172.21.15.67:0/2593246961 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:08.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d3fff640 1 --2- 172.21.15.67:0/2593246961 >> v2:172.21.15.176:3300/0 conn(0x7fe2d415aaf0 0x7fe2d415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798173396 0 0) 0x7fe2d40caa60 con 0x7fe2d406d460 2024-04-25T15:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2bc003530 con 0x7fe2d406d460 2024-04-25T15:54:08.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3928397230 0 0) 0x7fe2bc003530 con 0x7fe2d406d460 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 >> v2:172.21.15.176:3300/0 conn(0x7fe2d415aaf0 msgr2=0x7fe2d415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 --2- 172.21.15.67:0/2593246961 >> v2:172.21.15.176:3300/0 conn(0x7fe2d415aaf0 0x7fe2d415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 >> v1:172.21.15.176:6789/0 conn(0x7fe2d40736d0 legacy=0x7fe2d406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d40ca100 con 0x7fe2d406d460 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fe2c4001680 con 0x7fe2d406d460 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 >> v1:172.21.15.67:6789/0 conn(0x7fe2d406d460 legacy=0x7fe2d41547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0039c0 0x7fe2bc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2bc004810 0x7fe2bc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2bc0051d0 0x7fe2bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d4068590 con 0x7fe2bc0039c0 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe2bc00c170 con 0x7fe2bc0051d0 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe2bc00c370 con 0x7fe2bc004810 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d37fe640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0039c0 0x7fe2bc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d3fff640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2bc0051d0 0x7fe2bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d2ffd640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2bc004810 0x7fe2bc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d3fff640 1 -- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2bc004810 msgr2=0x7fe2bc004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d3fff640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2bc004810 0x7fe2bc004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d3fff640 1 -- 172.21.15.67:0/2593246961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0039c0 msgr2=0x7fe2bc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d3fff640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0039c0 0x7fe2bc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d3fff640 1 -- 172.21.15.67:0/2593246961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2bc003530 con 0x7fe2bc0051d0 2024-04-25T15:54:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.457+0000 7fe2d37fe640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0039c0 0x7fe2bc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.461+0000 7fe2d3fff640 1 -- 172.21.15.67:0/2593246961 reap_dead start 2024-04-25T15:54:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.461+0000 7fe2d3fff640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2bc0051d0 0x7fe2bc00b9d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009bd0 tx=0x7fe2cc020fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2928358f7908977f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.461+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0b6070 con 0x7fe2bc0051d0 2024-04-25T15:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.461+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc0a9cd0 con 0x7fe2bc0051d0 2024-04-25T15:54:08.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.461+0000 7fe2d27fc640 1 -- 172.21.15.67:0/2593246961 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0b6070 con 0x7fe2bc0051d0 2024-04-25T15:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.465+0000 7fe2d9a63640 1 -- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2bc0051d0 msgr2=0x7fe2bc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.465+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2bc0051d0 0x7fe2bc00b9d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009bd0 tx=0x7fe2cc020fd0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.465+0000 7fe2d9a63640 1 -- 172.21.15.67:0/2593246961 shutdown_connections 2024-04-25T15:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.465+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/2593246961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2bc0051d0 0x7fe2bc00b9d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.465+0000 7fe2d9a63640 1 -- 172.21.15.67:0/2593246961 >> 172.21.15.67:0/2593246961 conn(0x7fe2d40ce1c0 msgr2=0x7fe2d40ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.465+0000 7fe2d9a63640 1 -- 172.21.15.67:0/2593246961 shutdown_connections 2024-04-25T15:54:08.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.465+0000 7fe2d9a63640 1 -- 172.21.15.67:0/2593246961 wait complete. 2024-04-25T15:54:08.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.469+0000 7fe2d9a63640 1 Processor -- start 2024-04-25T15:54:08.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.469+0000 7fe2d9a63640 1 -- start start 2024-04-25T15:54:08.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d9a63640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0051d0 0x7fe2d41520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d9a63640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4153ed0 0x7fe2d41506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0051d0 0x7fe2d41520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0051d0 0x7fe2d41520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59958/0 (socket says 172.21.15.67:59958) 2024-04-25T15:54:08.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d9a63640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2d4145c30 0x7fe2d4146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d4154960 con 0x7fe2bc0051d0 2024-04-25T15:54:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 -- 172.21.15.67:0/3168127829 learned_addr learned my addr 172.21.15.67:0/3168127829 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d41547e0 con 0x7fe2d4153ed0 2024-04-25T15:54:08.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d4154ae0 con 0x7fe2d4145c30 2024-04-25T15:54:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d2ffd640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4153ed0 0x7fe2d41506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d3fff640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2d4145c30 0x7fe2d4146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 -- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2d4145c30 msgr2=0x7fe2d4146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2d4145c30 0x7fe2d4146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 -- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4153ed0 msgr2=0x7fe2d41506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4153ed0 0x7fe2d41506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d37fe640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d4152610 con 0x7fe2bc0051d0 2024-04-25T15:54:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.473+0000 7fe2d2ffd640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4153ed0 0x7fe2d41506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.477+0000 7fe2d37fe640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0051d0 0x7fe2d41520d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe2c40072c0 tx=0x7fe2c4007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.477+0000 7fe2bbfff640 1 -- 172.21.15.67:0/3168127829 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400f940 con 0x7fe2bc0051d0 2024-04-25T15:54:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.477+0000 7fe2bbfff640 1 -- 172.21.15.67:0/3168127829 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c4018070 con 0x7fe2bc0051d0 2024-04-25T15:54:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.477+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d414b180 con 0x7fe2bc0051d0 2024-04-25T15:54:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.477+0000 7fe2bbfff640 1 -- 172.21.15.67:0/3168127829 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400fce0 con 0x7fe2bc0051d0 2024-04-25T15:54:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.477+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d414b420 con 0x7fe2bc0051d0 2024-04-25T15:54:08.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.477+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d406a480 con 0x7fe2bc0051d0 2024-04-25T15:54:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.485+0000 7fe2bbfff640 1 -- 172.21.15.67:0/3168127829 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe2c401d460 con 0x7fe2bc0051d0 2024-04-25T15:54:08.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.485+0000 7fe2bbfff640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe2c006f400 0x7fe2c0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.485+0000 7fe2bbfff640 1 -- 172.21.15.67:0/3168127829 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5973+0+0 (secure 0 0 0) 0x7fe2c408f5e0 con 0x7fe2bc0051d0 2024-04-25T15:54:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.485+0000 7fe2d2ffd640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe2c006f400 0x7fe2c0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.485+0000 7fe2d2ffd640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe2c006f400 0x7fe2c0071910 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe2d414c070 tx=0x7fe2c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.500 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:08.497+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.501+0000 7fe2bbfff640 1 -- 172.21.15.67:0/3168127829 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe2c405c320 con 0x7fe2bc0051d0 2024-04-25T15:54:08.562 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:343 3.8 3:1115f3ca:::10000000002.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.554416304, 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.46089677899999998, 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 343 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006870+0000", 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9280000000000002e-06 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008363+0000", 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001493288 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008398+0000", 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5067000000000002e-05 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008416+0000", 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7782e-05 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008522+0000", 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106327 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008535+0000", 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2883e-05 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.387477+0000", 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.37894188200000001 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.392562+0000", 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050850870000000003 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445264+0000", 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052702037 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445382+0000", 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118433 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445603+0000", 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022077500000000001 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.465250+0000", 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019646976 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467658+0000", 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024073850000000002 2024-04-25T15:54:08.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467695+0000", 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7234999999999998e-05 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467763+0000", 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8694000000000003e-05 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:670 3.8 3:1323134e:::100000001f7.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:08.569 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.687346140999999, 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094982472999999998, 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 670 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873941+0000", 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0840000000000002e-06 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876488+0000", 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025470290000000001 2024-04-25T15:54:08.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876647+0000", 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015912000000000001 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876664+0000", 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7215e-05 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876782+0000", 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118082 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877222+0000", 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000439596 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877596+0000", 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037371700000000003 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.913106+0000", 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035509813000000001 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968792+0000", 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.055686739999999998 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968850+0000", 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8032000000000003e-05 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968919+0000", 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9045e-05 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:671 3.38 3:1c0639bf:::100000001f7.0000009b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.677181107000003, 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073197039000000005, 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 671 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884102+0000", 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.884106+0000", 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9709999999999998e-06 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886679+0000", 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025726770000000002 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886716+0000", 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.735e-05 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886733+0000", 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6697999999999998e-05 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.886843+0000", 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011074299999999999 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887031+0000", 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018804800000000001 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.887215+0000", 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018384599999999999 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.921326+0000", 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034110486000000002 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957180+0000", 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035854051999999997 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957227+0000", 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7290000000000003e-05 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.957299+0000", 2024-04-25T15:54:08.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1878000000000005e-05 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2313 3.1e 3:7a2a8d26:::100000001f8.00000505:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.696633769000002, 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091581081999999994, 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.580 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2313 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864653+0000", 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866695+0000", 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002042744 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866732+0000", 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7054000000000001e-05 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866765+0000", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2141999999999998e-05 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866853+0000", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7950999999999999e-05 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866986+0000", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013319800000000001 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.867189+0000", 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020333800000000001 2024-04-25T15:54:08.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.883954+0000", 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016764729999999999 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956056+0000", 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072101922999999998 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956116+0000", 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9815000000000002e-05 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956230+0000", 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114817 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2314 3.29 3:9452ed8f:::100000001f8.00000506:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.871348+0000", 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.689934681, 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.084828639999999997, 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2314 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.871348+0000", 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.871348+0000", 2024-04-25T15:54:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.871370+0000", 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1458e-05 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873408+0000", 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020381259999999999 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873460+0000", 2024-04-25T15:54:08.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2216999999999998e-05 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873478+0000", 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7481000000000001e-05 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873607+0000", 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012908599999999999 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.873821+0000", 2024-04-25T15:54:08.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000214322 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.874006+0000", 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018467099999999999 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.890192+0000", 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016186048000000001 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956045+0000", 2024-04-25T15:54:08.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.065853729999999999 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956112+0000", 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6701999999999999e-05 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956177+0000", 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4799000000000001e-05 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2315 3.19 3:99bca703:::100000001f8.00000507:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.879908+0000", 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.681375455000001, 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.076440714000000007, 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2315 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.879908+0000", 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.879908+0000", 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.879910+0000", 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.097e-06 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881780+0000", 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001870402 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881836+0000", 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5985e-05 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881855+0000", 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8909000000000001e-05 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.881937+0000", 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1948999999999998e-05 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.882115+0000", 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017834099999999999 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.882267+0000", 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151729 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.890850+0000", 2024-04-25T15:54:08.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0085832870000000002 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956251+0000", 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.065400729000000005 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956304+0000", 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3235000000000001e-05 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956348+0000", 2024-04-25T15:54:08.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4051000000000002e-05 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3207 3.7 3:e2e1ddc8:::100000001f8.00000883:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.813025+0000", 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.748258492, 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075993841000000006, 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.596 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3207 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.813025+0000", 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.813025+0000", 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.813026+0000", 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9099999999999999e-06 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.814571+0000", 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001544265 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.874682+0000", 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.060111285 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.874696+0000", 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4229e-05 2024-04-25T15:54:08.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.874788+0000", 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1318999999999998e-05 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.874911+0000", 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012380099999999999 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.875013+0000", 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000101334 2024-04-25T15:54:08.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.881978+0000", 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069657790000000001 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.888890+0000", 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069114190000000002 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.888972+0000", 2024-04-25T15:54:08.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1686000000000004e-05 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.889018+0000", 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6814000000000003e-05 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3921 3.29 3:95d78bcd:::100000001f8.0000014d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.0465482919999998, 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.156667944, 2024-04-25T15:54:08.601 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3921 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514737+0000", 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.029e-06 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517027+0000", 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022901739999999999 2024-04-25T15:54:08.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517155+0000", 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127763 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517189+0000", 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3847999999999998e-05 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517337+0000", 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014894999999999999 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517464+0000", 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126608 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517651+0000", 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018649000000000001 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.531319+0000", 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013667959 2024-04-25T15:54:08.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671222+0000", 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.139903946 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671326+0000", 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103785 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671403+0000", 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6391999999999996e-05 2024-04-25T15:54:08.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3935 3.23 3:c6f60763:::100000001f8.0000015b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.9752862230000003, 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091319014000000004, 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3935 2024-04-25T15:54:08.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.586000+0000", 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2789999999999999e-06 2024-04-25T15:54:08.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.587431+0000", 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00143063 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.662977+0000", 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075546586999999998 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663000+0000", 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2413999999999999e-05 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663082+0000", 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2659999999999998e-05 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663161+0000", 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8322e-05 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663262+0000", 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010089099999999999 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.674457+0000", 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011195544999999999 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677208+0000", 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027504769999999999 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677272+0000", 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4862000000000003e-05 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677316+0000", 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3959 3.7 3:e1bee111:::100000001f8.00000173:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7991542760000003, 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089007748999999997, 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3959 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762131+0000", 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0430000000000002e-06 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.764378+0000", 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022474560000000001 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786050+0000", 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021672166 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786065+0000", 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4652e-05 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786161+0000", 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5873999999999997e-05 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786278+0000", 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116684 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786462+0000", 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000184625 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.794936+0000", 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084740370000000002 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851010+0000", 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056073265999999997 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851079+0000", 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9556999999999998e-05 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851136+0000", 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7389000000000002e-05 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3963 3.19 3:9b5b1331:::100000001f8.00000177:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:08.617 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7806909400000004, 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.167075797, 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3963 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780596+0000", 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8079999999999998e-06 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786013+0000", 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054168410000000004 2024-04-25T15:54:08.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800764+0000", 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014751306 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800778+0000", 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4171999999999999e-05 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800924+0000", 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146286 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.801019+0000", 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4456000000000005e-05 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.801147+0000", 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127603 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.809771+0000", 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008624095 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947582+0000", 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13781142499999999 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947625+0000", 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2928e-05 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947668+0000", 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2877000000000003e-05 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3967 3.24 3:260ecc6d:::100000001f8.0000017b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.7622530990000005, 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18196963599999999, 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3967 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799032+0000", 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4040000000000002e-06 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800724+0000", 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001691889 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861343+0000", 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.060618914000000003 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861359+0000", 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6294999999999999e-05 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861444+0000", 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4251000000000004e-05 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861517+0000", 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3098999999999999e-05 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861617+0000", 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100603 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.878626+0000", 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017008836999999999 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980913+0000", 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.102287103 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980951+0000", 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7718e-05 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.981000+0000", 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8523000000000003e-05 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3980 3.8 3:13e1c2d8:::100000001f8.00000188:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.701379695, 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.111276876, 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3980 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859907+0000", 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7950000000000001e-06 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861306+0000", 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001398505 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893810+0000", 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032503907999999998 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893829+0000", 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9860999999999998e-05 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893916+0000", 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6890000000000003e-05 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.894052+0000", 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013542200000000001 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.894230+0000", 2024-04-25T15:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017861600000000001 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.902225+0000", 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079943059999999996 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971087+0000", 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068862265000000006 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971132+0000", 2024-04-25T15:54:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4903e-05 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971180+0000", 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8405000000000002e-05 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3987 3.1e 3:78ac7851:::100000001f8.0000018f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.892153+0000", 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.6691298200000002, 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088476949999999999, 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.633 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3987 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.892153+0000", 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.892153+0000", 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.892155+0000", 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.81e-06 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893771+0000", 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016161470000000001 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.909951+0000", 2024-04-25T15:54:08.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016180011000000001 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.909968+0000", 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6951000000000001e-05 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.910063+0000", 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4857000000000003e-05 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.910185+0000", 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012222200000000001 2024-04-25T15:54:08.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.910355+0000", 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000170137 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.920896+0000", 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010540570000000001 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980500+0000", 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.059603942 2024-04-25T15:54:08.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980570+0000", 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0372999999999999e-05 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980630+0000", 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9929999999999997e-05 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.638 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3990 3.19 3:9b35cac6:::100000001f8.00000192:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.6530237809999999, 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.093327906000000002, 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3990 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:08.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908262+0000", 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.258e-06 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.909909+0000", 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016468979999999999 2024-04-25T15:54:08.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987236+0000", 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.077326763000000007 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987250+0000", 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4248e-05 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987323+0000", 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2416000000000006e-05 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987413+0000", 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0482999999999994e-05 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987616+0000", 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020251899999999999 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.999192+0000", 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011576229 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001500+0000", 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023079099999999998 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001546+0000", 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6508999999999998e-05 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001587+0000", 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0673000000000002e-05 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.643 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4024 3.24 3:268a62b6:::100000001f8.000001b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4313267029999999, 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.101232684, 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4024 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.644 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129960+0000", 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1949999999999999e-06 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.131869+0000", 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001909151 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208635+0000", 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.076766361000000005 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208652+0000", 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7093999999999999e-05 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208741+0000", 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.899e-05 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208831+0000", 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9621000000000001e-05 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208947+0000", 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115788 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.225235+0000", 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016288894000000002 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.230862+0000", 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056270290000000004 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.230947+0000", 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5019999999999996e-05 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.231189+0000", 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024154100000000001 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4038 3.19 3:986f6147:::100000001f8.000001c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.206503+0000", 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3547803009999999, 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.08068475, 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.649 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4038 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.206503+0000", 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.206503+0000", 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.206505+0000", 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2409999999999998e-06 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208596+0000", 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002090668 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262189+0000", 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053593712000000002 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262208+0000", 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8349e-05 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262299+0000", 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1749000000000005e-05 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262410+0000", 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110189 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262560+0000", 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150202 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.274241+0000", 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011681261 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.287044+0000", 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012803386 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.287116+0000", 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1675999999999996e-05 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.287187+0000", 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1316999999999995e-05 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4062 3.24 3:26bdc0b7:::100000001f8.000001da:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.654 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.346477+0000", 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2148060909999998, 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.087538915999999994, 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4062 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.346477+0000", 2024-04-25T15:54:08.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.346477+0000", 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.346480+0000", 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7259999999999998e-06 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.348020+0000", 2024-04-25T15:54:08.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015406689999999999 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.397457+0000", 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049436435000000001 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.397470+0000", 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3684e-05 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.397552+0000", 2024-04-25T15:54:08.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1205999999999995e-05 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.397629+0000", 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7401e-05 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.397730+0000", 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100963 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.406994+0000", 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092639239999999998 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.433876+0000", 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026881961999999999 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.433950+0000", 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3980000000000007e-05 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.434016+0000", 2024-04-25T15:54:08.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5965999999999998e-05 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4072 3.8 3:131e2389:::100000001f8.000001e4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.395840+0000", 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.1654429940000002, 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088020573000000005, 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.660 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4072 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.395840+0000", 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.395840+0000", 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.395842+0000", 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2639999999999998e-06 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.397423+0000", 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015804549999999999 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.442804+0000", 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.045380942000000001 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.442829+0000", 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5361e-05 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.442943+0000", 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113761 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.443061+0000", 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011773900000000001 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.443207+0000", 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014639899999999999 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.459010+0000", 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015802712 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.483704+0000", 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024694226 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.483790+0000", 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5865999999999994e-05 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.483861+0000", 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0847999999999997e-05 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4113 3.2 3:43c5f71b:::100000001f8.0000020d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.619692+0000", 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.9415907670000001, 2024-04-25T15:54:08.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075623475999999995, 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4113 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.619692+0000", 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.619692+0000", 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.619695+0000", 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.751e-06 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.621210+0000", 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00151519 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.621244+0000", 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3810999999999999e-05 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.621262+0000", 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7878999999999999e-05 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.621353+0000", 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0989000000000006e-05 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:08.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.621448+0000", 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5236999999999996e-05 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.621574+0000", 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012640000000000001 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.629182+0000", 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0076072910000000004 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.695196+0000", 2024-04-25T15:54:08.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.066013848 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.695262+0000", 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6174999999999996e-05 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.695316+0000", 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3905e-05 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:08.670 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:08.671 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:08.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:54:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.617+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f66b80b21c0 con 0x7f66a406f450 2024-04-25T15:54:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f669ffff640 1 -- 172.21.15.67:0/1238575769 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254099 (secure 0 0 0) 0x7f66b80b21c0 con 0x7f66a406f450 2024-04-25T15:54:08.675 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66a406f450 msgr2=0x7f66a4071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66a406f450 0x7f66a4071960 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00a790 tx=0x7f66ac008040 comp rx=0 tx=0).stop 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b815a0b0 msgr2=0x7f66b814abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b815a0b0 0x7f66b814abf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f66b000b9e0 tx=0x7f66b000bea0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 shutdown_connections 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f66a406f450 0x7f66a4071960 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66b815a0b0 0x7f66b814abf0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66b8153a00 0x7f66b80c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 --2- 172.21.15.67:0/1238575769 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66b80c4010 0x7f66b80b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.633+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 >> 172.21.15.67:0/1238575769 conn(0x7f66b80ce1c0 msgr2=0x7f66b80ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.637+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 shutdown_connections 2024-04-25T15:54:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.637+0000 7f66bde84640 1 -- 172.21.15.67:0/1238575769 wait complete. 2024-04-25T15:54:08.678 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:08.669+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:08.679 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-04-25T15:54:07.873757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738194848,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":4839,"num_write_kb":16037746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":658513155,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4778,"ondisk_log_size":4778,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":21277076,"kb_used_data":20982208,"kb_used_omap":12,"kb_used_meta":274291,"kb_avail":733697644,"statfs":{"total":773094113280,"available":751306387456,"internally_reserved":0,"allocated":21485780992,"data_stored":21480694106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":280874546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":140,"apply_latency_ms":140,"commit_latency_ns":140000000,"apply_latency_ns":140000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":1355776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.797461"},"pg_stats":[{"pgid":"3.3f","version":"96'66","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.622402+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:54:02.622402+0000","last_peered":"2024-04-25T15:54:02.622402+0000","last_clean":"2024-04-25T15:54:02.622402+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:54:02.622402+0000","last_undegraded":"2024-04-25T15:54:02.622402+0000","last_fullsized":"2024-04-25T15:54:02.622402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":237568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"96'56","reported_seq":112,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.778077+0000","last_change":"2024-04-25T15:53:13.069367+0000","last_active":"2024-04-25T15:54:02.778077+0000","last_peered":"2024-04-25T15:54:02.778077+0000","last_clean":"2024-04-25T15:54:02.778077+0000","last_became_active":"2024-04-25T15:53:12.990829+0000","last_became_peered":"2024-04-25T15:53:12.990829+0000","last_unstale":"2024-04-25T15:54:02.778077+0000","last_undegraded":"2024-04-25T15:54:02.778077+0000","last_fullsized":"2024-04-25T15:54:02.778077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:46.276338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"96'67","reported_seq":187,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.191868+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:54:02.191868+0000","last_peered":"2024-04-25T15:54:02.191868+0000","last_clean":"2024-04-25T15:54:02.191868+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:54:02.191868+0000","last_undegraded":"2024-04-25T15:54:02.191868+0000","last_fullsized":"2024-04-25T15:54:02.191868+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"96'69","reported_seq":234,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.562481+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:54:02.562481+0000","last_peered":"2024-04-25T15:54:02.562481+0000","last_clean":"2024-04-25T15:54:02.562481+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:54:02.562481+0000","last_undegraded":"2024-04-25T15:54:02.562481+0000","last_fullsized":"2024-04-25T15:54:02.562481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":262144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"96'68","reported_seq":231,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.843188+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:54:01.843188+0000","last_peered":"2024-04-25T15:54:01.843188+0000","last_clean":"2024-04-25T15:54:01.843188+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:54:01.843188+0000","last_undegraded":"2024-04-25T15:54:01.843188+0000","last_fullsized":"2024-04-25T15:54:01.843188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"96'59","reported_seq":224,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.453283+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:54:02.453283+0000","last_peered":"2024-04-25T15:54:02.453283+0000","last_clean":"2024-04-25T15:54:02.453283+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:54:02.453283+0000","last_undegraded":"2024-04-25T15:54:02.453283+0000","last_fullsized":"2024-04-25T15:54:02.453283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":212992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"96'74","reported_seq":237,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.634204+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:54:02.634204+0000","last_peered":"2024-04-25T15:54:02.634204+0000","last_clean":"2024-04-25T15:54:02.634204+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:54:02.634204+0000","last_undegraded":"2024-04-25T15:54:02.634204+0000","last_fullsized":"2024-04-25T15:54:02.634204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":270336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"96'84","reported_seq":249,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.528713+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:54:02.528713+0000","last_peered":"2024-04-25T15:54:02.528713+0000","last_clean":"2024-04-25T15:54:02.528713+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:54:02.528713+0000","last_undegraded":"2024-04-25T15:54:02.528713+0000","last_fullsized":"2024-04-25T15:54:02.528713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"96'71","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.640673+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:54:02.640673+0000","last_peered":"2024-04-25T15:54:02.640673+0000","last_clean":"2024-04-25T15:54:02.640673+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:54:02.640673+0000","last_undegraded":"2024-04-25T15:54:02.640673+0000","last_fullsized":"2024-04-25T15:54:02.640673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"96'71","reported_seq":236,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.379292+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:54:02.379292+0000","last_peered":"2024-04-25T15:54:02.379292+0000","last_clean":"2024-04-25T15:54:02.379292+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:54:02.379292+0000","last_undegraded":"2024-04-25T15:54:02.379292+0000","last_fullsized":"2024-04-25T15:54:02.379292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71,"num_write_kb":253952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"96'77","reported_seq":242,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.842640+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:54:01.842640+0000","last_peered":"2024-04-25T15:54:01.842640+0000","last_clean":"2024-04-25T15:54:01.842640+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:54:01.842640+0000","last_undegraded":"2024-04-25T15:54:01.842640+0000","last_fullsized":"2024-04-25T15:54:01.842640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77,"num_write_kb":286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"98'79","reported_seq":133,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:07.724666+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:54:07.724666+0000","last_peered":"2024-04-25T15:54:07.724666+0000","last_clean":"2024-04-25T15:54:07.724666+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:54:07.724666+0000","last_undegraded":"2024-04-25T15:54:07.724666+0000","last_fullsized":"2024-04-25T15:54:07.724666+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"96'64","reported_seq":229,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.152566+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:54:02.152566+0000","last_peered":"2024-04-25T15:54:02.152566+0000","last_clean":"2024-04-25T15:54:02.152566+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:54:02.152566+0000","last_undegraded":"2024-04-25T15:54:02.152566+0000","last_fullsized":"2024-04-25T15:54:02.152566+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":233472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"96'65","reported_seq":231,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.656438+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:54:02.656438+0000","last_peered":"2024-04-25T15:54:02.656438+0000","last_clean":"2024-04-25T15:54:02.656438+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:54:02.656438+0000","last_undegraded":"2024-04-25T15:54:02.656438+0000","last_fullsized":"2024-04-25T15:54:02.656438+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":237568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"98'77","reported_seq":131,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:07.472794+0000","last_change":"2024-04-25T15:53:17.148839+0000","last_active":"2024-04-25T15:54:07.472794+0000","last_peered":"2024-04-25T15:54:07.472794+0000","last_clean":"2024-04-25T15:54:07.472794+0000","last_became_active":"2024-04-25T15:53:16.064416+0000","last_became_peered":"2024-04-25T15:53:16.064416+0000","last_unstale":"2024-04-25T15:54:07.472794+0000","last_undegraded":"2024-04-25T15:54:07.472794+0000","last_fullsized":"2024-04-25T15:54:07.472794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:58.798689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77,"num_write_kb":286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"96'70","reported_seq":235,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.329018+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:54:02.329018+0000","last_peered":"2024-04-25T15:54:02.329018+0000","last_clean":"2024-04-25T15:54:02.329018+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:54:02.329018+0000","last_undegraded":"2024-04-25T15:54:02.329018+0000","last_fullsized":"2024-04-25T15:54:02.329018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"96'59","reported_seq":157,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.453917+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:54:02.453917+0000","last_peered":"2024-04-25T15:54:02.453917+0000","last_clean":"2024-04-25T15:54:02.453917+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:54:02.453917+0000","last_undegraded":"2024-04-25T15:54:02.453917+0000","last_fullsized":"2024-04-25T15:54:02.453917+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"96'61","reported_seq":225,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.026433+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:54:02.026433+0000","last_peered":"2024-04-25T15:54:02.026433+0000","last_clean":"2024-04-25T15:54:02.026433+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:54:02.026433+0000","last_undegraded":"2024-04-25T15:54:02.026433+0000","last_fullsized":"2024-04-25T15:54:02.026433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":229376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"96'70","reported_seq":233,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.853293+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:54:01.853293+0000","last_peered":"2024-04-25T15:54:01.853293+0000","last_clean":"2024-04-25T15:54:01.853293+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:54:01.853293+0000","last_undegraded":"2024-04-25T15:54:01.853293+0000","last_fullsized":"2024-04-25T15:54:01.853293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":262144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"96'70","reported_seq":251,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.522886+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:54:02.522886+0000","last_peered":"2024-04-25T15:54:02.522886+0000","last_clean":"2024-04-25T15:54:02.522886+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:54:02.522886+0000","last_undegraded":"2024-04-25T15:54:02.522886+0000","last_fullsized":"2024-04-25T15:54:02.522886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":262144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"96'57","reported_seq":176,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190404+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:54:00.190404+0000","last_peered":"2024-04-25T15:54:00.190404+0000","last_clean":"2024-04-25T15:54:00.190404+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:54:00.190404+0000","last_undegraded":"2024-04-25T15:54:00.190404+0000","last_fullsized":"2024-04-25T15:54:00.190404+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"96'73","reported_seq":231,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.316307+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:54:02.316307+0000","last_peered":"2024-04-25T15:54:02.316307+0000","last_clean":"2024-04-25T15:54:02.316307+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:54:02.316307+0000","last_undegraded":"2024-04-25T15:54:02.316307+0000","last_fullsized":"2024-04-25T15:54:02.316307+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73,"num_write_kb":262144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"96'61","reported_seq":242,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.784885+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:54:02.784885+0000","last_peered":"2024-04-25T15:54:02.784885+0000","last_clean":"2024-04-25T15:54:02.784885+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:54:02.784885+0000","last_undegraded":"2024-04-25T15:54:02.784885+0000","last_fullsized":"2024-04-25T15:54:02.784885+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"96'78","reported_seq":243,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.956361+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:54:01.956361+0000","last_peered":"2024-04-25T15:54:01.956361+0000","last_clean":"2024-04-25T15:54:01.956361+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:54:01.956361+0000","last_undegraded":"2024-04-25T15:54:01.956361+0000","last_fullsized":"2024-04-25T15:54:01.956361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":282624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"96'57","reported_seq":94,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.191104+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:54:00.191104+0000","last_peered":"2024-04-25T15:54:00.191104+0000","last_clean":"2024-04-25T15:54:00.191104+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:54:00.191104+0000","last_undegraded":"2024-04-25T15:54:00.191104+0000","last_fullsized":"2024-04-25T15:54:00.191104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":208896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"96'92","reported_seq":257,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.665362+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:54:02.665362+0000","last_peered":"2024-04-25T15:54:02.665362+0000","last_clean":"2024-04-25T15:54:02.665362+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:54:02.665362+0000","last_undegraded":"2024-04-25T15:54:02.665362+0000","last_fullsized":"2024-04-25T15:54:02.665362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"96'61","reported_seq":207,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.253423+0000","last_change":"2024-04-25T15:53:03.808515+0000","last_active":"2024-04-25T15:54:02.253423+0000","last_peered":"2024-04-25T15:54:02.253423+0000","last_clean":"2024-04-25T15:54:02.253423+0000","last_became_active":"2024-04-25T15:53:03.367253+0000","last_became_peered":"2024-04-25T15:53:03.367253+0000","last_unstale":"2024-04-25T15:54:02.253423+0000","last_undegraded":"2024-04-25T15:54:02.253423+0000","last_fullsized":"2024-04-25T15:54:02.253423+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:05.644251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"96'62","reported_seq":262,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.055431+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:54:02.055431+0000","last_peered":"2024-04-25T15:54:02.055431+0000","last_clean":"2024-04-25T15:54:02.055431+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:54:02.055431+0000","last_undegraded":"2024-04-25T15:54:02.055431+0000","last_fullsized":"2024-04-25T15:54:02.055431+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"96'57","reported_seq":178,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.982713+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:54:01.982713+0000","last_peered":"2024-04-25T15:54:01.982713+0000","last_clean":"2024-04-25T15:54:01.982713+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:54:01.982713+0000","last_undegraded":"2024-04-25T15:54:01.982713+0000","last_fullsized":"2024-04-25T15:54:01.982713+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":196608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"96'57","reported_seq":166,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.633483+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:54:02.633483+0000","last_peered":"2024-04-25T15:54:02.633483+0000","last_clean":"2024-04-25T15:54:02.633483+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:54:02.633483+0000","last_undegraded":"2024-04-25T15:54:02.633483+0000","last_fullsized":"2024-04-25T15:54:02.633483+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":208896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"96'59","reported_seq":233,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.287510+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:54:02.287510+0000","last_peered":"2024-04-25T15:54:02.287510+0000","last_clean":"2024-04-25T15:54:02.287510+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:54:02.287510+0000","last_undegraded":"2024-04-25T15:54:02.287510+0000","last_fullsized":"2024-04-25T15:54:02.287510+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":208896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"95'71","reported_seq":236,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.079186+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:54:00.079186+0000","last_peered":"2024-04-25T15:54:00.079186+0000","last_clean":"2024-04-25T15:54:00.079186+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:54:00.079186+0000","last_undegraded":"2024-04-25T15:54:00.079186+0000","last_fullsized":"2024-04-25T15:54:00.079186+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"98'73","reported_seq":127,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:06.746630+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:54:06.746630+0000","last_peered":"2024-04-25T15:54:06.746630+0000","last_clean":"2024-04-25T15:54:06.746630+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:54:06.746630+0000","last_undegraded":"2024-04-25T15:54:06.746630+0000","last_fullsized":"2024-04-25T15:54:06.746630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":173,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190941+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:00.190941+0000","last_peered":"2024-04-25T15:54:00.190941+0000","last_clean":"2024-04-25T15:54:00.190941+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:00.190941+0000","last_undegraded":"2024-04-25T15:54:00.190941+0000","last_fullsized":"2024-04-25T15:54:00.190941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"98'75","reported_seq":129,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:06.778438+0000","last_change":"2024-04-25T15:53:16.510837+0000","last_active":"2024-04-25T15:54:06.778438+0000","last_peered":"2024-04-25T15:54:06.778438+0000","last_clean":"2024-04-25T15:54:06.778438+0000","last_became_active":"2024-04-25T15:53:16.041557+0000","last_became_peered":"2024-04-25T15:53:16.041557+0000","last_unstale":"2024-04-25T15:54:06.778438+0000","last_undegraded":"2024-04-25T15:54:06.778438+0000","last_fullsized":"2024-04-25T15:54:06.778438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:06.600570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":200,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.079372+0000","last_change":"2024-04-25T15:53:16.102712+0000","last_active":"2024-04-25T15:54:00.079372+0000","last_peered":"2024-04-25T15:54:00.079372+0000","last_clean":"2024-04-25T15:54:00.079372+0000","last_became_active":"2024-04-25T15:53:16.052729+0000","last_became_peered":"2024-04-25T15:53:16.052729+0000","last_unstale":"2024-04-25T15:54:00.079372+0000","last_undegraded":"2024-04-25T15:54:00.079372+0000","last_fullsized":"2024-04-25T15:54:00.079372+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:53.161289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":170,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190973+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:54:00.190973+0000","last_peered":"2024-04-25T15:54:00.190973+0000","last_clean":"2024-04-25T15:54:00.190973+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:54:00.190973+0000","last_undegraded":"2024-04-25T15:54:00.190973+0000","last_fullsized":"2024-04-25T15:54:00.190973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"96'72","reported_seq":132,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.775759+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:54:02.775759+0000","last_peered":"2024-04-25T15:54:02.775759+0000","last_clean":"2024-04-25T15:54:02.775759+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:54:02.775759+0000","last_undegraded":"2024-04-25T15:54:02.775759+0000","last_fullsized":"2024-04-25T15:54:02.775759+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":188,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.081498+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:00.081498+0000","last_peered":"2024-04-25T15:54:00.081498+0000","last_clean":"2024-04-25T15:54:00.081498+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:00.081498+0000","last_undegraded":"2024-04-25T15:54:00.081498+0000","last_fullsized":"2024-04-25T15:54:00.081498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"96'65","reported_seq":221,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.089398+0000","last_change":"2024-04-25T15:53:16.206656+0000","last_active":"2024-04-25T15:54:02.089398+0000","last_peered":"2024-04-25T15:54:02.089398+0000","last_clean":"2024-04-25T15:54:02.089398+0000","last_became_active":"2024-04-25T15:53:16.049908+0000","last_became_peered":"2024-04-25T15:53:16.049908+0000","last_unstale":"2024-04-25T15:54:02.089398+0000","last_undegraded":"2024-04-25T15:54:02.089398+0000","last_fullsized":"2024-04-25T15:54:02.089398+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.217418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":173,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.082994+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:54:00.082994+0000","last_peered":"2024-04-25T15:54:00.082994+0000","last_clean":"2024-04-25T15:54:00.082994+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:54:00.082994+0000","last_undegraded":"2024-04-25T15:54:00.082994+0000","last_fullsized":"2024-04-25T15:54:00.082994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"96'97","reported_seq":262,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.276349+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:54:02.276349+0000","last_peered":"2024-04-25T15:54:02.276349+0000","last_clean":"2024-04-25T15:54:02.276349+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:54:02.276349+0000","last_undegraded":"2024-04-25T15:54:02.276349+0000","last_fullsized":"2024-04-25T15:54:02.276349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":171,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.084971+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:00.084971+0000","last_peered":"2024-04-25T15:54:00.084971+0000","last_clean":"2024-04-25T15:54:00.084971+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:00.084971+0000","last_undegraded":"2024-04-25T15:54:00.084971+0000","last_fullsized":"2024-04-25T15:54:00.084971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"96'66","reported_seq":187,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.705506+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:54:02.705506+0000","last_peered":"2024-04-25T15:54:02.705506+0000","last_clean":"2024-04-25T15:54:02.705506+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:54:02.705506+0000","last_undegraded":"2024-04-25T15:54:02.705506+0000","last_fullsized":"2024-04-25T15:54:02.705506+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":233472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"88'8","reported_seq":127,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.085318+0000","last_change":"2024-04-25T15:53:03.435546+0000","last_active":"2024-04-25T15:54:00.085318+0000","last_peered":"2024-04-25T15:54:00.085318+0000","last_clean":"2024-04-25T15:54:00.085318+0000","last_became_active":"2024-04-25T15:53:03.387235+0000","last_became_peered":"2024-04-25T15:53:03.387235+0000","last_unstale":"2024-04-25T15:54:00.085318+0000","last_undegraded":"2024-04-25T15:54:00.085318+0000","last_fullsized":"2024-04-25T15:54:00.085318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:30.444213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1344,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"98'79","reported_seq":133,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:07.839438+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:54:07.839438+0000","last_peered":"2024-04-25T15:54:07.839438+0000","last_clean":"2024-04-25T15:54:07.839438+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:54:07.839438+0000","last_undegraded":"2024-04-25T15:54:07.839438+0000","last_fullsized":"2024-04-25T15:54:07.839438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"95'82","reported_seq":249,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.079919+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:54:00.079919+0000","last_peered":"2024-04-25T15:54:00.079919+0000","last_clean":"2024-04-25T15:54:00.079919+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:54:00.079919+0000","last_undegraded":"2024-04-25T15:54:00.079919+0000","last_fullsized":"2024-04-25T15:54:00.079919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178293,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"96'71","reported_seq":234,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.116902+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:54:02.116902+0000","last_peered":"2024-04-25T15:54:02.116902+0000","last_clean":"2024-04-25T15:54:02.116902+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:54:02.116902+0000","last_undegraded":"2024-04-25T15:54:02.116902+0000","last_fullsized":"2024-04-25T15:54:02.116902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:06.211877+0000","last_change":"2024-04-25T15:53:16.094342+0000","last_active":"2024-04-25T15:54:06.211877+0000","last_peered":"2024-04-25T15:54:06.211877+0000","last_clean":"2024-04-25T15:54:06.211877+0000","last_became_active":"2024-04-25T15:53:16.041484+0000","last_became_peered":"2024-04-25T15:53:16.041484+0000","last_unstale":"2024-04-25T15:54:06.211877+0000","last_undegraded":"2024-04-25T15:54:06.211877+0000","last_fullsized":"2024-04-25T15:54:06.211877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:42.531801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"96'71","reported_seq":190,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.838012+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:54:01.838012+0000","last_peered":"2024-04-25T15:54:01.838012+0000","last_clean":"2024-04-25T15:54:01.838012+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:54:01.838012+0000","last_undegraded":"2024-04-25T15:54:01.838012+0000","last_fullsized":"2024-04-25T15:54:01.838012+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":172,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.082568+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:00.082568+0000","last_peered":"2024-04-25T15:54:00.082568+0000","last_clean":"2024-04-25T15:54:00.082568+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:00.082568+0000","last_undegraded":"2024-04-25T15:54:00.082568+0000","last_fullsized":"2024-04-25T15:54:00.082568+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"96'65","reported_seq":248,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.960255+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:54:01.960255+0000","last_peered":"2024-04-25T15:54:01.960255+0000","last_clean":"2024-04-25T15:54:01.960255+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:54:01.960255+0000","last_undegraded":"2024-04-25T15:54:01.960255+0000","last_fullsized":"2024-04-25T15:54:01.960255+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":212,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.079339+0000","last_change":"2024-04-25T15:53:16.115440+0000","last_active":"2024-04-25T15:54:00.079339+0000","last_peered":"2024-04-25T15:54:00.079339+0000","last_clean":"2024-04-25T15:54:00.079339+0000","last_became_active":"2024-04-25T15:53:16.053229+0000","last_became_peered":"2024-04-25T15:53:16.053229+0000","last_unstale":"2024-04-25T15:54:00.079339+0000","last_undegraded":"2024-04-25T15:54:00.079339+0000","last_fullsized":"2024-04-25T15:54:00.079339+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:13.860370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3087,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"96'87","reported_seq":252,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.644170+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:54:02.644170+0000","last_peered":"2024-04-25T15:54:02.644170+0000","last_clean":"2024-04-25T15:54:02.644170+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:54:02.644170+0000","last_undegraded":"2024-04-25T15:54:02.644170+0000","last_fullsized":"2024-04-25T15:54:02.644170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":252,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.188776+0000","last_change":"2024-04-25T15:53:03.421455+0000","last_active":"2024-04-25T15:54:00.188776+0000","last_peered":"2024-04-25T15:54:00.188776+0000","last_clean":"2024-04-25T15:54:00.188776+0000","last_became_active":"2024-04-25T15:53:03.385909+0000","last_became_peered":"2024-04-25T15:53:03.385909+0000","last_unstale":"2024-04-25T15:54:00.188776+0000","last_undegraded":"2024-04-25T15:54:00.188776+0000","last_fullsized":"2024-04-25T15:54:00.188776+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:54.677309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"96'95","reported_seq":286,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.926968+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:54:01.926968+0000","last_peered":"2024-04-25T15:54:01.926968+0000","last_clean":"2024-04-25T15:54:01.926968+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:54:01.926968+0000","last_undegraded":"2024-04-25T15:54:01.926968+0000","last_fullsized":"2024-04-25T15:54:01.926968+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":327680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":58,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:06.212188+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:06.212188+0000","last_peered":"2024-04-25T15:54:06.212188+0000","last_clean":"2024-04-25T15:54:06.212188+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:06.212188+0000","last_undegraded":"2024-04-25T15:54:06.212188+0000","last_fullsized":"2024-04-25T15:54:06.212188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"96'62","reported_seq":241,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.453838+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:54:02.453838+0000","last_peered":"2024-04-25T15:54:02.453838+0000","last_clean":"2024-04-25T15:54:02.453838+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:54:02.453838+0000","last_undegraded":"2024-04-25T15:54:02.453838+0000","last_fullsized":"2024-04-25T15:54:02.453838+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":229376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":171,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190300+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:54:00.190300+0000","last_peered":"2024-04-25T15:54:00.190300+0000","last_clean":"2024-04-25T15:54:00.190300+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:54:00.190300+0000","last_undegraded":"2024-04-25T15:54:00.190300+0000","last_fullsized":"2024-04-25T15:54:00.190300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"96'59","reported_seq":242,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:01.857201+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:54:01.857201+0000","last_peered":"2024-04-25T15:54:01.857201+0000","last_clean":"2024-04-25T15:54:01.857201+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:54:01.857201+0000","last_undegraded":"2024-04-25T15:54:01.857201+0000","last_fullsized":"2024-04-25T15:54:01.857201+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":217088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":56,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:06.211737+0000","last_change":"2024-04-25T15:53:16.098410+0000","last_active":"2024-04-25T15:54:06.211737+0000","last_peered":"2024-04-25T15:54:06.211737+0000","last_clean":"2024-04-25T15:54:06.211737+0000","last_became_active":"2024-04-25T15:53:16.053979+0000","last_became_peered":"2024-04-25T15:53:16.053979+0000","last_unstale":"2024-04-25T15:54:06.211737+0000","last_undegraded":"2024-04-25T15:54:06.211737+0000","last_fullsized":"2024-04-25T15:54:06.211737+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:24.859180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"96'62","reported_seq":227,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.024900+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:54:02.024900+0000","last_peered":"2024-04-25T15:54:02.024900+0000","last_clean":"2024-04-25T15:54:02.024900+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:54:02.024900+0000","last_undegraded":"2024-04-25T15:54:02.024900+0000","last_fullsized":"2024-04-25T15:54:02.024900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":225280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":172,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.189915+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:54:00.189915+0000","last_peered":"2024-04-25T15:54:00.189915+0000","last_clean":"2024-04-25T15:54:00.189915+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:54:00.189915+0000","last_undegraded":"2024-04-25T15:54:00.189915+0000","last_fullsized":"2024-04-25T15:54:00.189915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"96'67","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.638991+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:54:02.638991+0000","last_peered":"2024-04-25T15:54:02.638991+0000","last_clean":"2024-04-25T15:54:02.638991+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:54:02.638991+0000","last_undegraded":"2024-04-25T15:54:02.638991+0000","last_fullsized":"2024-04-25T15:54:02.638991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":245760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":261,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190076+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:00.190076+0000","last_peered":"2024-04-25T15:54:00.190076+0000","last_clean":"2024-04-25T15:54:00.190076+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:00.190076+0000","last_undegraded":"2024-04-25T15:54:00.190076+0000","last_fullsized":"2024-04-25T15:54:00.190076+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":173,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.082008+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:00.082008+0000","last_peered":"2024-04-25T15:54:00.082008+0000","last_clean":"2024-04-25T15:54:00.082008+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:00.082008+0000","last_undegraded":"2024-04-25T15:54:00.082008+0000","last_fullsized":"2024-04-25T15:54:00.082008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"96'63","reported_seq":228,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.191136+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:54:00.191136+0000","last_peered":"2024-04-25T15:54:00.191136+0000","last_clean":"2024-04-25T15:54:00.191136+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:54:00.191136+0000","last_undegraded":"2024-04-25T15:54:00.191136+0000","last_fullsized":"2024-04-25T15:54:00.191136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":233472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":196,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.189997+0000","last_change":"2024-04-25T15:53:03.424512+0000","last_active":"2024-04-25T15:54:00.189997+0000","last_peered":"2024-04-25T15:54:00.189997+0000","last_clean":"2024-04-25T15:54:00.189997+0000","last_became_active":"2024-04-25T15:53:03.385988+0000","last_became_peered":"2024-04-25T15:53:03.385988+0000","last_unstale":"2024-04-25T15:54:00.189997+0000","last_undegraded":"2024-04-25T15:54:00.189997+0000","last_fullsized":"2024-04-25T15:54:00.189997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:55.224996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"98'79","reported_seq":133,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:07.396057+0000","last_change":"2024-04-25T15:53:16.680938+0000","last_active":"2024-04-25T15:54:07.396057+0000","last_peered":"2024-04-25T15:54:07.396057+0000","last_clean":"2024-04-25T15:54:07.396057+0000","last_became_active":"2024-04-25T15:53:16.041368+0000","last_became_peered":"2024-04-25T15:53:16.041368+0000","last_unstale":"2024-04-25T15:54:07.396057+0000","last_undegraded":"2024-04-25T15:54:07.396057+0000","last_fullsized":"2024-04-25T15:54:07.396057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:49.681769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79,"num_write_kb":278528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"88'9","reported_seq":177,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.189163+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:54:00.189163+0000","last_peered":"2024-04-25T15:54:00.189163+0000","last_clean":"2024-04-25T15:54:00.189163+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:54:00.189163+0000","last_undegraded":"2024-04-25T15:54:00.189163+0000","last_fullsized":"2024-04-25T15:54:00.189163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"94'55","reported_seq":200,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.188633+0000","last_change":"2024-04-25T15:53:03.530328+0000","last_active":"2024-04-25T15:54:00.188633+0000","last_peered":"2024-04-25T15:54:00.188633+0000","last_clean":"2024-04-25T15:54:00.188633+0000","last_became_active":"2024-04-25T15:53:03.382259+0000","last_became_peered":"2024-04-25T15:53:03.382259+0000","last_unstale":"2024-04-25T15:54:00.188633+0000","last_undegraded":"2024-04-25T15:54:00.188633+0000","last_fullsized":"2024-04-25T15:54:00.188633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:43.788748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":196608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":194,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.082876+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:00.082876+0000","last_peered":"2024-04-25T15:54:00.082876+0000","last_clean":"2024-04-25T15:54:00.082876+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:00.082876+0000","last_undegraded":"2024-04-25T15:54:00.082876+0000","last_fullsized":"2024-04-25T15:54:00.082876+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"98'73","reported_seq":194,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:06.688150+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:54:06.688150+0000","last_peered":"2024-04-25T15:54:06.688150+0000","last_clean":"2024-04-25T15:54:06.688150+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:54:06.688150+0000","last_undegraded":"2024-04-25T15:54:06.688150+0000","last_fullsized":"2024-04-25T15:54:06.688150+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":63,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190318+0000","last_change":"2024-04-25T15:53:03.381919+0000","last_active":"2024-04-25T15:54:00.190318+0000","last_peered":"2024-04-25T15:54:00.190318+0000","last_clean":"2024-04-25T15:54:00.190318+0000","last_became_active":"2024-04-25T15:53:03.381463+0000","last_became_peered":"2024-04-25T15:53:03.381463+0000","last_unstale":"2024-04-25T15:54:00.190318+0000","last_undegraded":"2024-04-25T15:54:00.190318+0000","last_fullsized":"2024-04-25T15:54:00.190318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:09.993891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"96'66","reported_seq":231,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.073944+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:54:02.073944+0000","last_peered":"2024-04-25T15:54:02.073944+0000","last_clean":"2024-04-25T15:54:02.073944+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:54:02.073944+0000","last_undegraded":"2024-04-25T15:54:02.073944+0000","last_fullsized":"2024-04-25T15:54:02.073944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":233472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":169,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.191086+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:00.191086+0000","last_peered":"2024-04-25T15:54:00.191086+0000","last_clean":"2024-04-25T15:54:00.191086+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:00.191086+0000","last_undegraded":"2024-04-25T15:54:00.191086+0000","last_fullsized":"2024-04-25T15:54:00.191086+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"96'58","reported_seq":189,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.189844+0000","last_change":"2024-04-25T15:53:03.480191+0000","last_active":"2024-04-25T15:54:00.189844+0000","last_peered":"2024-04-25T15:54:00.189844+0000","last_clean":"2024-04-25T15:54:00.189844+0000","last_became_active":"2024-04-25T15:53:03.380081+0000","last_became_peered":"2024-04-25T15:53:03.380081+0000","last_unstale":"2024-04-25T15:54:00.189844+0000","last_undegraded":"2024-04-25T15:54:00.189844+0000","last_fullsized":"2024-04-25T15:54:00.189844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:50.114177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58,"num_write_kb":212992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"88'20","reported_seq":181,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.081610+0000","last_change":"2024-04-25T15:53:16.083270+0000","last_active":"2024-04-25T15:54:00.081610+0000","last_peered":"2024-04-25T15:54:00.081610+0000","last_clean":"2024-04-25T15:54:00.081610+0000","last_became_active":"2024-04-25T15:53:16.045200+0000","last_became_peered":"2024-04-25T15:53:16.045200+0000","last_unstale":"2024-04-25T15:54:00.081610+0000","last_undegraded":"2024-04-25T15:54:00.081610+0000","last_fullsized":"2024-04-25T15:54:00.081610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:38.382894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"96'77","reported_seq":242,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.546240+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:54:02.546240+0000","last_peered":"2024-04-25T15:54:02.546240+0000","last_clean":"2024-04-25T15:54:02.546240+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:54:02.546240+0000","last_undegraded":"2024-04-25T15:54:02.546240+0000","last_fullsized":"2024-04-25T15:54:02.546240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77,"num_write_kb":262144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":174,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.083395+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:00.083395+0000","last_peered":"2024-04-25T15:54:00.083395+0000","last_clean":"2024-04-25T15:54:00.083395+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:00.083395+0000","last_undegraded":"2024-04-25T15:54:00.083395+0000","last_fullsized":"2024-04-25T15:54:00.083395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"98'86","reported_seq":140,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:07.554126+0000","last_change":"2024-04-25T15:53:16.863394+0000","last_active":"2024-04-25T15:54:07.554126+0000","last_peered":"2024-04-25T15:54:07.554126+0000","last_clean":"2024-04-25T15:54:07.554126+0000","last_became_active":"2024-04-25T15:53:16.048669+0000","last_became_peered":"2024-04-25T15:53:16.048669+0000","last_unstale":"2024-04-25T15:54:07.554126+0000","last_undegraded":"2024-04-25T15:54:07.554126+0000","last_fullsized":"2024-04-25T15:54:07.554126+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:53.132724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":323584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":167,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.189784+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:00.189784+0000","last_peered":"2024-04-25T15:54:00.189784+0000","last_clean":"2024-04-25T15:54:00.189784+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:00.189784+0000","last_undegraded":"2024-04-25T15:54:00.189784+0000","last_fullsized":"2024-04-25T15:54:00.189784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"98'82","reported_seq":136,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-04-25T15:54:06.346025+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:54:06.346025+0000","last_peered":"2024-04-25T15:54:06.346025+0000","last_clean":"2024-04-25T15:54:06.346025+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:54:06.346025+0000","last_undegraded":"2024-04-25T15:54:06.346025+0000","last_fullsized":"2024-04-25T15:54:06.346025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":282624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":140,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.191565+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:54:00.191565+0000","last_peered":"2024-04-25T15:54:00.191565+0000","last_clean":"2024-04-25T15:54:00.191565+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:54:00.191565+0000","last_undegraded":"2024-04-25T15:54:00.191565+0000","last_fullsized":"2024-04-25T15:54:00.191565+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"96'76","reported_seq":272,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.181052+0000","last_change":"2024-04-25T15:53:03.761415+0000","last_active":"2024-04-25T15:54:02.181052+0000","last_peered":"2024-04-25T15:54:02.181052+0000","last_clean":"2024-04-25T15:54:02.181052+0000","last_became_active":"2024-04-25T15:53:03.384666+0000","last_became_peered":"2024-04-25T15:53:03.384666+0000","last_unstale":"2024-04-25T15:54:02.181052+0000","last_undegraded":"2024-04-25T15:54:02.181052+0000","last_fullsized":"2024-04-25T15:54:02.181052+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:14.524576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76,"num_write_kb":262144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":114,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190996+0000","last_change":"2024-04-25T15:53:03.409633+0000","last_active":"2024-04-25T15:54:00.190996+0000","last_peered":"2024-04-25T15:54:00.190996+0000","last_clean":"2024-04-25T15:54:00.190996+0000","last_became_active":"2024-04-25T15:53:03.384646+0000","last_became_peered":"2024-04-25T15:53:03.384646+0000","last_unstale":"2024-04-25T15:54:00.190996+0000","last_undegraded":"2024-04-25T15:54:00.190996+0000","last_fullsized":"2024-04-25T15:54:00.190996+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:43.442571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"94'63","reported_seq":221,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.189472+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:54:00.189472+0000","last_peered":"2024-04-25T15:54:00.189472+0000","last_clean":"2024-04-25T15:54:00.189472+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:54:00.189472+0000","last_undegraded":"2024-04-25T15:54:00.189472+0000","last_fullsized":"2024-04-25T15:54:00.189472+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":192,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190991+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:00.190991+0000","last_peered":"2024-04-25T15:54:00.190991+0000","last_clean":"2024-04-25T15:54:00.190991+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:00.190991+0000","last_undegraded":"2024-04-25T15:54:00.190991+0000","last_fullsized":"2024-04-25T15:54:00.190991+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"96'54","reported_seq":245,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.495629+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:54:02.495629+0000","last_peered":"2024-04-25T15:54:02.495629+0000","last_clean":"2024-04-25T15:54:02.495629+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:54:02.495629+0000","last_undegraded":"2024-04-25T15:54:02.495629+0000","last_fullsized":"2024-04-25T15:54:02.495629+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"96'81","reported_seq":272,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.431523+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:54:02.431523+0000","last_peered":"2024-04-25T15:54:02.431523+0000","last_clean":"2024-04-25T15:54:02.431523+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:54:02.431523+0000","last_undegraded":"2024-04-25T15:54:02.431523+0000","last_fullsized":"2024-04-25T15:54:02.431523+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":170,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190614+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:00.190614+0000","last_peered":"2024-04-25T15:54:00.190614+0000","last_clean":"2024-04-25T15:54:00.190614+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:00.190614+0000","last_undegraded":"2024-04-25T15:54:00.190614+0000","last_fullsized":"2024-04-25T15:54:00.190614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"96'66","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.639480+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:54:02.639480+0000","last_peered":"2024-04-25T15:54:02.639480+0000","last_clean":"2024-04-25T15:54:02.639480+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:54:02.639480+0000","last_undegraded":"2024-04-25T15:54:02.639480+0000","last_fullsized":"2024-04-25T15:54:02.639480+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":233472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":178,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.190691+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:00.190691+0000","last_peered":"2024-04-25T15:54:00.190691+0000","last_clean":"2024-04-25T15:54:00.190691+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:00.190691+0000","last_undegraded":"2024-04-25T15:54:00.190691+0000","last_fullsized":"2024-04-25T15:54:00.190691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"96'55","reported_seq":176,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.068185+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:54:02.068185+0000","last_peered":"2024-04-25T15:54:02.068185+0000","last_clean":"2024-04-25T15:54:02.068185+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:54:02.068185+0000","last_undegraded":"2024-04-25T15:54:02.068185+0000","last_fullsized":"2024-04-25T15:54:02.068185+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":169,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.192127+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:00.192127+0000","last_peered":"2024-04-25T15:54:00.192127+0000","last_clean":"2024-04-25T15:54:00.192127+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:00.192127+0000","last_undegraded":"2024-04-25T15:54:00.192127+0000","last_fullsized":"2024-04-25T15:54:00.192127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"96'66","reported_seq":231,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:02.479511+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:54:02.479511+0000","last_peered":"2024-04-25T15:54:02.479511+0000","last_clean":"2024-04-25T15:54:02.479511+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:54:02.479511+0000","last_undegraded":"2024-04-25T15:54:02.479511+0000","last_fullsized":"2024-04-25T15:54:02.479511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":143,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T15:54:00.191485+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:54:00.191485+0000","last_peered":"2024-04-25T15:54:00.191485+0000","last_clean":"2024-04-25T15:54:00.191485+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:54:00.191485+0000","last_undegraded":"2024-04-25T15:54:00.191485+0000","last_fullsized":"2024-04-25T15:54:00.191485+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4427,"num_write_kb":16035840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":658505728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21474836480,"data_stored":21474836480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4428,"ondisk_log_size":4428,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":186240,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":311,"num_write_kb":278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":7427,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":622592,"data_stored":372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3465420,"kb_used_data":3430080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":90906420,"statfs":{"total":96636764160,"available":93088174080,"internally_reserved":0,"allocated":3512401920,"data_stored":3511761865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2506536,"kb_used_data":2471200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":91865304,"statfs":{"total":96636764160,"available":94070071296,"internally_reserved":0,"allocated":2530508800,"data_stored":2529881344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3657360,"kb_used_data":3617928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":90714480,"statfs":{"total":96636764160,"available":92891627520,"internally_reserved":0,"allocated":3704758272,"data_stored":3704111313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3427988,"kb_used_data":3380360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":90943852,"statfs":{"total":96636764160,"available":93126504448,"internally_reserved":0,"allocated":3461488640,"data_stored":3460851425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2809652,"kb_used_data":2770220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":91562188,"statfs":{"total":96636764160,"available":93759680512,"internally_reserved":0,"allocated":2836705280,"data_stored":2836065842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2256456,"kb_used_data":2225276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92115384,"statfs":{"total":96636764160,"available":94326153216,"internally_reserved":0,"allocated":2278682624,"data_stored":2278045685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675372,"kb_used_data":644192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696468,"statfs":{"total":96636764160,"available":95945183232,"internally_reserved":0,"allocated":659652608,"data_stored":659042914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2478292,"kb_used_data":2442952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":91893548,"statfs":{"total":96636764160,"available":94098993152,"internally_reserved":0,"allocated":2501582848,"data_stored":2500933718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":179830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":179524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2499805184,"data_stored":2499805184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":658505728,"data_stored":658505728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2277507072,"data_stored":2277507072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2835349504,"data_stored":2835349504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3460300800,"data_stored":3460300800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3703570432,"data_stored":3703570432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2529165312,"data_stored":2529165312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3510632448,"data_stored":3510632448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:54:08.683 INFO:tasks.thrashosds.thrasher:Setting 3.d to [7, 5, 2, 4] 2024-04-25T15:54:08.683 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.d', '7', '5', '2', '4'] 2024-04-25T15:54:08.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.d 7 5 2 4 2024-04-25T15:54:08.773 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:08.773+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:08.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.781+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fe2d40b4260 con 0x7fe2bc0051d0 2024-04-25T15:54:08.873 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:08.870+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:08.891 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:08.891 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:08.891 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:08.891 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:08.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:54:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.925+0000 7fc93d515640 1 Processor -- start 2024-04-25T15:54:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.925+0000 7fc93d515640 1 -- start start 2024-04-25T15:54:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc93d515640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc93815d160 0x7fc93815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc93d515640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93806ec70 0x7fc93806f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc93d515640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc938073460 con 0x7fc9381537e0 2024-04-25T15:54:08.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc93d515640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc938153bf0 con 0x7fc93815d160 2024-04-25T15:54:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc93d515640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc93806fe60 con 0x7fc93806ec70 2024-04-25T15:54:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc936ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc9381537e0 0x7fc938159f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:51874/0 (socket says 172.21.15.67:51874) 2024-04-25T15:54:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc936ffd640 1 -- 172.21.15.67:0/301305654 learned_addr learned my addr 172.21.15.67:0/301305654 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc92e7fc640 1 --2- 172.21.15.67:0/301305654 >> v2:172.21.15.176:3300/0 conn(0x7fc93815d160 0x7fc93815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc9377fe640 1 --2- 172.21.15.67:0/301305654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93806ec70 0x7fc93806f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc9367fc640 1 -- 172.21.15.67:0/301305654 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3607018480 0 0) 0x7fc938073460 con 0x7fc9381537e0 2024-04-25T15:54:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc9367fc640 1 -- 172.21.15.67:0/301305654 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc924003530 con 0x7fc9381537e0 2024-04-25T15:54:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc9377fe640 1 -- 172.21.15.67:0/301305654 >> v2:172.21.15.176:3300/0 conn(0x7fc93815d160 msgr2=0x7fc93815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc9377fe640 1 --2- 172.21.15.67:0/301305654 >> v2:172.21.15.176:3300/0 conn(0x7fc93815d160 0x7fc93815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc9377fe640 1 -- 172.21.15.67:0/301305654 >> v1:172.21.15.176:6789/0 conn(0x7fc9381537e0 legacy=0x7fc938159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.929+0000 7fc9377fe640 1 -- 172.21.15.67:0/301305654 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9380c9ea0 con 0x7fc93806ec70 2024-04-25T15:54:08.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc92e7fc640 1 --2- 172.21.15.67:0/301305654 >> v2:172.21.15.176:3300/0 conn(0x7fc93815d160 0x7fc93815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc9377fe640 1 --2- 172.21.15.67:0/301305654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93806ec70 0x7fc93806f080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc930009d90 tx=0x7fc930023580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc30dda867901a6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc9367fc640 1 -- 172.21.15.67:0/301305654 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9300b35f0 con 0x7fc93806ec70 2024-04-25T15:54:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc9367fc640 1 -- 172.21.15.67:0/301305654 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9300b8070 con 0x7fc93806ec70 2024-04-25T15:54:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc9367fc640 1 -- 172.21.15.67:0/301305654 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9300b39f0 con 0x7fc93806ec70 2024-04-25T15:54:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc93d515640 1 -- 172.21.15.67:0/301305654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93806ec70 msgr2=0x7fc93806f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc93d515640 1 --2- 172.21.15.67:0/301305654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93806ec70 0x7fc93806f080 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc930009d90 tx=0x7fc930023580 comp rx=0 tx=0).stop 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc93d515640 1 -- 172.21.15.67:0/301305654 shutdown_connections 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc93d515640 1 --2- 172.21.15.67:0/301305654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93806ec70 0x7fc93806f080 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc93d515640 1 --2- 172.21.15.67:0/301305654 >> v2:172.21.15.176:3300/0 conn(0x7fc93815d160 0x7fc93815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.933+0000 7fc93d515640 1 -- 172.21.15.67:0/301305654 >> 172.21.15.67:0/301305654 conn(0x7fc9380cebf0 msgr2=0x7fc9380cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.937+0000 7fc93d515640 1 -- 172.21.15.67:0/301305654 shutdown_connections 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.937+0000 7fc93d515640 1 -- 172.21.15.67:0/301305654 wait complete. 2024-04-25T15:54:08.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.937+0000 7fc93d515640 1 Processor -- start 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.937+0000 7fc93d515640 1 -- start start 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc93806ec70 0x7fc938154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9381537e0 0x7fc9381568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93815d160 0x7fc9380c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc93815cc40 con 0x7fc93806ec70 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc938073180 con 0x7fc9381537e0 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9380c9b10 con 0x7fc93815d160 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc93806ec70 0x7fc938154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc93806ec70 0x7fc938154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59972/0 (socket says 172.21.15.67:59972) 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 -- 172.21.15.67:0/2606619905 learned_addr learned my addr 172.21.15.67:0/2606619905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc92e7fc640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9381537e0 0x7fc9381568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc9377fe640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93815d160 0x7fc9380c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 -- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93815d160 msgr2=0x7fc9380c8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93815d160 0x7fc9380c8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 -- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9381537e0 msgr2=0x7fc9381568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9381537e0 0x7fc9381568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 -- 172.21.15.67:0/2606619905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc938155880 con 0x7fc93806ec70 2024-04-25T15:54:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc936ffd640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc93806ec70 0x7fc938154b90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc92800c3f0 tx=0x7fc92800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc92ffff640 1 -- 172.21.15.67:0/2606619905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc928017070 con 0x7fc93806ec70 2024-04-25T15:54:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc92ffff640 1 -- 172.21.15.67:0/2606619905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc928005cf0 con 0x7fc93806ec70 2024-04-25T15:54:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc92ffff640 1 -- 172.21.15.67:0/2606619905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9280072f0 con 0x7fc93806ec70 2024-04-25T15:54:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9381550d0 con 0x7fc93806ec70 2024-04-25T15:54:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.941+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc938155af0 con 0x7fc93806ec70 2024-04-25T15:54:08.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.945+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc93806a480 con 0x7fc93806ec70 2024-04-25T15:54:08.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.945+0000 7fc92ffff640 1 -- 172.21.15.67:0/2606619905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc92801a070 con 0x7fc93806ec70 2024-04-25T15:54:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.945+0000 7fc92ffff640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc91806f400 0x7fc918071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:08.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.945+0000 7fc92ffff640 1 -- 172.21.15.67:0/2606619905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5973+0+0 (secure 0 0 0) 0x7fc92808e6d0 con 0x7fc93806ec70 2024-04-25T15:54:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.945+0000 7fc92e7fc640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc91806f400 0x7fc918071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.949+0000 7fc92e7fc640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc91806f400 0x7fc918071910 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc9380c7f00 tx=0x7fc920009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:08.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:08.949+0000 7fc92ffff640 1 -- 172.21.15.67:0/2606619905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc92805ae70 con 0x7fc93806ec70 2024-04-25T15:54:09.011 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:09.010+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:09.076 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:09.074+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:09.165 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:09.165 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:09.165 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:09.165 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:09.165 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:09.179 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:09.177+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:09.198 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:54:09.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.221+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [7, 5, 2, 4]} v 0) -- 0x7fc9380ca650 con 0x7fc93806ec70 2024-04-25T15:54:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.249+0000 7fe2bbfff640 1 -- 172.21.15.67:0/3168127829 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v99) ==== 97+0+0 (secure 0 0 0) 0x7fe2c405bc20 con 0x7fe2bc0051d0 2024-04-25T15:54:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:54:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.249+0000 7fc92ffff640 1 -- 172.21.15.67:0/2606619905 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [7, 5, 2, 4]}]=0 set 3.d pg_upmap_items mapping to [7->5,2->4] v99) ==== 146+0+0 (secure 0 0 0) 0x7fc92805f170 con 0x7fc93806ec70 2024-04-25T15:54:09.252 INFO:teuthology.orchestra.run.smithi067.stderr:set 3.d pg_upmap_items mapping to [7->5,2->4] 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc91806f400 msgr2=0x7fc918071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc91806f400 0x7fc918071910 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc9380c7f00 tx=0x7fc920009190 comp rx=0 tx=0).stop 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc93806ec70 msgr2=0x7fc938154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc93806ec70 0x7fc938154b90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc92800c3f0 tx=0x7fc92800c8b0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 shutdown_connections 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc91806f400 0x7fc918071910 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc9380c7f00 tx=0x7fc920009190 comp rx=0 tx=0).stop 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc93815d160 0x7fc9380c8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9381537e0 0x7fc9381568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 --2- 172.21.15.67:0/2606619905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc93806ec70 0x7fc938154b90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.253+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 >> 172.21.15.67:0/2606619905 conn(0x7fc9380cebf0 msgr2=0x7fc9380cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:09.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 shutdown_connections 2024-04-25T15:54:09.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fc93d515640 1 -- 172.21.15.67:0/2606619905 wait complete. 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe2c006f400 msgr2=0x7fe2c0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe2c006f400 0x7fe2c0071910 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe2d414c070 tx=0x7fe2c8009040 comp rx=0 tx=0).stop 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0051d0 msgr2=0x7fe2d41520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0051d0 0x7fe2d41520d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fe2c40072c0 tx=0x7fe2c4007780 comp rx=0 tx=0).stop 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 shutdown_connections 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe2c006f400 0x7fe2c0071910 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2d4145c30 0x7fe2d4146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2d4153ed0 0x7fe2d41506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 --2- 172.21.15.67:0/3168127829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2bc0051d0 0x7fe2d41520d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:09.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 >> 172.21.15.67:0/3168127829 conn(0x7fe2d40ce1c0 msgr2=0x7fe2d4072f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:09.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 shutdown_connections 2024-04-25T15:54:09.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:09.257+0000 7fe2d9a63640 1 -- 172.21.15.67:0/3168127829 wait complete. 2024-04-25T15:54:09.287 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:09.286+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:09.378 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:09.377+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1777 3.1b 3:daa4f757:::100000001f8.000002ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:45.401998+0000", 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.049380837000001, 2024-04-25T15:54:09.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.116780538, 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1777 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.401998+0000", 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.401998+0000", 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.402025+0000", 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6185000000000001e-05 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.407170+0000", 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051459419999999997 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.496935+0000", 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.089764201000000002 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.496993+0000", 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8739000000000001e-05 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.497146+0000", 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015288500000000001 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.497267+0000", 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120788 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.497375+0000", 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010806000000000001 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.512753+0000", 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015378249 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.518595+0000", 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005842023 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.518687+0000", 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1265000000000001e-05 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.518779+0000", 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2200999999999994e-05 2024-04-25T15:54:09.458 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2272 3.13 e90/76 3:ca3e88be:::100000001f8.000004dc:head v 90'44, mlcod=90'42)", 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.812318+0000", 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.639061665, 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11464930399999999, 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.812318+0000", 2024-04-25T15:54:09.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.812318+0000", 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.812320+0000", 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.357e-06 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816423+0000", 2024-04-25T15:54:09.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041028100000000001 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816463+0000", 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0609999999999999e-05 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816488+0000", 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4808e-05 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816674+0000", 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0001861 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816729+0000", 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.448e-05 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.926331+0000", 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.109602326 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.926967+0000", 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063581300000000005 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2309 3.2c e90/76 3:354b7d43:::100000001f8.00000501:head v 90'43, mlcod=90'41)", 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.840375+0000", 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.611004307999998, 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11405333500000001, 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.463 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.840375+0000", 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.840375+0000", 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.840377+0000", 2024-04-25T15:54:09.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5919999999999999e-06 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844561+0000", 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041837319999999999 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844596+0000", 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5222999999999998e-05 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844620+0000", 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3467000000000001e-05 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844813+0000", 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019298699999999999 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844856+0000", 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3064e-05 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954245+0000", 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10938917300000001 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954428+0000", 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018309700000000001 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2294 3.11 e90/76 3:8bba9e6f:::100000001f8.000004f2:head v 90'68, mlcod=90'65)", 2024-04-25T15:54:09.467 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.871235+0000", 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 20.580144147999999, 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.083553868000000003, 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.871235+0000", 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.871235+0000", 2024-04-25T15:54:09.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.871237+0000", 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9889999999999999e-06 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.874573+0000", 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033363329999999999 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.874625+0000", 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1808000000000003e-05 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.874643+0000", 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7455000000000001e-05 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.874731+0000", 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8137e-05 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.874770+0000", 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9167e-05 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954724+0000", 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079954301000000005 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954789+0000", 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4678000000000007e-05 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.471 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3918 3.11 e97/76 3:8b260724:::100000001f8.0000014a:head v 97'96, mlcod=96'94)", 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518716+0000", 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.9326632989999997, 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13541348, 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518716+0000", 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518716+0000", 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518717+0000", 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.302e-06 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524601+0000", 2024-04-25T15:54:09.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058834430000000004 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524652+0000", 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0924000000000002e-05 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524723+0000", 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1606999999999994e-05 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524827+0000", 2024-04-25T15:54:09.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103522 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524861+0000", 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4541e-05 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654069+0000", 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12920799299999999 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654129+0000", 2024-04-25T15:54:09.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0148000000000001e-05 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3919 3.2c e97/76 3:37bce3fe:::100000001f8.0000014b:head v 97'73, mlcod=96'70)", 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.520193+0000", 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.9311863259999997, 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14953171500000001, 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520193+0000", 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520193+0000", 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520195+0000", 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.703e-06 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522166+0000", 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019710040000000002 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522206+0000", 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0635e-05 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522229+0000", 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2847999999999999e-05 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522327+0000", 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8157000000000002e-05 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522365+0000", 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7437000000000001e-05 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669684+0000", 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.147319441 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669725+0000", 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.049e-05 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3921 3.29 e97/76 3:95d78bcd:::100000001f8.0000014d:head v 97'62, mlcod=96'60)", 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.522303+0000", 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.9290766259999996, 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14725027600000001, 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.480 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522303+0000", 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522303+0000", 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522304+0000", 2024-04-25T15:54:09.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.031e-06 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531230+0000", 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089262600000000001 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531250+0000", 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0458e-05 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531271+0000", 2024-04-25T15:54:09.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0185000000000001e-05 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531359+0000", 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8338999999999996e-05 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531391+0000", 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2465999999999997e-05 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669508+0000", 2024-04-25T15:54:09.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.138116562 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669553+0000", 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4975000000000002e-05 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3927 3.34 3:2dd1f0a3:::100000001f8.00000153:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.542469+0000", 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.9089103229999997, 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12736851099999999, 2024-04-25T15:54:09.484 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3927 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542469+0000", 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542469+0000", 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542477+0000", 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8120000000000008e-06 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553089+0000", 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010612018000000001 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553128+0000", 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9477000000000002e-05 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553142+0000", 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.376e-05 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553223+0000", 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1075999999999994e-05 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553324+0000", 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101331 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553462+0000", 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000137856 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669674+0000", 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.116212207 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669735+0000", 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0303999999999997e-05 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669789+0000", 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4168e-05 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669837+0000", 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8501999999999997e-05 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.489 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3932 3.13 e97/76 3:cac515da:::100000001f8.00000158:head v 97'70, mlcod=97'68)", 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.579647+0000", 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.8717322080000001, 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.089741781000000007, 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.579647+0000", 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.579647+0000", 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.579649+0000", 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1459999999999999e-06 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.581101+0000", 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001452332 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.581126+0000", 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4029e-05 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.581140+0000", 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4296e-05 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.581266+0000", 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012605799999999999 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.581303+0000", 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7443e-05 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669333+0000", 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.088029360000000001 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669389+0000", 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6116999999999998e-05 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3963 3.19 e97/76 3:9b5b1331:::100000001f8.00000177:head v 97'70, mlcod=97'68)", 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.800007+0000", 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6513721690000001, 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14600712099999999, 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.494 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.800007+0000", 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.800007+0000", 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.800010+0000", 2024-04-25T15:54:09.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5679999999999998e-06 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804743+0000", 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047334430000000004 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804764+0000", 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1237e-05 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804797+0000", 2024-04-25T15:54:09.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3062999999999998e-05 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804866+0000", 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8603999999999996e-05 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804886+0000", 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0497999999999999e-05 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945932+0000", 2024-04-25T15:54:09.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.141045114 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.946014+0000", 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2594000000000002e-05 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3966 3.2c e97/76 3:36f7a988:::100000001f8.0000017a:head v 97'74, mlcod=97'72)", 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.805041+0000", 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6463377450000003, 2024-04-25T15:54:09.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.124131596, 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.805041+0000", 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.805041+0000", 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.805043+0000", 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0729999999999999e-06 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806829+0000", 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017861470000000001 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806845+0000", 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6404000000000001e-05 2024-04-25T15:54:09.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806859+0000", 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3749e-05 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806952+0000", 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3166999999999998e-05 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806982+0000", 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9612999999999999e-05 2024-04-25T15:54:09.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929100+0000", 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122118665 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929173+0000", 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2778e-05 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.502 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3972 3.a e97/76 3:506f1d9f:::100000001f8.00000180:head v 97'56, mlcod=96'54)", 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.830816+0000", 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6205636190000003, 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.149891782, 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830816+0000", 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830816+0000", 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830817+0000", 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1069999999999999e-06 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834726+0000", 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039096490000000003 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834762+0000", 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5710000000000002e-05 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834785+0000", 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3141000000000001e-05 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834948+0000", 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016240199999999999 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834980+0000", 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2116999999999998e-05 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.980644+0000", 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14566418 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.980707+0000", 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3476e-05 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.506 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3974 3.2c e97/76 3:35c2bab0:::100000001f8.00000182:head v 97'75, mlcod=97'72)", 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.838921+0000", 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6124577999999996, 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13982625600000001, 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.838921+0000", 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.838921+0000", 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.838924+0000", 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1440000000000001e-06 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.840999+0000", 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020758159999999999 2024-04-25T15:54:09.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841032+0000", 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2503999999999999e-05 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841056+0000", 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3901999999999999e-05 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841152+0000", 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5743999999999997e-05 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841220+0000", 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8163000000000005e-05 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978691+0000", 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13747087999999999 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978748+0000", 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7102999999999999e-05 2024-04-25T15:54:09.510 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3977 3.18 e97/76 3:1b05c35d:::100000001f8.00000185:head v 97'74, mlcod=96'72)", 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.843861+0000", 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6075179190000002, 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13525642099999999, 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.843861+0000", 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.843861+0000", 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.843863+0000", 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.559e-06 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850482+0000", 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066194130000000002 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850499+0000", 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7232e-05 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850512+0000", 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2152e-05 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850619+0000", 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107804 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850653+0000", 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3905999999999997e-05 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979060+0000", 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128406873 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979118+0000", 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7482000000000003e-05 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3976 3.2c e97/76 3:35666940:::100000001f8.00000184:head v 97'76, mlcod=97'72)", 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.846326+0000", 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.6050531469999996, 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13284864299999999, 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.515 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846326+0000", 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846326+0000", 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846327+0000", 2024-04-25T15:54:09.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1799999999999999e-06 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.849694+0000", 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003366834 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.849717+0000", 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2884e-05 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.849737+0000", 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.001e-05 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.855904+0000", 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061673459999999998 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.855939+0000", 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4866000000000001e-05 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979128+0000", 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12318931800000001 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979175+0000", 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6205000000000002e-05 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3967 3.24 e97/76 3:260ecc6d:::100000001f8.0000017b:head v 97'65, mlcod=96'63)", 2024-04-25T15:54:09.519 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.860511+0000", 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5908679210000001, 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.118943471, 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.860511+0000", 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.860511+0000", 2024-04-25T15:54:09.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.860513+0000", 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5650000000000001e-06 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864390+0000", 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038770699999999998 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864408+0000", 2024-04-25T15:54:09.521 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7723999999999998e-05 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864421+0000", 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3507e-05 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864487+0000", 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5382999999999994e-05 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864507+0000", 2024-04-25T15:54:09.522 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0160999999999999e-05 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979407+0000", 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.114900351 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979455+0000", 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7710000000000002e-05 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.523 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3981 3.1d e97/76 3:b93ad013:::100000001f8.00000189:head v 97'83, mlcod=96'81)", 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.871866+0000", 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5795130349999997, 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.107716402, 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871866+0000", 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.524 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871866+0000", 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871868+0000", 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8080000000000001e-06 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.875080+0000", 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003211855 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.525 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.875100+0000", 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0152999999999998e-05 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.875113+0000", 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2571e-05 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.875185+0000", 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2616999999999999e-05 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.526 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.875218+0000", 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3012000000000002e-05 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979539+0000", 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.104321185 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979583+0000", 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3201000000000002e-05 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.527 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3986 3.9 3:90f5a2a8:::100000001f8.0000018e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.886618+0000", 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5647607910000003, 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099890918999999995, 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3986 2024-04-25T15:54:09.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.886618+0000", 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.886618+0000", 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.886624+0000", 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.643e-06 2024-04-25T15:54:09.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.896189+0000", 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0095645000000000001 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.896236+0000", 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7205e-05 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.896258+0000", 2024-04-25T15:54:09.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2229999999999999e-05 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.896363+0000", 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104633 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.896518+0000", 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015529899999999999 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.896719+0000", 2024-04-25T15:54:09.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020116000000000001 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.920179+0000", 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.023459646000000001 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.986328+0000", 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.066149388000000003 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.986421+0000", 2024-04-25T15:54:09.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3317000000000001e-05 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.986509+0000", 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7898000000000004e-05 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3984 3.1c e97/76 3:39daebce:::100000001f8.0000018c:head v 97'70, mlcod=97'68)", 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.891385+0000", 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5599943449999998, 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.095884412000000002, 2024-04-25T15:54:09.533 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.891385+0000", 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.891385+0000", 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.891387+0000", 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4329999999999998e-06 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895407+0000", 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040200619999999996 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895434+0000", 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6166e-05 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895454+0000", 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0733e-05 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895547+0000", 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2639999999999994e-05 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895579+0000", 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2027999999999999e-05 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.987195+0000", 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.091615642999999997 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.987269+0000", 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4707000000000001e-05 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3985 3.11 e97/76 3:8a135217:::100000001f8.0000018d:head v 97'97, mlcod=96'95)", 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.895596+0000", 2024-04-25T15:54:09.537 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.5557827279999996, 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.091119942999999995, 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895596+0000", 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895596+0000", 2024-04-25T15:54:09.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.895598+0000", 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4580000000000001e-06 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.899120+0000", 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0035219169999999998 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.899146+0000", 2024-04-25T15:54:09.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5718e-05 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.899164+0000", 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8493999999999999e-05 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.899282+0000", 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011840099999999999 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.899316+0000", 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3974999999999999e-05 2024-04-25T15:54:09.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.986642+0000", 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087325157 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.986716+0000", 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4822999999999998e-05 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.541 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:09.543 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:54:09.543 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:09.538+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:09.644 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:09.642+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:09.745 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:09.742+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:09.817 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.817 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5018 3.c 3:302e3aab:::100000001f8.00000596:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:09.801579+0000", 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.015759039999999998, 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015813942000000001, 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5018 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.801579+0000", 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.801579+0000", 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.801583+0000", 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6490000000000001e-06 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.806539+0000", 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049565399999999997 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.806596+0000", 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7160999999999997e-05 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.806614+0000", 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7546000000000001e-05 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.806695+0000", 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0859000000000006e-05 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.806819+0000", 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012409399999999999 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.806995+0000", 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017631999999999999 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:09.821 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-04-25T15:54:09.822 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 1 2024-04-25T15:54:09.822 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:09.846 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:09.845+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:09.855 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:54:09.947 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:09.942+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:10.047 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:10.045+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:10.105 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:10.105 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:10.106 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:10.106 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:10.106 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:10.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:54:10.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:10.145+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:10.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:54:10.249 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:10.246+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:10.350 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:10.349+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:10.370 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 67.150201272000004, 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.371 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:54:10.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:54:10.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:54:10.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2305 3.2f 3:f6648653:::100000001f8.000004fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.566437635, 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16263440000000001, 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.378 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2305 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:10.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802795+0000", 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7530000000000001e-06 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815251+0000", 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012456312000000001 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815675+0000", 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042352399999999999 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815700+0000", 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5225e-05 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815859+0000", 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015856099999999999 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816239+0000", 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038042099999999998 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816444+0000", 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020472099999999999 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.906577+0000", 2024-04-25T15:54:10.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090133536 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965298+0000", 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058721044999999999 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965364+0000", 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6013000000000006e-05 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965425+0000", 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0288999999999998e-05 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2304 3.36 e90/18 3:6f9fe141:::100000001f8.000004fc:head v 90'48, mlcod=89'46)", 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.565111372000001, 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14476808899999999, 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.384 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804121+0000", 2024-04-25T15:54:10.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3950000000000003e-06 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818809+0000", 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014687881999999999 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818838+0000", 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9689e-05 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818859+0000", 2024-04-25T15:54:10.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0353999999999999e-05 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.819110+0000", 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025134700000000002 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.819181+0000", 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1094999999999996e-05 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.948829+0000", 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12964726300000001 2024-04-25T15:54:10.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.948885+0000", 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6063999999999997e-05 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2306 3.5 3:a0445fe6:::100000001f8.000004fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.553965246000001, 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.1339195, 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.388 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2306 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:10.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815265+0000", 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2759999999999999e-06 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830149+0000", 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014884298000000001 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830753+0000", 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060401599999999997 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830772+0000", 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9017000000000001e-05 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.831046+0000", 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027419800000000002 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.832603+0000", 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015568450000000001 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833182+0000", 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057851500000000004 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.859487+0000", 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026305354 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949024+0000", 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.089536491999999995 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949121+0000", 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7589000000000004e-05 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949182+0000", 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0900000000000003e-05 2024-04-25T15:54:10.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2308 3.4 e90/68 3:2343a40c:::100000001f8.00000500:head v 90'35, mlcod=90'33)", 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.546030809000001, 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12610180200000001, 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823200+0000", 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4590000000000001e-06 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833167+0000", 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0099672500000000004 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833193+0000", 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6231999999999999e-05 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833214+0000", 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0942e-05 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833316+0000", 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101772 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833360+0000", 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3835999999999998e-05 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949250+0000", 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11589087000000001 2024-04-25T15:54:10.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949299+0000", 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8440999999999999e-05 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2307 3.35 e90/18 3:ad417e89:::100000001f8.000004ff:head v 90'49, mlcod=90'47)", 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.828366+0000", 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 21.540862132000001, 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.121096469, 2024-04-25T15:54:10.398 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.828366+0000", 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.828366+0000", 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.828368+0000", 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2000000000000001e-06 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833667+0000", 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052990789999999999 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833694+0000", 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6800000000000001e-05 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833715+0000", 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379e-05 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.841285+0000", 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075698600000000003 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.841333+0000", 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8004999999999999e-05 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949416+0000", 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10808335099999999 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949462+0000", 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5794999999999998e-05 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.402 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3916 3.c 3:3043a2b3:::100000001f8.00000148:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.493213+0000", 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.8760151900000004, 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15984567799999999, 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3916 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493213+0000", 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493213+0000", 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493219+0000", 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0730000000000001e-06 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503369+0000", 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010149867 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503412+0000", 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3680000000000002e-05 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503429+0000", 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6395000000000001e-05 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503520+0000", 2024-04-25T15:54:10.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1250999999999998e-05 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503645+0000", 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125463 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503826+0000", 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180543 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.530142+0000", 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026315782999999999 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.652922+0000", 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122779765 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.652979+0000", 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7485999999999998e-05 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653058+0000", 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9371999999999998e-05 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3915 3.35 e97/18 3:ac1713fb:::100000001f8.00000147:head v 97'80, mlcod=96'78)", 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.501267+0000", 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.8679613479999997, 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15192006999999999, 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.408 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501267+0000", 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501267+0000", 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501268+0000", 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.674e-06 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504879+0000", 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003611104 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504900+0000", 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0581000000000002e-05 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504920+0000", 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9905999999999999e-05 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.505199+0000", 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000279399 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.505242+0000", 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3078000000000003e-05 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653126+0000", 2024-04-25T15:54:10.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14788367899999999 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653187+0000", 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0649000000000002e-05 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3922 3.d e97/18 3:b3f0d00e:::100000001f8.0000014e:head v 97'72, mlcod=97'69)", 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.529755+0000", 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.8394729769999998, 2024-04-25T15:54:10.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13872857699999999, 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529755+0000", 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529755+0000", 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529757+0000", 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.996e-06 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.536951+0000", 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00719384 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.536973+0000", 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1857000000000001e-05 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.536990+0000", 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7382999999999999e-05 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.537138+0000", 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014826999999999999 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.537160+0000", 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1552e-05 2024-04-25T15:54:10.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668418+0000", 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13125822300000001 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668483+0000", 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5456000000000005e-05 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.416 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3923 3.35 e97/18 3:ac352473:::100000001f8.0000014f:head v 97'81, mlcod=96'78)", 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.532588+0000", 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.836639828, 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13704064699999999, 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532588+0000", 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532588+0000", 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532590+0000", 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2979999999999999e-06 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535339+0000", 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027488389999999999 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535368+0000", 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8747000000000001e-05 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535389+0000", 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1192e-05 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535503+0000", 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011338200000000001 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535538+0000", 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5169000000000002e-05 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669570+0000", 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13403183299999999 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669629+0000", 2024-04-25T15:54:10.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9187000000000001e-05 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3925 3.28 3:143638ba:::100000001f8.00000151:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.534277+0000", 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.8349508370000001, 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13462772200000001, 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.421 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3925 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.534277+0000", 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.534277+0000", 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.534281+0000", 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0010000000000003e-06 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541593+0000", 2024-04-25T15:54:10.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073119880000000002 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541641+0000", 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7651999999999997e-05 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541660+0000", 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9785000000000001e-05 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541838+0000", 2024-04-25T15:54:10.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177877 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542057+0000", 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021911399999999999 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542366+0000", 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030894100000000002 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.578475+0000", 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036108636999999999 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668789+0000", 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090313702999999995 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668851+0000", 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2022999999999999e-05 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668905+0000", 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4001e-05 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3926 3.d e97/18 3:b20dc027:::100000001f8.00000152:head v 97'73, mlcod=97'69)", 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.543642+0000", 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.8255863080000001, 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12602907699999999, 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.427 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543642+0000", 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543642+0000", 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543643+0000", 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5540000000000001e-06 2024-04-25T15:54:10.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546584+0000", 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002940397 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546605+0000", 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1013e-05 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546619+0000", 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4219000000000001e-05 2024-04-25T15:54:10.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546713+0000", 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4411000000000001e-05 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546738+0000", 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4433000000000001e-05 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669633+0000", 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122895448 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669671+0000", 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7601999999999997e-05 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3929 3.c 3:3222fa1b:::100000001f8.00000155:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:10.431 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.552567+0000", 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.8166609510000002, 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.116475469, 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3929 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.552567+0000", 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.552567+0000", 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.552572+0000", 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2429999999999996e-06 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.557508+0000", 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049355980000000002 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.557558+0000", 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0052999999999997e-05 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.557580+0000", 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2039999999999999e-05 2024-04-25T15:54:10.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.557691+0000", 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110782 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.557823+0000", 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013229499999999999 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:10.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.558014+0000", 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000191175 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.583797+0000", 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025783325999999999 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668950+0000", 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.085152248999999999 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669000+0000", 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9954000000000003e-05 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669042+0000", 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2753999999999998e-05 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3970 3.16 3:694e1180:::100000001f8.0000017e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.813653+0000", 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.555574708, 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16449430000000001, 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.438 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3970 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.813653+0000", 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.813653+0000", 2024-04-25T15:54:10.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.813655+0000", 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2290000000000002e-06 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817783+0000", 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041279630000000001 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817818+0000", 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5043000000000003e-05 2024-04-25T15:54:10.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817832+0000", 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4054e-05 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818594+0000", 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076106100000000005 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818664+0000", 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0543000000000005e-05 2024-04-25T15:54:10.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818770+0000", 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010637300000000001 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945674+0000", 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12690309999999999 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978006+0000", 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032332847999999997 2024-04-25T15:54:10.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978086+0000", 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9950000000000005e-05 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978147+0000", 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1136000000000006e-05 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.443 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3973 3.3a 3:5d7d7b7b:::100000001f8.00000181:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.825096+0000", 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.5441322609999997, 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.153213244, 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3973 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.825096+0000", 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.825096+0000", 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.825098+0000", 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4600000000000002e-06 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832630+0000", 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075320489999999999 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832678+0000", 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8359999999999998e-05 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832702+0000", 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3402999999999999e-05 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832810+0000", 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107723 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832947+0000", 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013714900000000001 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.833088+0000", 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014118099999999999 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846427+0000", 2024-04-25T15:54:10.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013338645 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978210+0000", 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13178303999999999 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978263+0000", 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3341000000000003e-05 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978309+0000", 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5893000000000003e-05 2024-04-25T15:54:10.448 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3981 3.1d 3:b93ad013:::100000001f8.00000189:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.867488+0000", 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 6.5017398120000003, 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11268381700000001, 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.449 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3981 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.867488+0000", 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.867488+0000", 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.867499+0000", 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1114000000000001e-05 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871222+0000", 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037233069999999999 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871259+0000", 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6921000000000001e-05 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871275+0000", 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5204999999999999e-05 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871371+0000", 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5916999999999997e-05 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871499+0000", 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012859699999999999 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.871700+0000", 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020040499999999999 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979683+0000", 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.107983565 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.980080+0000", 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039648599999999998 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.980132+0000", 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1983e-05 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.980172+0000", 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0317e-05 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5054 3.3a 3:5eb75885:::100000001f8.000005ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:10.454 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:09.986550+0000", 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.382677461, 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12179451500000001, 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5054 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.986550+0000", 2024-04-25T15:54:10.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.986550+0000", 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.986552+0000", 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.809e-06 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991418+0000", 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048656230000000003 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991614+0000", 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019635900000000001 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991630+0000", 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5789000000000001e-05 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991727+0000", 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6520000000000004e-05 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991853+0000", 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000126542 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.992031+0000", 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017744700000000001 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.009109+0000", 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017078945000000002 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108172+0000", 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099062997 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108255+0000", 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2055e-05 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108345+0000", 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0428999999999998e-05 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5055 3.1c 3:394277a5:::100000001f8.000005bb:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:09.991443+0000", 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.37778457300000001, 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11705053999999999, 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.460 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5055 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991443+0000", 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991443+0000", 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991449+0000", 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9170000000000003e-06 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.996473+0000", 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050235059999999996 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.996509+0000", 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6687999999999998e-05 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.996531+0000", 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1903999999999999e-05 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.996627+0000", 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6120999999999997e-05 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.996726+0000", 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9011000000000005e-05 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.996877+0000", 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015065799999999999 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.106661+0000", 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.109784285 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108391+0000", 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017300519999999999 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108447+0000", 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5494000000000001e-05 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108494+0000", 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6903999999999997e-05 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5056 3.38 e99/18 3:1d039b80:::100000001f8.000005bc:head v 99'105, mlcod=98'103)", 2024-04-25T15:54:10.465 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:09.997790+0000", 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.37143798900000002, 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.110838151, 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.997790+0000", 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.997790+0000", 2024-04-25T15:54:10.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.997792+0000", 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6479999999999999e-06 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.003669+0000", 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058779069999999999 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.003699+0000", 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9692999999999998e-05 2024-04-25T15:54:10.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.003718+0000", 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8944999999999999e-05 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.003801+0000", 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3267000000000001e-05 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.003825+0000", 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.376e-05 2024-04-25T15:54:10.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108566+0000", 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.104741002 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108628+0000", 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1928999999999996e-05 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.469 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5062 3.2f 3:f4e1ea89:::100000001f8.000005c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.024415+0000", 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.34481276999999999, 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12491956899999999, 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5062 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.024415+0000", 2024-04-25T15:54:10.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.024415+0000", 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.024417+0000", 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0269999999999998e-06 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:10.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028822+0000", 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044052029999999999 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028859+0000", 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6504000000000002e-05 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028878+0000", 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9363000000000001e-05 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028992+0000", 2024-04-25T15:54:10.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011408099999999999 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.029609+0000", 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061717599999999999 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.029826+0000", 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000216257 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.109119+0000", 2024-04-25T15:54:10.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079293001000000002 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149258+0000", 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040138807999999998 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149300+0000", 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2765999999999997e-05 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:10.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149335+0000", 2024-04-25T15:54:10.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4382999999999999e-05 2024-04-25T15:54:10.475 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.475 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.475 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.475 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:10.475 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:10.475 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:10.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:54:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.453+0000 7f0523a97640 1 Processor -- start 2024-04-25T15:54:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.453+0000 7f0523a97640 1 -- start start 2024-04-25T15:54:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.453+0000 7f0523a97640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.453+0000 7f0523a97640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f051c0b5660 0x7f051c0b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0523a97640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f051c15fe00 0x7f051c160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0523a97640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f051c15d510 con 0x7f051c0b5660 2024-04-25T15:54:10.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0523a97640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f051c06d780 con 0x7f051c15fe00 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0523a97640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f051c160be0 con 0x7f051c06ca60 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521833640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521833640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c06ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:38810/0 (socket says 172.21.15.67:38810) 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521833640 1 -- 172.21.15.67:0/2744695932 learned_addr learned my addr 172.21.15.67:0/2744695932 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521032640 1 --2- 172.21.15.67:0/2744695932 >> v2:172.21.15.67:3300/0 conn(0x7f051c0b5660 0x7f051c0b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521032640 1 -- 172.21.15.67:0/2744695932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 msgr2=0x7f051c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521032640 1 --2- 172.21.15.67:0/2744695932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521032640 1 -- 172.21.15.67:0/2744695932 >> v2:172.21.15.176:3300/0 conn(0x7f051c15fe00 msgr2=0x7f051c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521032640 1 --2- 172.21.15.67:0/2744695932 >> v2:172.21.15.176:3300/0 conn(0x7f051c15fe00 0x7f051c160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521032640 1 -- 172.21.15.67:0/2744695932 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051c15cbe0 con 0x7f051c0b5660 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0521032640 1 --2- 172.21.15.67:0/2744695932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c0b5660 0x7f051c0b5a70 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f051c0cf690 tx=0x7f050c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f590b44c082032a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0520831640 1 -- 172.21.15.67:0/2744695932 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c0b0070 con 0x7f051c0b5660 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0520831640 1 -- 172.21.15.67:0/2744695932 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f050c0ab600 con 0x7f051c0b5660 2024-04-25T15:54:10.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.457+0000 7f0520831640 1 -- 172.21.15.67:0/2744695932 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c0ab9d0 con 0x7f051c0b5660 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 -- 172.21.15.67:0/2744695932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c0b5660 msgr2=0x7f051c0b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 --2- 172.21.15.67:0/2744695932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c0b5660 0x7f051c0b5a70 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f051c0cf690 tx=0x7f050c01c480 comp rx=0 tx=0).stop 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 -- 172.21.15.67:0/2744695932 shutdown_connections 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 --2- 172.21.15.67:0/2744695932 >> v2:172.21.15.176:3300/0 conn(0x7f051c15fe00 0x7f051c160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 --2- 172.21.15.67:0/2744695932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c0b5660 0x7f051c0b5a70 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 --2- 172.21.15.67:0/2744695932 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 -- 172.21.15.67:0/2744695932 >> 172.21.15.67:0/2744695932 conn(0x7f051c06e420 msgr2=0x7f051c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 -- 172.21.15.67:0/2744695932 shutdown_connections 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 -- 172.21.15.67:0/2744695932 wait complete. 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 Processor -- start 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.461+0000 7f0523a97640 1 -- start start 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0523a97640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c14a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0523a97640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f051c0b5660 0x7f051c15efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0523a97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c15fe00 0x7f051c147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0523a97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f051c0b6e90 con 0x7f051c15fe00 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0523a97640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f051c15c890 con 0x7f051c0b5660 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0523a97640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f051c0b7010 con 0x7f051c06ca60 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0522034640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c15fe00 0x7f051c147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c14a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c14a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:38824/0 (socket says 172.21.15.67:38824) 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 -- 172.21.15.67:0/3205387949 learned_addr learned my addr 172.21.15.67:0/3205387949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521032640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f051c0b5660 0x7f051c15efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f051c0b5660 msgr2=0x7f051c15efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f051c0b5660 0x7f051c15efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c15fe00 msgr2=0x7f051c147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f051c15fe00 0x7f051c147b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.465+0000 7f0521833640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051c14a920 con 0x7f051c06ca60 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.469+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c14a3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f051800e8a0 tx=0x7f051800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.469+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0518017070 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.469+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f051800c940 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.469+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051c1f9780 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.469+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051800cd10 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.469+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f051c1f9d00 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.469+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f051c06a480 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f051801a070 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f05127fc640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f050006f400 0x7f0500071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(99..99 src has 1..99) ==== 6002+0+0 (secure 0 0 0) 0x7f051808f930 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v99) ==== 54+0+16 (secure 0 0 0) 0x7f05180900c0 con 0x7f051c06ca60 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f051c150440 0x7f051c0d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f051c076ec0 con 0x7f051c150440 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f0522034640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f051c150440 0x7f051c0d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f0522034640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f051c150440 0x7f051c0d0ea0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.486 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:10.473+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:10.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.477+0000 7f0521032640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f050006f400 0x7f0500071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.477+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c076ec0 con 0x7f051c150440 2024-04-25T15:54:10.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.477+0000 7f0521032640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f050006f400 0x7f0500071910 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f050c0ad000 tx=0x7f050c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.505+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c0b4540 con 0x7f051c150440 2024-04-25T15:54:10.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.505+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c0b4540 con 0x7f051c150440 2024-04-25T15:54:10.508 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:10.508 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:10.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.505+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f051c0cfa00 0x7f051c0cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.505+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f051c14ab30 con 0x7f051c0cfa00 2024-04-25T15:54:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.505+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f051c0cfa00 0x7f051c0cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.509+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f051c0cfa00 0x7f051c0cfe10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.509+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c14ab30 con 0x7f051c0cfa00 2024-04-25T15:54:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.525+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c154fb0 con 0x7f051c0cfa00 2024-04-25T15:54:10.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.533+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c154fb0 con 0x7f051c0cfa00 2024-04-25T15:54:10.533 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:10.533 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:10.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.533+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f051c063090 0x7f051c0634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.533+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f051c0d0860 con 0x7f051c063090 2024-04-25T15:54:10.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.533+0000 7f0521032640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f051c063090 0x7f051c0634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.533+0000 7f0521032640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f051c063090 0x7f051c0634a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.533+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c0d0860 con 0x7f051c063090 2024-04-25T15:54:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.553+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c0d1a60 con 0x7f051c063090 2024-04-25T15:54:10.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.553+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c0d1a60 con 0x7f051c063090 2024-04-25T15:54:10.556 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.553+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f051c15a700 0x7f051c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.553+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f051c063ef0 con 0x7f051c15a700 2024-04-25T15:54:10.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.557+0000 7f0522034640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f051c15a700 0x7f051c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.557+0000 7f0522034640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f051c15a700 0x7f051c15ab10 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.557+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c063ef0 con 0x7f051c15a700 2024-04-25T15:54:10.577 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:10.577+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:10.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.577+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c063ef0 con 0x7f051c15a700 2024-04-25T15:54:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.581+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c063ef0 con 0x7f051c15a700 2024-04-25T15:54:10.583 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:10.583 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.581+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f051c15b380 0x7f051c02cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.581+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f051c02d430 con 0x7f051c15b380 2024-04-25T15:54:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.581+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f051c15b380 0x7f051c02cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.585+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f051c15b380 0x7f051c02cef0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.585+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c02d430 con 0x7f051c15b380 2024-04-25T15:54:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.605+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c02d430 con 0x7f051c15b380 2024-04-25T15:54:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.609+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c02d430 con 0x7f051c15b380 2024-04-25T15:54:10.609 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:10.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.609+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f051c02dab0 0x7f051c20c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.609+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f051c02dec0 con 0x7f051c02dab0 2024-04-25T15:54:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.609+0000 7f0521032640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f051c02dab0 0x7f051c20c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.609+0000 7f0521032640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f051c02dab0 0x7f051c20c970 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.609+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c02dec0 con 0x7f051c02dab0 2024-04-25T15:54:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.637+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c02dec0 con 0x7f051c02dab0 2024-04-25T15:54:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.641+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c02dec0 con 0x7f051c02dab0 2024-04-25T15:54:10.642 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:10.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.641+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f051c20d3c0 0x7f051c20f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.641+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f051c2125b0 con 0x7f051c20d3c0 2024-04-25T15:54:10.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.641+0000 7f0522034640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f051c20d3c0 0x7f051c20f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.641+0000 7f0522034640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f051c20d3c0 0x7f051c20f7e0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.645+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c2125b0 con 0x7f051c20d3c0 2024-04-25T15:54:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.669+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c20fd20 con 0x7f051c20d3c0 2024-04-25T15:54:10.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.669+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c20fd20 con 0x7f051c20d3c0 2024-04-25T15:54:10.671 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:10.672 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:10.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.669+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f051c2103a0 0x7f051c2127c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:10.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.669+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f051c215640 con 0x7f051c2103a0 2024-04-25T15:54:10.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.669+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f051c2103a0 0x7f051c2127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:10.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.673+0000 7f0521833640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f051c2103a0 0x7f051c2127c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:10.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.673+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f051c215640 con 0x7f051c2103a0 2024-04-25T15:54:10.678 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:10.674+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:10.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.693+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f051c212d80 con 0x7f051c2103a0 2024-04-25T15:54:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f05127fc640 1 -- 172.21.15.67:0/3205387949 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f051c212d80 con 0x7f051c2103a0 2024-04-25T15:54:10.697 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:10.697 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f051c150440 msgr2=0x7f051c0d0ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f051c150440 0x7f051c0d0ea0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f051c0cfa00 msgr2=0x7f051c0cfe10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f051c0cfa00 0x7f051c0cfe10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f051c063090 msgr2=0x7f051c0634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f051c063090 0x7f051c0634a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0522034640 1 -- 172.21.15.67:0/3205387949 reap_dead start 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f051c15a700 msgr2=0x7f051c15ab10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f051c15a700 0x7f051c15ab10 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f051c15b380 msgr2=0x7f051c02cef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f051c15b380 0x7f051c02cef0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f051c02dab0 msgr2=0x7f051c20c970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f051c02dab0 0x7f051c20c970 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f051c20d3c0 msgr2=0x7f051c20f7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f051c20d3c0 0x7f051c20f7e0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f051c2103a0 msgr2=0x7f051c2127c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f051c2103a0 0x7f051c2127c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0522034640 1 -- 172.21.15.67:0/3205387949 reap_dead start 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f050006f400 msgr2=0x7f0500071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f050006f400 0x7f0500071910 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f050c0ad000 tx=0x7f050c008040 comp rx=0 tx=0).stop 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 msgr2=0x7f051c14a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c14a3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f051800e8a0 tx=0x7f051800ed60 comp rx=0 tx=0).stop 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 shutdown_connections 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f050006f400 0x7f0500071910 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 --2- 172.21.15.67:0/3205387949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f051c06ca60 0x7f051c14a3e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:10.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 >> 172.21.15.67:0/3205387949 conn(0x7f051c06e420 msgr2=0x7f051c075cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:10.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 shutdown_connections 2024-04-25T15:54:10.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:10.697+0000 7f0523a97640 1 -- 172.21.15.67:0/3205387949 wait complete. 2024-04-25T15:54:10.721 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:10.721 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:10.743 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:54:10.788 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:10.786+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:10.880 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:10.877+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:10.978 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:10.978 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:10.981 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:10.978+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:11.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:54:11.082 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:11.081+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:11.183 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:11.182+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:11.254 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:11.254 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:11.284 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:11.281+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:11.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:54:11.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:54:11.385 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:11.385+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:11.485 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:11.482+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7e351640 1 Processor -- start 2024-04-25T15:54:11.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7e351640 1 -- start start 2024-04-25T15:54:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7e351640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5d780caf60 0x7f5d7815d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7e351640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d780c77a0 con 0x7f5d78073bf0 2024-04-25T15:54:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7e351640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d780c7a60 con 0x7f5d7815d790 2024-04-25T15:54:11.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7e351640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5d780cab60 con 0x7f5d780caf60 2024-04-25T15:54:11.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7c8ee640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5d78073bf0 0x7f5d78163ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36832/0 (socket says 172.21.15.67:36832) 2024-04-25T15:54:11.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d7c8ee640 1 -- 172.21.15.67:0/3299479716 learned_addr learned my addr 172.21.15.67:0/3299479716 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3113599012 0 0) 0x7f5d780c77a0 con 0x7f5d78073bf0 2024-04-25T15:54:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d60003530 con 0x7f5d78073bf0 2024-04-25T15:54:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2996315086 0 0) 0x7f5d60003530 con 0x7f5d78073bf0 2024-04-25T15:54:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 >> v2:172.21.15.176:3300/0 conn(0x7f5d780caf60 msgr2=0x7f5d7815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 --2- 172.21.15.67:0/3299479716 >> v2:172.21.15.176:3300/0 conn(0x7f5d780caf60 0x7f5d7815d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 >> v1:172.21.15.176:6789/0 conn(0x7f5d7815d790 legacy=0x7f5d7806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d780c7d40 con 0x7f5d78073bf0 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f5d70001a00 con 0x7f5d78073bf0 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 >> v1:172.21.15.67:6789/0 conn(0x7f5d78073bf0 legacy=0x7f5d78163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d600039c0 0x7f5d60003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d60004810 0x7f5d60004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.501+0000 7f5d76ffd640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d6000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6000c2a0 con 0x7f5d60005360 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6000c450 con 0x7f5d60004810 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d6000c650 con 0x7f5d600039c0 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d6000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 -- 172.21.15.67:0/3299479716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d600039c0 msgr2=0x7f5d60003e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d600039c0 0x7f5d60003e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 -- 172.21.15.67:0/3299479716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d60004810 msgr2=0x7f5d60004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d60004810 0x7f5d60004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:11.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 -- 172.21.15.67:0/3299479716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d60003530 con 0x7f5d60005360 2024-04-25T15:54:11.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 -- 172.21.15.67:0/3299479716 reap_dead start 2024-04-25T15:54:11.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d7c8ee640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d6000b9f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000a830 tx=0x7f5d7001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a22d33dc036a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7000a480 con 0x7f5d60005360 2024-04-25T15:54:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d700ac620 con 0x7f5d60005360 2024-04-25T15:54:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.505+0000 7f5d76ffd640 1 -- 172.21.15.67:0/3299479716 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d700ac9d0 con 0x7f5d60005360 2024-04-25T15:54:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.509+0000 7f5d7e351640 1 -- 172.21.15.67:0/3299479716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 msgr2=0x7f5d6000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.509+0000 7f5d7e351640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d6000b9f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000a830 tx=0x7f5d7001d360 comp rx=0 tx=0).stop 2024-04-25T15:54:11.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.509+0000 7f5d7e351640 1 -- 172.21.15.67:0/3299479716 shutdown_connections 2024-04-25T15:54:11.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.509+0000 7f5d7e351640 1 --2- 172.21.15.67:0/3299479716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d6000b9f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:11.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.509+0000 7f5d7e351640 1 -- 172.21.15.67:0/3299479716 >> 172.21.15.67:0/3299479716 conn(0x7f5d78070db0 msgr2=0x7f5d780ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:11.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.509+0000 7f5d7e351640 1 -- 172.21.15.67:0/3299479716 shutdown_connections 2024-04-25T15:54:11.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.509+0000 7f5d7e351640 1 -- 172.21.15.67:0/3299479716 wait complete. 2024-04-25T15:54:11.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 Processor -- start 2024-04-25T15:54:11.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 -- start start 2024-04-25T15:54:11.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d780c20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d78156c70 0x7f5d780c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d780c5130 0x7f5d780bc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7815a3f0 con 0x7f5d60005360 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78159620 con 0x7f5d780c5130 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7e351640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d7815a570 con 0x7f5d78156c70 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d780c20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d780c20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60008/0 (socket says 172.21.15.67:60008) 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 -- 172.21.15.67:0/1277579847 learned_addr learned my addr 172.21.15.67:0/1277579847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d777fe640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d78156c70 0x7f5d780c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d7c8ee640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d780c5130 0x7f5d780bc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 -- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d78156c70 msgr2=0x7f5d780c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d78156c70 0x7f5d780c2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 -- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d780c5130 msgr2=0x7f5d780bc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d780c5130 0x7f5d780bc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:11.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.513+0000 7f5d77fff640 1 -- 172.21.15.67:0/1277579847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d780baf90 con 0x7f5d60005360 2024-04-25T15:54:11.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d77fff640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d780c20d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800b460 tx=0x7f5d6800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:11.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d74ff9640 1 -- 172.21.15.67:0/1277579847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68015020 con 0x7f5d60005360 2024-04-25T15:54:11.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d780bb230 con 0x7f5d60005360 2024-04-25T15:54:11.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d74ff9640 1 -- 172.21.15.67:0/1277579847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d68011890 con 0x7f5d60005360 2024-04-25T15:54:11.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d74ff9640 1 -- 172.21.15.67:0/1277579847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d68011c80 con 0x7f5d60005360 2024-04-25T15:54:11.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d780bb7d0 con 0x7f5d60005360 2024-04-25T15:54:11.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7806a480 con 0x7f5d60005360 2024-04-25T15:54:11.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d74ff9640 1 -- 172.21.15.67:0/1277579847 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5d68018020 con 0x7f5d60005360 2024-04-25T15:54:11.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d74ff9640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5d6406f400 0x7f5d64071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:11.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.517+0000 7f5d74ff9640 1 -- 172.21.15.67:0/1277579847 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6002+0+0 (secure 0 0 0) 0x7f5d6808f480 con 0x7f5d60005360 2024-04-25T15:54:11.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.521+0000 7f5d777fe640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5d6406f400 0x7f5d64071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:11.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.521+0000 7f5d777fe640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5d6406f400 0x7f5d64071910 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5d780bbce0 tx=0x7f5d6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.529+0000 7f5d74ff9640 1 -- 172.21.15.67:0/1277579847 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5d6801f020 con 0x7f5d60005360 2024-04-25T15:54:11.540 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:11.540 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:11.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:54:11.611 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:11.610+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:11.687 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:11.686+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:11.779 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:11.779 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:11.788 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:11.785+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:11.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:11.797+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f5d78159db0 con 0x7f5d60005360 2024-04-25T15:54:11.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:54:11.891 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:11.890+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:11.990 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:11.989+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:12.040 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:12.040 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:12.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:54:12.127 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:12.126+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:12.192 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:12.190+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:12.292 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:12.290+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:12.308 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:12.308 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:12.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:54:12.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.365+0000 7f5d74ff9640 1 -- 172.21.15.67:0/1277579847 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v100) ==== 91+0+0 (secure 0 0 0) 0x7f5d6800d040 con 0x7f5d60005360 2024-04-25T15:54:12.366 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:54:12.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5d6406f400 msgr2=0x7f5d64071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:12.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5d6406f400 0x7f5d64071910 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5d780bbce0 tx=0x7f5d6c009040 comp rx=0 tx=0).stop 2024-04-25T15:54:12.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 msgr2=0x7f5d780c20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:12.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d780c20d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800b460 tx=0x7f5d6800b920 comp rx=0 tx=0).stop 2024-04-25T15:54:12.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 shutdown_connections 2024-04-25T15:54:12.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5d6406f400 0x7f5d64071910 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:12.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d780c5130 0x7f5d780bc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:12.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d78156c70 0x7f5d780c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:12.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 --2- 172.21.15.67:0/1277579847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d60005360 0x7f5d780c20d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:12.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 >> 172.21.15.67:0/1277579847 conn(0x7f5d78070db0 msgr2=0x7f5d78157d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:12.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 shutdown_connections 2024-04-25T15:54:12.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.369+0000 7f5d7e351640 1 -- 172.21.15.67:0/1277579847 wait complete. 2024-04-25T15:54:12.393 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:12.393+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:12.494 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:12.493+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:12.562 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:12.562 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:12.596 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:12.594+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:12.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:54:12.696 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:12.694+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:12.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:54:12.797 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:12.794+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:12.868 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:12.868 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:12.898 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:12.897+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:12.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:54:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.929+0000 7f06d67cc640 1 Processor -- start 2024-04-25T15:54:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d67cc640 1 -- start start 2024-04-25T15:54:12.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d67cc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06d0162ec0 0x7f06d01652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:12.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d67cc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d00c77a0 con 0x7f06d00caf60 2024-04-25T15:54:12.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d67cc640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d00c7a60 con 0x7f06d0073bf0 2024-04-25T15:54:12.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d67cc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f06d00cab60 con 0x7f06d0162ec0 2024-04-25T15:54:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d4d69640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06d0162ec0 0x7f06d01652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06cf7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f06d0073bf0 0x7f06d006d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:39820/0 (socket says 172.21.15.67:39820) 2024-04-25T15:54:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06cf7fe640 1 -- 172.21.15.67:0/2893487420 learned_addr learned my addr 172.21.15.67:0/2893487420 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06cffff640 1 --1- 172.21.15.67:0/2893487420 >> v1:172.21.15.67:6789/0 conn(0x7f06d00caf60 0x7f06d015d250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50348/0 (socket says 172.21.15.67:50348) 2024-04-25T15:54:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d4d69640 1 -- 172.21.15.67:0/2893487420 >> v1:172.21.15.176:6789/0 conn(0x7f06d0073bf0 legacy=0x7f06d006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:12.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d4d69640 1 -- 172.21.15.67:0/2893487420 >> v1:172.21.15.67:6789/0 conn(0x7f06d00caf60 legacy=0x7f06d015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:12.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.933+0000 7f06d4d69640 1 -- 172.21.15.67:0/2893487420 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d00c7d40 con 0x7f06d0162ec0 2024-04-25T15:54:12.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06d4d69640 1 --2- 172.21.15.67:0/2893487420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d01652e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009d90 tx=0x7f06c801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fbef32de3a544b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:12.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06ceffd640 1 -- 172.21.15.67:0/2893487420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b2070 con 0x7f06d0162ec0 2024-04-25T15:54:12.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06ceffd640 1 -- 172.21.15.67:0/2893487420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c80a5730 con 0x7f06d0162ec0 2024-04-25T15:54:12.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06ceffd640 1 -- 172.21.15.67:0/2893487420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80a5b00 con 0x7f06d0162ec0 2024-04-25T15:54:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06d67cc640 1 -- 172.21.15.67:0/2893487420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 msgr2=0x7f06d01652e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06d67cc640 1 --2- 172.21.15.67:0/2893487420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d01652e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f06c8009d90 tx=0x7f06c801c970 comp rx=0 tx=0).stop 2024-04-25T15:54:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06d67cc640 1 -- 172.21.15.67:0/2893487420 shutdown_connections 2024-04-25T15:54:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06d67cc640 1 --2- 172.21.15.67:0/2893487420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d01652e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.937+0000 7f06d67cc640 1 -- 172.21.15.67:0/2893487420 >> 172.21.15.67:0/2893487420 conn(0x7f06d0070db0 msgr2=0x7f06d00ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.941+0000 7f06d67cc640 1 -- 172.21.15.67:0/2893487420 shutdown_connections 2024-04-25T15:54:12.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.941+0000 7f06d67cc640 1 -- 172.21.15.67:0/2893487420 wait complete. 2024-04-25T15:54:12.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.941+0000 7f06d67cc640 1 Processor -- start 2024-04-25T15:54:12.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.941+0000 7f06d67cc640 1 -- start start 2024-04-25T15:54:12.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d67cc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06d0073bf0 0x7f06d014d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:12.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d67cc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06d00caf60 0x7f06d0163c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:12.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d67cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d014af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:12.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d67cc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06d015a3f0 con 0x7f06d0162ec0 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d67cc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f06d0159620 con 0x7f06d0073bf0 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d67cc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f06d015a570 con 0x7f06d00caf60 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d014af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d014af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43952/0 (socket says 172.21.15.67:43952) 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 learned_addr learned my addr 172.21.15.67:0/3699563494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06d0073bf0 0x7f06d014d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06d00caf60 0x7f06d0163c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06d00caf60 msgr2=0x7f06d0163c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06d00caf60 0x7f06d0163c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06d0073bf0 msgr2=0x7f06d014d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06d0073bf0 0x7f06d014d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d014d600 con 0x7f06d0162ec0 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06d00caf60 0x7f06d0163c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:12.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06d4d69640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d014af90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f06c80b0040 tx=0x7f06c8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80b2050 con 0x7f06d0162ec0 2024-04-25T15:54:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c80be070 con 0x7f06d0162ec0 2024-04-25T15:54:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.945+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c80ad850 con 0x7f06d0162ec0 2024-04-25T15:54:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06d67cc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d01f6e60 con 0x7f06d0162ec0 2024-04-25T15:54:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06d67cc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d014c5e0 con 0x7f06d0162ec0 2024-04-25T15:54:12.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f06a8005280 con 0x7f06d0162ec0 2024-04-25T15:54:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f06c80ad9f0 con 0x7f06d0162ec0 2024-04-25T15:54:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06ccff9640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f06bc06f430 0x7f06bc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:12.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6002+0+0 (secure 0 0 0) 0x7f06c812adb0 con 0x7f06d0162ec0 2024-04-25T15:54:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v100) ==== 54+0+16 (secure 0 0 0) 0x7f06c8016560 con 0x7f06d0162ec0 2024-04-25T15:54:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f06a80045f0 0x7f06a8004a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06a8005560 con 0x7f06a80045f0 2024-04-25T15:54:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f06bc06f430 0x7f06bc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:12.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.949+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f06a80045f0 0x7f06a8004a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.953+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f06a80045f0 0x7f06a8004a00 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:12.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.953+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f06bc06f430 0x7f06bc071940 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06c0000f70 tx=0x7f06c0007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:12.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.953+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a8005560 con 0x7f06a80045f0 2024-04-25T15:54:12.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.977+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a8001280 con 0x7f06a80045f0 2024-04-25T15:54:12.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.981+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a8001280 con 0x7f06a80045f0 2024-04-25T15:54:12.983 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:12.983 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:12.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.981+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f06a8001b90 0x7f06a8002020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:12.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.981+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f06a8003dd0 con 0x7f06a8001b90 2024-04-25T15:54:12.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.981+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f06a8001b90 0x7f06a8002020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:12.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.981+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f06a8001b90 0x7f06a8002020 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:12.985+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a8003dd0 con 0x7f06a8001b90 2024-04-25T15:54:12.998 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:12.997+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:13.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.005+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a8002dc0 con 0x7f06a8001b90 2024-04-25T15:54:13.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.009+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a8002dc0 con 0x7f06a8001b90 2024-04-25T15:54:13.012 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.013+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f06a8003840 0x7f06a800bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.013+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f06a800edc0 con 0x7f06a8003840 2024-04-25T15:54:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.013+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f06a8003840 0x7f06a800bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:13.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.013+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f06a8003840 0x7f06a800bd20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:13.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.013+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a800edc0 con 0x7f06a8003840 2024-04-25T15:54:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.033+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a800ddb0 con 0x7f06a8003840 2024-04-25T15:54:13.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.037+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a800ddb0 con 0x7f06a8003840 2024-04-25T15:54:13.039 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:13.040 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:13.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.037+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f06a800e830 0x7f06a800c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:13.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.037+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f06a800cbf0 con 0x7f06a800e830 2024-04-25T15:54:13.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.037+0000 7f06d4d69640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f06a800e830 0x7f06a800c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:13.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.037+0000 7f06d4d69640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f06a800e830 0x7f06a800c3e0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:13.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.041+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a800cbf0 con 0x7f06a800e830 2024-04-25T15:54:13.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.061+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a800d100 con 0x7f06a800e830 2024-04-25T15:54:13.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.061+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a800d100 con 0x7f06a800e830 2024-04-25T15:54:13.064 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:13.065 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:13.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.065+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f06a800d5a0 0x7f06a8012f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:13.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.065+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f06a8015fc0 con 0x7f06a800d5a0 2024-04-25T15:54:13.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.069+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f06a800d5a0 0x7f06a8012f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:13.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.069+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f06a800d5a0 0x7f06a8012f90 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:13.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.073+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a8015fc0 con 0x7f06a800d5a0 2024-04-25T15:54:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.097+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a8014d00 con 0x7f06a800d5a0 2024-04-25T15:54:13.099 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:13.098+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.101+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a8014d00 con 0x7f06a800d5a0 2024-04-25T15:54:13.101 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.101+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f06a8015420 0x7f06a8015830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.101+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f06a8019390 con 0x7f06a8015420 2024-04-25T15:54:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.101+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f06a8015420 0x7f06a8015830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.101+0000 7f06cf7fe640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f06a8015420 0x7f06a8015830 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:13.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.105+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a8019390 con 0x7f06a8015420 2024-04-25T15:54:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.133+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a8018290 con 0x7f06a8015420 2024-04-25T15:54:13.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.137+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a8018290 con 0x7f06a8015420 2024-04-25T15:54:13.138 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.137+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f06a80189a0 0x7f06a8018db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.137+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f06a801c760 con 0x7f06a80189a0 2024-04-25T15:54:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.137+0000 7f06d4d69640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f06a80189a0 0x7f06a8018db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:13.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.141+0000 7f06d4d69640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f06a80189a0 0x7f06a8018db0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:13.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.145+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a801c760 con 0x7f06a80189a0 2024-04-25T15:54:13.163 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5623 3.22 3:471c22da:::100000001f8.000007f3:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.150317+0000", 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.014745744, 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014780598000000001, 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5623 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.150317+0000", 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.150317+0000", 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.150320+0000", 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1499999999999999e-06 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.152471+0000", 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021507760000000001 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.152509+0000", 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8203e-05 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.152521+0000", 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2126e-05 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.152613+0000", 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1879999999999994e-05 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.152744+0000", 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130748 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.152941+0000", 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019706300000000001 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5625 3.32 3:4d717b28:::100000001f8.000007f5:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.162428+0000", 2024-04-25T15:54:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0026350610000000002, 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002772446, 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5625 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.162428+0000", 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.162428+0000", 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.162431+0000", 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0599999999999999e-06 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.164819+0000", 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002387573 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.164863+0000", 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4474999999999997e-05 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.164883+0000", 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9755999999999999e-05 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.164987+0000", 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104151 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.165092+0000", 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000104929 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 2 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.169+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a801b5e0 con 0x7f06a80189a0 2024-04-25T15:54:13.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.177+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a801b5e0 con 0x7f06a80189a0 2024-04-25T15:54:13.177 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.177+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f06a801bc60 0x7f06a801c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.177+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f06a801fb20 con 0x7f06a801bc60 2024-04-25T15:54:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.177+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f06a801bc60 0x7f06a801c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:13.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.177+0000 7f06cffff640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f06a801bc60 0x7f06a801c0b0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:13.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.181+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06a801fb20 con 0x7f06a801bc60 2024-04-25T15:54:13.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:54:13.202 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:13.198+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.205+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f06a80135c0 con 0x7f06a801bc60 2024-04-25T15:54:13.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.217+0000 7f06ccff9640 1 -- 172.21.15.67:0/3699563494 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06a80135c0 con 0x7f06a801bc60 2024-04-25T15:54:13.221 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:13.221 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:13.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f06a80045f0 msgr2=0x7f06a8004a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f06a80045f0 0x7f06a8004a00 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f06a8001b90 msgr2=0x7f06a8002020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f06a8001b90 0x7f06a8002020 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f06a8003840 msgr2=0x7f06a800bd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f06a8003840 0x7f06a800bd20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f06a800e830 msgr2=0x7f06a800c3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f06a800e830 0x7f06a800c3e0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 reap_dead start 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f06a800d5a0 msgr2=0x7f06a8012f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f06a800d5a0 0x7f06a8012f90 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 reap_dead start 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f06a8015420 msgr2=0x7f06a8015830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f06a8015420 0x7f06a8015830 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f06a80189a0 msgr2=0x7f06a8018db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f06a80189a0 0x7f06a8018db0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f06a801bc60 msgr2=0x7f06a801c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f06a801bc60 0x7f06a801c0b0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f06bc06f430 msgr2=0x7f06bc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f06bc06f430 0x7f06bc071940 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06c0000f70 tx=0x7f06c0007e90 comp rx=0 tx=0).stop 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 msgr2=0x7f06d014af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 --2- 172.21.15.67:0/3699563494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06d0162ec0 0x7f06d014af90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f06c80b0040 tx=0x7f06c8019710 comp rx=0 tx=0).stop 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 reap_dead start 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06d4d69640 1 -- 172.21.15.67:0/3699563494 reap_dead start 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 shutdown_connections 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 >> 172.21.15.67:0/3699563494 conn(0x7f06d0070db0 msgr2=0x7f06d0158db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 shutdown_connections 2024-04-25T15:54:13.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:13.221+0000 7f06b67fc640 1 -- 172.21.15.67:0/3699563494 wait complete. 2024-04-25T15:54:13.301 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:13.301+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:13.402 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:13.398+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:13.422 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:13.422 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:13.422 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:13.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:54:13.502 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:13.498+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:13.604 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:13.601+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 211.443689939, 2024-04-25T15:54:13.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:13.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.699 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:676 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.761651608000001, 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053839007000000001, 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.700 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929776+0000", 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.743e-06 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931715+0000", 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019394130000000001 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931754+0000", 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8779000000000003e-05 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931769+0000", 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5231999999999999e-05 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931867+0000", 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7485000000000006e-05 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932043+0000", 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017616100000000001 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932181+0000", 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013769100000000001 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.942677+0000", 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010495931 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983477+0000", 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040800621000000002 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983546+0000", 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9138000000000001e-05 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983611+0000", 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4813000000000004e-05 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:733 3.22 3:44586a4d:::100000001f7.000000d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 49.383185560000001, 2024-04-25T15:54:13.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037465947999999999, 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 733 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308242+0000", 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.399e-06 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309826+0000", 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015840349999999999 2024-04-25T15:54:13.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309863+0000", 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7122000000000001e-05 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309880+0000", 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6888000000000002e-05 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309986+0000", 2024-04-25T15:54:13.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105979 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310169+0000", 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018349699999999999 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310353+0000", 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018415799999999999 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.342377+0000", 2024-04-25T15:54:13.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032023169999999997 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345569+0000", 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031928680000000002 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345644+0000", 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4140999999999994e-05 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345704+0000", 2024-04-25T15:54:13.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0690999999999999e-05 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1589 3.1a e88/76 3:590c0ff7:::100000001f8.00000231:head v 88'30, mlcod=88'28)", 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.444600559000001, 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032177728000000003, 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246826+0000", 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1900000000000002e-06 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045171680000000002 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251364+0000", 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1444999999999999e-05 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251375+0000", 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.115e-05 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.713 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251458+0000", 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3190999999999997e-05 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251485+0000", 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6661000000000001e-05 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.278940+0000", 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027455222000000001 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.279001+0000", 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0701e-05 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1595 3.18 3:1adf79c2:::100000001f8.00000237:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.409239522, 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038955741000000002, 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.715 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1595 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282187+0000", 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1569999999999998e-06 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284089+0000", 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001902672 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284127+0000", 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7663000000000002e-05 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284141+0000", 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284236+0000", 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4895999999999994e-05 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284417+0000", 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018094500000000001 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284602+0000", 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018422800000000001 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.293831+0000", 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092292650000000004 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321052+0000", 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027220978 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321098+0000", 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6520000000000002e-05 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321140+0000", 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1850000000000001e-05 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2263 3.18 3:1b2b5a9e:::100000001f8.000004d3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.550992+0000", 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.14043195, 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02958715, 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2263 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.550992+0000", 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.550992+0000", 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.550995+0000", 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8150000000000002e-06 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.552761+0000", 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017658839999999999 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.552980+0000", 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00021939299999999999 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.552994+0000", 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4416999999999999e-05 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.553100+0000", 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106028 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.553258+0000", 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000157159 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.553400+0000", 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000142323 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.559455+0000", 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006054785 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.580489+0000", 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021034177000000001 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.580531+0000", 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1777999999999999e-05 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.580579+0000", 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8390999999999998e-05 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2389 3.18 3:1bdc9869:::100000001f8.00000551:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.302188319999999, 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030905229999999999, 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.726 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2389 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389238+0000", 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7099999999999999e-06 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391140+0000", 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001901359 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391174+0000", 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4165000000000002e-05 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391187+0000", 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3188e-05 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391277+0000", 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0267999999999997e-05 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391455+0000", 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017728700000000001 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391618+0000", 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163308 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.419720+0000", 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028102374999999999 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420001+0000", 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028060900000000002 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420083+0000", 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2010999999999998e-05 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420141+0000", 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7949999999999999e-05 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2392 3.32 3:4cdbb504:::100000001f8.00000554:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.285489561999999, 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048266518000000001, 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2392 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405937+0000", 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6759999999999999e-06 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408696+0000", 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027593589999999999 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408735+0000", 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8516000000000002e-05 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408749+0000", 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4117000000000001e-05 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408844+0000", 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4891999999999999e-05 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409024+0000", 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018000199999999999 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409200+0000", 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017591800000000001 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.427538+0000", 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018337728000000001 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454069+0000", 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026531451000000001 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454141+0000", 2024-04-25T15:54:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2316000000000004e-05 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454201+0000", 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9543000000000002e-05 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2393 3.22 3:45fb7032:::100000001f8.00000555:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:13.737 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.279096990999999, 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052582106000000003, 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2393 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412330+0000", 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3119999999999998e-06 2024-04-25T15:54:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413928+0000", 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597521 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413971+0000", 2024-04-25T15:54:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3086e-05 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413988+0000", 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7336000000000001e-05 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414091+0000", 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103099 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414245+0000", 2024-04-25T15:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015350300000000001 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414359+0000", 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011410500000000001 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.446661+0000", 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032302571000000002 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464781+0000", 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018119283999999999 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464856+0000", 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5153000000000004e-05 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464909+0000", 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3136000000000001e-05 2024-04-25T15:54:13.743 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2395 3.18 3:183ba26c:::100000001f8.00000557:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.266696883000002, 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029801055999999999, 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.744 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2395 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424735+0000", 2024-04-25T15:54:13.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1780000000000003e-06 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426303+0000", 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015679170000000001 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426349+0000", 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6343e-05 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426370+0000", 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329000000000001e-05 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426458+0000", 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.831e-05 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426609+0000", 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150629 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426735+0000", 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012625900000000001 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.441627+0000", 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014891824999999999 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454417+0000", 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012789994000000001 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454476+0000", 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8752000000000003e-05 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454528+0000", 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2519999999999999e-05 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3145 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:13.749 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.256738149, 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030601595999999998, 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3145 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434689+0000", 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.005e-06 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436062+0000", 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001373682 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436096+0000", 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3797000000000002e-05 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436107+0000", 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436177+0000", 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0069999999999998e-05 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436346+0000", 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169168 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436510+0000", 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163807 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.753 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.443699+0000", 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007189394 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465141+0000", 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021441386999999999 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465223+0000", 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1920000000000002e-05 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465287+0000", 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4692999999999999e-05 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3841 3.18 3:1a4016cc:::100000001f8.000000fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.605741315, 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043270667999999998, 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3841 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085686+0000", 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4510000000000001e-06 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088301+0000", 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002614987 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088348+0000", 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7057e-05 2024-04-25T15:54:13.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088362+0000", 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4352000000000001e-05 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088457+0000", 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4794000000000001e-05 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088583+0000", 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012605400000000001 2024-04-25T15:54:13.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088776+0000", 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019268700000000001 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.107828+0000", 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019051702 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128788+0000", 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020960433000000001 2024-04-25T15:54:13.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128888+0000", 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9493000000000004e-05 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128953+0000", 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5658000000000001e-05 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3977 3.18 3:1b05c35d:::100000001f8.00000185:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.8492027330000003, 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13853472, 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3977 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842224+0000", 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1130000000000001e-06 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844135+0000", 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019102920000000001 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844181+0000", 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6616000000000002e-05 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844202+0000", 2024-04-25T15:54:13.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0542000000000001e-05 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844358+0000", 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156379 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844555+0000", 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000196651 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844759+0000", 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020436799999999999 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.853781+0000", 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090217130000000006 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980620+0000", 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.126839061 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980690+0000", 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9596000000000002e-05 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980756+0000", 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4039 3.32 3:4c7e4113:::100000001f8.000001c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.212124+0000", 2024-04-25T15:54:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.4792999229999992, 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028539355999999998, 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4039 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.212124+0000", 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.212124+0000", 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.212127+0000", 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3050000000000001e-06 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.214190+0000", 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020630090000000002 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.214226+0000", 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5981e-05 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.214245+0000", 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.838e-05 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.214350+0000", 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010566 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.214470+0000", 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119417 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.214657+0000", 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018719799999999999 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.231733+0000", 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017075798999999999 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.240568+0000", 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0088357509999999993 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.240624+0000", 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5887000000000002e-05 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.240663+0000", 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8968999999999999e-05 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4099 3.18 3:181e4796:::100000001f8.000001ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1561305270000002, 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030713761999999999, 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.772 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4099 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535297+0000", 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2049999999999998e-06 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536942+0000", 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00164504 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536980+0000", 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8736999999999999e-05 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536997+0000", 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6653000000000001e-05 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537091+0000", 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3786e-05 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537191+0000", 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9840000000000006e-05 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537331+0000", 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014034900000000001 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.556229+0000", 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018898278000000001 2024-04-25T15:54:13.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565921+0000", 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096916850000000002 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565961+0000", 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0290999999999998e-05 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.566007+0000", 2024-04-25T15:54:13.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5898e-05 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4103 3.32 3:4d482082:::100000001f8.00000203:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.559048+0000", 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.1323762209999995, 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028897519999999999, 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.778 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4103 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.559048+0000", 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.559048+0000", 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.559055+0000", 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0299999999999996e-06 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.560772+0000", 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001717613 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.560818+0000", 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5500999999999997e-05 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.560836+0000", 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8136999999999999e-05 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.560924+0000", 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7732000000000006e-05 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.561026+0000", 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102246 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.561186+0000", 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015970200000000001 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.585403+0000", 2024-04-25T15:54:13.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.024217590000000001 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.587806+0000", 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024031130000000001 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.587883+0000", 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6630000000000003e-05 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.587945+0000", 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2225999999999997e-05 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4201 3.32 3:4d6fc8c1:::100000001f8.00000265:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:05.079701+0000", 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.6117231620000005, 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029110897, 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4201 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.079701+0000", 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.079701+0000", 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.079705+0000", 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8580000000000002e-06 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.082351+0000", 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026465249999999998 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.082400+0000", 2024-04-25T15:54:13.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8451000000000001e-05 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.082415+0000", 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5857000000000001e-05 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.082510+0000", 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4123000000000007e-05 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.082595+0000", 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5589999999999999e-05 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.082722+0000", 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012712699999999999 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.108583+0000", 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025860806 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.108665+0000", 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1477999999999995e-05 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.108736+0000", 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1456000000000002e-05 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:05.108812+0000", 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5625999999999997e-05 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.789 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4812 3.18 3:194fa8d8:::100000001f8.000004c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.0077163120000003, 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038646409, 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4812 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.790 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683711+0000", 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7170000000000002e-06 2024-04-25T15:54:13.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685822+0000", 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105619999999999 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685876+0000", 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3798999999999998e-05 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685910+0000", 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4727999999999997e-05 2024-04-25T15:54:13.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686022+0000", 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111549 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686150+0000", 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826800000000001 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686332+0000", 2024-04-25T15:54:13.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181538 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.693171+0000", 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068389519999999997 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722249+0000", 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029078171 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722294+0000", 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4756000000000003e-05 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722354+0000", 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0368999999999997e-05 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4949 3.18 3:1bdc9869:::100000001f8.00000551:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.448766+0000", 2024-04-25T15:54:13.795 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.2426574979999998, 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029743667000000001, 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4949 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.448766+0000", 2024-04-25T15:54:13.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.448766+0000", 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.448770+0000", 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.036e-06 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.450079+0000", 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00130904 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.450114+0000", 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4251999999999997e-05 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.450127+0000", 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3223e-05 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.450197+0000", 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0086999999999995e-05 2024-04-25T15:54:13.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.450275+0000", 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7591999999999998e-05 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.450377+0000", 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010264600000000001 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.460664+0000", 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0102868 2024-04-25T15:54:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.478380+0000", 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017715504999999999 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.478451+0000", 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1276e-05 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.478510+0000", 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9209999999999997e-05 2024-04-25T15:54:13.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4967 3.22 3:478cb4f4:::100000001f8.00000563:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.1497825989999999, 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035932619999999998, 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:13.801 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4967 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:13.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541672+0000", 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0978000000000001e-05 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543306+0000", 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016338310000000001 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543342+0000", 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5994000000000001e-05 2024-04-25T15:54:13.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543360+0000", 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543470+0000", 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110319 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543624+0000", 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153549 2024-04-25T15:54:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543799+0000", 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017463299999999999 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.555562+0000", 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011763177999999999 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577476+0000", 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021913818000000002 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577529+0000", 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3711000000000001e-05 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577574+0000", 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4678999999999997e-05 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:13.806 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:13.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:54:13.809 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:13.805+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:13.908 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:13.909+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:14.009 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:14.009 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:14.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.009 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5771 3.21 3:842f8913:::100000001f8.00000887:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.009 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.993009+0000", 2024-04-25T15:54:14.009 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.017430075999999999, 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017464456, 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5771 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.993009+0000", 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.993009+0000", 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.993012+0000", 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.196e-06 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.994651+0000", 2024-04-25T15:54:14.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016388310000000001 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.994695+0000", 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3679e-05 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.994714+0000", 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9239999999999999e-05 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.994831+0000", 2024-04-25T15:54:14.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116712 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.994951+0000", 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120232 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.995126+0000", 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000175668 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.007916+0000", 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012789174 2024-04-25T15:54:14.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5772 3.37 3:ef1ef352:::100000001f8.00000888:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.997827+0000", 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.012612073, 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012715545999999999, 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.014 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5772 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.997827+0000", 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.997827+0000", 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.997830+0000", 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9849999999999998e-06 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.999685+0000", 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018551730000000001 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.999725+0000", 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9662000000000001e-05 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.999744+0000", 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9539999999999999e-05 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.999837+0000", 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2317000000000004e-05 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.999945+0000", 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108175 2024-04-25T15:54:14.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.000087+0000", 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014208800000000001 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5773 3.2e 3:7774ab2b:::100000001f8.00000889:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.003007+0000", 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0074320549999999999, 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075899280000000001, 2024-04-25T15:54:14.018 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "waiting for sub ops", 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5773 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.003007+0000", 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.003007+0000", 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.003010+0000", 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7159999999999999e-06 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004714+0000", 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001704573 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004744+0000", 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9578999999999999e-05 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004756+0000", 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2116999999999999e-05 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004841+0000", 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4741999999999996e-05 2024-04-25T15:54:14.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004950+0000", 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109109 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.005073+0000", 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123008 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.022 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:14.023 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 3 2024-04-25T15:54:14.023 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:14.023 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:14.009+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:14.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:54:14.110 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:14.106+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:14.211 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:14.210+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:14.289 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:14.289 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:14.289 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:14.289 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:14.289 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:14.303 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:54:14.303 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:54:14.303 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-04-25T15:54:14.303 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:54:14.303 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:54:14.303 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-04-25T15:54:14.312 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:14.313+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:14.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:54:14.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:54:14.413 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:14.410+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:14.515 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:14.514+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:14.530 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:14.530 INFO:teuthology.orchestra.run.smithi176.stdout: "success": "" 2024-04-25T15:54:14.530 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:14.561 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:14.561 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:14.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:14.561 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:14.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.561 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:148 3.9 3:91a8729a:::10000000002.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:54:14.561 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 86.002008911000004, 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.432712019, 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 148 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558221+0000", 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3929999999999998e-06 2024-04-25T15:54:14.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559704+0000", 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001483807 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975277+0000", 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.41557216400000002 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975295+0000", 2024-04-25T15:54:14.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8408999999999999e-05 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975372+0000", 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7235000000000002e-05 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975484+0000", 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111499 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975582+0000", 2024-04-25T15:54:14.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8589000000000001e-05 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.980957+0000", 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053747600000000001 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990855+0000", 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098981350000000006 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990894+0000", 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8436000000000003e-05 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990930+0000", 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6591999999999998e-05 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1473 3.0 3:008165f2:::100000001f8.000001bd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.068007444999999, 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.084007844999999998, 2024-04-25T15:54:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1473 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492220+0000", 2024-04-25T15:54:14.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.492222+0000", 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6910000000000002e-06 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.494870+0000", 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026480620000000001 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557687+0000", 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.062816831000000004 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557707+0000", 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9262e-05 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557796+0000", 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9209000000000006e-05 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.557984+0000", 2024-04-25T15:54:14.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000188118 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.558165+0000", 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181292 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.565264+0000", 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0070987309999999996 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576094+0000", 2024-04-25T15:54:14.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010829920999999999 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576165+0000", 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0951999999999994e-05 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.576228+0000", 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2775999999999997e-05 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1494 3.37 3:efe547ea:::100000001f8.000001d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.915824362999999, 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089208711999999996, 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1494 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644406+0000", 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2229999999999999e-06 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.646356+0000", 2024-04-25T15:54:14.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019497990000000001 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718358+0000", 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072002624000000001 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718376+0000", 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7649999999999999e-05 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718458+0000", 2024-04-25T15:54:14.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1984999999999995e-05 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718600+0000", 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000141922 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718731+0000", 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131488 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.724805+0000", 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006074032 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733498+0000", 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086928439999999999 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733549+0000", 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1137000000000002e-05 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733611+0000", 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2008000000000007e-05 2024-04-25T15:54:14.577 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1504 3.2a 3:54fb97d1:::100000001f8.000001dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.843530927, 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121933077, 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1504 2024-04-25T15:54:14.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716699+0000", 2024-04-25T15:54:14.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4250000000000001e-06 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718317+0000", 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016187440000000001 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.815989+0000", 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.097671735999999995 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816008+0000", 2024-04-25T15:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9131e-05 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816090+0000", 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1717000000000004e-05 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816244+0000", 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015372199999999999 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816348+0000", 2024-04-25T15:54:14.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010470499999999999 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.826448+0000", 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010099281999999999 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838528+0000", 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012080206 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838583+0000", 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5105000000000002e-05 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838629+0000", 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6304000000000003e-05 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1528 3.3e 3:7f4f4afd:::100000001f8.000001f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:14.583 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.696057080999999, 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.133348521, 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1528 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864172+0000", 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.204e-06 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865554+0000", 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013821899999999999 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986006+0000", 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12045179 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986022+0000", 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999e-05 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986101+0000", 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9251000000000004e-05 2024-04-25T15:54:14.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986224+0000", 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012263899999999999 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986330+0000", 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106164 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.996058+0000", 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097274319999999994 2024-04-25T15:54:14.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997353+0000", 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001294882 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997450+0000", 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7446000000000002e-05 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997519+0000", 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8523999999999996e-05 2024-04-25T15:54:14.588 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1548 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.576591373999999, 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096155170999999998, 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.589 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1548 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983639+0000", 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1750000000000001e-06 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.985960+0000", 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023213790000000002 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058434+0000", 2024-04-25T15:54:14.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072473913000000001 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058454+0000", 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9329000000000001e-05 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058541+0000", 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7702999999999997e-05 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058708+0000", 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016712300000000001 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058864+0000", 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015600100000000001 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.065501+0000", 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066362299999999999 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079696+0000", 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014195228000000001 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079745+0000", 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8980000000000002e-05 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079791+0000", 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6109999999999997e-05 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1561 3.2e 3:75253c31:::100000001f8.00000215:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:14.594 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.492196843999999, 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099620287000000002, 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1561 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068033+0000", 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1240000000000001e-06 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.070277+0000", 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002243676 2024-04-25T15:54:14.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143785+0000", 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073508283999999993 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143804+0000", 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8808e-05 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143881+0000", 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6379999999999997e-05 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144063+0000", 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018198500000000001 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144168+0000", 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105253 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.154374+0000", 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010206374000000001 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167481+0000", 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013106405999999999 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167580+0000", 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9329999999999999e-05 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167651+0000", 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0667000000000006e-05 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.599 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2308 3.4 3:2343a40c:::100000001f8.00000500:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.739142531999999, 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.129707092, 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2308 2024-04-25T15:54:14.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821088+0000", 2024-04-25T15:54:14.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1599999999999998e-06 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823092+0000", 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020043909999999999 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823143+0000", 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0791999999999998e-05 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823162+0000", 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9332999999999999e-05 2024-04-25T15:54:14.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823254+0000", 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2169000000000005e-05 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823637+0000", 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038265100000000002 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823849+0000", 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211879 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.834007+0000", 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010158377 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950654+0000", 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.116646229 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950721+0000", 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7020000000000005e-05 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950792+0000", 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1091000000000001e-05 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4976 3.2a 3:57c1fe5a:::100000001f8.0000056c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.9691062439999998, 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15973960500000001, 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.605 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4976 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591123+0000", 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.306e-06 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.592621+0000", 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014980250000000001 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703776+0000", 2024-04-25T15:54:14.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.111154948 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703790+0000", 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3915e-05 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703878+0000", 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7527000000000004e-05 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703996+0000", 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118602 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.704166+0000", 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017002799999999999 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.710386+0000", 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062202239999999999 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750704+0000", 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040317106999999998 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750795+0000", 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1880999999999997e-05 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750860+0000", 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5042000000000005e-05 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5006 3.2a 3:566ba4fe:::100000001f8.0000058a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.8205961840000002, 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094006696000000001, 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.610 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5006 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:14.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739634+0000", 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999998e-06 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.740943+0000", 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013092780000000001 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.612 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791183+0000", 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050239791999999998 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791197+0000", 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4250000000000001e-05 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791287+0000", 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9627000000000001e-05 2024-04-25T15:54:14.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791404+0000", 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117663 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791560+0000", 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155997 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.801092+0000", 2024-04-25T15:54:14.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095317030000000007 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833424+0000", 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032332514999999999 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833527+0000", 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102997 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833638+0000", 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110142 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5024 3.2a 3:55e8c2c8:::100000001f8.0000059c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.730731295, 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14725615, 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.616 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5024 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829499+0000", 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0740000000000001e-06 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.831085+0000", 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001586241 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932577+0000", 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.10149145399999999 2024-04-25T15:54:14.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932592+0000", 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4934e-05 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932681+0000", 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9140000000000004e-05 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932803+0000", 2024-04-25T15:54:14.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012215800000000001 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932954+0000", 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151328 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.943046+0000", 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010092261999999999 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976650+0000", 2024-04-25T15:54:14.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033603840000000003 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976710+0000", 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9348999999999997e-05 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976752+0000", 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2370000000000003e-05 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.621 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5049 3.2e 3:740912a7:::100000001f8.000005b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.596823326, 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.162097727, 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5049 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963406+0000", 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4279999999999999e-06 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.965277+0000", 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018703210000000001 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021994+0000", 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056717005000000001 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022018+0000", 2024-04-25T15:54:14.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4043e-05 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022095+0000", 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6959999999999995e-05 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022198+0000", 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103167 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022356+0000", 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015838899999999999 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.052459+0000", 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030102694999999999 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125349+0000", 2024-04-25T15:54:14.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072890606999999996 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125439+0000", 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9587999999999997e-05 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125502+0000", 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2524e-05 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.627 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5064 3.21 3:845d14c1:::100000001f8.000005c4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.5258300590000005, 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14591528400000001, 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5064 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034400+0000", 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3079999999999999e-06 2024-04-25T15:54:14.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.036710+0000", 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023096000000000002 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075414+0000", 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038703923000000001 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075434+0000", 2024-04-25T15:54:14.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9712e-05 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075522+0000", 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8472000000000003e-05 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075609+0000", 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7200000000000005e-05 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075766+0000", 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015715400000000001 2024-04-25T15:54:14.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.089149+0000", 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013382338000000001 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180138+0000", 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.090989451999999998 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180200+0000", 2024-04-25T15:54:14.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2098000000000001e-05 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180312+0000", 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112027 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5072 3.2e 3:75b854b3:::100000001f8.000005cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4866307230000002, 2024-04-25T15:54:14.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.106817279, 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5072 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:14.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073599+0000", 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2759999999999999e-06 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.635 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075372+0000", 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001773574 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125855+0000", 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050482955000000003 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125865+0000", 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0029e-05 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125933+0000", 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8100000000000002e-05 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.126051+0000", 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117333 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.126222+0000", 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000171609 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.133559+0000", 2024-04-25T15:54:14.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0073365649999999998 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180298+0000", 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046739050999999997 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180358+0000", 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0479000000000002e-05 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180414+0000", 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5308e-05 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5081 3.2e 3:75f1801c:::100000001f8.000005d5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.128682+0000", 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.4315451489999997, 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.087623594999999999, 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.639 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5081 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.128682+0000", 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.128682+0000", 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.128685+0000", 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1099999999999999e-06 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.130187+0000", 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001501966 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.203365+0000", 2024-04-25T15:54:14.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073178213000000006 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.203379+0000", 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4051e-05 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.203459+0000", 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9803999999999997e-05 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.203539+0000", 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9710999999999996e-05 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.203659+0000", 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120643 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.211367+0000", 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0077075570000000003 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.216181+0000", 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0048142569999999997 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.216236+0000", 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4727e-05 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.216306+0000", 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9555999999999996e-05 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.644 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5601 3.2e 3:75930891:::100000001f8.000007dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.546160464, 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088347508000000005, 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5601 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.645 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014070+0000", 2024-04-25T15:54:14.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0869999999999998e-06 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.015590+0000", 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015200649999999999 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086402+0000", 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070812293999999998 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086419+0000", 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7088999999999999e-05 2024-04-25T15:54:14.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086530+0000", 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011049599999999999 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086686+0000", 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015618899999999999 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086917+0000", 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000231198 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.095623+0000", 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087063049999999993 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102292+0000", 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066687609999999996 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102356+0000", 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3958e-05 2024-04-25T15:54:14.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102414+0000", 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8066000000000003e-05 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5729 3.2e 3:77f75ed6:::100000001f8.0000085d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.80387525900000001, 2024-04-25T15:54:14.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.114341208, 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5729 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756354+0000", 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.576e-06 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757773+0000", 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014189560000000001 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757806+0000", 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2428000000000002e-05 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757820+0000", 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4279e-05 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757910+0000", 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0138999999999999e-05 2024-04-25T15:54:14.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.758024+0000", 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114158 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.758197+0000", 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172931 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.769252+0000", 2024-04-25T15:54:14.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011054342 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870574+0000", 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.10132216099999999 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870628+0000", 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3996999999999998e-05 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870693+0000", 2024-04-25T15:54:14.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5240999999999994e-05 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5739 3.2e 3:757c9716:::100000001f8.00000867:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.75474836000000001, 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099555849000000002, 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.656 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5739 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805482+0000", 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9670000000000001e-06 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807186+0000", 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017046520000000001 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807217+0000", 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0892999999999997e-05 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807230+0000", 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2235999999999999e-05 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807314+0000", 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4481999999999995e-05 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807445+0000", 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130977 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807554+0000", 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108805 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.824323+0000", 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016769217999999999 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904744+0000", 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080421406000000001 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904929+0000", 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018458600000000001 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.905035+0000", 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105627 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5742 3.3e 3:7c485d5a:::100000001f8.0000086a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.822962+0000", 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.73726503899999996, 2024-04-25T15:54:14.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.081911291999999997, 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5742 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.822962+0000", 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.822962+0000", 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.822979+0000", 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7407000000000001e-05 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.825519+0000", 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025399120000000001 2024-04-25T15:54:14.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.825560+0000", 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0871999999999998e-05 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.825580+0000", 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9550000000000001e-05 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.825662+0000", 2024-04-25T15:54:14.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1808e-05 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.825782+0000", 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120337 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.825956+0000", 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017362599999999999 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.834388+0000", 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084321080000000007 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904730+0000", 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070342032999999998 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904805+0000", 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5511000000000003e-05 2024-04-25T15:54:14.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904873+0000", 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8127999999999996e-05 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5773 3.2e 3:7774ab2b:::100000001f8.00000889:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.003007+0000", 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.55722016500000005, 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.079469437000000004, 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:14.667 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5773 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.003007+0000", 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.003007+0000", 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.003010+0000", 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7159999999999999e-06 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004714+0000", 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001704573 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004744+0000", 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9578999999999999e-05 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004756+0000", 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2116999999999999e-05 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004841+0000", 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4741999999999996e-05 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.004950+0000", 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000109109 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.005073+0000", 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000123008 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.016241+0000", 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011167712 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.082332+0000", 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.066091442 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.082410+0000", 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8047999999999995e-05 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.082476+0000", 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6390999999999995e-05 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.672 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.673 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:14.673 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:14.673 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:14.674 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7742fef640 1 Processor -- start 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7742fef640 1 -- start start 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7742fef640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f773c15dd00 0x7f773c068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7742fef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7742fef640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773c160990 con 0x7f773c156db0 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7742fef640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f773c15e110 con 0x7f773c15dd00 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7742fef640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f773c160c70 con 0x7f773c1569a0 2024-04-25T15:54:14.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f774158c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f773c156db0 0x7f773c15c870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50350/0 (socket says 172.21.15.67:50350) 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f774158c640 1 -- 172.21.15.67:0/1398207386 learned_addr learned my addr 172.21.15.67:0/1398207386 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7740d8b640 1 --2- 172.21.15.67:0/1398207386 >> v2:172.21.15.67:3300/0 conn(0x7f773c15dd00 0x7f773c068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f773b7fe640 1 -- 172.21.15.67:0/1398207386 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 143035388 0 0) 0x7f773c160990 con 0x7f773c156db0 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f773b7fe640 1 -- 172.21.15.67:0/1398207386 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f772c003530 con 0x7f773c156db0 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7740d8b640 1 -- 172.21.15.67:0/1398207386 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 msgr2=0x7f773c06c640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7740d8b640 1 --2- 172.21.15.67:0/1398207386 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c06c640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7740d8b640 1 -- 172.21.15.67:0/1398207386 >> v1:172.21.15.67:6789/0 conn(0x7f773c156db0 legacy=0x7f773c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.613+0000 7f7740d8b640 1 -- 172.21.15.67:0/1398207386 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773c0c3bf0 con 0x7f773c15dd00 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.617+0000 7f7740d8b640 1 --2- 172.21.15.67:0/1398207386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c15dd00 0x7f773c068830 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f77280097c0 tx=0x7f772801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efe79ef4df687d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.617+0000 7f773b7fe640 1 -- 172.21.15.67:0/1398207386 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280b0070 con 0x7f773c15dd00 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.617+0000 7f773b7fe640 1 -- 172.21.15.67:0/1398207386 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77280ab600 con 0x7f773c15dd00 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.617+0000 7f773b7fe640 1 -- 172.21.15.67:0/1398207386 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280ab9b0 con 0x7f773c15dd00 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 -- 172.21.15.67:0/1398207386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c15dd00 msgr2=0x7f773c068830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 --2- 172.21.15.67:0/1398207386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c15dd00 0x7f773c068830 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f77280097c0 tx=0x7f772801c4c0 comp rx=0 tx=0).stop 2024-04-25T15:54:14.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 -- 172.21.15.67:0/1398207386 shutdown_connections 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 --2- 172.21.15.67:0/1398207386 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c06c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 --2- 172.21.15.67:0/1398207386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c15dd00 0x7f773c068830 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 -- 172.21.15.67:0/1398207386 >> 172.21.15.67:0/1398207386 conn(0x7f773c0ce1c0 msgr2=0x7f773c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 -- 172.21.15.67:0/1398207386 shutdown_connections 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 -- 172.21.15.67:0/1398207386 wait complete. 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 Processor -- start 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.621+0000 7f7742fef640 1 -- start start 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7742fef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7742fef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773c156db0 0x7f773c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7742fef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c0b30a0 0x7f773c15b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7742fef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f773c06d7e0 con 0x7f773c0b30a0 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7742fef640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f773c06d660 con 0x7f773c156db0 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7742fef640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f773c0c3740 con 0x7f773c1569a0 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c1587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:56914/0 (socket says 172.21.15.67:56914) 2024-04-25T15:54:14.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 -- 172.21.15.67:0/3373988887 learned_addr learned my addr 172.21.15.67:0/3373988887 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f773bfff640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773c156db0 0x7f773c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f774158c640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c0b30a0 0x7f773c15b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 -- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773c156db0 msgr2=0x7f773c0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773c156db0 0x7f773c0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 -- 172.21.15.67:0/3373988887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c0b30a0 msgr2=0x7f773c15b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c0b30a0 0x7f773c15b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f7740d8b640 1 -- 172.21.15.67:0/3373988887 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773c0b3ab0 con 0x7f773c1569a0 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f773bfff640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773c156db0 0x7f773c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.625+0000 7f774158c640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c0b30a0 0x7f773c15b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f7740d8b640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c1587a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f772801ab70 tx=0x7f7728019260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f77397fa640 1 -- 172.21.15.67:0/3373988887 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280ba070 con 0x7f773c1569a0 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f77397fa640 1 -- 172.21.15.67:0/3373988887 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77280ab9b0 con 0x7f773c1569a0 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f77397fa640 1 -- 172.21.15.67:0/3373988887 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280b0040 con 0x7f773c1569a0 2024-04-25T15:54:14.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773c158ce0 con 0x7f773c1569a0 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f773c0b3810 con 0x7f773c1569a0 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f773c06a480 con 0x7f773c1569a0 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.629+0000 7f77397fa640 1 -- 172.21.15.67:0/3373988887 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7728003620 con 0x7f773c1569a0 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.633+0000 7f77397fa640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f772006f400 0x7f7720071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.633+0000 7f77397fa640 1 -- 172.21.15.67:0/3373988887 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(100..100 src has 1..100) ==== 6002+0+0 (secure 0 0 0) 0x7f7728128240 con 0x7f773c1569a0 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.633+0000 7f773bfff640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f772006f400 0x7f7720071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.633+0000 7f773bfff640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f772006f400 0x7f7720071910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7730006fd0 tx=0x7f7730008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:14.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.641+0000 7f77397fa640 1 -- 172.21.15.67:0/3373988887 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f77280160b0 con 0x7f773c1569a0 2024-04-25T15:54:14.683 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:14.670+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:14.776 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:14.773+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:14.876 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:14.873+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:14.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:14.893+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f773c152ad0 con 0x7f773c1569a0 2024-04-25T15:54:14.902 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:14.903 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:14.903 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:14.903 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:14.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:54:14.977 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:14.974+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:15.078 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:15.074+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:15.178 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:15.174+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:15.188 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:15.188 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:15.188 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:15.189 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:15.189 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:15.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:54:15.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:54:15.279 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:15.274+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:15.380 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:15.377+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:15.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.409+0000 7f77397fa640 1 -- 172.21.15.67:0/3373988887 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v101) ==== 101+0+0 (secure 0 0 0) 0x7f77280f5200 con 0x7f773c1569a0 2024-04-25T15:54:15.412 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:54:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f772006f400 msgr2=0x7f7720071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f772006f400 0x7f7720071910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7730006fd0 tx=0x7f7730008040 comp rx=0 tx=0).stop 2024-04-25T15:54:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 msgr2=0x7f773c1587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c1587a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f772801ab70 tx=0x7f7728019260 comp rx=0 tx=0).stop 2024-04-25T15:54:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 shutdown_connections 2024-04-25T15:54:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f772006f400 0x7f7720071910 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f773c0b30a0 0x7f773c15b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f773c156db0 0x7f773c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 --2- 172.21.15.67:0/3373988887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f773c1569a0 0x7f773c1587a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 >> 172.21.15.67:0/3373988887 conn(0x7f773c0ce1c0 msgr2=0x7f773c155e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:15.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 shutdown_connections 2024-04-25T15:54:15.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.413+0000 7f7742fef640 1 -- 172.21.15.67:0/3373988887 wait complete. 2024-04-25T15:54:15.443 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:15.443 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:15.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:15.443 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:15.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.443 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:343 3.8 3:1115f3ca:::10000000002.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.435585079000006, 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.46089677899999998, 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 343 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006870+0000", 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9280000000000002e-06 2024-04-25T15:54:15.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008363+0000", 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001493288 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008398+0000", 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5067000000000002e-05 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008416+0000", 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7782e-05 2024-04-25T15:54:15.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008522+0000", 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106327 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008535+0000", 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2883e-05 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.387477+0000", 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.37894188200000001 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.392562+0000", 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050850870000000003 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445264+0000", 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052702037 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445382+0000", 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118433 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445603+0000", 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022077500000000001 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.465250+0000", 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019646976 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467658+0000", 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024073850000000002 2024-04-25T15:54:15.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467695+0000", 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7234999999999998e-05 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467763+0000", 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8694000000000003e-05 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.450 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:670 3.8 3:1323134e:::100000001f7.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.568514915999998, 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094982472999999998, 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 670 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:15.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873941+0000", 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0840000000000002e-06 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876488+0000", 2024-04-25T15:54:15.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025470290000000001 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876647+0000", 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015912000000000001 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876664+0000", 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7215e-05 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876782+0000", 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118082 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877222+0000", 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000439596 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877596+0000", 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037371700000000003 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.913106+0000", 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035509813000000001 2024-04-25T15:54:15.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968792+0000", 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.055686739999999998 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968850+0000", 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8032000000000003e-05 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968919+0000", 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9045e-05 2024-04-25T15:54:15.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2313 3.1e 3:7a2a8d26:::100000001f8.00000505:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.577802544000001, 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091581081999999994, 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.456 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2313 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864653+0000", 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866695+0000", 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002042744 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866732+0000", 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7054000000000001e-05 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866765+0000", 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2141999999999998e-05 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866853+0000", 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7950999999999999e-05 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866986+0000", 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013319800000000001 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.867189+0000", 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020333800000000001 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.883954+0000", 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016764729999999999 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956056+0000", 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072101922999999998 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956116+0000", 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9815000000000002e-05 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956230+0000", 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114817 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3921 3.29 3:95d78bcd:::100000001f8.0000014d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.927717067, 2024-04-25T15:54:15.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.156667944, 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3921 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514737+0000", 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.029e-06 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517027+0000", 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022901739999999999 2024-04-25T15:54:15.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517155+0000", 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127763 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517189+0000", 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3847999999999998e-05 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517337+0000", 2024-04-25T15:54:15.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014894999999999999 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517464+0000", 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126608 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517651+0000", 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018649000000000001 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.531319+0000", 2024-04-25T15:54:15.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013667959 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671222+0000", 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.139903946 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671326+0000", 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103785 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671403+0000", 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6391999999999996e-05 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3935 3.23 3:c6f60763:::100000001f8.0000015b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.856454998, 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091319014000000004, 2024-04-25T15:54:15.467 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3935 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.586000+0000", 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2789999999999999e-06 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.587431+0000", 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00143063 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.662977+0000", 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075546586999999998 2024-04-25T15:54:15.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663000+0000", 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2413999999999999e-05 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663082+0000", 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2659999999999998e-05 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663161+0000", 2024-04-25T15:54:15.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8322e-05 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663262+0000", 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010089099999999999 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.674457+0000", 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011195544999999999 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677208+0000", 2024-04-25T15:54:15.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027504769999999999 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677272+0000", 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4862000000000003e-05 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677316+0000", 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3959 3.7 3:e1bee111:::100000001f8.00000173:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.680323051, 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089007748999999997, 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3959 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.473 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762131+0000", 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0430000000000002e-06 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.764378+0000", 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022474560000000001 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786050+0000", 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021672166 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786065+0000", 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4652e-05 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786161+0000", 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5873999999999997e-05 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786278+0000", 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116684 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.476 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786462+0000", 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000184625 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.794936+0000", 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084740370000000002 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851010+0000", 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056073265999999997 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.477 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851079+0000", 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9556999999999998e-05 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851136+0000", 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7389000000000002e-05 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.478 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3963 3.19 3:9b5b1331:::100000001f8.00000177:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.661859715, 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.167075797, 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3963 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780596+0000", 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8079999999999998e-06 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786013+0000", 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054168410000000004 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800764+0000", 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014751306 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800778+0000", 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4171999999999999e-05 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800924+0000", 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146286 2024-04-25T15:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.801019+0000", 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4456000000000005e-05 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.801147+0000", 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127603 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.809771+0000", 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008624095 2024-04-25T15:54:15.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947582+0000", 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13781142499999999 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947625+0000", 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2928e-05 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947668+0000", 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2877000000000003e-05 2024-04-25T15:54:15.486 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3967 3.24 3:260ecc6d:::100000001f8.0000017b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.643421874, 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18196963599999999, 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3967 2024-04-25T15:54:15.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799032+0000", 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4040000000000002e-06 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800724+0000", 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001691889 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861343+0000", 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.060618914000000003 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861359+0000", 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6294999999999999e-05 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861444+0000", 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4251000000000004e-05 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861517+0000", 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3098999999999999e-05 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861617+0000", 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100603 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.878626+0000", 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017008836999999999 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980913+0000", 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.102287103 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980951+0000", 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7718e-05 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.491 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.981000+0000", 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8523000000000003e-05 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3980 3.8 3:13e1c2d8:::100000001f8.00000188:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.582548470000001, 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.111276876, 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3980 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859907+0000", 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7950000000000001e-06 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861306+0000", 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001398505 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893810+0000", 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032503907999999998 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893829+0000", 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9860999999999998e-05 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893916+0000", 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6890000000000003e-05 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.894052+0000", 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013542200000000001 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.894230+0000", 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017861600000000001 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.902225+0000", 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079943059999999996 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971087+0000", 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068862265000000006 2024-04-25T15:54:15.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971132+0000", 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4903e-05 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971180+0000", 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8405000000000002e-05 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3990 3.19 3:9b35cac6:::100000001f8.00000192:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.534192556000001, 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.093327906000000002, 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3990 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908262+0000", 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.258e-06 2024-04-25T15:54:15.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.909909+0000", 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016468979999999999 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987236+0000", 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.077326763000000007 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987250+0000", 2024-04-25T15:54:15.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4248e-05 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987323+0000", 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2416000000000006e-05 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987413+0000", 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0482999999999994e-05 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:15.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987616+0000", 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020251899999999999 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.999192+0000", 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011576229 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001500+0000", 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023079099999999998 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001546+0000", 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6508999999999998e-05 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001587+0000", 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0673000000000002e-05 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4024 3.24 3:268a62b6:::100000001f8.000001b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:15.503 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.312495478000001, 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.101232684, 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4024 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129960+0000", 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1949999999999999e-06 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.131869+0000", 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001909151 2024-04-25T15:54:15.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208635+0000", 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.076766361000000005 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208652+0000", 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7093999999999999e-05 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208741+0000", 2024-04-25T15:54:15.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.899e-05 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208831+0000", 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9621000000000001e-05 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208947+0000", 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115788 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.225235+0000", 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016288894000000002 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.230862+0000", 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056270290000000004 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.230947+0000", 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5019999999999996e-05 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.231189+0000", 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024154100000000001 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5056 3.38 3:1d039b80:::100000001f8.000005bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.996489+0000", 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4459625760000003, 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.113765959, 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.509 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5056 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.996489+0000", 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.996489+0000", 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.996492+0000", 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7690000000000001e-06 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998555+0000", 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020628059999999999 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998596+0000", 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0855000000000001e-05 2024-04-25T15:54:15.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998611+0000", 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5204e-05 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998717+0000", 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010653 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998847+0000", 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001292 2024-04-25T15:54:15.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998993+0000", 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146682 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.010434+0000", 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011441066999999999 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.110173+0000", 2024-04-25T15:54:15.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099738744000000004 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.110217+0000", 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3467999999999998e-05 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.110255+0000", 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8634000000000003e-05 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5722 3.38 3:1deaa7c3:::100000001f8.00000856:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.719324+0000", 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.7231282569999999, 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096971450000000001, 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5722 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.719324+0000", 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.719324+0000", 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.719327+0000", 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1530000000000001e-06 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.720925+0000", 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597916 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.720963+0000", 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000002e-05 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.720977+0000", 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774000000000001e-05 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.721072+0000", 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5136999999999994e-05 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.721202+0000", 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130068 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.721379+0000", 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017673 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.732921+0000", 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011542202999999999 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.816192+0000", 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.083271077999999998 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.816234+0000", 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.206e-05 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.816295+0000", 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0951e-05 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5724 3.2 3:4214d263:::100000001f8.00000858:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.731630+0000", 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.7108219739999999, 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13283815800000001, 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5724 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.731630+0000", 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.731630+0000", 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.731633+0000", 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6910000000000002e-06 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734039+0000", 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024061310000000002 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734069+0000", 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0539e-05 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734086+0000", 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7218999999999999e-05 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734154+0000", 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7228000000000001e-05 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734237+0000", 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3102000000000005e-05 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:15.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734348+0000", 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111759 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.744430+0000", 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010081639999999999 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.864304+0000", 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.11987374200000001 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.864393+0000", 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9310999999999999e-05 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.864468+0000", 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4795999999999993e-05 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5728 3.29 3:94338435:::100000001f8.0000085c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.750848+0000", 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.6916040749999999, 2024-04-25T15:54:15.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.107473081, 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5728 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.750848+0000", 2024-04-25T15:54:15.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.750848+0000", 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.750851+0000", 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8329999999999998e-06 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752255+0000", 2024-04-25T15:54:15.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014045430000000001 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752301+0000", 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6385999999999999e-05 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752318+0000", 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.643e-05 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752398+0000", 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9745000000000003e-05 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752477+0000", 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9290999999999997e-05 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752596+0000", 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119268 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.769023+0000", 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016426473 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.858162+0000", 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089139180999999998 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.858247+0000", 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4647000000000005e-05 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.858321+0000", 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4283999999999996e-05 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5731 3.1e 3:7ae5ac58:::100000001f8.0000085f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.765834+0000", 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.676617807, 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.105229766, 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.531 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5731 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.765834+0000", 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.765834+0000", 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.765836+0000", 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.99e-06 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767215+0000", 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001378881 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767254+0000", 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8868000000000001e-05 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767273+0000", 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9230000000000001e-05 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767349+0000", 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6217000000000006e-05 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767424+0000", 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4597000000000004e-05 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767540+0000", 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116661 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.774440+0000", 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068993980000000002 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870906+0000", 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096465683999999996 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870985+0000", 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9066999999999994e-05 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871064+0000", 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9172999999999996e-05 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5733 3.1e 3:79c5b821:::100000001f8.00000861:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.776276+0000", 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.666176168, 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094987504, 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5733 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.776276+0000", 2024-04-25T15:54:15.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.776276+0000", 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.776278+0000", 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8059999999999999e-06 2024-04-25T15:54:15.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.777883+0000", 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001604205 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.777926+0000", 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2985999999999998e-05 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.777940+0000", 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4107000000000001e-05 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.778025+0000", 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5066000000000002e-05 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.778105+0000", 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0506000000000005e-05 2024-04-25T15:54:15.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.778219+0000", 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113596 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.790528+0000", 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012308723000000001 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871143+0000", 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080615081000000005 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871211+0000", 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8670000000000005e-05 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871263+0000", 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1758000000000001e-05 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5777 3.2 3:42907a88:::100000001f8.0000088d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.542 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.024509+0000", 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.4179430900000001, 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.133883539, 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5777 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.024509+0000", 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.024509+0000", 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.024512+0000", 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.027e-06 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026152+0000", 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640308 2024-04-25T15:54:15.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026193+0000", 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0967999999999998e-05 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026211+0000", 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8139e-05 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026288+0000", 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7043000000000001e-05 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026368+0000", 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0118999999999996e-05 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026512+0000", 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014386600000000001 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.034500+0000", 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079875799999999993 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.158295+0000", 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.123794761 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.158350+0000", 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5235000000000002e-05 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.158392+0000", 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2493000000000001e-05 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5783 3.1e 3:79270052:::100000001f8.00000893:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.053917+0000", 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.3885351209999999, 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096657987000000001, 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.548 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5783 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.053917+0000", 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.053917+0000", 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.053919+0000", 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4490000000000002e-06 2024-04-25T15:54:15.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055598+0000", 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001679304 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055632+0000", 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3905000000000001e-05 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055646+0000", 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3720000000000001e-05 2024-04-25T15:54:15.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055740+0000", 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4035000000000003e-05 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055854+0000", 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011380200000000001 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.056027+0000", 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172919 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.061846+0000", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058194650000000002 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.150432+0000", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088585632999999997 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.150511+0000", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9191999999999997e-05 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.150575+0000", 2024-04-25T15:54:15.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3563000000000001e-05 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5787 3.19 3:9bb5c4e2:::100000001f8.00000897:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.070830+0000", 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 1.3716216530000001, 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.102205173, 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:15.553 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5787 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.070830+0000", 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.070830+0000", 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.070833+0000", 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8499999999999998e-06 2024-04-25T15:54:15.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072599+0000", 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017661770000000001 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072637+0000", 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7438999999999998e-05 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072655+0000", 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8552000000000001e-05 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072814+0000", 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015876900000000001 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.073065+0000", 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025143700000000003 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.073253+0000", 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018773499999999999 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.094308+0000", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021054812999999999 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.172902+0000", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.078594302000000005 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.172977+0000", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5195000000000001e-05 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.173035+0000", 2024-04-25T15:54:15.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7904e-05 2024-04-25T15:54:15.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:15.558 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:15.558 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:15.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d30896640 1 Processor -- start 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d30896640 1 -- start start 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d30896640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 0x7f4d2c161740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d30896640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d2c0c9b50 con 0x7f4d2c0c35a0 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d30896640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d2c068bf0 con 0x7f4d2c06c640 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d30896640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2c161c80 con 0x7f4d2c0c9740 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d2a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d2c0c35a0 0x7f4d2c15ae00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50356/0 (socket says 172.21.15.67:50356) 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d2a59c640 1 -- 172.21.15.67:0/564937344 learned_addr learned my addr 172.21.15.67:0/564937344 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.493+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 0x7f4d2c161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2406061293 0 0) 0x7f4d2c0c9b50 con 0x7f4d2c0c35a0 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d18003530 con 0x7f4d2c0c35a0 2024-04-25T15:54:15.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302125991 0 0) 0x7f4d2c068bf0 con 0x7f4d2c06c640 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d2c0c9b50 con 0x7f4d2c06c640 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3500933682 0 0) 0x7f4d18003530 con 0x7f4d2c0c35a0 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 msgr2=0x7f4d2c161740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 0x7f4d2c161740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 >> v1:172.21.15.176:6789/0 conn(0x7f4d2c06c640 legacy=0x7f4d2c16b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2c068630 con 0x7f4d2c0c35a0 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f4d1c001600 con 0x7f4d2c0c35a0 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 >> v1:172.21.15.67:6789/0 conn(0x7f4d2c0c35a0 legacy=0x7f4d2c15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18003b80 0x7f4d18003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18004810 0x7f4d18004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 0x7f4d180052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c161c80 con 0x7f4d18004810 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1800bbd0 con 0x7f4d18003b80 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d1800bdd0 con 0x7f4d2c0c9740 2024-04-25T15:54:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18004810 0x7f4d18004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 0x7f4d180052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18003b80 0x7f4d18003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d21d9b640 1 -- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 msgr2=0x7f4d180052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c0c9740 0x7f4d180052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d21d9b640 1 -- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18003b80 msgr2=0x7f4d18003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18003b80 0x7f4d18003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d21d9b640 1 -- 172.21.15.67:0/564937344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d18003530 con 0x7f4d18004810 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.497+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18004810 0x7f4d18004c20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4d14009a90 tx=0x7f4d1401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf8a2e8f13f0ff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14009690 con 0x7f4d18004810 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d140059a0 con 0x7f4d18004810 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d29d9b640 1 -- 172.21.15.67:0/564937344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14005d90 con 0x7f4d18004810 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d30896640 1 -- 172.21.15.67:0/564937344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18004810 msgr2=0x7f4d18004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d30896640 1 --2- 172.21.15.67:0/564937344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18004810 0x7f4d18004c20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4d14009a90 tx=0x7f4d1401ca50 comp rx=0 tx=0).stop 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d2ad9d640 1 -- 172.21.15.67:0/564937344 reap_dead start 2024-04-25T15:54:15.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d30896640 1 -- 172.21.15.67:0/564937344 shutdown_connections 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d30896640 1 -- 172.21.15.67:0/564937344 >> 172.21.15.67:0/564937344 conn(0x7f4d2c0ce1c0 msgr2=0x7f4d2c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d30896640 1 -- 172.21.15.67:0/564937344 shutdown_connections 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.501+0000 7f4d30896640 1 -- 172.21.15.67:0/564937344 wait complete. 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 Processor -- start 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 -- start start 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18004810 0x7f4d2c159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d2c1553b0 0x7f4d2c1fe070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c15de80 0x7f4d2c1fe5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c06ce60 con 0x7f4d2c1553b0 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c0c9220 con 0x7f4d18004810 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d30896640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2c0ca970 con 0x7f4d2c15de80 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2a59c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18004810 0x7f4d2c159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d21d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d2c1553b0 0x7f4d2c1fe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2ad9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c15de80 0x7f4d2c1fe5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d21d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d2c1553b0 0x7f4d2c1fe070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44008/0 (socket says 172.21.15.67:44008) 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2ad9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c15de80 0x7f4d2c1fe5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:56972/0 (socket says 172.21.15.67:56972) 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d21d9b640 1 -- 172.21.15.67:0/1216142236 learned_addr learned my addr 172.21.15.67:0/1216142236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2ad9d640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18004810 msgr2=0x7f4d2c159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18004810 0x7f4d2c159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2ad9d640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d2c1553b0 msgr2=0x7f4d2c1fe070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d2c1553b0 0x7f4d2c1fe070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2ad9d640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2c159790 con 0x7f4d2c15de80 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d2c1553b0 0x7f4d2c1fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.505+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d18004810 0x7f4d2c159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c15de80 0x7f4d2c1fe5b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800bfa0 tx=0x7f4d2400eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24003980 con 0x7f4d2c15de80 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d24018070 con 0x7f4d2c15de80 2024-04-25T15:54:15.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2c0c8290 con 0x7f4d2c15de80 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24003d80 con 0x7f4d2c15de80 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2c159480 con 0x7f4d2c15de80 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f4d2c06a480 con 0x7f4d2c15de80 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4d240050d0 con 0x7f4d2c15de80 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.509+0000 7f4d237fe640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4d0c06f430 0x7f4d0c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(101..101 src has 1..101) ==== 6002+0+0 (secure 0 0 0) 0x7f4d2408ef10 con 0x7f4d2c15de80 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v101) ==== 54+0+16 (secure 0 0 0) 0x7f4d24091590 con 0x7f4d2c15de80 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4d0c06f430 0x7f4d0c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4d2c1feaf0 0x7f4d2c15fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c15a990 con 0x7f4d2c1feaf0 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4d2c1feaf0 0x7f4d2c15fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4d0c06f430 0x7f4d0c071940 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c001ec0 tx=0x7f4d1c007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4d2c1feaf0 0x7f4d2c15fc10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.513+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c15a990 con 0x7f4d2c1feaf0 2024-04-25T15:54:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.533+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c0ce820 con 0x7f4d2c1feaf0 2024-04-25T15:54:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.537+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c0ce820 con 0x7f4d2c1feaf0 2024-04-25T15:54:15.572 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.537+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4d2c151270 0x7f4d2c151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.537+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c151bc0 con 0x7f4d2c151270 2024-04-25T15:54:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.537+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4d2c151270 0x7f4d2c151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.537+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4d2c151270 0x7f4d2c151680 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.541+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c151bc0 con 0x7f4d2c151270 2024-04-25T15:54:15.574 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:15.557+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:15.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.561+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c0cb9d0 con 0x7f4d2c151270 2024-04-25T15:54:15.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.561+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c0cb9d0 con 0x7f4d2c151270 2024-04-25T15:54:15.574 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.561+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4d2c0635a0 0x7f4d2c0639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.561+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c0c4b30 con 0x7f4d2c0635a0 2024-04-25T15:54:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.561+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4d2c0635a0 0x7f4d2c0639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.565+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4d2c0635a0 0x7f4d2c0639b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.565+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c0c4b30 con 0x7f4d2c0635a0 2024-04-25T15:54:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.589+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c063ef0 con 0x7f4d2c0635a0 2024-04-25T15:54:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.589+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c063ef0 con 0x7f4d2c0635a0 2024-04-25T15:54:15.592 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.589+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4d2c0c3ec0 0x7f4d2c0c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.589+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c063ef0 con 0x7f4d2c0c3ec0 2024-04-25T15:54:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.589+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4d2c0c3ec0 0x7f4d2c0c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.593+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4d2c0c3ec0 0x7f4d2c0c42d0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.593+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c063ef0 con 0x7f4d2c0c3ec0 2024-04-25T15:54:15.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.617+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c063ef0 con 0x7f4d2c0c3ec0 2024-04-25T15:54:15.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.617+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c063ef0 con 0x7f4d2c0c3ec0 2024-04-25T15:54:15.620 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:15.620 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:15.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.617+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4d2c02cef0 0x7f4d2c02d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.617+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c0c8460 con 0x7f4d2c02cef0 2024-04-25T15:54:15.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.621+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4d2c02cef0 0x7f4d2c02d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.621+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4d2c02cef0 0x7f4d2c02d300 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.621+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c0c8460 con 0x7f4d2c02cef0 2024-04-25T15:54:15.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.645+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c0c8460 con 0x7f4d2c02cef0 2024-04-25T15:54:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.653+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c0c8460 con 0x7f4d2c02cef0 2024-04-25T15:54:15.656 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.657+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4d2c20f600 0x7f4d2c211a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.657+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c02de00 con 0x7f4d2c20f600 2024-04-25T15:54:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.657+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4d2c20f600 0x7f4d2c211a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.659 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:15.657+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:15.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.661+0000 7f4d21d9b640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4d2c20f600 0x7f4d2c211a20 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.661+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c02de00 con 0x7f4d2c20f600 2024-04-25T15:54:15.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.685+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c0c69e0 con 0x7f4d2c20f600 2024-04-25T15:54:15.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.689+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c0c69e0 con 0x7f4d2c20f600 2024-04-25T15:54:15.690 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.689+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4d2c212470 0x7f4d2c214890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.689+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c0c6b70 con 0x7f4d2c212470 2024-04-25T15:54:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.689+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4d2c212470 0x7f4d2c214890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.689+0000 7f4d2ad9d640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4d2c212470 0x7f4d2c214890 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.693+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c0c6b70 con 0x7f4d2c212470 2024-04-25T15:54:15.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.713+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c214dd0 con 0x7f4d2c212470 2024-04-25T15:54:15.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.717+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c214dd0 con 0x7f4d2c212470 2024-04-25T15:54:15.719 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.717+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4d2c2154f0 0x7f4d2c215900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.717+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f4d2c21a8a0 con 0x7f4d2c2154f0 2024-04-25T15:54:15.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.717+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4d2c2154f0 0x7f4d2c215900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:15.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.721+0000 7f4d2a59c640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4d2c2154f0 0x7f4d2c215900 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:15.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.721+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d2c21a8a0 con 0x7f4d2c2154f0 2024-04-25T15:54:15.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.741+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f4d2c2196a0 con 0x7f4d2c2154f0 2024-04-25T15:54:15.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d237fe640 1 -- 172.21.15.67:0/1216142236 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f4d2c2196a0 con 0x7f4d2c2154f0 2024-04-25T15:54:15.745 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:15.747 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4d2c1feaf0 msgr2=0x7f4d2c15fc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f4d2c1feaf0 0x7f4d2c15fc10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4d2c151270 msgr2=0x7f4d2c151680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f4d2c151270 0x7f4d2c151680 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4d2c0635a0 msgr2=0x7f4d2c0639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f4d2c0635a0 0x7f4d2c0639b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d2ad9d640 1 -- 172.21.15.67:0/1216142236 reap_dead start 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4d2c0c3ec0 msgr2=0x7f4d2c0c42d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4d2c0c3ec0 0x7f4d2c0c42d0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4d2c02cef0 msgr2=0x7f4d2c02d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f4d2c02cef0 0x7f4d2c02d300 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4d2c20f600 msgr2=0x7f4d2c211a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f4d2c20f600 0x7f4d2c211a20 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4d2c212470 msgr2=0x7f4d2c214890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f4d2c212470 0x7f4d2c214890 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4d2c2154f0 msgr2=0x7f4d2c215900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f4d2c2154f0 0x7f4d2c215900 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d2ad9d640 1 -- 172.21.15.67:0/1216142236 reap_dead start 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4d0c06f430 msgr2=0x7f4d0c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4d0c06f430 0x7f4d0c071940 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c001ec0 tx=0x7f4d1c007040 comp rx=0 tx=0).stop 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c15de80 msgr2=0x7f4d2c1fe5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c15de80 0x7f4d2c1fe5b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800bfa0 tx=0x7f4d2400eb90 comp rx=0 tx=0).stop 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 shutdown_connections 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4d0c06f430 0x7f4d0c071940 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 --2- 172.21.15.67:0/1216142236 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d2c15de80 0x7f4d2c1fe5b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:15.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 >> 172.21.15.67:0/1216142236 conn(0x7f4d2c0ce1c0 msgr2=0x7f4d2c0cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:15.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 shutdown_connections 2024-04-25T15:54:15.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:15.745+0000 7f4d30896640 1 -- 172.21.15.67:0/1216142236 wait complete. 2024-04-25T15:54:15.760 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:15.758+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:15.763 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:15.763 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:15.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:54:15.875 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:15.874+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:15.962 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:15.961+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:16.040 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:16.040 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:16.063 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:16.062+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:16.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:54:16.164 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:16.162+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:16.265 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:16.265+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:16.266 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:16.266 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:16.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:54:16.366 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:16.365+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:16.467 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:16.462+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:16.468 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:54:16.469 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 rewriters = 705297.06 kB/sec 2024-04-25T15:54:16.469 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 rewriters = 705245.64 kB/sec 2024-04-25T15:54:16.469 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 705297.06 kB/sec 2024-04-25T15:54:16.469 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 705297.06 kB/sec 2024-04-25T15:54:16.469 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 705297.06 kB/sec 2024-04-25T15:54:16.469 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-04-25T15:54:16.487 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:16.487 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:16.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:54:16.567 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:16.565+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:16.668 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:16.666+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:16.701 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:16.701 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:16.719 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:54:16.770 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:16.769+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:16.870 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:16.866+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:16.911 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:16.911 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:16.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:54:16.971 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:16.969+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:17.072 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:17.070+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:17.114 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:17.114 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:17.132 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:54:17.175 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:17.174+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:17.274 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:17.270+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:17.313 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:17.313 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:17.330 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:54:17.374 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:17.370+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:17.474 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:17.470+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:17.517 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:17.517 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:17.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:54:17.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:54:17.605 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:17.605+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:17.676 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:17.677+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:17.725 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:17.725 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.725+0000 7f5443106640 1 Processor -- start 2024-04-25T15:54:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5443106640 1 -- start start 2024-04-25T15:54:17.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5443106640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f543c152b00 0x7f543c0c4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5443106640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f543c06c640 0x7f543c06ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5443106640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f543c158d20 con 0x7f543c0c8680 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5443106640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f543c0722c0 con 0x7f543c152b00 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5443106640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f543c158fe0 con 0x7f543c06c640 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5440ea2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f543c152b00 0x7f543c0c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f54416a3640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f543c0c8680 0x7f543c0c8a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50360/0 (socket says 172.21.15.67:50360) 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f54416a3640 1 -- 172.21.15.67:0/1987533019 learned_addr learned my addr 172.21.15.67:0/1987533019 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f543bfff640 1 --2- 172.21.15.67:0/1987533019 >> v2:172.21.15.176:3300/0 conn(0x7f543c06c640 0x7f543c06ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5440ea2640 1 -- 172.21.15.67:0/1987533019 >> v2:172.21.15.176:3300/0 conn(0x7f543c06c640 msgr2=0x7f543c06ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5440ea2640 1 --2- 172.21.15.67:0/1987533019 >> v2:172.21.15.176:3300/0 conn(0x7f543c06c640 0x7f543c06ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5440ea2640 1 -- 172.21.15.67:0/1987533019 >> v1:172.21.15.67:6789/0 conn(0x7f543c0c8680 legacy=0x7f543c0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.729+0000 7f5440ea2640 1 -- 172.21.15.67:0/1987533019 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c1592c0 con 0x7f543c152b00 2024-04-25T15:54:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5440ea2640 1 --2- 172.21.15.67:0/1987533019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c0c4420 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f543c0c5be0 tx=0x7f543001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9f5137caa0ea1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f543b7fe640 1 -- 172.21.15.67:0/1987533019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300b0070 con 0x7f543c152b00 2024-04-25T15:54:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f543b7fe640 1 -- 172.21.15.67:0/1987533019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54300ab600 con 0x7f543c152b00 2024-04-25T15:54:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f543b7fe640 1 -- 172.21.15.67:0/1987533019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300ab9d0 con 0x7f543c152b00 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 -- 172.21.15.67:0/1987533019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 msgr2=0x7f543c0c4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 --2- 172.21.15.67:0/1987533019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c0c4420 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f543c0c5be0 tx=0x7f543001c480 comp rx=0 tx=0).stop 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 -- 172.21.15.67:0/1987533019 shutdown_connections 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 --2- 172.21.15.67:0/1987533019 >> v2:172.21.15.176:3300/0 conn(0x7f543c06c640 0x7f543c06ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 --2- 172.21.15.67:0/1987533019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c0c4420 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 -- 172.21.15.67:0/1987533019 >> 172.21.15.67:0/1987533019 conn(0x7f543c06fe70 msgr2=0x7f543c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 -- 172.21.15.67:0/1987533019 shutdown_connections 2024-04-25T15:54:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 -- 172.21.15.67:0/1987533019 wait complete. 2024-04-25T15:54:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 Processor -- start 2024-04-25T15:54:17.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.733+0000 7f5443106640 1 -- start start 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f543c06c640 0x7f543c072e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f543c0c8680 0x7f543c13df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c13e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f543c0c4960 con 0x7f543c152b00 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f543c152f10 con 0x7f543c0c8680 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f543c0c4ae0 con 0x7f543c06c640 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c13e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c13e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44020/0 (socket says 172.21.15.67:44020) 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 -- 172.21.15.67:0/1323467515 learned_addr learned my addr 172.21.15.67:0/1323467515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5440ea2640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f543c06c640 0x7f543c072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f543bfff640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f543c0c8680 0x7f543c13df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 -- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f543c06c640 msgr2=0x7f543c072e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f543c06c640 0x7f543c072e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 -- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f543c0c8680 msgr2=0x7f543c13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f543c0c8680 0x7f543c13df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 -- 172.21.15.67:0/1323467515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c13e9d0 con 0x7f543c152b00 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f543bfff640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f543c0c8680 0x7f543c13df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54416a3640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c13e490 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f543400cad0 tx=0x7f543400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54397fa640 1 -- 172.21.15.67:0/1323467515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5434017070 con 0x7f543c152b00 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54397fa640 1 -- 172.21.15.67:0/1323467515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f543400ac30 con 0x7f543c152b00 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543c074770 con 0x7f543c152b00 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f54397fa640 1 -- 172.21.15.67:0/1323467515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5434012400 con 0x7f543c152b00 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f543c144a50 con 0x7f543c152b00 2024-04-25T15:54:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.737+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543c076730 con 0x7f543c152b00 2024-04-25T15:54:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.741+0000 7f54397fa640 1 -- 172.21.15.67:0/1323467515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f543401a070 con 0x7f543c152b00 2024-04-25T15:54:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.741+0000 7f54397fa640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f542006f400 0x7f5420071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.741+0000 7f54397fa640 1 -- 172.21.15.67:0/1323467515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6002+0+0 (secure 0 0 0) 0x7f543408f6a0 con 0x7f543c152b00 2024-04-25T15:54:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.741+0000 7f5440ea2640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f542006f400 0x7f5420071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.741+0000 7f5440ea2640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f542006f400 0x7f5420071910 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f54300ad000 tx=0x7f5430008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.743 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:54:17.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.745+0000 7f54397fa640 1 -- 172.21.15.67:0/1323467515 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f543405bf80 con 0x7f543c152b00 2024-04-25T15:54:17.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:54:17.791 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:17.790+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:17.879 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:17.874+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:17.892 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:17.892 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:17.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:54:17.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.921+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f543c073a90 con 0x7f543c152b00 2024-04-25T15:54:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.921+0000 7f1e684ca640 1 Processor -- start 2024-04-25T15:54:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.921+0000 7f1e684ca640 1 -- start start 2024-04-25T15:54:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e684ca640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1e6016d6c0 0x7f1e60072430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e684ca640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e600c5ad0 con 0x7f1e6006c640 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e684ca640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e600c5db0 con 0x7f1e600c5620 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e684ca640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1e600c9240 con 0x7f1e6016d6c0 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e66a67640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1e6006c640 0x7f1e6006ca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50362/0 (socket says 172.21.15.67:50362) 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e66a67640 1 -- 172.21.15.67:0/1385977278 learned_addr learned my addr 172.21.15.67:0/1385977278 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e65264640 1 -- 172.21.15.67:0/1385977278 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 91505794 0 0) 0x7f1e600c5ad0 con 0x7f1e6006c640 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.925+0000 7f1e65a65640 1 --2- 172.21.15.67:0/1385977278 >> v2:172.21.15.176:3300/0 conn(0x7f1e6016d6c0 0x7f1e60072430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.929+0000 7f1e65264640 1 -- 172.21.15.67:0/1385977278 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e4c003dc0 con 0x7f1e6006c640 2024-04-25T15:54:17.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.929+0000 7f1e65264640 1 -- 172.21.15.67:0/1385977278 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3934460584 0 0) 0x7f1e600c5db0 con 0x7f1e600c5620 2024-04-25T15:54:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.929+0000 7f1e65264640 1 -- 172.21.15.67:0/1385977278 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e600c5ad0 con 0x7f1e600c5620 2024-04-25T15:54:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.929+0000 7f1e65a65640 1 -- 172.21.15.67:0/1385977278 >> v1:172.21.15.176:6789/0 conn(0x7f1e600c5620 legacy=0x7f1e600c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.929+0000 7f1e65a65640 1 -- 172.21.15.67:0/1385977278 >> v1:172.21.15.67:6789/0 conn(0x7f1e6006c640 legacy=0x7f1e6006ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.929+0000 7f1e65a65640 1 -- 172.21.15.67:0/1385977278 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e600c8340 con 0x7f1e6016d6c0 2024-04-25T15:54:17.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e65a65640 1 --2- 172.21.15.67:0/1385977278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e6016d6c0 0x7f1e60072430 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1e50005dc0 tx=0x7f1e5001efd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=819216e235f293c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e65264640 1 -- 172.21.15.67:0/1385977278 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e500b5070 con 0x7f1e6016d6c0 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e65264640 1 -- 172.21.15.67:0/1385977278 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5000c040 con 0x7f1e6016d6c0 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e65264640 1 -- 172.21.15.67:0/1385977278 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e500b0440 con 0x7f1e6016d6c0 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e684ca640 1 -- 172.21.15.67:0/1385977278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e6016d6c0 msgr2=0x7f1e60072430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e684ca640 1 --2- 172.21.15.67:0/1385977278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e6016d6c0 0x7f1e60072430 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1e50005dc0 tx=0x7f1e5001efd0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e684ca640 1 -- 172.21.15.67:0/1385977278 shutdown_connections 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e684ca640 1 --2- 172.21.15.67:0/1385977278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e6016d6c0 0x7f1e60072430 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e684ca640 1 -- 172.21.15.67:0/1385977278 >> 172.21.15.67:0/1385977278 conn(0x7f1e6006fe70 msgr2=0x7f1e600cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e684ca640 1 -- 172.21.15.67:0/1385977278 shutdown_connections 2024-04-25T15:54:17.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.933+0000 7f1e684ca640 1 -- 172.21.15.67:0/1385977278 wait complete. 2024-04-25T15:54:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 Processor -- start 2024-04-25T15:54:17.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 -- start start 2024-04-25T15:54:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e6006c640 0x7f1e601ffce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e600c5620 0x7f1e601603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e6016d6c0 0x7f1e60159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6016dad0 con 0x7f1e6006c640 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1e600c8990 con 0x7f1e600c5620 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1e600c6090 con 0x7f1e6016d6c0 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e6006c640 0x7f1e601ffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e6006c640 0x7f1e601ffce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44028/0 (socket says 172.21.15.67:44028) 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 -- 172.21.15.67:0/4075359700 learned_addr learned my addr 172.21.15.67:0/4075359700 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66a67640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e6016d6c0 0x7f1e60159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e65a65640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e600c5620 0x7f1e601603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e6016d6c0 msgr2=0x7f1e60159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e6016d6c0 0x7f1e60159140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e600c5620 msgr2=0x7f1e601603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e600c5620 0x7f1e601603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e602033d0 con 0x7f1e6006c640 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e6006c640 0x7f1e601ffce0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c00c430 tx=0x7f1e5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c017070 con 0x7f1e6006c640 2024-04-25T15:54:17.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5c0052f0 con 0x7f1e6006c640 2024-04-25T15:54:17.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c003840 con 0x7f1e6006c640 2024-04-25T15:54:17.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e602036b0 con 0x7f1e6006c640 2024-04-25T15:54:17.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e601f2d30 con 0x7f1e6006c640 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.937+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1e6006a480 con 0x7f1e6006c640 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1e5c01a070 con 0x7f1e6006c640 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e56ffd640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1e4406f400 0x7f1e44071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6002+0+0 (secure 0 0 0) 0x7f1e5c08ea80 con 0x7f1e6006c640 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v101) ==== 54+0+16 (secure 0 0 0) 0x7f1e5c021020 con 0x7f1e6006c640 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e65a65640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1e4406f400 0x7f1e44071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1e601f51f0 0x7f1e60159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e6015bcd0 con 0x7f1e601f51f0 2024-04-25T15:54:17.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e66a67640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1e601f51f0 0x7f1e60159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e66a67640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1e601f51f0 0x7f1e60159680 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e65a65640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1e4406f400 0x7f1e44071910 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1e500b2000 tx=0x7f1e500aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.941+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e58001f10 con 0x7f1e601f51f0 2024-04-25T15:54:17.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.957+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e600c9e50 con 0x7f1e601f51f0 2024-04-25T15:54:17.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.957+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e600c9e50 con 0x7f1e601f51f0 2024-04-25T15:54:17.960 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.957+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1e60162c40 0x7f1e60163050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.957+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1e60074b20 con 0x7f1e60162c40 2024-04-25T15:54:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.957+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1e60162c40 0x7f1e60163050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.961+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1e60162c40 0x7f1e60163050 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.961+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e60074b20 con 0x7f1e60162c40 2024-04-25T15:54:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f54397fa640 1 -- 172.21.15.67:0/1323467515 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v102) ==== 87+0+0 (secure 0 0 0) 0x7f5434060280 con 0x7f543c152b00 2024-04-25T15:54:17.973 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:54:17.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f542006f400 msgr2=0x7f5420071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f542006f400 0x7f5420071910 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f54300ad000 tx=0x7f5430008040 comp rx=0 tx=0).stop 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 msgr2=0x7f543c13e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c13e490 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f543400cad0 tx=0x7f543400cf90 comp rx=0 tx=0).stop 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 shutdown_connections 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f542006f400 0x7f5420071910 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f543c152b00 0x7f543c13e490 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f543c0c8680 0x7f543c13df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 --2- 172.21.15.67:0/1323467515 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f543c06c640 0x7f543c072e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 >> 172.21.15.67:0/1323467515 conn(0x7f543c06fe70 msgr2=0x7f543c146070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 shutdown_connections 2024-04-25T15:54:17.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f5443106640 1 -- 172.21.15.67:0/1323467515 wait complete. 2024-04-25T15:54:17.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.973+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e600cba60 con 0x7f1e60162c40 2024-04-25T15:54:17.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.977+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e600cba60 con 0x7f1e60162c40 2024-04-25T15:54:17.979 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:17.979 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:17.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.977+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1e60063090 0x7f1e600634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.977+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1e60163aa0 con 0x7f1e60063090 2024-04-25T15:54:17.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.977+0000 7f1e65a65640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1e60063090 0x7f1e600634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.977+0000 7f1e65a65640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1e60063090 0x7f1e600634a0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:17.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.981+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e60163aa0 con 0x7f1e60063090 2024-04-25T15:54:17.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.993+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e6015ce20 con 0x7f1e60063090 2024-04-25T15:54:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.997+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e6015ce20 con 0x7f1e60063090 2024-04-25T15:54:17.998 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.997+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1e60063da0 0x7f1e6015e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.997+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1e600cb780 con 0x7f1e60063da0 2024-04-25T15:54:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.997+0000 7f1e66a67640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1e60063da0 0x7f1e6015e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:17.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.997+0000 7f1e66a67640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1e60063da0 0x7f1e6015e070 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:18.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:17.997+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e600cb780 con 0x7f1e60063da0 2024-04-25T15:54:18.000 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:17.998+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:18.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.013+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e6015e880 con 0x7f1e60063da0 2024-04-25T15:54:18.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.013+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e6015e880 con 0x7f1e60063da0 2024-04-25T15:54:18.015 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:18.015 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:18.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.013+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1e6002cef0 0x7f1e6002d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.013+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1e6015e800 con 0x7f1e6002cef0 2024-04-25T15:54:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.013+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1e6002cef0 0x7f1e6002d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.013+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1e6002cef0 0x7f1e6002d300 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.017+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e6015e800 con 0x7f1e6002cef0 2024-04-25T15:54:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.029+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e60077070 con 0x7f1e6002cef0 2024-04-25T15:54:18.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.033+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e60077070 con 0x7f1e6002cef0 2024-04-25T15:54:18.035 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.033+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1e6015e5b0 0x7f1e602165e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.033+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1e60077070 con 0x7f1e6015e5b0 2024-04-25T15:54:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.033+0000 7f1e65a65640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1e6015e5b0 0x7f1e602165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.033+0000 7f1e65a65640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1e6015e5b0 0x7f1e602165e0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:18.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.037+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e60077070 con 0x7f1e6015e5b0 2024-04-25T15:54:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.049+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e60077070 con 0x7f1e6015e5b0 2024-04-25T15:54:18.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.053+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e60077070 con 0x7f1e6015e5b0 2024-04-25T15:54:18.056 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:18.056 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:18.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.053+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1e60217030 0x7f1e60219450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.053+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1e6002deb0 con 0x7f1e60217030 2024-04-25T15:54:18.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.053+0000 7f1e66a67640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1e60217030 0x7f1e60219450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.057+0000 7f1e66a67640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1e60217030 0x7f1e60219450 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.057+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e6002deb0 con 0x7f1e60217030 2024-04-25T15:54:18.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.069+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e60219990 con 0x7f1e60217030 2024-04-25T15:54:18.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.077+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e60219990 con 0x7f1e60217030 2024-04-25T15:54:18.077 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.077+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1e60219ea0 0x7f1e6021c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.077+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1e6021f280 con 0x7f1e60219ea0 2024-04-25T15:54:18.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.077+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1e60219ea0 0x7f1e6021c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.077+0000 7f1e66266640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1e60219ea0 0x7f1e6021c2c0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:18.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.077+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e6021f280 con 0x7f1e60219ea0 2024-04-25T15:54:18.080 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:18.077+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:18.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.093+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1e60000940 con 0x7f1e60219ea0 2024-04-25T15:54:18.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e56ffd640 1 -- 172.21.15.67:0/4075359700 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1e60000940 con 0x7f1e60219ea0 2024-04-25T15:54:18.098 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:18.099 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1e601f51f0 msgr2=0x7f1e60159680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1e601f51f0 0x7f1e60159680 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1e60162c40 msgr2=0x7f1e60163050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1e60162c40 0x7f1e60163050 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1e60063090 msgr2=0x7f1e600634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1e60063090 0x7f1e600634a0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1e60063da0 msgr2=0x7f1e6015e070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e66a67640 1 -- 172.21.15.67:0/4075359700 reap_dead start 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1e60063da0 0x7f1e6015e070 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1e6002cef0 msgr2=0x7f1e6002d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1e6002cef0 0x7f1e6002d300 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1e6015e5b0 msgr2=0x7f1e602165e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1e6015e5b0 0x7f1e602165e0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1e60217030 msgr2=0x7f1e60219450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1e60217030 0x7f1e60219450 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1e60219ea0 msgr2=0x7f1e6021c2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1e60219ea0 0x7f1e6021c2c0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e66a67640 1 -- 172.21.15.67:0/4075359700 reap_dead start 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1e4406f400 msgr2=0x7f1e44071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1e4406f400 0x7f1e44071910 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1e500b2000 tx=0x7f1e500aa070 comp rx=0 tx=0).stop 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e6006c640 msgr2=0x7f1e601ffce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e6006c640 0x7f1e601ffce0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c00c430 tx=0x7f1e5c00c8f0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 shutdown_connections 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1e4406f400 0x7f1e44071910 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 --2- 172.21.15.67:0/4075359700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e6006c640 0x7f1e601ffce0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 >> 172.21.15.67:0/4075359700 conn(0x7f1e6006fe70 msgr2=0x7f1e600c93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 shutdown_connections 2024-04-25T15:54:18.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:18.097+0000 7f1e684ca640 1 -- 172.21.15.67:0/4075359700 wait complete. 2024-04-25T15:54:18.105 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 239.00353472399999, 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.106 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:54:18.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:54:18.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.789660190999996, 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.112 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:18.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:54:18.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 54.518982239000003, 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:18.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:54:18.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2266 3.11 3:89528692:::100000001f8.000004d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.530815523000001, 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.38717112199999998, 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2266 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.124 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571186+0000", 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4569999999999999e-06 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784888+0000", 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.213702742 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785181+0000", 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029301099999999998 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785201+0000", 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9683000000000001e-05 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785333+0000", 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013224600000000001 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785779+0000", 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044548299999999999 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.786106+0000", 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032678999999999998 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870593+0000", 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084487524999999994 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958234+0000", 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087640933000000004 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958283+0000", 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8473999999999997e-05 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958354+0000", 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1778000000000003e-05 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2272 3.13 3:ca3e88be:::100000001f8.000004dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.306066874999999, 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13173032000000001, 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2272 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.130 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795936+0000", 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.754e-06 2024-04-25T15:54:18.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805167+0000", 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092308890000000008 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805557+0000", 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039059400000000002 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805588+0000", 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0865999999999999e-05 2024-04-25T15:54:18.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805759+0000", 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017085599999999999 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806172+0000", 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000413416 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806552+0000", 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000379417 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905757+0000", 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099205023000000003 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927568+0000", 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021810800000000002 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927616+0000", 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:54:18.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927662+0000", 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6535000000000001e-05 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2273 3.39 3:9fcbcc01:::100000001f8.000004dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.29681476, 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15983650499999999, 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.135 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2273 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805188+0000", 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816847+0000", 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011659119000000001 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817576+0000", 2024-04-25T15:54:18.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00072975000000000004 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817601+0000", 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4450000000000001e-05 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817741+0000", 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014058500000000001 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818074+0000", 2024-04-25T15:54:18.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033291000000000001 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818385+0000", 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031034999999999998 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870618+0000", 2024-04-25T15:54:18.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052233743999999999 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964814+0000", 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094195216999999998 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964945+0000", 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013086799999999999 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965021+0000", 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6032000000000007e-05 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2275 3.12 3:492b2588:::100000001f8.000004df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.285139381, 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14794684399999999, 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.143 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2275 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816863+0000", 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4149999999999999e-06 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829311+0000", 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012448222 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829569+0000", 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025766800000000001 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829598+0000", 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.968e-05 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829786+0000", 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187746 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830129+0000", 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034284099999999998 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830479+0000", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034961500000000001 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867146+0000", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036667733000000001 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964688+0000", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097541577000000004 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964748+0000", 2024-04-25T15:54:18.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9626000000000001e-05 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964806+0000", 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8721000000000002e-05 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2278 3.2b 3:d79b9024:::100000001f8.000004e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.272674123000002, 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135647779, 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.148 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2278 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829328+0000", 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7309999999999999e-06 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843821+0000", 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014493012 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844787+0000", 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096513199999999995 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844876+0000", 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9147999999999994e-05 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844972+0000", 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6566999999999999e-05 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845195+0000", 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222699 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845422+0000", 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022700699999999999 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870319+0000", 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024897319000000001 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964892+0000", 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094572233000000006 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964938+0000", 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6646e-05 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964972+0000", 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4285000000000001e-05 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2305 3.2f e90/51 3:f6648653:::100000001f8.000004fd:head v 90'33, mlcod=90'30)", 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.264740332999999, 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12780160600000001, 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837261+0000", 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1679999999999998e-06 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839587+0000", 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00232607 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839620+0000", 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3503000000000001e-05 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839642+0000", 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1302e-05 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839754+0000", 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112454 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839798+0000", 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4240999999999999e-05 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965003+0000", 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12520515600000001 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965060+0000", 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6712000000000002e-05 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3917 3.27 3:e652e8b8:::100000001f8.00000149:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.603929829, 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16896387500000001, 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3917 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498071+0000", 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4779999999999998e-06 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509296+0000", 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011224433000000001 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509572+0000", 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027576500000000003 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509591+0000", 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9466e-05 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509677+0000", 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5607999999999998e-05 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509773+0000", 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6136000000000002e-05 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509942+0000", 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016951999999999999 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.610111+0000", 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.100168231 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666922+0000", 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056811077000000001 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666991+0000", 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9758000000000005e-05 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667033+0000", 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1403000000000003e-05 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3918 3.11 3:8b260724:::100000001f8.0000014a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.592693616, 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15748047100000001, 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.163 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3918 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509307+0000", 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.12e-06 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515132+0000", 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058250860000000002 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515164+0000", 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1563000000000003e-05 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515179+0000", 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5102e-05 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515262+0000", 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2917999999999995e-05 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515360+0000", 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7532000000000001e-05 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:18.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515512+0000", 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152442 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654409+0000", 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13889727499999999 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666644+0000", 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012234584999999999 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666730+0000", 2024-04-25T15:54:18.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5793999999999999e-05 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666786+0000", 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6054000000000002e-05 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3922 3.d 3:b3f0d00e:::100000001f8.0000014e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.583331695, 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.150212927, 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.168 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3922 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518670+0000", 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1039999999999998e-06 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529034+0000", 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010364237 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529073+0000", 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8407e-05 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529092+0000", 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9411000000000001e-05 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529203+0000", 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011113 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529330+0000", 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012611499999999999 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:18.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529517+0000", 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187639 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667032+0000", 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13751481300000001 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668779+0000", 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017469740000000001 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668835+0000", 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6251000000000001e-05 2024-04-25T15:54:18.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668880+0000", 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4845999999999997e-05 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3920 3.26 e97/18 3:6422fb70:::100000001f8.0000014c:head v 97'93, mlcod=96'91)", 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.578308547000001, 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14317563999999999, 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.173 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523692+0000", 2024-04-25T15:54:18.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8050000000000001e-06 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525897+0000", 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022045290000000002 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525923+0000", 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6225e-05 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525941+0000", 2024-04-25T15:54:18.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8304e-05 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526045+0000", 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104158 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526076+0000", 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0454e-05 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666788+0000", 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140712106 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666866+0000", 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8059000000000006e-05 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3926 3.d 3:b20dc027:::100000001f8.00000152:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.563822, 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131821096, 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.177 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3926 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538180+0000", 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4690000000000002e-06 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542940+0000", 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047600250000000002 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542988+0000", 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7633999999999998e-05 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543006+0000", 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8114999999999999e-05 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543103+0000", 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6723000000000002e-05 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543283+0000", 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018030100000000001 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543476+0000", 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019295200000000001 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668734+0000", 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12525763200000001 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669901+0000", 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011670890000000001 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669955+0000", 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4373999999999998e-05 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669998+0000", 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3971 3.3f 3:febcb5ea:::100000001f8.0000017f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.284298244, 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15452237999999999, 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.182 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3971 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817703+0000", 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822518+0000", 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048147939999999998 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822563+0000", 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4515000000000003e-05 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822577+0000", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.432e-05 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822664+0000", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7440000000000003e-05 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822786+0000", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012110800000000001 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822960+0000", 2024-04-25T15:54:18.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017468900000000001 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945290+0000", 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122329907 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972057+0000", 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026767102000000001 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972139+0000", 2024-04-25T15:54:18.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2070000000000005e-05 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972223+0000", 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3565000000000004e-05 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5059 3.26 e99/18 3:65b3bea5:::100000001f8.000005bf:head v 99'112, mlcod=99'109)", 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.0846090729999993, 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131194432, 2024-04-25T15:54:18.187 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017392+0000", 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4650000000000001e-06 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021054+0000", 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036613240000000001 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021076+0000", 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2067000000000001e-05 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021092+0000", 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6209999999999999e-05 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021178+0000", 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6260000000000004e-05 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021208+0000", 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0046999999999999e-05 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148491+0000", 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12728336500000001 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148584+0000", 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2694000000000004e-05 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5063 3.13 3:c9096c28:::100000001f8.000005c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:18.191 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.0730569170000006, 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13114490600000001, 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5063 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028946+0000", 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0520000000000003e-06 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035431+0000", 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064851509999999998 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035461+0000", 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9547999999999999e-05 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035474+0000", 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2880999999999999e-05 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035550+0000", 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6027999999999998e-05 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035622+0000", 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2366999999999993e-05 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035727+0000", 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105561 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.150510+0000", 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11478287099999999 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159903+0000", 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093925880000000003 2024-04-25T15:54:18.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159983+0000", 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0122000000000003e-05 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160087+0000", 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103737 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5730 3.e 3:7357dd18:::100000001f8.0000085e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:18.196 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3427606980000002, 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14322902300000001, 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5730 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759240+0000", 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3910000000000001e-06 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765109+0000", 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058681660000000002 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765139+0000", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9981e-05 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765149+0000", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0125e-05 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765215+0000", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6700999999999997e-05 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765293+0000", 2024-04-25T15:54:18.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7883e-05 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765394+0000", 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100869 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.893428+0000", 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128033758 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902331+0000", 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089030610000000003 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902403+0000", 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1891999999999995e-05 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902467+0000", 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4196000000000007e-05 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5769 3.d 3:b350774d:::100000001f8.00000885:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:18.201 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.1100429529999998, 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12752495899999999, 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5769 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:18.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991958+0000", 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998738+0000", 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067804069999999996 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998782+0000", 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3738e-05 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998800+0000", 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7907e-05 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998944+0000", 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144089 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999140+0000", 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195465 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999415+0000", 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027557100000000001 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.070872+0000", 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071456602999999994 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119350+0000", 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048478354000000001 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119421+0000", 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0970999999999995e-05 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119481+0000", 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9802000000000001e-05 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.206 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5780 3.f 3:f20b812e:::100000001f8.00000890:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.061861875, 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13282865699999999, 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5780 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:18.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040139+0000", 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9570000000000001e-06 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:18.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047681+0000", 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075419279999999998 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047720+0000", 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8794999999999997e-05 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047740+0000", 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0091000000000001e-05 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047828+0000", 2024-04-25T15:54:18.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8745000000000005e-05 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047922+0000", 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3076000000000001e-05 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048095+0000", 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000173134 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.170485+0000", 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122390033 2024-04-25T15:54:18.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172900+0000", 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002415482 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172933+0000", 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3021000000000001e-05 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172966+0000", 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2394999999999997e-05 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.211 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.212 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:18.212 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:18.212 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:18.213 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:18.210+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:18.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:54:18.314 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:18.310+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:18.415 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:18.410+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:18.415 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:18.415 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:18.415 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:18.415 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:18.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:54:18.516 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:18.514+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:18.580 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:18.580 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:18.580 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:18.580 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:18.580 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:18.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:54:18.644 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:18.645+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:18.718 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:18.714+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:668 3.1 e81/76 3:83ba1de1:::100000001f7.00000098:head v 81'14, mlcod=81'12)", 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:18.754 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.880357535999998, 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.045409913000000003, 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872468+0000", 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3659999999999999e-06 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879062+0000", 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065941039999999999 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879116+0000", 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4372000000000001e-05 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879152+0000", 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6139000000000001e-05 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879318+0000", 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016595499999999999 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879365+0000", 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6897999999999997e-05 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917803+0000", 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038437773000000001 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917875+0000", 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2305999999999995e-05 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:637 3.20 e81/18 3:07263727:::100000001f7.00000079:head v 81'17, mlcod=81'15)", 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 54.870869044999999, 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040244958999999997, 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881957+0000", 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3890000000000001e-06 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888368+0000", 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006410682 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888395+0000", 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7477000000000001e-05 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888414+0000", 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9046e-05 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888512+0000", 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7876999999999998e-05 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888546+0000", 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3204000000000003e-05 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922131+0000", 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033585168999999998 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922199+0000", 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8114999999999994e-05 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1709 3.b e89/21 3:d0c4adc9:::100000001f8.000002a9:head v 89'35, mlcod=89'33)", 2024-04-25T15:54:18.762 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.753893058000003, 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041562524000000003, 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:18.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998932+0000", 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7570000000000001e-06 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000395+0000", 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014629389999999999 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000414+0000", 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9908999999999999e-05 2024-04-25T15:54:18.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000432+0000", 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249999999999999e-05 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000518+0000", 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6501000000000004e-05 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000548+0000", 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9777e-05 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040424+0000", 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039875586999999997 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040492+0000", 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8804000000000001e-05 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1710 3.b e89/21 3:d079ce42:::100000001f8.000002aa:head v 89'36, mlcod=89'33)", 2024-04-25T15:54:18.766 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 33.747464340999997, 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040049727, 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005360+0000", 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9290000000000001e-06 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006358+0000", 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099724699999999993 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006377+0000", 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9656e-05 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006392+0000", 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4207e-05 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006477+0000", 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4989999999999998e-05 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006507+0000", 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0786e-05 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045329+0000", 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038821829000000002 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045408+0000", 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9083000000000002e-05 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2273 3.39 e90/18 3:9fcbcc01:::100000001f8.000004dd:head v 90'54, mlcod=90'52)", 2024-04-25T15:54:18.770 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.927315096000001, 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035128116000000001, 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825510+0000", 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837142+0000", 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011632136 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837175+0000", 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3166000000000001e-05 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837197+0000", 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1495e-05 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837293+0000", 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6353000000000004e-05 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837329+0000", 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5901000000000001e-05 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860257+0000", 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022927804999999999 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860636+0000", 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000379327 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2392 3.32 e90/40 3:4cdbb504:::100000001f8.00000554:head v 90'45, mlcod=90'43)", 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:18.774 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.343500335000002, 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044535299, 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409325+0000", 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2220000000000001e-06 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410846+0000", 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001521454 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410866+0000", 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0129999999999999e-05 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410882+0000", 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5452e-05 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410962+0000", 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9833999999999995e-05 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410991+0000", 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.968e-05 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453793+0000", 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042801621999999998 2024-04-25T15:54:18.777 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453858+0000", 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4905000000000003e-05 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2394 3.6 e90/76 3:63a5c66e:::100000001f8.00000556:head v 90'46, mlcod=90'44)", 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.328805211999999, 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038940880999999997, 2024-04-25T15:54:18.778 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.779 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424019+0000", 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6160000000000001e-06 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428140+0000", 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041208529999999998 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428169+0000", 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8824999999999999e-05 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428190+0000", 2024-04-25T15:54:18.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0894999999999999e-05 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428283+0000", 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3277999999999997e-05 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428318+0000", 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5191000000000002e-05 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462890+0000", 2024-04-25T15:54:18.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034571283000000001 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462959+0000", 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000001e-05 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2396 3.3d e90/40 3:bc4ba9bc:::100000001f8.00000558:head v 90'44, mlcod=90'42)", 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.318999990999998, 2024-04-25T15:54:18.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033872159999999998, 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433825+0000", 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.951e-06 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435239+0000", 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014141729999999999 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435260+0000", 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0775e-05 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435278+0000", 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8317000000000001e-05 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435363+0000", 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4441e-05 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435395+0000", 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1939e-05 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467549+0000", 2024-04-25T15:54:18.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032154061999999997 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467695+0000", 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014650199999999999 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3866 3.1 e97/76 3:81b781a2:::100000001f8.00000116:head v 97'69, mlcod=96'67)", 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.508818054000001, 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035311905999999997, 2024-04-25T15:54:18.786 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244007+0000", 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249687+0000", 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056798860000000003 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249709+0000", 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2393999999999999e-05 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249729+0000", 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9536000000000001e-05 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249849+0000", 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120068 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249888+0000", 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9103999999999997e-05 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279251+0000", 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029363087999999999 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279317+0000", 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5535000000000002e-05 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3865 3.20 e97/18 3:0689c4f5:::100000001f8.00000115:head v 97'72, mlcod=94'70)", 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:18.790 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.506009357, 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032711928000000001, 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246817+0000", 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.27e-06 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252975+0000", 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006157872 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252993+0000", 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8070000000000001e-05 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253009+0000", 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6169999999999999e-05 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253090+0000", 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1312999999999999e-05 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253111+0000", 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279471+0000", 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026359693 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279525+0000", 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4530999999999997e-05 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.794 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4046 3.3a e97/18 3:5c489b63:::100000001f8.000001ca:head v 97'64, mlcod=97'62)", 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.494175417999999, 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037080704999999999, 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258649+0000", 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.637e-06 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262631+0000", 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039815079999999999 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.796 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262647+0000", 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6745999999999999e-05 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262663+0000", 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6116e-05 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262779+0000", 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011513700000000001 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262807+0000", 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8200000000000001e-05 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295660+0000", 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032853392000000002 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295728+0000", 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7968999999999999e-05 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4727 3.1b e98/76 3:db35eb1b:::100000001f8.00000473:head v 98'76, mlcod=98'74)", 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.611730203, 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036260694000000003, 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141095+0000", 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9870000000000002e-06 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145531+0000", 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044364670000000004 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.800 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145560+0000", 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354999999999999e-05 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145578+0000", 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8715e-05 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145701+0000", 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012246800000000001 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145732+0000", 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1615000000000001e-05 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177281+0000", 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031549063000000002 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177353+0000", 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2024e-05 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.802 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4899 3.2b e98/40 3:d5e979f0:::100000001f8.0000051f:head v 98'72, mlcod=98'70)", 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.5725565760000002, 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034607091, 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180267+0000", 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6499999999999998e-07 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184620+0000", 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043528660000000004 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184641+0000", 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1126000000000001e-05 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184660+0000", 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9014000000000001e-05 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184747+0000", 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6829999999999994e-05 2024-04-25T15:54:18.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184776+0000", 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9473999999999999e-05 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214781+0000", 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030004797 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214873+0000", 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2219000000000006e-05 2024-04-25T15:54:18.806 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5058 3.17 e99/40 3:eaf2c8b6:::100000001f8.000005be:head v 99'80, mlcod=99'78)", 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7381470169999993, 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035602108, 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014679+0000", 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.095e-06 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021382+0000", 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067033750000000001 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021413+0000", 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0846000000000002e-05 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021435+0000", 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1378000000000001e-05 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021583+0000", 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014862400000000001 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021620+0000", 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6718999999999999e-05 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050194+0000", 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028573676999999999 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050278+0000", 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4394000000000005e-05 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5611 3.20 e100/18 3:07896b3f:::100000001f8.000007e7:head v 100'100, mlcod=99'98)", 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6692939579999999, 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034695030000000002, 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.811 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083531+0000", 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1179999999999999e-06 2024-04-25T15:54:18.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092805+0000", 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.009273524 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092830+0000", 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5894e-05 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092847+0000", 2024-04-25T15:54:18.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6762999999999999e-05 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092944+0000", 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6612000000000003e-05 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092974+0000", 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0568000000000003e-05 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118174+0000", 2024-04-25T15:54:18.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025199813000000001 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118224+0000", 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9737999999999997e-05 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5680 3.10 e100/18 3:0bf23c77:::100000001f8.0000082c:head v 100'114, mlcod=100'112)", 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.2640109170000002, 2024-04-25T15:54:18.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039581641000000001, 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488821+0000", 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9060000000000008e-06 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494301+0000", 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054798900000000003 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494321+0000", 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0271999999999998e-05 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494338+0000", 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249e-05 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494420+0000", 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2107999999999994e-05 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494443+0000", 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3071e-05 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528334+0000", 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033890497999999998 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528394+0000", 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9647e-05 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5986 3.10 e100/18 3:09f7ffbc:::100000001f8.0000095e:head v 100'117, mlcod=100'115)", 2024-04-25T15:54:18.819 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.5297253189999998, 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047845334000000003, 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:18.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223099+0000", 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230025+0000", 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069255319999999999 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230048+0000", 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3135000000000001e-05 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230062+0000", 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4119e-05 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230143+0000", 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0508999999999998e-05 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230165+0000", 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2314999999999999e-05 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270851+0000", 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040685763999999999 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270943+0000", 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2326999999999999e-05 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.823 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5978 3.b e100/21 3:d3280d2b:::100000001f8.00000956:head v 100'98, mlcod=100'96)", 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.5217651669999999, 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051283404999999997, 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231060+0000", 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1169999999999998e-06 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233300+0000", 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022402749999999999 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233324+0000", 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4261e-05 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233347+0000", 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.298e-05 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233488+0000", 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000140359 2024-04-25T15:54:18.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233523+0000", 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5040000000000003e-05 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282285+0000", 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048762358999999998 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282341+0000", 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6014000000000003e-05 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.827 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5988 3.3d e100/40 3:bdc62163:::100000001f8.00000960:head v 100'102, mlcod=100'100)", 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.5127687189999999, 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043901969999999998, 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:18.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240056+0000", 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.674e-06 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242229+0000", 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021732269999999998 2024-04-25T15:54:18.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242254+0000", 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4661999999999999e-05 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242271+0000", 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.681e-05 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242362+0000", 2024-04-25T15:54:18.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1411999999999998e-05 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242396+0000", 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3983000000000003e-05 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283805+0000", 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041409488000000001 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283956+0000", 2024-04-25T15:54:18.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015071400000000001 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5987 3.2b e100/40 3:d716bd7b:::100000001f8.0000095f:head v 100'88, mlcod=100'86)", 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 3.5087103060000002, 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038273439999999999, 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244115+0000", 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0020000000000001e-06 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248434+0000", 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043194920000000003 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248458+0000", 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3941e-05 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248477+0000", 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8995999999999999e-05 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248566+0000", 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9331000000000002e-05 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248596+0000", 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9332e-05 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282347+0000", 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033750952000000001 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282386+0000", 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9394000000000003e-05 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:18.836 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:18.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:54:18.838 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:18.834+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:18.939 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:18.937+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:19.036 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:19.036 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:19.036 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:19.039 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:19.038+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:19.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:54:19.145 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:19.145+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:19.240 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:19.241+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:19.257 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:19.257 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:19.257 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:19.257 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:19.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:54:19.365 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:19.365+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:19.441 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:19.438+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:19.479 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:19.479 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 217.22948734299999, 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.480 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.481 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.482 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.483 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.484 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:676 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.547449012000001, 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053839007000000001, 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.485 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.486 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929776+0000", 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.743e-06 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931715+0000", 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019394130000000001 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931754+0000", 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8779000000000003e-05 2024-04-25T15:54:19.487 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931769+0000", 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5231999999999999e-05 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931867+0000", 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7485000000000006e-05 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.488 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932043+0000", 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017616100000000001 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932181+0000", 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013769100000000001 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.942677+0000", 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010495931 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.489 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983477+0000", 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040800621000000002 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983546+0000", 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9138000000000001e-05 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983611+0000", 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4813000000000004e-05 2024-04-25T15:54:19.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:733 3.22 3:44586a4d:::100000001f7.000000d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 55.168982964000001, 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037465947999999999, 2024-04-25T15:54:19.491 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 733 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.492 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308242+0000", 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.399e-06 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309826+0000", 2024-04-25T15:54:19.493 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015840349999999999 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309863+0000", 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7122000000000001e-05 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309880+0000", 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6888000000000002e-05 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309986+0000", 2024-04-25T15:54:19.494 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105979 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310169+0000", 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018349699999999999 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310353+0000", 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018415799999999999 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.495 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.342377+0000", 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032023169999999997 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345569+0000", 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031928680000000002 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345644+0000", 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4140999999999994e-05 2024-04-25T15:54:19.496 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345704+0000", 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0690999999999999e-05 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1589 3.1a e88/76 3:590c0ff7:::100000001f8.00000231:head v 88'30, mlcod=88'28)", 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.230397963000001, 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032177728000000003, 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.497 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.498 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246826+0000", 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1900000000000002e-06 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045171680000000002 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251364+0000", 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1444999999999999e-05 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.499 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251375+0000", 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.115e-05 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251458+0000", 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3190999999999997e-05 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251485+0000", 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6661000000000001e-05 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.278940+0000", 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027455222000000001 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.279001+0000", 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0701e-05 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1595 3.18 3:1adf79c2:::100000001f8.00000237:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.195036926, 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038955741000000002, 2024-04-25T15:54:19.501 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1595 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282187+0000", 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1569999999999998e-06 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284089+0000", 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001902672 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284127+0000", 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7663000000000002e-05 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284141+0000", 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284236+0000", 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4895999999999994e-05 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284417+0000", 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018094500000000001 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284602+0000", 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018422800000000001 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.293831+0000", 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092292650000000004 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321052+0000", 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027220978 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321098+0000", 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6520000000000002e-05 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321140+0000", 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1850000000000001e-05 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.506 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2389 3.18 3:1bdc9869:::100000001f8.00000551:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.087985723999999, 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030905229999999999, 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2389 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389238+0000", 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7099999999999999e-06 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391140+0000", 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001901359 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391174+0000", 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4165000000000002e-05 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391187+0000", 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3188e-05 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391277+0000", 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0267999999999997e-05 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391455+0000", 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017728700000000001 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391618+0000", 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163308 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.419720+0000", 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028102374999999999 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420001+0000", 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028060900000000002 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420083+0000", 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2010999999999998e-05 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420141+0000", 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7949999999999999e-05 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2392 3.32 3:4cdbb504:::100000001f8.00000554:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.071286965999999, 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048266518000000001, 2024-04-25T15:54:19.512 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2392 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405937+0000", 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6759999999999999e-06 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408696+0000", 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027593589999999999 2024-04-25T15:54:19.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408735+0000", 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8516000000000002e-05 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408749+0000", 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4117000000000001e-05 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408844+0000", 2024-04-25T15:54:19.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4891999999999999e-05 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409024+0000", 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018000199999999999 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409200+0000", 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017591800000000001 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.427538+0000", 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018337728000000001 2024-04-25T15:54:19.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454069+0000", 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026531451000000001 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454141+0000", 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2316000000000004e-05 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454201+0000", 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9543000000000002e-05 2024-04-25T15:54:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2393 3.22 3:45fb7032:::100000001f8.00000555:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.064894395, 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052582106000000003, 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2393 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412330+0000", 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3119999999999998e-06 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413928+0000", 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597521 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413971+0000", 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3086e-05 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413988+0000", 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7336000000000001e-05 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414091+0000", 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103099 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414245+0000", 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015350300000000001 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414359+0000", 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011410500000000001 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.446661+0000", 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032302571000000002 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464781+0000", 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018119283999999999 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464856+0000", 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5153000000000004e-05 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464909+0000", 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3136000000000001e-05 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2395 3.18 3:183ba26c:::100000001f8.00000557:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.052494286999998, 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029801055999999999, 2024-04-25T15:54:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2395 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424735+0000", 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1780000000000003e-06 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426303+0000", 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015679170000000001 2024-04-25T15:54:19.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426349+0000", 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6343e-05 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426370+0000", 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329000000000001e-05 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426458+0000", 2024-04-25T15:54:19.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.831e-05 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426609+0000", 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150629 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426735+0000", 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012625900000000001 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.441627+0000", 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014891824999999999 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454417+0000", 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012789994000000001 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454476+0000", 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8752000000000003e-05 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454528+0000", 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2519999999999999e-05 2024-04-25T15:54:19.529 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3145 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.042535553, 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030601595999999998, 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.530 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3145 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.531 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434689+0000", 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.005e-06 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436062+0000", 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001373682 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436096+0000", 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3797000000000002e-05 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436107+0000", 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436177+0000", 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0069999999999998e-05 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436346+0000", 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169168 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436510+0000", 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163807 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.443699+0000", 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007189394 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465141+0000", 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021441386999999999 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465223+0000", 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1920000000000002e-05 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465287+0000", 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4692999999999999e-05 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.535 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3841 3.18 3:1a4016cc:::100000001f8.000000fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 16.391538719, 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043270667999999998, 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3841 2024-04-25T15:54:19.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085686+0000", 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4510000000000001e-06 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088301+0000", 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002614987 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088348+0000", 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7057e-05 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088362+0000", 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4352000000000001e-05 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088457+0000", 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4794000000000001e-05 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088583+0000", 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012605400000000001 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088776+0000", 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019268700000000001 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.107828+0000", 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019051702 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128788+0000", 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020960433000000001 2024-04-25T15:54:19.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128888+0000", 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9493000000000004e-05 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128953+0000", 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5658000000000001e-05 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3977 3.18 3:1b05c35d:::100000001f8.00000185:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.635000137, 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13853472, 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3977 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.542 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842224+0000", 2024-04-25T15:54:19.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1130000000000001e-06 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844135+0000", 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019102920000000001 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844181+0000", 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6616000000000002e-05 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844202+0000", 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0542000000000001e-05 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844358+0000", 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156379 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844555+0000", 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000196651 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844759+0000", 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020436799999999999 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.853781+0000", 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090217130000000006 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980620+0000", 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.126839061 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980690+0000", 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9596000000000002e-05 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980756+0000", 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4099 3.18 3:181e4796:::100000001f8.000001ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.941927931, 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030713761999999999, 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.547 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4099 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:19.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535297+0000", 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2049999999999998e-06 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536942+0000", 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00164504 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536980+0000", 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8736999999999999e-05 2024-04-25T15:54:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536997+0000", 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6653000000000001e-05 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537091+0000", 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3786e-05 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537191+0000", 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9840000000000006e-05 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537331+0000", 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014034900000000001 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.556229+0000", 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018898278000000001 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565921+0000", 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096916850000000002 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565961+0000", 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0290999999999998e-05 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.566007+0000", 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5898e-05 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4812 3.18 3:194fa8d8:::100000001f8.000004c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.793513716, 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038646409, 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.554 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4812 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683711+0000", 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7170000000000002e-06 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685822+0000", 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105619999999999 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685876+0000", 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3798999999999998e-05 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685910+0000", 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4727999999999997e-05 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686022+0000", 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111549 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686150+0000", 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826800000000001 2024-04-25T15:54:19.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686332+0000", 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181538 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.693171+0000", 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068389519999999997 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722249+0000", 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029078171 2024-04-25T15:54:19.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722294+0000", 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4756000000000003e-05 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722354+0000", 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0368999999999997e-05 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4967 3.22 3:478cb4f4:::100000001f8.00000563:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.9355800030000001, 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035932619999999998, 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4967 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.560 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541672+0000", 2024-04-25T15:54:19.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0978000000000001e-05 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543306+0000", 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016338310000000001 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543342+0000", 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5994000000000001e-05 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543360+0000", 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543470+0000", 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110319 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543624+0000", 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153549 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543799+0000", 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017463299999999999 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.555562+0000", 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011763177999999999 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577476+0000", 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021913818000000002 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577529+0000", 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3711000000000001e-05 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577574+0000", 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4678999999999997e-05 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.565 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5705 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.8528565390000002, 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080110828999999995, 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5705 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624368+0000", 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.117e-06 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.567 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626198+0000", 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001829968 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626234+0000", 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6220000000000002e-05 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626249+0000", 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4742999999999999e-05 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626341+0000", 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2477000000000002e-05 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626462+0000", 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121191 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626636+0000", 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017351000000000001 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.635055+0000", 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084194100000000004 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704341+0000", 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.069285804000000006 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704412+0000", 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0674999999999997e-05 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704476+0000", 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3713999999999993e-05 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5751 3.18 3:1b3fa6ee:::100000001f8.00000873:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.6007906570000001, 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037922083000000002, 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5751 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.572 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876434+0000", 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-04-25T15:54:19.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878480+0000", 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020457800000000001 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878539+0000", 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8640000000000001e-05 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878587+0000", 2024-04-25T15:54:19.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7746e-05 2024-04-25T15:54:19.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878724+0000", 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137572 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878868+0000", 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014331899999999999 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.879018+0000", 2024-04-25T15:54:19.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015069000000000001 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.886482+0000", 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007464048 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914229+0000", 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027746804 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.579 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914300+0000", 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0952999999999996e-05 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914353+0000", 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.274e-05 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5755 3.32 3:4c40e2d2:::100000001f8.00000877:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:19.580 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.5817545019999999, 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033152545999999998, 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5755 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895470+0000", 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7839999999999999e-06 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898117+0000", 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026470959999999998 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898163+0000", 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6406000000000003e-05 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898183+0000", 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0092999999999999e-05 2024-04-25T15:54:19.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898295+0000", 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011160800000000001 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898422+0000", 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012762099999999999 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898601+0000", 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017858500000000001 2024-04-25T15:54:19.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914502+0000", 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015900724000000001 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928454+0000", 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013952585999999999 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928552+0000", 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7299000000000005e-05 2024-04-25T15:54:19.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928619+0000", 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7743999999999994e-05 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5966 3.18 3:1b2b62aa:::100000001f8.0000094a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.3771111090000003, 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043485786999999998, 2024-04-25T15:54:19.586 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5966 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100114+0000", 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.315e-06 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.101977+0000", 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018630109999999999 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102022+0000", 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5636000000000001e-05 2024-04-25T15:54:19.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102040+0000", 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8e-05 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102140+0000", 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9720000000000001e-05 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102270+0000", 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130409 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102415+0000", 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144492 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.118110+0000", 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015694725999999999 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143459+0000", 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025349846999999998 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143534+0000", 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4839999999999995e-05 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143596+0000", 2024-04-25T15:54:19.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1791000000000005e-05 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:6009 3.18 3:1b7f6e54:::100000001f8.00000975:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 4.1114412680000001, 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030441923999999999, 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:19.592 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 6009 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:19.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365782+0000", 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3889999999999999e-06 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367595+0000", 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001812976 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367633+0000", 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7185999999999999e-05 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367647+0000", 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367735+0000", 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7474999999999999e-05 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367814+0000", 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9326000000000006e-05 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367936+0000", 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122077 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.375787+0000", 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078512010000000004 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396095+0000", 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02030732 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396164+0000", 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9549999999999996e-05 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396222+0000", 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7741999999999997e-05 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.597 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.598 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.598 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.598 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:19.598 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:19.598 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:54:19.598 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:54:19.598 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-25T15:54:19.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-25T15:54:19.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:54:19.600 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:19.597+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:19.700 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:19.697+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:19.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde675c5640 1 Processor -- start 2024-04-25T15:54:19.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde675c5640 1 -- start start 2024-04-25T15:54:19.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde675c5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde6006c640 0x7fde60154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:19.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde675c5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fde6015e8b0 0x7fde60164f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:19.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde675c5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde600749d0 con 0x7fde6015e4a0 2024-04-25T15:54:19.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde675c5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fde6006e510 con 0x7fde6006c640 2024-04-25T15:54:19.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde675c5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fde6015dca0 con 0x7fde6015e8b0 2024-04-25T15:54:19.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65361640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde6006c640 0x7fde60154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:19.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65361640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde6006c640 0x7fde60154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44044/0 (socket says 172.21.15.67:44044) 2024-04-25T15:54:19.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65361640 1 -- 172.21.15.67:0/3062350206 learned_addr learned my addr 172.21.15.67:0/3062350206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:19.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65b62640 1 --2- 172.21.15.67:0/3062350206 >> v2:172.21.15.176:3300/0 conn(0x7fde6015e8b0 0x7fde60164f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:19.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65361640 1 -- 172.21.15.67:0/3062350206 >> v2:172.21.15.176:3300/0 conn(0x7fde6015e8b0 msgr2=0x7fde60164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:19.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65361640 1 --2- 172.21.15.67:0/3062350206 >> v2:172.21.15.176:3300/0 conn(0x7fde6015e8b0 0x7fde60164f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:19.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65361640 1 -- 172.21.15.67:0/3062350206 >> v1:172.21.15.176:6789/0 conn(0x7fde6015e4a0 legacy=0x7fde600c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:19.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.797+0000 7fde65361640 1 -- 172.21.15.67:0/3062350206 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6015df80 con 0x7fde6006c640 2024-04-25T15:54:19.802 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:19.801+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:19.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde65361640 1 --2- 172.21.15.67:0/3062350206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde60154a60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fde600cf6c0 tx=0x7fde5001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af6db3e9d8cfa86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:19.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde57fff640 1 -- 172.21.15.67:0/3062350206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500b1070 con 0x7fde6006c640 2024-04-25T15:54:19.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde57fff640 1 -- 172.21.15.67:0/3062350206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde500ac600 con 0x7fde6006c640 2024-04-25T15:54:19.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde57fff640 1 -- 172.21.15.67:0/3062350206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500ac9d0 con 0x7fde6006c640 2024-04-25T15:54:19.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 -- 172.21.15.67:0/3062350206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 msgr2=0x7fde60154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 --2- 172.21.15.67:0/3062350206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde60154a60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fde600cf6c0 tx=0x7fde5001c450 comp rx=0 tx=0).stop 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 -- 172.21.15.67:0/3062350206 shutdown_connections 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 --2- 172.21.15.67:0/3062350206 >> v2:172.21.15.176:3300/0 conn(0x7fde6015e8b0 0x7fde60164f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 --2- 172.21.15.67:0/3062350206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde60154a60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 -- 172.21.15.67:0/3062350206 >> 172.21.15.67:0/3062350206 conn(0x7fde60070db0 msgr2=0x7fde600ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 -- 172.21.15.67:0/3062350206 shutdown_connections 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.801+0000 7fde675c5640 1 -- 172.21.15.67:0/3062350206 wait complete. 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:19.804 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:19.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 Processor -- start 2024-04-25T15:54:19.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 -- start start 2024-04-25T15:54:19.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde600c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:19.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde6015e4a0 0x7fde6014d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:19.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde6015e8b0 0x7fde6014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:19.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fde600cad60 con 0x7fde6006c640 2024-04-25T15:54:19.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fde600c86d0 con 0x7fde6015e8b0 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde60155490 con 0x7fde6015e4a0 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde600c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde600c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44052/0 (socket says 172.21.15.67:44052) 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 -- 172.21.15.67:0/2220098231 learned_addr learned my addr 172.21.15.67:0/2220098231 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde64b60640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde6015e4a0 0x7fde6014d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65b62640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde6015e8b0 0x7fde6014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 -- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde6015e4a0 msgr2=0x7fde6014d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde6015e4a0 0x7fde6014d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 -- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde6015e8b0 msgr2=0x7fde6014d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde6015e8b0 0x7fde6014d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 -- 172.21.15.67:0/2220098231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde60154fc0 con 0x7fde6006c640 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65b62640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde6015e8b0 0x7fde6014d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde64b60640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde6015e4a0 0x7fde6014d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:19.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde65361640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde600c5c20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fde5001ccf0 tx=0x7fde5001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:19.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde55ffb640 1 -- 172.21.15.67:0/2220098231 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500b1040 con 0x7fde6006c640 2024-04-25T15:54:19.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde55ffb640 1 -- 172.21.15.67:0/2220098231 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde500bc070 con 0x7fde6006c640 2024-04-25T15:54:19.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde60156c90 con 0x7fde6006c640 2024-04-25T15:54:19.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde55ffb640 1 -- 172.21.15.67:0/2220098231 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde500b78b0 con 0x7fde6006c640 2024-04-25T15:54:19.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.805+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde600c7300 con 0x7fde6006c640 2024-04-25T15:54:19.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.809+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6006a480 con 0x7fde6006c640 2024-04-25T15:54:19.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.809+0000 7fde55ffb640 1 -- 172.21.15.67:0/2220098231 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde500ac480 con 0x7fde6006c640 2024-04-25T15:54:19.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.809+0000 7fde55ffb640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde4406f400 0x7fde44071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:19.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.809+0000 7fde55ffb640 1 -- 172.21.15.67:0/2220098231 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6002+0+0 (secure 0 0 0) 0x7fde50129610 con 0x7fde6006c640 2024-04-25T15:54:19.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.809+0000 7fde64b60640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde4406f400 0x7fde44071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:19.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.809+0000 7fde64b60640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde4406f400 0x7fde44071910 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fde601f3440 tx=0x7fde5c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:19.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.813+0000 7fde55ffb640 1 -- 172.21.15.67:0/2220098231 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fde500c6020 con 0x7fde6006c640 2024-04-25T15:54:19.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:54:19.913 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:19.913+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:19.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.981+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fde600c9860 con 0x7fde6006c640 2024-04-25T15:54:19.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.981+0000 7fde55ffb640 1 -- 172.21.15.67:0/2220098231 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v102) ==== 81+0+35850 (secure 0 0 0) 0x7fde500160d0 con 0x7fde6006c640 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "epoch": 102, 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "fd5b4894-828e-4cb5-af55-b54a791f04c8", 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "created": "2024-04-25T15:50:05.181255+0000", 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "modified": "2024-04-25T15:54:17.973793+0000", 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "last_up_change": "2024-04-25T15:50:17.054376+0000", 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "last_in_change": "2024-04-25T15:53:14.990625+0000", 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T15:54:19.983 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_num": 5801984, 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_set": [ 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "noscrub", 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "pglog_hardlimit", 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "purged_snapdirs", 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_deletes", 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "sortbitwise" 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_version": 4, 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "full_ratio": 0.94999998807907104, 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_snapshot": "", 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_max": 3, 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "max_osd": 8, 2024-04-25T15:54:19.984 INFO:teuthology.orchestra.run.smithi067.stdout: "require_min_compat_client": "luminous", 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "min_compat_client": "luminous", 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "require_osd_release": "squid", 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "allow_crimson": false, 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "pools": [ 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 1, 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": ".mgr", 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:17.846623+0000", 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:54:19.985 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 1, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 1, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 1, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 1, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 1, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:54:19.986 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "16", 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:54:19.987 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:54:19.988 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:54:19.989 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_max": 32, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 1 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": {} 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 6.9000000953674316, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 6.9000000953674316, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 0.28999999165534973, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:54:19.990 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 2, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_metadata", 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:28.408547+0000", 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:54:19.991 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 32, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 32, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 32, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 32, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 32, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:54:19.992 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "19", 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:54:19.993 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:54:19.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_bias": 4, 2024-04-25T15:54:19.995 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 16, 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_priority": 5 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": "cephfs" 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.75, 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.75, 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 2, 2024-04-25T15:54:19.996 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 2, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "pool": 3, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_name": "cephfs_data", 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "create_time": "2024-04-25T15:50:29.569527+0000", 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flags": 1, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flags_names": "hashpspool", 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type": 1, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 2, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "min_size": 1, 2024-04-25T15:54:19.997 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_rule": 0, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_count": 0, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_target": 0, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "object_hash": 2, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_autoscale_mode": "off", 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num": 64, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num": 64, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_placement_num_target": 64, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_target": 64, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_pending": 64, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "last_pg_merge_meta": { 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "source_pgid": "0.0", 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "ready_epoch": 0, 2024-04-25T15:54:19.998 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_started": 0, 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "last_epoch_clean": 0, 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "source_version": "0'0", 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "target_version": "0'0" 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "last_change": "20", 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend": "0", 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "auid": 0, 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_mode": "selfmanaged", 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_seq": 0, 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "snap_epoch": 0, 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "pool_snaps": [], 2024-04-25T15:54:19.999 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps": "[]", 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_bytes": 0, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "quota_max_objects": 0, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "tiers": [], 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "tier_of": -1, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "read_tier": -1, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "write_tier": -1, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_mode": "none", 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_bytes": 0, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_objects": 0, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_flush_age": 0, 2024-04-25T15:54:20.000 INFO:teuthology.orchestra.run.smithi067.stdout: "cache_min_evict_age": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profile": "", 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_params": { 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "none" 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_period": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_count": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "use_gmt_hitset": true, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "min_read_recency_for_promote": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "min_write_recency_for_promote": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "hit_set_search_last_n": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "grade_table": [], 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "stripe_width": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "expected_num_objects": 0, 2024-04-25T15:54:20.001 INFO:teuthology.orchestra.run.smithi067.stdout: "fast_read": false, 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "options": { 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_num_min": 64, 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "target_size_ratio": 0.89999997615814209 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "application_metadata": { 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs": { 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "data": "cephfs" 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "read_balance": { 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "score_type": "Fair distribution", 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "score_acting": 1.4299999475479126, 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "score_stable": 1.4299999475479126, 2024-04-25T15:54:20.002 INFO:teuthology.orchestra.run.smithi067.stdout: "optimal_score": 1.1399999856948853, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_acting": 1.6299999952316284, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "raw_score_stable": 1.6299999952316284, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity_weighted": 0.875, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity": 1, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "2ae04396-30a5-4c1d-86e5-63bc4f26f4c0", 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.003 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 0, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 35, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6800", 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6801", 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6802", 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6803", 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.006 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6806", 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6807", 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.007 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6804", 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6805", 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 851806635 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6801/851806635", 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6803/851806635", 2024-04-25T15:54:20.008 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6807/851806635", 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6805/851806635", 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "efacae28-481c-42e9-9d26-5a019cb078b1", 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 0.2953948974609375, 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.009 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6809", 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.010 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6812", 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6814", 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6817", 2024-04-25T15:54:20.011 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6825", 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6828", 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6820", 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6822", 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2360705508 2024-04-25T15:54:20.013 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6812/2360705508", 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6817/2360705508", 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6828/2360705508", 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6822/2360705508", 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:54:20.014 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "60e87127-223e-4fd7-aefb-f00326f9deca", 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 73, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.015 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6810", 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6813", 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6816", 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6819", 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.017 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6830", 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6831", 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6823", 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6827", 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2903206440 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6813/2903206440", 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6819/2903206440", 2024-04-25T15:54:20.019 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6831/2903206440", 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6827/2903206440", 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "85ccf60b-0609-40a0-aae5-1e6a37eebdf2", 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.020 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6808", 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6811", 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.021 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6815", 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6818", 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.022 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6826", 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6829", 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.023 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6821", 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:6824", 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 1688455876 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.024 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.67:6811/1688455876", 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.67:6818/1688455876", 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.67:6829/1688455876", 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.67:6824/1688455876", 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "ce4919c0-d2fb-4210-b281-e843e84c94c1", 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 92, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6808", 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6810", 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6812", 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6814", 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6820", 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.028 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6822", 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6816", 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.029 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6818", 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 696101260 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6810/696101260", 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6814/696101260", 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6822/696101260", 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6818/696101260", 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.030 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "47dcf3c9-5c23-4584-922c-b774345f2175", 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.031 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6824", 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6825", 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6826", 2024-04-25T15:54:20.032 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6827", 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6830", 2024-04-25T15:54:20.033 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6831", 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6828", 2024-04-25T15:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6829", 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2964881609 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6825/2964881609", 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6827/2964881609", 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6831/2964881609", 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6829/2964881609", 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "312d2166-f5ad-4cf3-8339-16a49bf48b19", 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 76, 2024-04-25T15:54:20.036 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6809", 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6811", 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.037 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6813", 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6815", 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.038 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6821", 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6823", 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.039 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6817", 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6819", 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 211167612 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6811/211167612", 2024-04-25T15:54:20.040 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6815/211167612", 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6823/211167612", 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6819/211167612", 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "uuid": "922d3949-fd7b-4d1e-8b59-7311998bbc0b", 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "up": 1, 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "in": 1, 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "weight": 1, 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_affinity": 1, 2024-04-25T15:54:20.041 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_begin": 0, 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "last_clean_end": 0, 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "up_from": 13, 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "up_thru": 85, 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "down_at": 0, 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "lost_at": 0, 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrs": { 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6800", 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.042 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6801", 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addrs": { 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6802", 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6803", 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.043 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addrs": { 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6806", 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6807", 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addrs": { 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "addrvec": [ 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v2", 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6804", 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.176:6805", 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 604707640 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.045 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "172.21.15.176:6801/604707640", 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "172.21.15.176:6803/604707640", 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_back_addr": "172.21.15.176:6807/604707640", 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_front_addr": "172.21.15.176:6805/604707640", 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "state": [ 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "exists", 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "up" 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_xinfo": [ 2024-04-25T15:54:20.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 0, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.418696+0000", 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 1, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.047 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.395641+0000", 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 2, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.361155+0000", 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.048 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 3, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.573487+0000", 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 4, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.449326+0000", 2024-04-25T15:54:20.049 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 5, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.423627+0000", 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 6, 2024-04-25T15:54:20.050 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.434102+0000", 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": 7, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "down_stamp": "0.000000", 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_probability": 0, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "laggy_interval": 0, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "features": 4540701547738038271, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "old_weight": 0, 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "last_purged_snaps_scrub": "2024-04-25T15:50:13.384476+0000", 2024-04-25T15:54:20.051 INFO:teuthology.orchestra.run.smithi067.stdout: "dead_epoch": 0 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap": [ 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.18", 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: 1, 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: 4 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.27", 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: "osds": [ 2024-04-25T15:54:20.052 INFO:teuthology.orchestra.run.smithi067.stdout: 7, 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: 6 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_items": [ 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.3", 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 0, 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.b", 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 3, 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.d", 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:54:20.054 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 5 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.22", 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 1 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.055 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.2a", 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 5, 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.37", 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 7, 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.056 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: "pgid": "3.3e", 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: "mappings": [ 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: "from": 0, 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: "to": 2 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_upmap_primaries": [], 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: "pg_temp": [], 2024-04-25T15:54:20.057 INFO:teuthology.orchestra.run.smithi067.stdout: "primary_temp": [], 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklist": {}, 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "range_blocklist": {}, 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_profiles": { 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "default": { 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "crush-failure-domain": "osd", 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "k": "2", 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "m": "1", 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin": "jerasure", 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "technique": "reed_sol_van" 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "removed_snaps_queue": [], 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "new_removed_snaps": [], 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "new_purged_snaps": [], 2024-04-25T15:54:20.058 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_node_flags": {}, 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: "device_class_flags": {}, 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode": { 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_enabled": false, 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_bucket_count": 0, 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: "degraded_stretch_mode": 0, 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: "recovering_stretch_mode": 0, 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: "stretch_mode_bucket": 0 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.059 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:20.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:54:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde4406f400 msgr2=0x7fde44071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde4406f400 0x7fde44071910 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fde601f3440 tx=0x7fde5c008040 comp rx=0 tx=0).stop 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 msgr2=0x7fde600c5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde600c5c20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fde5001ccf0 tx=0x7fde5001ac90 comp rx=0 tx=0).stop 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 shutdown_connections 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fde4406f400 0x7fde44071910 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde6015e8b0 0x7fde6014d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde6015e4a0 0x7fde6014d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 --2- 172.21.15.67:0/2220098231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde6006c640 0x7fde600c5c20 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 >> 172.21.15.67:0/2220098231 conn(0x7fde60070db0 msgr2=0x7fde6006d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 shutdown_connections 2024-04-25T15:54:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:19.985+0000 7fde675c5640 1 -- 172.21.15.67:0/2220098231 wait complete. 2024-04-25T15:54:20.063 INFO:tasks.thrashosds.thrasher:j is {'epoch': 102, 'fsid': 'fd5b4894-828e-4cb5-af55-b54a791f04c8', 'created': '2024-04-25T15:50:05.181255+0000', 'modified': '2024-04-25T15:54:17.973793+0000', 'last_up_change': '2024-04-25T15:50:17.054376+0000', 'last_in_change': '2024-04-25T15:53:14.990625+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4299999475479126, 'score_stable': 1.4299999475479126, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '2ae04396-30a5-4c1d-86e5-63bc4f26f4c0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6800', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6801', 'nonce': 851806635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6802', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6803', 'nonce': 851806635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6806', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6807', 'nonce': 851806635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6804', 'nonce': 851806635}, {'type': 'v1', 'addr': '172.21.15.67:6805', 'nonce': 851806635}]}, 'public_addr': '172.21.15.67:6801/851806635', 'cluster_addr': '172.21.15.67:6803/851806635', 'heartbeat_back_addr': '172.21.15.67:6807/851806635', 'heartbeat_front_addr': '172.21.15.67:6805/851806635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'efacae28-481c-42e9-9d26-5a019cb078b1', 'up': 1, 'in': 1, 'weight': 0.2953948974609375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6809', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6812', 'nonce': 2360705508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6814', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6817', 'nonce': 2360705508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6825', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6828', 'nonce': 2360705508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6820', 'nonce': 2360705508}, {'type': 'v1', 'addr': '172.21.15.67:6822', 'nonce': 2360705508}]}, 'public_addr': '172.21.15.67:6812/2360705508', 'cluster_addr': '172.21.15.67:6817/2360705508', 'heartbeat_back_addr': '172.21.15.67:6828/2360705508', 'heartbeat_front_addr': '172.21.15.67:6822/2360705508', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '60e87127-223e-4fd7-aefb-f00326f9deca', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6810', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6813', 'nonce': 2903206440}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6816', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6819', 'nonce': 2903206440}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6830', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6831', 'nonce': 2903206440}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6823', 'nonce': 2903206440}, {'type': 'v1', 'addr': '172.21.15.67:6827', 'nonce': 2903206440}]}, 'public_addr': '172.21.15.67:6813/2903206440', 'cluster_addr': '172.21.15.67:6819/2903206440', 'heartbeat_back_addr': '172.21.15.67:6831/2903206440', 'heartbeat_front_addr': '172.21.15.67:6827/2903206440', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '85ccf60b-0609-40a0-aae5-1e6a37eebdf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6808', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6811', 'nonce': 1688455876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6815', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6818', 'nonce': 1688455876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6826', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6829', 'nonce': 1688455876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6821', 'nonce': 1688455876}, {'type': 'v1', 'addr': '172.21.15.67:6824', 'nonce': 1688455876}]}, 'public_addr': '172.21.15.67:6811/1688455876', 'cluster_addr': '172.21.15.67:6818/1688455876', 'heartbeat_back_addr': '172.21.15.67:6829/1688455876', 'heartbeat_front_addr': '172.21.15.67:6824/1688455876', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'ce4919c0-d2fb-4210-b281-e843e84c94c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6808', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 696101260}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6812', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 696101260}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6820', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6822', 'nonce': 696101260}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6816', 'nonce': 696101260}, {'type': 'v1', 'addr': '172.21.15.176:6818', 'nonce': 696101260}]}, 'public_addr': '172.21.15.176:6810/696101260', 'cluster_addr': '172.21.15.176:6814/696101260', 'heartbeat_back_addr': '172.21.15.176:6822/696101260', 'heartbeat_front_addr': '172.21.15.176:6818/696101260', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '47dcf3c9-5c23-4584-922c-b774345f2175', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6824', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6825', 'nonce': 2964881609}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6826', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6827', 'nonce': 2964881609}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6830', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6831', 'nonce': 2964881609}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6828', 'nonce': 2964881609}, {'type': 'v1', 'addr': '172.21.15.176:6829', 'nonce': 2964881609}]}, 'public_addr': '172.21.15.176:6825/2964881609', 'cluster_addr': '172.21.15.176:6827/2964881609', 'heartbeat_back_addr': '172.21.15.176:6831/2964881609', 'heartbeat_front_addr': '172.21.15.176:6829/2964881609', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '312d2166-f5ad-4cf3-8339-16a49bf48b19', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6809', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 211167612}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6813', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 211167612}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6821', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6823', 'nonce': 211167612}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6817', 'nonce': 211167612}, {'type': 'v1', 'addr': '172.21.15.176:6819', 'nonce': 211167612}]}, 'public_addr': '172.21.15.176:6811/211167612', 'cluster_addr': '172.21.15.176:6815/211167612', 'heartbeat_back_addr': '172.21.15.176:6823/211167612', 'heartbeat_front_addr': '172.21.15.176:6819/211167612', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '922d3949-fd7b-4d1e-8b59-7311998bbc0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 85, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6800', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 604707640}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6802', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 604707640}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6806', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 604707640}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.176:6804', 'nonce': 604707640}, {'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 604707640}]}, 'public_addr': '172.21.15.176:6801/604707640', 'cluster_addr': '172.21.15.176:6803/604707640', 'heartbeat_back_addr': '172.21.15.176:6807/604707640', 'heartbeat_front_addr': '172.21.15.176:6805/604707640', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.418696+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.395641+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.361155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.573487+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.449326+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.423627+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.434102+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-25T15:50:13.384476+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [1, 4]}, {'pgid': '3.27', 'osds': [7, 6]}], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '3.b', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '3.d', 'mappings': [{'from': 7, 'to': 5}]}, {'pgid': '3.22', 'mappings': [{'from': 5, 'to': 1}]}, {'pgid': '3.2a', 'mappings': [{'from': 5, 'to': 2}]}, {'pgid': '3.37', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '3.3e', 'mappings': [{'from': 0, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-25T15:54:20.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:54:20.064 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:20.064 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:20.064 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:20.065 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:20.065 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:20.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:54:20.065 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:20.061+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:20.161 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:20.158+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:20.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:54:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58b5962640 1 Processor -- start 2024-04-25T15:54:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58b5962640 1 -- start start 2024-04-25T15:54:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58b5962640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b0068830 0x7f58b006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58b5962640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58b0068c40 con 0x7f58b015dd00 2024-04-25T15:54:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58b5962640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58b0160990 con 0x7f58b006cb80 2024-04-25T15:54:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58b5962640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58b015e110 con 0x7f58b0068830 2024-04-25T15:54:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58aeffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f58b015dd00 0x7f58b01569a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50374/0 (socket says 172.21.15.67:50374) 2024-04-25T15:54:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58aeffd640 1 -- 172.21.15.67:0/3970861578 learned_addr learned my addr 172.21.15.67:0/3970861578 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58ae7fc640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b0068830 0x7f58b006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696267703 0 0) 0x7f58a0001120 con 0x7f58b015dd00 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.249+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5898003530 con 0x7f58b015dd00 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3665920422 0 0) 0x7f5898003530 con 0x7f58b015dd00 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b0068830 msgr2=0x7f58b006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b0068830 0x7f58b006c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 >> v1:172.21.15.176:6789/0 conn(0x7f58b006cb80 legacy=0x7f58b0167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b0160c70 con 0x7f58b015dd00 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f58a0001850 con 0x7f58b015dd00 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 >> v1:172.21.15.67:6789/0 conn(0x7f58b015dd00 legacy=0x7f58b01569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58980039f0 0x7f5898003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5898004810 0x7f5898004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5898005160 0x7f589800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58b015e110 con 0x7f58980039f0 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f589800c040 con 0x7f5898004810 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f589800c240 con 0x7f5898005160 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58aeffd640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58980039f0 0x7f5898003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58ae7fc640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5898004810 0x7f5898004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58af7fe640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5898005160 0x7f589800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58aeffd640 1 -- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5898005160 msgr2=0x7f589800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58aeffd640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5898005160 0x7f589800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58aeffd640 1 -- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5898004810 msgr2=0x7f5898004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58aeffd640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5898004810 0x7f5898004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58aeffd640 1 -- 172.21.15.67:0/3970861578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5898003530 con 0x7f58980039f0 2024-04-25T15:54:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58ae7fc640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5898004810 0x7f5898004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58aeffd640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58980039f0 0x7f5898003e20 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f58a000c4c0 tx=0x7f58a001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd7eb0929b14277 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a000c0e0 con 0x7f58980039f0 2024-04-25T15:54:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a00a9020 con 0x7f58980039f0 2024-04-25T15:54:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.253+0000 7f58adffb640 1 -- 172.21.15.67:0/3970861578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0005ae0 con 0x7f58980039f0 2024-04-25T15:54:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 -- 172.21.15.67:0/3970861578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58980039f0 msgr2=0x7f5898003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 --2- 172.21.15.67:0/3970861578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58980039f0 0x7f5898003e20 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f58a000c4c0 tx=0x7f58a001cc80 comp rx=0 tx=0).stop 2024-04-25T15:54:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58af7fe640 1 -- 172.21.15.67:0/3970861578 reap_dead start 2024-04-25T15:54:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 -- 172.21.15.67:0/3970861578 shutdown_connections 2024-04-25T15:54:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 -- 172.21.15.67:0/3970861578 >> 172.21.15.67:0/3970861578 conn(0x7f58b00ce1c0 msgr2=0x7f58b00ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 -- 172.21.15.67:0/3970861578 shutdown_connections 2024-04-25T15:54:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 -- 172.21.15.67:0/3970861578 wait complete. 2024-04-25T15:54:20.258 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:20.258 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:20.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:20.258 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:148 3.9 3:91a8729a:::10000000002.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 91.698672360000003, 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.432712019, 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 148 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558221+0000", 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3929999999999998e-06 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559704+0000", 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001483807 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975277+0000", 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.41557216400000002 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975295+0000", 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8408999999999999e-05 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975372+0000", 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7235000000000002e-05 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975484+0000", 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111499 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975582+0000", 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8589000000000001e-05 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.980957+0000", 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053747600000000001 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990855+0000", 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098981350000000006 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990894+0000", 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8436000000000003e-05 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990930+0000", 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6591999999999998e-05 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1494 3.37 3:efe547ea:::100000001f8.000001d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.612487811999998, 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089208711999999996, 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.264 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1494 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644406+0000", 2024-04-25T15:54:20.265 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2229999999999999e-06 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.646356+0000", 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019497990000000001 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718358+0000", 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072002624000000001 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718376+0000", 2024-04-25T15:54:20.266 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7649999999999999e-05 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718458+0000", 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1984999999999995e-05 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718600+0000", 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000141922 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718731+0000", 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131488 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.267 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.724805+0000", 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006074032 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733498+0000", 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086928439999999999 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733549+0000", 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1137000000000002e-05 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.268 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733611+0000", 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2008000000000007e-05 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1504 3.2a 3:54fb97d1:::100000001f8.000001dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.540194376000002, 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121933077, 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.269 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1504 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.270 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716699+0000", 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4250000000000001e-06 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718317+0000", 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016187440000000001 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.815989+0000", 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.097671735999999995 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.271 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816008+0000", 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9131e-05 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816090+0000", 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1717000000000004e-05 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816244+0000", 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015372199999999999 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.272 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816348+0000", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010470499999999999 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.826448+0000", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010099281999999999 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838528+0000", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012080206 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838583+0000", 2024-04-25T15:54:20.273 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5105000000000002e-05 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838629+0000", 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6304000000000003e-05 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1528 3.3e 3:7f4f4afd:::100000001f8.000001f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.392720529999998, 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.133348521, 2024-04-25T15:54:20.274 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1528 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.275 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864172+0000", 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.204e-06 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865554+0000", 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013821899999999999 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986006+0000", 2024-04-25T15:54:20.276 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12045179 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986022+0000", 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999e-05 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986101+0000", 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9251000000000004e-05 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986224+0000", 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012263899999999999 2024-04-25T15:54:20.277 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986330+0000", 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106164 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.996058+0000", 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097274319999999994 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997353+0000", 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001294882 2024-04-25T15:54:20.278 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997450+0000", 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7446000000000002e-05 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997519+0000", 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8523999999999996e-05 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1548 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:20.279 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.273254823000002, 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096155170999999998, 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1548 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.280 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983639+0000", 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1750000000000001e-06 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.985960+0000", 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023213790000000002 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.281 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058434+0000", 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072473913000000001 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058454+0000", 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9329000000000001e-05 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058541+0000", 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7702999999999997e-05 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.282 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058708+0000", 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016712300000000001 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058864+0000", 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015600100000000001 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.065501+0000", 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066362299999999999 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.283 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079696+0000", 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014195228000000001 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079745+0000", 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8980000000000002e-05 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079791+0000", 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6109999999999997e-05 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.284 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1561 3.2e 3:75253c31:::100000001f8.00000215:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 36.188860292999998, 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099620287000000002, 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1561 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.285 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068033+0000", 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1240000000000001e-06 2024-04-25T15:54:20.286 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.070277+0000", 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002243676 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143785+0000", 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073508283999999993 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.287 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143804+0000", 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8808e-05 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143881+0000", 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6379999999999997e-05 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144063+0000", 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018198500000000001 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.288 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144168+0000", 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105253 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.154374+0000", 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010206374000000001 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167481+0000", 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013106405999999999 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.289 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167580+0000", 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9329999999999999e-05 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167651+0000", 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0667000000000006e-05 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.290 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2308 3.4 3:2343a40c:::100000001f8.00000500:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.435805981000001, 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.129707092, 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2308 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:20.291 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821088+0000", 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1599999999999998e-06 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823092+0000", 2024-04-25T15:54:20.292 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020043909999999999 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823143+0000", 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0791999999999998e-05 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823162+0000", 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9332999999999999e-05 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823254+0000", 2024-04-25T15:54:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2169000000000005e-05 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823637+0000", 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038265100000000002 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823849+0000", 2024-04-25T15:54:20.294 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211879 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.834007+0000", 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010158377 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950654+0000", 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.116646229 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950721+0000", 2024-04-25T15:54:20.295 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7020000000000005e-05 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950792+0000", 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1091000000000001e-05 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4976 3.2a 3:57c1fe5a:::100000001f8.0000056c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.665769693, 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15973960500000001, 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.296 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4976 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.297 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591123+0000", 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.306e-06 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.592621+0000", 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014980250000000001 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.298 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703776+0000", 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.111154948 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703790+0000", 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3915e-05 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703878+0000", 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7527000000000004e-05 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.299 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703996+0000", 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118602 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.704166+0000", 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017002799999999999 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.710386+0000", 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062202239999999999 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750704+0000", 2024-04-25T15:54:20.300 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040317106999999998 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750795+0000", 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1880999999999997e-05 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750860+0000", 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5042000000000005e-05 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.301 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5006 3.2a 3:566ba4fe:::100000001f8.0000058a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.517259633, 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094006696000000001, 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5006 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.302 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739634+0000", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999998e-06 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.740943+0000", 2024-04-25T15:54:20.303 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013092780000000001 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791183+0000", 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050239791999999998 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791197+0000", 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4250000000000001e-05 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791287+0000", 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9627000000000001e-05 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791404+0000", 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117663 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791560+0000", 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155997 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.801092+0000", 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095317030000000007 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.305 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833424+0000", 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032332514999999999 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833527+0000", 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102997 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833638+0000", 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110142 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.306 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5024 3.2a 3:55e8c2c8:::100000001f8.0000059c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.427394744000001, 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14725615, 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5024 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.307 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829499+0000", 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0740000000000001e-06 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.308 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.831085+0000", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001586241 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932577+0000", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.10149145399999999 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932592+0000", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4934e-05 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932681+0000", 2024-04-25T15:54:20.309 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9140000000000004e-05 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932803+0000", 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012215800000000001 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932954+0000", 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151328 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.943046+0000", 2024-04-25T15:54:20.310 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010092261999999999 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976650+0000", 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033603840000000003 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976710+0000", 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9348999999999997e-05 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976752+0000", 2024-04-25T15:54:20.311 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2370000000000003e-05 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5049 3.2e 3:740912a7:::100000001f8.000005b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.293486775, 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.162097727, 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5049 2024-04-25T15:54:20.312 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963406+0000", 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4279999999999999e-06 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.313 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.965277+0000", 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018703210000000001 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021994+0000", 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056717005000000001 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022018+0000", 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4043e-05 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.314 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022095+0000", 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6959999999999995e-05 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022198+0000", 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103167 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022356+0000", 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015838899999999999 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.315 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.052459+0000", 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030102694999999999 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125349+0000", 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072890606999999996 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125439+0000", 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9587999999999997e-05 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125502+0000", 2024-04-25T15:54:20.316 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2524e-05 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5064 3.21 3:845d14c1:::100000001f8.000005c4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.222493507999999, 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14591528400000001, 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.317 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5064 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.318 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034400+0000", 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3079999999999999e-06 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.036710+0000", 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023096000000000002 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075414+0000", 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038703923000000001 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.319 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075434+0000", 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9712e-05 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075522+0000", 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8472000000000003e-05 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075609+0000", 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7200000000000005e-05 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:20.320 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075766+0000", 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015715400000000001 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.089149+0000", 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013382338000000001 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180138+0000", 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.090989451999999998 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.321 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180200+0000", 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2098000000000001e-05 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180312+0000", 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112027 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5072 3.2e 3:75b854b3:::100000001f8.000005cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.183294172, 2024-04-25T15:54:20.322 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.106817279, 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5072 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.323 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073599+0000", 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2759999999999999e-06 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075372+0000", 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001773574 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.324 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125855+0000", 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050482955000000003 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125865+0000", 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0029e-05 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125933+0000", 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8100000000000002e-05 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.126051+0000", 2024-04-25T15:54:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117333 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.126222+0000", 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000171609 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.133559+0000", 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0073365649999999998 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.326 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180298+0000", 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046739050999999997 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180358+0000", 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0479000000000002e-05 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180414+0000", 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5308e-05 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.327 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5601 3.2e 3:75930891:::100000001f8.000007dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 7.2428239129999996, 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088347508000000005, 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5601 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.328 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014070+0000", 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0869999999999998e-06 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.329 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.015590+0000", 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015200649999999999 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086402+0000", 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070812293999999998 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086419+0000", 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7088999999999999e-05 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.330 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086530+0000", 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011049599999999999 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086686+0000", 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015618899999999999 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086917+0000", 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000231198 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.095623+0000", 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087063049999999993 2024-04-25T15:54:20.331 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102292+0000", 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066687609999999996 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102356+0000", 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3958e-05 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102414+0000", 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8066000000000003e-05 2024-04-25T15:54:20.332 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5729 3.2e 3:77f75ed6:::100000001f8.0000085d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.5005387079999997, 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.114341208, 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5729 2024-04-25T15:54:20.333 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756354+0000", 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.576e-06 2024-04-25T15:54:20.334 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757773+0000", 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014189560000000001 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757806+0000", 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2428000000000002e-05 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757820+0000", 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4279e-05 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.335 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757910+0000", 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0138999999999999e-05 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.758024+0000", 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114158 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.758197+0000", 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172931 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.769252+0000", 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011054342 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870574+0000", 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.10132216099999999 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870628+0000", 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3996999999999998e-05 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.337 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870693+0000", 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5240999999999994e-05 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5739 3.2e 3:757c9716:::100000001f8.00000867:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 6.4514118089999997, 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099555849000000002, 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.338 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5739 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.339 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805482+0000", 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9670000000000001e-06 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807186+0000", 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017046520000000001 2024-04-25T15:54:20.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807217+0000", 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0892999999999997e-05 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807230+0000", 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2235999999999999e-05 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807314+0000", 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4481999999999995e-05 2024-04-25T15:54:20.341 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807445+0000", 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130977 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807554+0000", 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108805 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.824323+0000", 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016769217999999999 2024-04-25T15:54:20.342 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904744+0000", 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080421406000000001 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904929+0000", 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018458600000000001 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.905035+0000", 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105627 2024-04-25T15:54:20.343 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5870 3.4 3:2197a314:::100000001f8.000008ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.549685+0000", 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.7072060130000004, 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14750286300000001, 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.344 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5870 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.549685+0000", 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.549685+0000", 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.549687+0000", 2024-04-25T15:54:20.345 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1780000000000002e-06 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.551318+0000", 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001631179 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641743+0000", 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.090425249999999999 2024-04-25T15:54:20.346 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641758+0000", 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5299999999999999e-05 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641875+0000", 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116809 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641952+0000", 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7021000000000007e-05 2024-04-25T15:54:20.347 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.642075+0000", 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012264 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.647613+0000", 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005538472 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.697082+0000", 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049468828999999999 2024-04-25T15:54:20.348 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.697136+0000", 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3616999999999998e-05 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.697187+0000", 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1567999999999998e-05 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.349 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5915 3.2e 3:7446f81c:::100000001f8.00000917:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.822335+0000", 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.4345553710000001, 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.25176186900000003, 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.350 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5915 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.822335+0000", 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.822335+0000", 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.822338+0000", 2024-04-25T15:54:20.351 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8329999999999998e-06 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.823856+0000", 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001518092 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035519+0000", 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21166296100000001 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035533+0000", 2024-04-25T15:54:20.352 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.346e-05 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035609+0000", 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6087999999999994e-05 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035684+0000", 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5240999999999993e-05 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035788+0000", 2024-04-25T15:54:20.353 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010430899999999999 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.043303+0000", 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075152530000000004 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.073955+0000", 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030651226 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.074031+0000", 2024-04-25T15:54:20.354 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6631000000000005e-05 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.074097+0000", 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5775e-05 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5965 3.4 3:2356ea06:::100000001f8.00000949:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.092631+0000", 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.1642594910000001, 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096216884000000003, 2024-04-25T15:54:20.355 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5965 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.092631+0000", 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.356 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.092631+0000", 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.092634+0000", 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.915e-06 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.094975+0000", 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023405980000000002 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.357 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164137+0000", 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.069162491000000006 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164157+0000", 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9545e-05 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164250+0000", 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2872000000000001e-05 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.358 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164324+0000", 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4405000000000003e-05 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164469+0000", 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014499000000000001 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.174678+0000", 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010209197999999999 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.359 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.188695+0000", 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014016768000000001 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.188780+0000", 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5573000000000002e-05 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.188848+0000", 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7528999999999997e-05 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.360 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5978 3.b 3:d3280d2b:::100000001f8.00000956:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.167843+0000", 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 5.08904785, 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.11479445200000001, 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5978 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:20.361 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.167843+0000", 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.167843+0000", 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.167845+0000", 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6740000000000001e-06 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:20.362 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.169444+0000", 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015984149999999999 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230486+0000", 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.061041911999999997 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230500+0000", 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4758e-05 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230585+0000", 2024-04-25T15:54:20.363 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4509e-05 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230698+0000", 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011307999999999999 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230897+0000", 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019934900000000001 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.273195+0000", 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042297522999999997 2024-04-25T15:54:20.364 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282533+0000", 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093378950000000006 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282587+0000", 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282637+0000", 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9823000000000001e-05 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.365 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:20.366 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:20.366 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 Processor -- start 2024-04-25T15:54:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 -- start start 2024-04-25T15:54:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b015ad60 0x7f58b00cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.257+0000 7f58b5962640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b00cd3f0 0x7f58b015c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58b5962640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b015ca20 0x7f58b00733b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58b5962640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f58b0074a80 con 0x7f58b015ad60 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58b5962640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f58b0074900 con 0x7f58b015ca20 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58b5962640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58b006d660 con 0x7f58b00cd3f0 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b015ad60 0x7f58b00cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b015ad60 0x7f58b00cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44074/0 (socket says 172.21.15.67:44074) 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 -- 172.21.15.67:0/1508581998 learned_addr learned my addr 172.21.15.67:0/1508581998 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58af7fe640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b015ca20 0x7f58b00733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58ae7fc640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b00cd3f0 0x7f58b015c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 -- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b00cd3f0 msgr2=0x7f58b015c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b00cd3f0 0x7f58b015c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 -- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b015ca20 msgr2=0x7f58b00733b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b015ca20 0x7f58b00733b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 -- 172.21.15.67:0/1508581998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b015d710 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58ae7fc640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b00cd3f0 0x7f58b015c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58aeffd640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b015ad60 0x7f58b00cceb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f58a001c6f0 tx=0x7f5898005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f5897fff640 1 -- 172.21.15.67:0/1508581998 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a00048b0 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f5897fff640 1 -- 172.21.15.67:0/1508581998 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a00b2070 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f5897fff640 1 -- 172.21.15.67:0/1508581998 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0004c50 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b0072620 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b015d430 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.261+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b006a480 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7f5897fff640 1 -- 172.21.15.67:0/1508581998 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f58a00a5680 con 0x7f58b015ad60 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7f5897fff640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f589c06f430 0x7f589c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7f5897fff640 1 -- 172.21.15.67:0/1508581998 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6002+0+0 (secure 0 0 0) 0x7f58a0128420 con 0x7f58b015ad60 2024-04-25T15:54:20.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff4132ba640 1 Processor -- start 2024-04-25T15:54:20.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff4132ba640 1 -- start start 2024-04-25T15:54:20.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7f58ae7fc640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f589c06f430 0x7f589c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7f58ae7fc640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f589c06f430 0x7f589c071940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f58a40090e0 tx=0x7f58a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.372 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:20.366+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff4132ba640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff40c15d300 0x7ff40c0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff4132ba640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff40c165150 0x7ff40c167570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff4132ba640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff40c156fd0 con 0x7ff40c06d030 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff4132ba640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c073350 con 0x7ff40c15d300 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff4132ba640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c073630 con 0x7ff40c165150 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff411056640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff40c15d300 0x7ff40c0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff410855640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff40c06d030 0x7ff40c06d440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50390/0 (socket says 172.21.15.67:50390) 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff410855640 1 -- 172.21.15.67:0/1821125979 learned_addr learned my addr 172.21.15.67:0/1821125979 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff411857640 1 --2- 172.21.15.67:0/1821125979 >> v2:172.21.15.176:3300/0 conn(0x7ff40c165150 0x7ff40c167570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff411056640 1 -- 172.21.15.67:0/1821125979 >> v2:172.21.15.176:3300/0 conn(0x7ff40c165150 msgr2=0x7ff40c167570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:20.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff411056640 1 --2- 172.21.15.67:0/1821125979 >> v2:172.21.15.176:3300/0 conn(0x7ff40c165150 0x7ff40c167570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff411056640 1 -- 172.21.15.67:0/1821125979 >> v1:172.21.15.67:6789/0 conn(0x7ff40c06d030 legacy=0x7ff40c06d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.265+0000 7ff411056640 1 -- 172.21.15.67:0/1821125979 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c073be0 con 0x7ff40c15d300 2024-04-25T15:54:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.269+0000 7ff411056640 1 --2- 172.21.15.67:0/1821125979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c0c92d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0c63e0 tx=0x7ff40801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3b4f91f8e08a0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.273+0000 7ff403fff640 1 -- 172.21.15.67:0/1821125979 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4080b0070 con 0x7ff40c15d300 2024-04-25T15:54:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.273+0000 7f5897fff640 1 -- 172.21.15.67:0/1508581998 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f58b0068c40 con 0x7f58b015ad60 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.273+0000 7ff403fff640 1 -- 172.21.15.67:0/1821125979 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4080ab600 con 0x7ff40c15d300 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.273+0000 7ff403fff640 1 -- 172.21.15.67:0/1821125979 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4080ab9d0 con 0x7ff40c15d300 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.273+0000 7ff4132ba640 1 -- 172.21.15.67:0/1821125979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 msgr2=0x7ff40c0c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.273+0000 7ff4132ba640 1 --2- 172.21.15.67:0/1821125979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c0c92d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0c63e0 tx=0x7ff40801c480 comp rx=0 tx=0).stop 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 -- 172.21.15.67:0/1821125979 shutdown_connections 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 --2- 172.21.15.67:0/1821125979 >> v2:172.21.15.176:3300/0 conn(0x7ff40c165150 0x7ff40c167570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 --2- 172.21.15.67:0/1821125979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c0c92d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 -- 172.21.15.67:0/1821125979 >> 172.21.15.67:0/1821125979 conn(0x7ff40c070db0 msgr2=0x7ff40c0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 -- 172.21.15.67:0/1821125979 shutdown_connections 2024-04-25T15:54:20.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 -- 172.21.15.67:0/1821125979 wait complete. 2024-04-25T15:54:20.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 Processor -- start 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.277+0000 7ff4132ba640 1 -- start start 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff4132ba640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff40c06d030 0x7ff40c166c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff4132ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c165730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff4132ba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff40c165150 0x7ff40c149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff4132ba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c15d710 con 0x7ff40c15d300 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff4132ba640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c072ff0 con 0x7ff40c06d030 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff4132ba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff40c0749a0 con 0x7ff40c165150 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c165730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c165730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44086/0 (socket says 172.21.15.67:44086) 2024-04-25T15:54:20.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 -- 172.21.15.67:0/3454935410 learned_addr learned my addr 172.21.15.67:0/3454935410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff411056640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff40c06d030 0x7ff40c166c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff411857640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff40c165150 0x7ff40c149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 -- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff40c165150 msgr2=0x7ff40c149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff40c165150 0x7ff40c149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 -- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff40c06d030 msgr2=0x7ff40c166c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff40c06d030 0x7ff40c166c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 -- 172.21.15.67:0/3454935410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c0c6600 con 0x7ff40c15d300 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff410855640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c165730 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00c370 tx=0x7ff3fc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.281+0000 7ff401ffb640 1 -- 172.21.15.67:0/3454935410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc017070 con 0x7ff40c15d300 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.285+0000 7ff401ffb640 1 -- 172.21.15.67:0/3454935410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3fc005cf0 con 0x7ff40c15d300 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.285+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40c0c77f0 con 0x7ff40c15d300 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.285+0000 7ff401ffb640 1 -- 172.21.15.67:0/3454935410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc0072f0 con 0x7ff40c15d300 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.285+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40c1f77b0 con 0x7ff40c15d300 2024-04-25T15:54:20.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.285+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff40c06a480 con 0x7ff40c15d300 2024-04-25T15:54:20.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.285+0000 7ff401ffb640 1 -- 172.21.15.67:0/3454935410 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff3fc01a070 con 0x7ff40c15d300 2024-04-25T15:54:20.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.289+0000 7ff401ffb640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3f006f400 0x7ff3f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.289+0000 7ff401ffb640 1 -- 172.21.15.67:0/3454935410 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6002+0+0 (secure 0 0 0) 0x7ff3fc08e650 con 0x7ff40c15d300 2024-04-25T15:54:20.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.289+0000 7ff411056640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3f006f400 0x7ff3f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.289+0000 7ff411056640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3f006f400 0x7ff3f0071910 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff4080ad000 tx=0x7ff4080a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.297+0000 7ff401ffb640 1 -- 172.21.15.67:0/3454935410 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff3fc05ae50 con 0x7ff40c15d300 2024-04-25T15:54:20.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f362bff8640 1 Processor -- start 2024-04-25T15:54:20.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f362bff8640 1 -- start start 2024-04-25T15:54:20.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f362bff8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f362406c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f362bff8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3624068c40 con 0x7f362415dd00 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f362bff8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3624160990 con 0x7f362406cb80 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f362bff8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f362415e110 con 0x7f3624068830 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f3629d94640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f362415dd00 0x7f36241569a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50402/0 (socket says 172.21.15.67:50402) 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f3629d94640 1 -- 172.21.15.67:0/1289929378 learned_addr learned my addr 172.21.15.67:0/1289929378 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.373+0000 7f3629593640 1 --2- 172.21.15.67:0/1289929378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f362406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3628d92640 1 -- 172.21.15.67:0/1289929378 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4195036447 0 0) 0x7f3620001120 con 0x7f362415dd00 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3628d92640 1 -- 172.21.15.67:0/1289929378 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3610003530 con 0x7f362415dd00 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3629593640 1 -- 172.21.15.67:0/1289929378 >> v1:172.21.15.176:6789/0 conn(0x7f362406cb80 legacy=0x7f3624167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3628d92640 1 -- 172.21.15.67:0/1289929378 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100502404 0 0) 0x7f3624160990 con 0x7f362406cb80 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3629593640 1 -- 172.21.15.67:0/1289929378 >> v1:172.21.15.67:6789/0 conn(0x7f362415dd00 legacy=0x7f36241569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3629593640 1 -- 172.21.15.67:0/1289929378 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3624160c70 con 0x7f3624068830 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3629593640 1 --2- 172.21.15.67:0/1289929378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f362406c640 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36140097d0 tx=0x7f361401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=270d3cef10029571 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3628d92640 1 -- 172.21.15.67:0/1289929378 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36140b0070 con 0x7f3624068830 2024-04-25T15:54:20.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3628d92640 1 -- 172.21.15.67:0/1289929378 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36140ab600 con 0x7f3624068830 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.377+0000 7f3628d92640 1 -- 172.21.15.67:0/1289929378 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36140ab9b0 con 0x7f3624068830 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.381+0000 7f362bff8640 1 -- 172.21.15.67:0/1289929378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 msgr2=0x7f362406c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.381+0000 7f362bff8640 1 --2- 172.21.15.67:0/1289929378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f362406c640 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f36140097d0 tx=0x7f361401c4c0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.381+0000 7f362bff8640 1 -- 172.21.15.67:0/1289929378 shutdown_connections 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.381+0000 7f362bff8640 1 --2- 172.21.15.67:0/1289929378 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f362406c640 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.381+0000 7f362bff8640 1 -- 172.21.15.67:0/1289929378 >> 172.21.15.67:0/1289929378 conn(0x7f36240ce1c0 msgr2=0x7f36240ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.381+0000 7f362bff8640 1 -- 172.21.15.67:0/1289929378 shutdown_connections 2024-04-25T15:54:20.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.381+0000 7f362bff8640 1 -- 172.21.15.67:0/1289929378 wait complete. 2024-04-25T15:54:20.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 Processor -- start 2024-04-25T15:54:20.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 -- start start 2024-04-25T15:54:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f3624072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f362406cb80 0x7f362415ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f362415dd00 0x7f362415d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3624074a80 con 0x7f362406cb80 2024-04-25T15:54:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3624074900 con 0x7f362415dd00 2024-04-25T15:54:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.385+0000 7f362bff8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f362406d660 con 0x7f3624068830 2024-04-25T15:54:20.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f362406cb80 0x7f362415ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f362406cb80 0x7f362415ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44100/0 (socket says 172.21.15.67:44100) 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 -- 172.21.15.67:0/3734252608 learned_addr learned my addr 172.21.15.67:0/3734252608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f3624072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f362a595640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f362415dd00 0x7f362415d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 msgr2=0x7f3624072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3624068830 0x7f3624072aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f362415dd00 msgr2=0x7f362415d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f362415dd00 0x7f362415d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36240725f0 con 0x7f362406cb80 2024-04-25T15:54:20.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f3629593640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f362406cb80 0x7f362415ccd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f36140097d0 tx=0x7f3614019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36140b0040 con 0x7f362406cb80 2024-04-25T15:54:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36140b9070 con 0x7f362406cb80 2024-04-25T15:54:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3624072fe0 con 0x7f362406cb80 2024-04-25T15:54:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614006dc0 con 0x7f362406cb80 2024-04-25T15:54:20.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3624157e10 con 0x7f362406cb80 2024-04-25T15:54:20.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.389+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f36241f9ef0 con 0x7f362406cb80 2024-04-25T15:54:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f36140b4630 con 0x7f362406cb80 2024-04-25T15:54:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f361a7fc640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f360806f400 0x7f3608071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6002+0+0 (secure 0 0 0) 0x7f3614129a60 con 0x7f362406cb80 2024-04-25T15:54:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v102) ==== 54+0+16 (secure 0 0 0) 0x7f36140160b0 con 0x7f362406cb80 2024-04-25T15:54:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f360806f400 0x7f3608071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f362415b6e0 0x7f362415baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3624153f90 con 0x7f362415b6e0 2024-04-25T15:54:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f362a595640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f362415b6e0 0x7f362415baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f360806f400 0x7f3608071910 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3620005890 tx=0x7f3620009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.393+0000 7f362a595640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f362415b6e0 0x7f362415baf0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.397+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3624153f90 con 0x7f362415b6e0 2024-04-25T15:54:20.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.413+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f362415f950 con 0x7f362415b6e0 2024-04-25T15:54:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.413+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f362415f950 con 0x7f362415b6e0 2024-04-25T15:54:20.415 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.413+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f362415a640 0x7f362415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.413+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3624151d40 con 0x7f362415a640 2024-04-25T15:54:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.413+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f362415a640 0x7f362415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.417+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f362415a640 0x7f362415aa50 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.417+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3624151d40 con 0x7f362415a640 2024-04-25T15:54:20.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.457+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f362415af90 con 0x7f362415a640 2024-04-25T15:54:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.457+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f362415af90 con 0x7f362415a640 2024-04-25T15:54:20.460 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.461+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3624146a40 0x7f3624146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.461+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3624147390 con 0x7f3624146a40 2024-04-25T15:54:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.461+0000 7f3629593640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3624146a40 0x7f3624146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.461+0000 7f3629593640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3624146a40 0x7f3624146e50 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.461+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3624147390 con 0x7f3624146a40 2024-04-25T15:54:20.468 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:20.465+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:20.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.477+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f36240632e0 con 0x7f3624146a40 2024-04-25T15:54:20.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.477+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff40c06cc00 con 0x7ff40c15d300 2024-04-25T15:54:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.481+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f36240632e0 con 0x7f3624146a40 2024-04-25T15:54:20.481 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:20.481 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.481+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3624063dc0 0x7f362415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.481+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f36240632e0 con 0x7f3624063dc0 2024-04-25T15:54:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.481+0000 7f362a595640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3624063dc0 0x7f362415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.481+0000 7f362a595640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3624063dc0 0x7f362415e2b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.481+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36240632e0 con 0x7f3624063dc0 2024-04-25T15:54:20.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.497+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f36240632e0 con 0x7f3624063dc0 2024-04-25T15:54:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.497+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f36240632e0 con 0x7f3624063dc0 2024-04-25T15:54:20.500 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:20.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.497+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f362402c9e0 0x7f362402cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.497+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f362415f030 con 0x7f362402c9e0 2024-04-25T15:54:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.501+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f362402c9e0 0x7f362402cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.501+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f362402c9e0 0x7f362402cdf0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.501+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f362415f030 con 0x7f362402c9e0 2024-04-25T15:54:20.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.513+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58b0156530 con 0x7f589c06f430 2024-04-25T15:54:20.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.517+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f362415f030 con 0x7f362402c9e0 2024-04-25T15:54:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.521+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f362415f030 con 0x7f362402c9e0 2024-04-25T15:54:20.523 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:20.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.521+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f362402d840 0x7f362420c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.521+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f362415f270 con 0x7f362402d840 2024-04-25T15:54:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.521+0000 7f3629593640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f362402d840 0x7f362420c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.525+0000 7f3629593640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f362402d840 0x7f362420c990 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.525+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36140f61a0 con 0x7f362402d840 2024-04-25T15:54:20.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.533+0000 7f5897fff640 1 -- 172.21.15.67:0/1508581998 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254450 (secure 0 0 0) 0x7f58b0156530 con 0x7f589c06f430 2024-04-25T15:54:20.537 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:20.538 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:54:20.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f589c06f430 msgr2=0x7f589c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f589c06f430 0x7f589c071940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f58a40090e0 tx=0x7f58a4008040 comp rx=0 tx=0).stop 2024-04-25T15:54:20.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b015ad60 msgr2=0x7f58b00cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b015ad60 0x7f58b00cceb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f58a001c6f0 tx=0x7f5898005840 comp rx=0 tx=0).stop 2024-04-25T15:54:20.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 shutdown_connections 2024-04-25T15:54:20.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f589c06f430 0x7f589c071940 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b015ca20 0x7f58b00733b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b00cd3f0 0x7f58b015c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 --2- 172.21.15.67:0/1508581998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58b015ad60 0x7f58b00cceb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 >> 172.21.15.67:0/1508581998 conn(0x7f58b00ce1c0 msgr2=0x7f58b00c31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 shutdown_connections 2024-04-25T15:54:20.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.537+0000 7f58b5962640 1 -- 172.21.15.67:0/1508581998 wait complete. 2024-04-25T15:54:20.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.541+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f362402dc50 con 0x7f362402d840 2024-04-25T15:54:20.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.545+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f362402dc50 con 0x7f362402d840 2024-04-25T15:54:20.547 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:20.548 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:20.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.545+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f362420d3e0 0x7f362420f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.545+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f36242125d0 con 0x7f362420d3e0 2024-04-25T15:54:20.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.545+0000 7f362a595640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f362420d3e0 0x7f362420f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.549+0000 7f362a595640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f362420d3e0 0x7f362420f800 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.549+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36242125d0 con 0x7f362420d3e0 2024-04-25T15:54:20.566 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-04-25T15:54:19.877138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10738198242,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":68,"num_read_kb":57,"num_write":7070,"num_write_kb":25167735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":658513155,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7009,"ondisk_log_size":7009,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":21265392,"kb_used_data":20982672,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":733709328,"statfs":{"total":773094113280,"available":751318351872,"internally_reserved":0,"allocated":21486256128,"data_stored":21480936396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":171,"apply_latency_ms":171,"commit_latency_ns":171000000,"apply_latency_ns":171000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3394,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":5996549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.503836"},"pg_stats":[{"pgid":"3.3f","version":"101'100","reported_seq":273,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.490655+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:54:15.490655+0000","last_peered":"2024-04-25T15:54:15.490655+0000","last_clean":"2024-04-25T15:54:15.490655+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:54:15.490655+0000","last_undegraded":"2024-04-25T15:54:15.490655+0000","last_fullsized":"2024-04-25T15:54:15.490655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"101'90","reported_seq":156,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.427731+0000","last_change":"2024-04-25T15:53:13.069367+0000","last_active":"2024-04-25T15:54:15.427731+0000","last_peered":"2024-04-25T15:54:15.427731+0000","last_clean":"2024-04-25T15:54:15.427731+0000","last_became_active":"2024-04-25T15:53:12.990829+0000","last_became_peered":"2024-04-25T15:53:12.990829+0000","last_unstale":"2024-04-25T15:54:15.427731+0000","last_undegraded":"2024-04-25T15:54:15.427731+0000","last_fullsized":"2024-04-25T15:54:15.427731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:46.276338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3d","version":"100'103","reported_seq":234,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.427730+0000","last_change":"2024-04-25T15:51:59.934624+0000","last_active":"2024-04-25T15:54:15.427730+0000","last_peered":"2024-04-25T15:54:15.427730+0000","last_clean":"2024-04-25T15:54:15.427730+0000","last_became_active":"2024-04-25T15:51:59.934111+0000","last_became_peered":"2024-04-25T15:51:59.934111+0000","last_unstale":"2024-04-25T15:54:15.427730+0000","last_undegraded":"2024-04-25T15:54:15.427730+0000","last_fullsized":"2024-04-25T15:54:15.427730+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:52.931463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3c","version":"101'115","reported_seq":290,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.418152+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:54:15.418152+0000","last_peered":"2024-04-25T15:54:15.418152+0000","last_clean":"2024-04-25T15:54:15.418152+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:54:15.418152+0000","last_undegraded":"2024-04-25T15:54:15.418152+0000","last_fullsized":"2024-04-25T15:54:15.418152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"101'105","reported_seq":278,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.884747+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:54:15.884747+0000","last_peered":"2024-04-25T15:54:15.884747+0000","last_clean":"2024-04-25T15:54:15.884747+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:54:15.884747+0000","last_undegraded":"2024-04-25T15:54:15.884747+0000","last_fullsized":"2024-04-25T15:54:15.884747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"101'93","reported_seq":268,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.780838+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:54:15.780838+0000","last_peered":"2024-04-25T15:54:15.780838+0000","last_clean":"2024-04-25T15:54:15.780838+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:54:15.780838+0000","last_undegraded":"2024-04-25T15:54:15.780838+0000","last_fullsized":"2024-04-25T15:54:15.780838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"101'114","reported_seq":287,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.490964+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:54:15.490964+0000","last_peered":"2024-04-25T15:54:15.490964+0000","last_clean":"2024-04-25T15:54:15.490964+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:54:15.490964+0000","last_undegraded":"2024-04-25T15:54:15.490964+0000","last_fullsized":"2024-04-25T15:54:15.490964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"101'126","reported_seq":301,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.958772+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:54:15.958772+0000","last_peered":"2024-04-25T15:54:15.958772+0000","last_clean":"2024-04-25T15:54:15.958772+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:54:15.958772+0000","last_undegraded":"2024-04-25T15:54:15.958772+0000","last_fullsized":"2024-04-25T15:54:15.958772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"101'103","reported_seq":271,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.558884+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:54:15.558884+0000","last_peered":"2024-04-25T15:54:15.558884+0000","last_clean":"2024-04-25T15:54:15.558884+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:54:15.558884+0000","last_undegraded":"2024-04-25T15:54:15.558884+0000","last_fullsized":"2024-04-25T15:54:15.558884+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"101'103","reported_seq":278,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.695751+0000","last_change":"2024-04-25T15:50:31.183152+0000","last_active":"2024-04-25T15:54:15.695751+0000","last_peered":"2024-04-25T15:54:15.695751+0000","last_clean":"2024-04-25T15:54:15.695751+0000","last_became_active":"2024-04-25T15:50:31.180684+0000","last_became_peered":"2024-04-25T15:50:31.180684+0000","last_unstale":"2024-04-25T15:54:15.695751+0000","last_undegraded":"2024-04-25T15:54:15.695751+0000","last_fullsized":"2024-04-25T15:54:15.695751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:16.471334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"101'129","reported_seq":304,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.487278+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:54:15.487278+0000","last_peered":"2024-04-25T15:54:15.487278+0000","last_clean":"2024-04-25T15:54:15.487278+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:54:15.487278+0000","last_undegraded":"2024-04-25T15:54:15.487278+0000","last_fullsized":"2024-04-25T15:54:15.487278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"101'101","reported_seq":161,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.642310+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:54:15.642310+0000","last_peered":"2024-04-25T15:54:15.642310+0000","last_clean":"2024-04-25T15:54:15.642310+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:54:15.642310+0000","last_undegraded":"2024-04-25T15:54:15.642310+0000","last_fullsized":"2024-04-25T15:54:15.642310+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"101'101","reported_seq":276,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.446601+0000","last_change":"2024-04-25T15:50:31.193438+0000","last_active":"2024-04-25T15:54:15.446601+0000","last_peered":"2024-04-25T15:54:15.446601+0000","last_clean":"2024-04-25T15:54:15.446601+0000","last_became_active":"2024-04-25T15:50:31.191131+0000","last_became_peered":"2024-04-25T15:50:31.191131+0000","last_unstale":"2024-04-25T15:54:15.446601+0000","last_undegraded":"2024-04-25T15:54:15.446601+0000","last_fullsized":"2024-04-25T15:54:15.446601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:46.094391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"101'101","reported_seq":277,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.582087+0000","last_change":"2024-04-25T15:51:59.931428+0000","last_active":"2024-04-25T15:54:15.582087+0000","last_peered":"2024-04-25T15:54:15.582087+0000","last_clean":"2024-04-25T15:54:15.582087+0000","last_became_active":"2024-04-25T15:51:59.930935+0000","last_became_peered":"2024-04-25T15:51:59.930935+0000","last_unstale":"2024-04-25T15:54:15.582087+0000","last_undegraded":"2024-04-25T15:54:15.582087+0000","last_fullsized":"2024-04-25T15:54:15.582087+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:38:20.224690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"101'103","reported_seq":163,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.750722+0000","last_change":"2024-04-25T15:53:17.148839+0000","last_active":"2024-04-25T15:54:15.750722+0000","last_peered":"2024-04-25T15:54:15.750722+0000","last_clean":"2024-04-25T15:54:15.750722+0000","last_became_active":"2024-04-25T15:53:16.064416+0000","last_became_peered":"2024-04-25T15:53:16.064416+0000","last_unstale":"2024-04-25T15:54:15.750722+0000","last_undegraded":"2024-04-25T15:54:15.750722+0000","last_fullsized":"2024-04-25T15:54:15.750722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:58.798689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"100'102","reported_seq":277,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.416803+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:54:15.416803+0000","last_peered":"2024-04-25T15:54:15.416803+0000","last_clean":"2024-04-25T15:54:15.416803+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:54:15.416803+0000","last_undegraded":"2024-04-25T15:54:15.416803+0000","last_fullsized":"2024-04-25T15:54:15.416803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"101'96","reported_seq":204,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417306+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:54:15.417306+0000","last_peered":"2024-04-25T15:54:15.417306+0000","last_clean":"2024-04-25T15:54:15.417306+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:54:15.417306+0000","last_undegraded":"2024-04-25T15:54:15.417306+0000","last_fullsized":"2024-04-25T15:54:15.417306+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"101'103","reported_seq":277,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:16.087274+0000","last_change":"2024-04-25T15:50:31.189962+0000","last_active":"2024-04-25T15:54:16.087274+0000","last_peered":"2024-04-25T15:54:16.087274+0000","last_clean":"2024-04-25T15:54:16.087274+0000","last_became_active":"2024-04-25T15:50:31.189118+0000","last_became_peered":"2024-04-25T15:50:31.189118+0000","last_unstale":"2024-04-25T15:54:16.087274+0000","last_undegraded":"2024-04-25T15:54:16.087274+0000","last_fullsized":"2024-04-25T15:54:16.087274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:41.216143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"101'116","reported_seq":289,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.929563+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:54:15.929563+0000","last_peered":"2024-04-25T15:54:15.929563+0000","last_clean":"2024-04-25T15:54:15.929563+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:54:15.929563+0000","last_undegraded":"2024-04-25T15:54:15.929563+0000","last_fullsized":"2024-04-25T15:54:15.929563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"101'108","reported_seq":299,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.731671+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:54:15.731671+0000","last_peered":"2024-04-25T15:54:15.731671+0000","last_clean":"2024-04-25T15:54:15.731671+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:54:15.731671+0000","last_undegraded":"2024-04-25T15:54:15.731671+0000","last_fullsized":"2024-04-25T15:54:15.731671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"101'91","reported_seq":221,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:16.053767+0000","last_change":"2024-04-25T15:51:59.932316+0000","last_active":"2024-04-25T15:54:16.053767+0000","last_peered":"2024-04-25T15:54:16.053767+0000","last_clean":"2024-04-25T15:54:16.053767+0000","last_became_active":"2024-04-25T15:51:59.932024+0000","last_became_peered":"2024-04-25T15:51:59.932024+0000","last_unstale":"2024-04-25T15:54:16.053767+0000","last_undegraded":"2024-04-25T15:54:16.053767+0000","last_fullsized":"2024-04-25T15:54:16.053767+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:23.435895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2a","version":"101'113","reported_seq":281,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.916298+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:54:15.916298+0000","last_peered":"2024-04-25T15:54:15.916298+0000","last_clean":"2024-04-25T15:54:15.916298+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:54:15.916298+0000","last_undegraded":"2024-04-25T15:54:15.916298+0000","last_fullsized":"2024-04-25T15:54:15.916298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"100'90","reported_seq":281,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.446364+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:54:15.446364+0000","last_peered":"2024-04-25T15:54:15.446364+0000","last_clean":"2024-04-25T15:54:15.446364+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:54:15.446364+0000","last_undegraded":"2024-04-25T15:54:15.446364+0000","last_fullsized":"2024-04-25T15:54:15.446364+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'125","reported_seq":300,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.416950+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:54:15.416950+0000","last_peered":"2024-04-25T15:54:15.416950+0000","last_clean":"2024-04-25T15:54:15.416950+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:54:15.416950+0000","last_undegraded":"2024-04-25T15:54:15.416950+0000","last_fullsized":"2024-04-25T15:54:15.416950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"101'94","reported_seq":141,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.720571+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:54:15.720571+0000","last_peered":"2024-04-25T15:54:15.720571+0000","last_clean":"2024-04-25T15:54:15.720571+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:54:15.720571+0000","last_undegraded":"2024-04-25T15:54:15.720571+0000","last_fullsized":"2024-04-25T15:54:15.720571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"101'135","reported_seq":310,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.853588+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:54:15.853588+0000","last_peered":"2024-04-25T15:54:15.853588+0000","last_clean":"2024-04-25T15:54:15.853588+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:54:15.853588+0000","last_undegraded":"2024-04-25T15:54:15.853588+0000","last_fullsized":"2024-04-25T15:54:15.853588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"101'98","reported_seq":254,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.674736+0000","last_change":"2024-04-25T15:53:03.808515+0000","last_active":"2024-04-25T15:54:15.674736+0000","last_peered":"2024-04-25T15:54:15.674736+0000","last_clean":"2024-04-25T15:54:15.674736+0000","last_became_active":"2024-04-25T15:53:03.367253+0000","last_became_peered":"2024-04-25T15:53:03.367253+0000","last_unstale":"2024-04-25T15:54:15.674736+0000","last_undegraded":"2024-04-25T15:54:15.674736+0000","last_fullsized":"2024-04-25T15:54:15.674736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:05.644251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"101'94","reported_seq":304,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.774303+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:54:15.774303+0000","last_peered":"2024-04-25T15:54:15.774303+0000","last_clean":"2024-04-25T15:54:15.774303+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:54:15.774303+0000","last_undegraded":"2024-04-25T15:54:15.774303+0000","last_fullsized":"2024-04-25T15:54:15.774303+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"101'83","reported_seq":215,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.446386+0000","last_change":"2024-04-25T15:51:59.932288+0000","last_active":"2024-04-25T15:54:15.446386+0000","last_peered":"2024-04-25T15:54:15.446386+0000","last_clean":"2024-04-25T15:54:15.446386+0000","last_became_active":"2024-04-25T15:51:59.931796+0000","last_became_peered":"2024-04-25T15:51:59.931796+0000","last_unstale":"2024-04-25T15:54:15.446386+0000","last_undegraded":"2024-04-25T15:54:15.446386+0000","last_fullsized":"2024-04-25T15:54:15.446386+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.939226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.22","version":"101'88","reported_seq":208,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.730735+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:54:15.730735+0000","last_peered":"2024-04-25T15:54:15.730735+0000","last_clean":"2024-04-25T15:54:15.730735+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:54:15.730735+0000","last_undegraded":"2024-04-25T15:54:15.730735+0000","last_fullsized":"2024-04-25T15:54:15.730735+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"101'90","reported_seq":274,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:16.037883+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:54:16.037883+0000","last_peered":"2024-04-25T15:54:16.037883+0000","last_clean":"2024-04-25T15:54:16.037883+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:54:16.037883+0000","last_undegraded":"2024-04-25T15:54:16.037883+0000","last_fullsized":"2024-04-25T15:54:16.037883+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"101'106","reported_seq":281,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.418629+0000","last_change":"2024-04-25T15:50:31.182866+0000","last_active":"2024-04-25T15:54:15.418629+0000","last_peered":"2024-04-25T15:54:15.418629+0000","last_clean":"2024-04-25T15:54:15.418629+0000","last_became_active":"2024-04-25T15:50:31.181837+0000","last_became_peered":"2024-04-25T15:50:31.181837+0000","last_unstale":"2024-04-25T15:54:15.418629+0000","last_undegraded":"2024-04-25T15:54:15.418629+0000","last_fullsized":"2024-04-25T15:54:15.418629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:16:28.497150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"101'88","reported_seq":148,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.419721+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:54:15.419721+0000","last_peered":"2024-04-25T15:54:15.419721+0000","last_clean":"2024-04-25T15:54:15.419721+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:54:15.419721+0000","last_undegraded":"2024-04-25T15:54:15.419721+0000","last_fullsized":"2024-04-25T15:54:15.419721+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":183,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.446729+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:15.446729+0000","last_peered":"2024-04-25T15:54:15.446729+0000","last_clean":"2024-04-25T15:54:15.446729+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:15.446729+0000","last_undegraded":"2024-04-25T15:54:15.446729+0000","last_fullsized":"2024-04-25T15:54:15.446729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"101'94","reported_seq":154,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.420512+0000","last_change":"2024-04-25T15:53:16.510837+0000","last_active":"2024-04-25T15:54:15.420512+0000","last_peered":"2024-04-25T15:54:15.420512+0000","last_clean":"2024-04-25T15:54:15.420512+0000","last_became_active":"2024-04-25T15:53:16.041557+0000","last_became_peered":"2024-04-25T15:53:16.041557+0000","last_unstale":"2024-04-25T15:54:15.420512+0000","last_undegraded":"2024-04-25T15:54:15.420512+0000","last_fullsized":"2024-04-25T15:54:15.420512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:06.600570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":210,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.418727+0000","last_change":"2024-04-25T15:53:16.102712+0000","last_active":"2024-04-25T15:54:15.418727+0000","last_peered":"2024-04-25T15:54:15.418727+0000","last_clean":"2024-04-25T15:54:15.418727+0000","last_became_active":"2024-04-25T15:53:16.052729+0000","last_became_peered":"2024-04-25T15:53:16.052729+0000","last_unstale":"2024-04-25T15:54:15.418727+0000","last_undegraded":"2024-04-25T15:54:15.418727+0000","last_fullsized":"2024-04-25T15:54:15.418727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:27:53.161289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":180,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.446479+0000","last_change":"2024-04-25T15:50:30.191382+0000","last_active":"2024-04-25T15:54:15.446479+0000","last_peered":"2024-04-25T15:54:15.446479+0000","last_clean":"2024-04-25T15:54:15.446479+0000","last_became_active":"2024-04-25T15:50:30.190142+0000","last_became_peered":"2024-04-25T15:50:30.190142+0000","last_unstale":"2024-04-25T15:54:15.446479+0000","last_undegraded":"2024-04-25T15:54:15.446479+0000","last_fullsized":"2024-04-25T15:54:15.446479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:27.113644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"101'115","reported_seq":185,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.396400+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:54:15.396400+0000","last_peered":"2024-04-25T15:54:15.396400+0000","last_clean":"2024-04-25T15:54:15.396400+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:54:15.396400+0000","last_undegraded":"2024-04-25T15:54:15.396400+0000","last_fullsized":"2024-04-25T15:54:15.396400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":198,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417008+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:15.417008+0000","last_peered":"2024-04-25T15:54:15.417008+0000","last_clean":"2024-04-25T15:54:15.417008+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:15.417008+0000","last_undegraded":"2024-04-25T15:54:15.417008+0000","last_fullsized":"2024-04-25T15:54:15.417008+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"101'104","reported_seq":270,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.446209+0000","last_change":"2024-04-25T15:53:16.206656+0000","last_active":"2024-04-25T15:54:15.446209+0000","last_peered":"2024-04-25T15:54:15.446209+0000","last_clean":"2024-04-25T15:54:15.446209+0000","last_became_active":"2024-04-25T15:53:16.049908+0000","last_became_peered":"2024-04-25T15:53:16.049908+0000","last_unstale":"2024-04-25T15:54:15.446209+0000","last_undegraded":"2024-04-25T15:54:15.446209+0000","last_fullsized":"2024-04-25T15:54:15.446209+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.217418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180355072,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":183,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.425255+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:54:15.425255+0000","last_peered":"2024-04-25T15:54:15.425255+0000","last_clean":"2024-04-25T15:54:15.425255+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:54:15.425255+0000","last_undegraded":"2024-04-25T15:54:15.425255+0000","last_fullsized":"2024-04-25T15:54:15.425255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"101'157","reported_seq":332,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:16.151553+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:54:16.151553+0000","last_peered":"2024-04-25T15:54:16.151553+0000","last_clean":"2024-04-25T15:54:16.151553+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:54:16.151553+0000","last_undegraded":"2024-04-25T15:54:16.151553+0000","last_fullsized":"2024-04-25T15:54:16.151553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260046848,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":157,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":181,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417389+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:15.417389+0000","last_peered":"2024-04-25T15:54:15.417389+0000","last_clean":"2024-04-25T15:54:15.417389+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:15.417389+0000","last_undegraded":"2024-04-25T15:54:15.417389+0000","last_fullsized":"2024-04-25T15:54:15.417389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"101'97","reported_seq":228,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.928300+0000","last_change":"2024-04-25T15:51:59.934342+0000","last_active":"2024-04-25T15:54:15.928300+0000","last_peered":"2024-04-25T15:54:15.928300+0000","last_clean":"2024-04-25T15:54:15.928300+0000","last_became_active":"2024-04-25T15:51:59.934033+0000","last_became_peered":"2024-04-25T15:51:59.934033+0000","last_unstale":"2024-04-25T15:54:15.928300+0000","last_undegraded":"2024-04-25T15:54:15.928300+0000","last_fullsized":"2024-04-25T15:54:15.928300+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:02:58.000735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"88'8","reported_seq":137,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417428+0000","last_change":"2024-04-25T15:53:03.435546+0000","last_active":"2024-04-25T15:54:15.417428+0000","last_peered":"2024-04-25T15:54:15.417428+0000","last_clean":"2024-04-25T15:54:15.417428+0000","last_became_active":"2024-04-25T15:53:03.387235+0000","last_became_peered":"2024-04-25T15:53:03.387235+0000","last_unstale":"2024-04-25T15:54:15.417428+0000","last_undegraded":"2024-04-25T15:54:15.417428+0000","last_fullsized":"2024-04-25T15:54:15.417428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:25:30.444213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1344,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"101'99","reported_seq":159,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.490152+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:54:15.490152+0000","last_peered":"2024-04-25T15:54:15.490152+0000","last_clean":"2024-04-25T15:54:15.490152+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:54:15.490152+0000","last_undegraded":"2024-04-25T15:54:15.490152+0000","last_fullsized":"2024-04-25T15:54:15.490152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"101'84","reported_seq":261,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:16.134346+0000","last_change":"2024-04-25T15:50:30.197530+0000","last_active":"2024-04-25T15:54:16.134346+0000","last_peered":"2024-04-25T15:54:16.134346+0000","last_clean":"2024-04-25T15:54:16.134346+0000","last_became_active":"2024-04-25T15:50:30.196794+0000","last_became_peered":"2024-04-25T15:50:30.196794+0000","last_unstale":"2024-04-25T15:54:16.134346+0000","last_undegraded":"2024-04-25T15:54:16.134346+0000","last_fullsized":"2024-04-25T15:54:16.134346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:58.058087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181687,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"100'106","reported_seq":279,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414879+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:54:15.414879+0000","last_peered":"2024-04-25T15:54:15.414879+0000","last_clean":"2024-04-25T15:54:15.414879+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:54:15.414879+0000","last_undegraded":"2024-04-25T15:54:15.414879+0000","last_fullsized":"2024-04-25T15:54:15.414879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":72,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.420595+0000","last_change":"2024-04-25T15:53:16.094342+0000","last_active":"2024-04-25T15:54:15.420595+0000","last_peered":"2024-04-25T15:54:15.420595+0000","last_clean":"2024-04-25T15:54:15.420595+0000","last_became_active":"2024-04-25T15:53:16.041484+0000","last_became_peered":"2024-04-25T15:53:16.041484+0000","last_unstale":"2024-04-25T15:54:15.420595+0000","last_undegraded":"2024-04-25T15:54:15.420595+0000","last_fullsized":"2024-04-25T15:54:15.420595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:42.531801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"101'108","reported_seq":237,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.656663+0000","last_change":"2024-04-25T15:51:59.929086+0000","last_active":"2024-04-25T15:54:15.656663+0000","last_peered":"2024-04-25T15:54:15.656663+0000","last_clean":"2024-04-25T15:54:15.656663+0000","last_became_active":"2024-04-25T15:51:59.928818+0000","last_became_peered":"2024-04-25T15:51:59.928818+0000","last_unstale":"2024-04-25T15:54:15.656663+0000","last_undegraded":"2024-04-25T15:54:15.656663+0000","last_fullsized":"2024-04-25T15:54:15.656663+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:31.155137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":182,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.416944+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:15.416944+0000","last_peered":"2024-04-25T15:54:15.416944+0000","last_clean":"2024-04-25T15:54:15.416944+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:15.416944+0000","last_undegraded":"2024-04-25T15:54:15.416944+0000","last_fullsized":"2024-04-25T15:54:15.416944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"101'109","reported_seq":302,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.490995+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:54:15.490995+0000","last_peered":"2024-04-25T15:54:15.490995+0000","last_clean":"2024-04-25T15:54:15.490995+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:54:15.490995+0000","last_undegraded":"2024-04-25T15:54:15.490995+0000","last_fullsized":"2024-04-25T15:54:15.490995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197132288,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":222,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.418671+0000","last_change":"2024-04-25T15:53:16.115440+0000","last_active":"2024-04-25T15:54:15.418671+0000","last_peered":"2024-04-25T15:54:15.418671+0000","last_clean":"2024-04-25T15:54:15.418671+0000","last_became_active":"2024-04-25T15:53:16.053229+0000","last_became_peered":"2024-04-25T15:53:16.053229+0000","last_unstale":"2024-04-25T15:54:15.418671+0000","last_undegraded":"2024-04-25T15:54:15.418671+0000","last_fullsized":"2024-04-25T15:54:15.418671+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:13.860370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3087,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"101'119","reported_seq":294,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.700950+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:54:15.700950+0000","last_peered":"2024-04-25T15:54:15.700950+0000","last_clean":"2024-04-25T15:54:15.700950+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:54:15.700950+0000","last_undegraded":"2024-04-25T15:54:15.700950+0000","last_fullsized":"2024-04-25T15:54:15.700950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":262,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.426815+0000","last_change":"2024-04-25T15:53:03.421455+0000","last_active":"2024-04-25T15:54:15.426815+0000","last_peered":"2024-04-25T15:54:15.426815+0000","last_clean":"2024-04-25T15:54:15.426815+0000","last_became_active":"2024-04-25T15:53:03.385909+0000","last_became_peered":"2024-04-25T15:53:03.385909+0000","last_unstale":"2024-04-25T15:54:15.426815+0000","last_undegraded":"2024-04-25T15:54:15.426815+0000","last_fullsized":"2024-04-25T15:54:15.426815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:54.677309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":336,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"101'137","reported_seq":338,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.553044+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:54:15.553044+0000","last_peered":"2024-04-25T15:54:15.553044+0000","last_clean":"2024-04-25T15:54:15.553044+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:54:15.553044+0000","last_undegraded":"2024-04-25T15:54:15.553044+0000","last_fullsized":"2024-04-25T15:54:15.553044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192937984,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":64,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.419884+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:15.419884+0000","last_peered":"2024-04-25T15:54:15.419884+0000","last_clean":"2024-04-25T15:54:15.419884+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:15.419884+0000","last_undegraded":"2024-04-25T15:54:15.419884+0000","last_fullsized":"2024-04-25T15:54:15.419884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"101'100","reported_seq":289,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.623229+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:54:15.623229+0000","last_peered":"2024-04-25T15:54:15.623229+0000","last_clean":"2024-04-25T15:54:15.623229+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:54:15.623229+0000","last_undegraded":"2024-04-25T15:54:15.623229+0000","last_fullsized":"2024-04-25T15:54:15.623229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":181,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.428219+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:54:15.428219+0000","last_peered":"2024-04-25T15:54:15.428219+0000","last_clean":"2024-04-25T15:54:15.428219+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:54:15.428219+0000","last_undegraded":"2024-04-25T15:54:15.428219+0000","last_fullsized":"2024-04-25T15:54:15.428219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"101'92","reported_seq":285,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417729+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:54:15.417729+0000","last_peered":"2024-04-25T15:54:15.417729+0000","last_clean":"2024-04-25T15:54:15.417729+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:54:15.417729+0000","last_undegraded":"2024-04-25T15:54:15.417729+0000","last_fullsized":"2024-04-25T15:54:15.417729+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.423786+0000","last_change":"2024-04-25T15:53:16.098410+0000","last_active":"2024-04-25T15:54:15.423786+0000","last_peered":"2024-04-25T15:54:15.423786+0000","last_clean":"2024-04-25T15:54:15.423786+0000","last_became_active":"2024-04-25T15:53:16.053979+0000","last_became_peered":"2024-04-25T15:53:16.053979+0000","last_unstale":"2024-04-25T15:54:15.423786+0000","last_undegraded":"2024-04-25T15:54:15.423786+0000","last_fullsized":"2024-04-25T15:54:15.423786+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:24.859180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"101'99","reported_seq":274,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.418017+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:54:15.418017+0000","last_peered":"2024-04-25T15:54:15.418017+0000","last_clean":"2024-04-25T15:54:15.418017+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:54:15.418017+0000","last_undegraded":"2024-04-25T15:54:15.418017+0000","last_fullsized":"2024-04-25T15:54:15.418017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":182,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.427883+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:54:15.427883+0000","last_peered":"2024-04-25T15:54:15.427883+0000","last_clean":"2024-04-25T15:54:15.427883+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:54:15.427883+0000","last_undegraded":"2024-04-25T15:54:15.427883+0000","last_fullsized":"2024-04-25T15:54:15.427883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"101'103","reported_seq":276,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414786+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:54:15.414786+0000","last_peered":"2024-04-25T15:54:15.414786+0000","last_clean":"2024-04-25T15:54:15.414786+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:54:15.414786+0000","last_undegraded":"2024-04-25T15:54:15.414786+0000","last_fullsized":"2024-04-25T15:54:15.414786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":271,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414585+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:15.414585+0000","last_peered":"2024-04-25T15:54:15.414585+0000","last_clean":"2024-04-25T15:54:15.414585+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:15.414585+0000","last_undegraded":"2024-04-25T15:54:15.414585+0000","last_fullsized":"2024-04-25T15:54:15.414585+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":183,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417036+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:15.417036+0000","last_peered":"2024-04-25T15:54:15.417036+0000","last_clean":"2024-04-25T15:54:15.417036+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:15.417036+0000","last_undegraded":"2024-04-25T15:54:15.417036+0000","last_fullsized":"2024-04-25T15:54:15.417036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"101'106","reported_seq":281,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.929173+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:54:15.929173+0000","last_peered":"2024-04-25T15:54:15.929173+0000","last_clean":"2024-04-25T15:54:15.929173+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:54:15.929173+0000","last_undegraded":"2024-04-25T15:54:15.929173+0000","last_fullsized":"2024-04-25T15:54:15.929173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":206,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414504+0000","last_change":"2024-04-25T15:53:03.424512+0000","last_active":"2024-04-25T15:54:15.414504+0000","last_peered":"2024-04-25T15:54:15.414504+0000","last_clean":"2024-04-25T15:54:15.414504+0000","last_became_active":"2024-04-25T15:53:03.385988+0000","last_became_peered":"2024-04-25T15:53:03.385988+0000","last_unstale":"2024-04-25T15:54:15.414504+0000","last_undegraded":"2024-04-25T15:54:15.414504+0000","last_fullsized":"2024-04-25T15:54:15.414504+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:55.224996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"101'95","reported_seq":155,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.963183+0000","last_change":"2024-04-25T15:53:16.680938+0000","last_active":"2024-04-25T15:54:15.963183+0000","last_peered":"2024-04-25T15:54:15.963183+0000","last_clean":"2024-04-25T15:54:15.963183+0000","last_became_active":"2024-04-25T15:53:16.041368+0000","last_became_peered":"2024-04-25T15:53:16.041368+0000","last_unstale":"2024-04-25T15:54:15.963183+0000","last_undegraded":"2024-04-25T15:54:15.963183+0000","last_fullsized":"2024-04-25T15:54:15.963183+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:05:49.681769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"88'9","reported_seq":187,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.427156+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:54:15.427156+0000","last_peered":"2024-04-25T15:54:15.427156+0000","last_clean":"2024-04-25T15:54:15.427156+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:54:15.427156+0000","last_undegraded":"2024-04-25T15:54:15.427156+0000","last_fullsized":"2024-04-25T15:54:15.427156+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"101'89","reported_seq":244,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.664479+0000","last_change":"2024-04-25T15:53:03.530328+0000","last_active":"2024-04-25T15:54:15.664479+0000","last_peered":"2024-04-25T15:54:15.664479+0000","last_clean":"2024-04-25T15:54:15.664479+0000","last_became_active":"2024-04-25T15:53:03.382259+0000","last_became_peered":"2024-04-25T15:53:03.382259+0000","last_unstale":"2024-04-25T15:54:15.664479+0000","last_undegraded":"2024-04-25T15:54:15.664479+0000","last_fullsized":"2024-04-25T15:54:15.664479+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:43.788748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142606336,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":204,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417113+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:15.417113+0000","last_peered":"2024-04-25T15:54:15.417113+0000","last_clean":"2024-04-25T15:54:15.417113+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:15.417113+0000","last_undegraded":"2024-04-25T15:54:15.417113+0000","last_fullsized":"2024-04-25T15:54:15.417113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"101'98","reported_seq":225,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.490158+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:54:15.490158+0000","last_peered":"2024-04-25T15:54:15.490158+0000","last_clean":"2024-04-25T15:54:15.490158+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:54:15.490158+0000","last_undegraded":"2024-04-25T15:54:15.490158+0000","last_fullsized":"2024-04-25T15:54:15.490158+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167772160,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":73,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414690+0000","last_change":"2024-04-25T15:53:03.381919+0000","last_active":"2024-04-25T15:54:15.414690+0000","last_peered":"2024-04-25T15:54:15.414690+0000","last_clean":"2024-04-25T15:54:15.414690+0000","last_became_active":"2024-04-25T15:53:03.381463+0000","last_became_peered":"2024-04-25T15:53:03.381463+0000","last_unstale":"2024-04-25T15:54:15.414690+0000","last_undegraded":"2024-04-25T15:54:15.414690+0000","last_fullsized":"2024-04-25T15:54:15.414690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:09.993891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"101'101","reported_seq":276,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.730461+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:54:15.730461+0000","last_peered":"2024-04-25T15:54:15.730461+0000","last_clean":"2024-04-25T15:54:15.730461+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:54:15.730461+0000","last_undegraded":"2024-04-25T15:54:15.730461+0000","last_fullsized":"2024-04-25T15:54:15.730461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":179,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.415035+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:15.415035+0000","last_peered":"2024-04-25T15:54:15.415035+0000","last_clean":"2024-04-25T15:54:15.415035+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:15.415035+0000","last_undegraded":"2024-04-25T15:54:15.415035+0000","last_fullsized":"2024-04-25T15:54:15.415035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"101'96","reported_seq":237,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.930492+0000","last_change":"2024-04-25T15:53:03.480191+0000","last_active":"2024-04-25T15:54:15.930492+0000","last_peered":"2024-04-25T15:54:15.930492+0000","last_clean":"2024-04-25T15:54:15.930492+0000","last_became_active":"2024-04-25T15:53:03.380081+0000","last_became_peered":"2024-04-25T15:53:03.380081+0000","last_unstale":"2024-04-25T15:54:15.930492+0000","last_undegraded":"2024-04-25T15:54:15.930492+0000","last_fullsized":"2024-04-25T15:54:15.930492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:10:50.114177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"88'20","reported_seq":191,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.424741+0000","last_change":"2024-04-25T15:53:16.083270+0000","last_active":"2024-04-25T15:54:15.424741+0000","last_peered":"2024-04-25T15:54:15.424741+0000","last_clean":"2024-04-25T15:54:15.424741+0000","last_became_active":"2024-04-25T15:53:16.045200+0000","last_became_peered":"2024-04-25T15:53:16.045200+0000","last_unstale":"2024-04-25T15:54:15.424741+0000","last_undegraded":"2024-04-25T15:54:15.424741+0000","last_fullsized":"2024-04-25T15:54:15.424741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:38.382894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"101'105","reported_seq":280,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.886855+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:54:15.886855+0000","last_peered":"2024-04-25T15:54:15.886855+0000","last_clean":"2024-04-25T15:54:15.886855+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:54:15.886855+0000","last_undegraded":"2024-04-25T15:54:15.886855+0000","last_fullsized":"2024-04-25T15:54:15.886855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":184,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.425299+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:15.425299+0000","last_peered":"2024-04-25T15:54:15.425299+0000","last_clean":"2024-04-25T15:54:15.425299+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:15.425299+0000","last_undegraded":"2024-04-25T15:54:15.425299+0000","last_fullsized":"2024-04-25T15:54:15.425299+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"101'111","reported_seq":171,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.420232+0000","last_change":"2024-04-25T15:53:16.863394+0000","last_active":"2024-04-25T15:54:15.420232+0000","last_peered":"2024-04-25T15:54:15.420232+0000","last_clean":"2024-04-25T15:54:15.420232+0000","last_became_active":"2024-04-25T15:53:16.048669+0000","last_became_peered":"2024-04-25T15:53:16.048669+0000","last_unstale":"2024-04-25T15:54:15.420232+0000","last_undegraded":"2024-04-25T15:54:15.420232+0000","last_fullsized":"2024-04-25T15:54:15.420232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:53.132724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":177,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414406+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:15.414406+0000","last_peered":"2024-04-25T15:54:15.414406+0000","last_clean":"2024-04-25T15:54:15.414406+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:15.414406+0000","last_undegraded":"2024-04-25T15:54:15.414406+0000","last_fullsized":"2024-04-25T15:54:15.414406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"101'101","reported_seq":161,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.420518+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:54:15.420518+0000","last_peered":"2024-04-25T15:54:15.420518+0000","last_clean":"2024-04-25T15:54:15.420518+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:54:15.420518+0000","last_undegraded":"2024-04-25T15:54:15.420518+0000","last_fullsized":"2024-04-25T15:54:15.420518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130023424,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":150,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.396272+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:54:15.396272+0000","last_peered":"2024-04-25T15:54:15.396272+0000","last_clean":"2024-04-25T15:54:15.396272+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:54:15.396272+0000","last_undegraded":"2024-04-25T15:54:15.396272+0000","last_fullsized":"2024-04-25T15:54:15.396272+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"101'108","reported_seq":314,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.915404+0000","last_change":"2024-04-25T15:53:03.761415+0000","last_active":"2024-04-25T15:54:15.915404+0000","last_peered":"2024-04-25T15:54:15.915404+0000","last_clean":"2024-04-25T15:54:15.915404+0000","last_became_active":"2024-04-25T15:53:03.384666+0000","last_became_peered":"2024-04-25T15:53:03.384666+0000","last_unstale":"2024-04-25T15:54:15.915404+0000","last_undegraded":"2024-04-25T15:54:15.915404+0000","last_fullsized":"2024-04-25T15:54:15.915404+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:14.524576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":124,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.446477+0000","last_change":"2024-04-25T15:53:03.409633+0000","last_active":"2024-04-25T15:54:15.446477+0000","last_peered":"2024-04-25T15:54:15.446477+0000","last_clean":"2024-04-25T15:54:15.446477+0000","last_became_active":"2024-04-25T15:53:03.384646+0000","last_became_peered":"2024-04-25T15:53:03.384646+0000","last_unstale":"2024-04-25T15:54:15.446477+0000","last_undegraded":"2024-04-25T15:54:15.446477+0000","last_fullsized":"2024-04-25T15:54:15.446477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:46:43.442571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"101'99","reported_seq":267,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.427537+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:54:15.427537+0000","last_peered":"2024-04-25T15:54:15.427537+0000","last_clean":"2024-04-25T15:54:15.427537+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:54:15.427537+0000","last_undegraded":"2024-04-25T15:54:15.427537+0000","last_fullsized":"2024-04-25T15:54:15.427537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":202,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.415053+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:15.415053+0000","last_peered":"2024-04-25T15:54:15.415053+0000","last_clean":"2024-04-25T15:54:15.415053+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:15.415053+0000","last_undegraded":"2024-04-25T15:54:15.415053+0000","last_fullsized":"2024-04-25T15:54:15.415053+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"100'86","reported_seq":287,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.418578+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:54:15.418578+0000","last_peered":"2024-04-25T15:54:15.418578+0000","last_clean":"2024-04-25T15:54:15.418578+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:54:15.418578+0000","last_undegraded":"2024-04-25T15:54:15.418578+0000","last_fullsized":"2024-04-25T15:54:15.418578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"101'123","reported_seq":324,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.773560+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:54:15.773560+0000","last_peered":"2024-04-25T15:54:15.773560+0000","last_clean":"2024-04-25T15:54:15.773560+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:54:15.773560+0000","last_undegraded":"2024-04-25T15:54:15.773560+0000","last_fullsized":"2024-04-25T15:54:15.773560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":180,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414933+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:15.414933+0000","last_peered":"2024-04-25T15:54:15.414933+0000","last_clean":"2024-04-25T15:54:15.414933+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:15.414933+0000","last_undegraded":"2024-04-25T15:54:15.414933+0000","last_fullsized":"2024-04-25T15:54:15.414933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"101'97","reported_seq":315,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417863+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:54:15.417863+0000","last_peered":"2024-04-25T15:54:15.417863+0000","last_clean":"2024-04-25T15:54:15.417863+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:54:15.417863+0000","last_undegraded":"2024-04-25T15:54:15.417863+0000","last_fullsized":"2024-04-25T15:54:15.417863+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146800640,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":188,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.414850+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:15.414850+0000","last_peered":"2024-04-25T15:54:15.414850+0000","last_clean":"2024-04-25T15:54:15.414850+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:15.414850+0000","last_undegraded":"2024-04-25T15:54:15.414850+0000","last_fullsized":"2024-04-25T15:54:15.414850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"101'91","reported_seq":223,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:16.061346+0000","last_change":"2024-04-25T15:51:59.934567+0000","last_active":"2024-04-25T15:54:16.061346+0000","last_peered":"2024-04-25T15:54:16.061346+0000","last_clean":"2024-04-25T15:54:16.061346+0000","last_became_active":"2024-04-25T15:51:59.934167+0000","last_became_peered":"2024-04-25T15:51:59.934167+0000","last_unstale":"2024-04-25T15:54:16.061346+0000","last_undegraded":"2024-04-25T15:54:16.061346+0000","last_fullsized":"2024-04-25T15:54:16.061346+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:02.242212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":179,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.447017+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:15.447017+0000","last_peered":"2024-04-25T15:54:15.447017+0000","last_clean":"2024-04-25T15:54:15.447017+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:15.447017+0000","last_undegraded":"2024-04-25T15:54:15.447017+0000","last_fullsized":"2024-04-25T15:54:15.447017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"101'105","reported_seq":280,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.746471+0000","last_change":"2024-04-25T15:50:31.201522+0000","last_active":"2024-04-25T15:54:15.746471+0000","last_peered":"2024-04-25T15:54:15.746471+0000","last_clean":"2024-04-25T15:54:15.746471+0000","last_became_active":"2024-04-25T15:50:31.196967+0000","last_became_peered":"2024-04-25T15:50:31.196967+0000","last_unstale":"2024-04-25T15:54:15.746471+0000","last_undegraded":"2024-04-25T15:54:15.746471+0000","last_fullsized":"2024-04-25T15:54:15.746471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:48.396381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":154,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-25T15:54:15.417814+0000","last_change":"2024-04-25T15:51:59.931380+0000","last_active":"2024-04-25T15:54:15.417814+0000","last_peered":"2024-04-25T15:54:15.417814+0000","last_clean":"2024-04-25T15:54:15.417814+0000","last_became_active":"2024-04-25T15:51:59.930941+0000","last_became_peered":"2024-04-25T15:51:59.930941+0000","last_unstale":"2024-04-25T15:54:15.417814+0000","last_undegraded":"2024-04-25T15:54:15.417814+0000","last_fullsized":"2024-04-25T15:54:15.417814+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:30:53.757347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":10737418240,"num_objects":2560,"num_object_clones":0,"num_object_copies":5120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6656,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":658505728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21474836480,"data_stored":21474836480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6657,"ondisk_log_size":6657,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":189634,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":313,"num_write_kb":283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":7427,"num_keys_recovered":40,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":379268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3465480,"kb_used_data":3430140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":90906360,"statfs":{"total":96636764160,"available":93088112640,"internally_reserved":0,"allocated":3512463360,"data_stored":3511792846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2506604,"kb_used_data":2471264,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":91865236,"statfs":{"total":96636764160,"available":94070001664,"internally_reserved":0,"allocated":2530574336,"data_stored":2529915719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574900,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3653328,"kb_used_data":3617988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":90718512,"statfs":{"total":96636764160,"available":92895756288,"internally_reserved":0,"allocated":3704819712,"data_stored":3704142294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3419896,"kb_used_data":3380396,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":90951944,"statfs":{"total":96636764160,"available":93134790656,"internally_reserved":0,"allocated":3461525504,"data_stored":3460870060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2805624,"kb_used_data":2770284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":91566216,"statfs":{"total":96636764160,"available":93763805184,"internally_reserved":0,"allocated":2836770816,"data_stored":2836100217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2260676,"kb_used_data":2225336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":92111164,"statfs":{"total":96636764160,"available":94321831936,"internally_reserved":0,"allocated":2278744064,"data_stored":2278076666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675432,"kb_used_data":644252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696408,"statfs":{"total":96636764160,"available":95945121792,"internally_reserved":0,"allocated":659714048,"data_stored":659073895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2478352,"kb_used_data":2443012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":91893488,"statfs":{"total":96636764160,"available":94098931712,"internally_reserved":0,"allocated":2501644288,"data_stored":2500964699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":183224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":182918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2499805184,"data_stored":2499805184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":658505728,"data_stored":658505728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2277507072,"data_stored":2277507072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2835349504,"data_stored":2835349504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3460300800,"data_stored":3460300800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3703570432,"data_stored":3703570432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2529165312,"data_stored":2529165312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3510632448,"data_stored":3510632448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:54:20.570 INFO:tasks.thrashosds.thrasher:Setting 2.a to [2, 6] 2024-04-25T15:54:20.570 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.a', '2', '6'] 2024-04-25T15:54:20.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.a 2 6 2024-04-25T15:54:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.565+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f362420fd40 con 0x7f362420d3e0 2024-04-25T15:54:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.569+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f362420fd40 con 0x7f362420d3e0 2024-04-25T15:54:20.571 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.569+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f36242103c0 0x7f36242127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.569+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f3624215840 con 0x7f36242103c0 2024-04-25T15:54:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.569+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f36242103c0 0x7f36242127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.573+0000 7f3629d94640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f36242103c0 0x7f36242127e0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.573 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:20.570+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.573+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3624215840 con 0x7f36242103c0 2024-04-25T15:54:20.582 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:20.582 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:20.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.589+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f3624214580 con 0x7f36242103c0 2024-04-25T15:54:20.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f361a7fc640 1 -- 172.21.15.67:0/3734252608 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3624214580 con 0x7f36242103c0 2024-04-25T15:54:20.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f362415b6e0 msgr2=0x7f362415baf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f362415b6e0 0x7f362415baf0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f362415a640 msgr2=0x7f362415aa50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f362415a640 0x7f362415aa50 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3624146a40 msgr2=0x7f3624146e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f3624146a40 0x7f3624146e50 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3624063dc0 msgr2=0x7f362415e2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f3624063dc0 0x7f362415e2b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f362402c9e0 msgr2=0x7f362402cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f362402c9e0 0x7f362402cdf0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f362402d840 msgr2=0x7f362420c990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f362402d840 0x7f362420c990 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f362420d3e0 msgr2=0x7f362420f800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f362420d3e0 0x7f362420f800 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362a595640 1 -- 172.21.15.67:0/3734252608 reap_dead start 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f36242103c0 msgr2=0x7f36242127e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f36242103c0 0x7f36242127e0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362a595640 1 -- 172.21.15.67:0/3734252608 reap_dead start 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362a595640 1 -- 172.21.15.67:0/3734252608 reap_dead start 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f360806f400 msgr2=0x7f3608071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f360806f400 0x7f3608071910 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3620005890 tx=0x7f3620009040 comp rx=0 tx=0).stop 2024-04-25T15:54:20.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f362406cb80 msgr2=0x7f362415ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 --2- 172.21.15.67:0/3734252608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f362406cb80 0x7f362415ccd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f36140097d0 tx=0x7f3614019260 comp rx=0 tx=0).stop 2024-04-25T15:54:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362a595640 1 -- 172.21.15.67:0/3734252608 reap_dead start 2024-04-25T15:54:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 shutdown_connections 2024-04-25T15:54:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 >> 172.21.15.67:0/3734252608 conn(0x7f36240ce1c0 msgr2=0x7f3624150e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 shutdown_connections 2024-04-25T15:54:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.593+0000 7f362bff8640 1 -- 172.21.15.67:0/3734252608 wait complete. 2024-04-25T15:54:20.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:54:20.673 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:20.670+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:20.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.717+0000 7fda91464640 1 Processor -- start 2024-04-25T15:54:20.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.717+0000 7fda91464640 1 -- start start 2024-04-25T15:54:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda91464640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 0x7fda8c168810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda91464640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c160fa0 con 0x7fda8c156f10 2024-04-25T15:54:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda91464640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda8c0c6d10 con 0x7fda8c0cbfc0 2024-04-25T15:54:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda91464640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda8c157320 con 0x7fda8c1663f0 2024-04-25T15:54:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fda8c156f10 0x7fda8c15d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50408/0 (socket says 172.21.15.67:50408) 2024-04-25T15:54:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 -- 172.21.15.67:0/3013280832 learned_addr learned my addr 172.21.15.67:0/3013280832 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:20.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8b7fe640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 0x7fda8c168810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833583626 0 0) 0x7fda8c160fa0 con 0x7fda8c156f10 2024-04-25T15:54:20.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda74003530 con 0x7fda8c156f10 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1105046677 0 0) 0x7fda74003530 con 0x7fda8c156f10 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 msgr2=0x7fda8c168810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 0x7fda8c168810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 >> v1:172.21.15.176:6789/0 conn(0x7fda8c0cbfc0 legacy=0x7fda8c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c160420 con 0x7fda8c156f10 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fda7c001680 con 0x7fda8c156f10 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 >> v1:172.21.15.67:6789/0 conn(0x7fda8c156f10 legacy=0x7fda8c15d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740039f0 0x7fda74003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740043a0 0x7fda740048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 0x7fda74005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c157320 con 0x7fda740039f0 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda7400ba20 con 0x7fda740043a0 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda7400bc20 con 0x7fda8c1663f0 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740039f0 0x7fda74003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8a7fc640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740043a0 0x7fda740048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8b7fe640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 0x7fda74005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 -- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 msgr2=0x7fda74005130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c1663f0 0x7fda74005130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 -- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740043a0 msgr2=0x7fda740048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740043a0 0x7fda740048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.721+0000 7fda8affd640 1 -- 172.21.15.67:0/3013280832 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c160420 con 0x7fda740039f0 2024-04-25T15:54:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda8affd640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740039f0 0x7fda74003e20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c370 tx=0x7fda7c01cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23e0f3787bd34b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c007e00 con 0x7fda740039f0 2024-04-25T15:54:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c0a9030 con 0x7fda740039f0 2024-04-25T15:54:20.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda89ffb640 1 -- 172.21.15.67:0/3013280832 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0058e0 con 0x7fda740039f0 2024-04-25T15:54:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 -- 172.21.15.67:0/3013280832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740039f0 msgr2=0x7fda74003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 --2- 172.21.15.67:0/3013280832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740039f0 0x7fda74003e20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c370 tx=0x7fda7c01cdc0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda8b7fe640 1 -- 172.21.15.67:0/3013280832 reap_dead start 2024-04-25T15:54:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 -- 172.21.15.67:0/3013280832 shutdown_connections 2024-04-25T15:54:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 -- 172.21.15.67:0/3013280832 >> 172.21.15.67:0/3013280832 conn(0x7fda8c068c70 msgr2=0x7fda8c071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 -- 172.21.15.67:0/3013280832 shutdown_connections 2024-04-25T15:54:20.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 -- 172.21.15.67:0/3013280832 wait complete. 2024-04-25T15:54:20.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 Processor -- start 2024-04-25T15:54:20.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.725+0000 7fda91464640 1 -- start start 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740039f0 0x7fda8c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c0ceb80 0x7fda8c1f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda8c1593f0 0x7fda8c159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c1609e0 con 0x7fda8c1593f0 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda8c0cc3d0 con 0x7fda740039f0 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda8c160cc0 con 0x7fda8c0ceb80 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda8c1593f0 0x7fda8c159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda8c1593f0 0x7fda8c159b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44120/0 (socket says 172.21.15.67:44120) 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 -- 172.21.15.67:0/4091077684 learned_addr learned my addr 172.21.15.67:0/4091077684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8affd640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740039f0 0x7fda8c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8a7fc640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c0ceb80 0x7fda8c1f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 -- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c0ceb80 msgr2=0x7fda8c1f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c0ceb80 0x7fda8c1f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 -- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740039f0 msgr2=0x7fda8c158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740039f0 0x7fda8c158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 -- 172.21.15.67:0/4091077684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8c1f6f90 con 0x7fda8c1593f0 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8affd640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740039f0 0x7fda8c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8a7fc640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c0ceb80 0x7fda8c1f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda8b7fe640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda8c1593f0 0x7fda8c159b10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fda8c073be0 tx=0x7fda8400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda737fe640 1 -- 172.21.15.67:0/4091077684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840039a0 con 0x7fda8c1593f0 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda737fe640 1 -- 172.21.15.67:0/4091077684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8401c070 con 0x7fda8c1593f0 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda737fe640 1 -- 172.21.15.67:0/4091077684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84003d80 con 0x7fda8c1593f0 2024-04-25T15:54:20.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8c157880 con 0x7fda8c1593f0 2024-04-25T15:54:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.729+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda8c0cff40 con 0x7fda8c1593f0 2024-04-25T15:54:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.733+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8c06a480 con 0x7fda8c1593f0 2024-04-25T15:54:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.733+0000 7fda737fe640 1 -- 172.21.15.67:0/4091077684 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fda84004030 con 0x7fda8c1593f0 2024-04-25T15:54:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.733+0000 7fda737fe640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7806f400 0x7fda78071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.733+0000 7fda737fe640 1 -- 172.21.15.67:0/4091077684 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 6002+0+0 (secure 0 0 0) 0x7fda84090740 con 0x7fda8c1593f0 2024-04-25T15:54:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.733+0000 7fda8affd640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7806f400 0x7fda78071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.733+0000 7fda8affd640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7806f400 0x7fda78071910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0ac000 tx=0x7fda7c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.737+0000 7fda737fe640 1 -- 172.21.15.67:0/4091077684 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fda84010040 con 0x7fda8c1593f0 2024-04-25T15:54:20.775 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:20.773+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:20.806 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:20.806 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:20.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:54:20.877 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:20.877+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.913+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "2.a", "id": [2, 6]} v 0) -- 0x7fda8c157610 con 0x7fda8c1593f0 2024-04-25T15:54:20.977 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:20.974+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:20.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.993+0000 7ff401ffb640 1 -- 172.21.15.67:0/3454935410 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v103) ==== 97+0+0 (secure 0 0 0) 0x7ff3fc05f150 con 0x7ff40c15d300 2024-04-25T15:54:20.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.993+0000 7fda737fe640 1 -- 172.21.15.67:0/4091077684 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "2.a", "id": [2, 6]}]=0 set 2.a pg_upmap mapping to [2,6] v103) ==== 122+0+0 (secure 0 0 0) 0x7fda8405d020 con 0x7fda8c1593f0 2024-04-25T15:54:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:54:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:set 2.a pg_upmap mapping to [2,6] 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7806f400 msgr2=0x7fda78071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7806f400 0x7fda78071910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fda7c0ac000 tx=0x7fda7c0a7000 comp rx=0 tx=0).stop 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda8c1593f0 msgr2=0x7fda8c159b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda8c1593f0 0x7fda8c159b10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fda8c073be0 tx=0x7fda8400f580 comp rx=0 tx=0).stop 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 shutdown_connections 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7806f400 0x7fda78071910 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda8c1593f0 0x7fda8c159b10 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8c0ceb80 0x7fda8c1f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 --2- 172.21.15.67:0/4091077684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda740039f0 0x7fda8c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 >> 172.21.15.67:0/4091077684 conn(0x7fda8c068c70 msgr2=0x7fda8c15d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 shutdown_connections 2024-04-25T15:54:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:20.997+0000 7fda91464640 1 -- 172.21.15.67:0/4091077684 wait complete. 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3f006f400 msgr2=0x7ff3f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3f006f400 0x7ff3f0071910 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff4080ad000 tx=0x7ff4080a6000 comp rx=0 tx=0).stop 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 msgr2=0x7ff40c165730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c165730 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00c370 tx=0x7ff3fc00c830 comp rx=0 tx=0).stop 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 shutdown_connections 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3f006f400 0x7ff3f0071910 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff40c165150 0x7ff40c149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:21.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff40c15d300 0x7ff40c165730 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 --2- 172.21.15.67:0/3454935410 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff40c06d030 0x7ff40c166c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 >> 172.21.15.67:0/3454935410 conn(0x7ff40c070db0 msgr2=0x7ff40c0cb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 shutdown_connections 2024-04-25T15:54:21.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:21.001+0000 7ff4132ba640 1 -- 172.21.15.67:0/3454935410 wait complete. 2024-04-25T15:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:21.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:54:21.091 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:21.090+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:21.178 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:21.174+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:21.204 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:21.204 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:21.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:54:21.311 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:21.310+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:21.380 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:21.381+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:21.413 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:21.413 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:21.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:54:21.519 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:21.518+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:21.538 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:54:21.539 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 readers = 8311036.00 kB/sec 2024-04-25T15:54:21.539 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 readers = 8307533.92 kB/sec 2024-04-25T15:54:21.539 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 8311036.00 kB/sec 2024-04-25T15:54:21.539 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 8311036.00 kB/sec 2024-04-25T15:54:21.539 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 8311036.00 kB/sec 2024-04-25T15:54:21.539 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-04-25T15:54:21.583 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:21.581+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:21.620 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:21.620 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:21.640 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:54:21.683 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:21.682+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:21.785 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:21.782+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:21.835 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:21.835 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:21.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:54:21.895 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:21.894+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:21.986 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:21.985+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:22.054 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:22.054 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:22.074 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:54:22.087 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:22.085+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:22.188 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:22.185+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:22.219 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:22.219 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:22.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:54:22.289 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:22.286+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:22.381 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:22.381 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:22.390 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:22.389+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:22.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:54:22.491 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:22.489+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:22.588 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:22.588 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:22.593 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:22.590+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:22.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:54:22.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:54:22.694 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:22.693+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:22.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 Processor -- start 2024-04-25T15:54:22.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 -- start start 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f872c06d460 0x7f872c0736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c1547e0 0x7f872c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f872c06cb80 0x7f872c15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f872c068590 con 0x7f872c06d460 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f872c0caa60 con 0x7f872c06cb80 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f87324df640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f872c15a5f0 con 0x7f872c1547e0 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f872c06d460 0x7f872c0736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f872c06d460 0x7f872c0736d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54790/0 (socket says 172.21.15.67:54790) 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 -- 172.21.15.67:0/3302670288 learned_addr learned my addr 172.21.15.67:0/3302670288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872b7fe640 1 --2- 172.21.15.67:0/3302670288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c1547e0 0x7f872c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f8730a7c640 1 --2- 172.21.15.67:0/3302670288 >> v2:172.21.15.176:3300/0 conn(0x7f872c06cb80 0x7f872c15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 -- 172.21.15.67:0/3302670288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c1547e0 msgr2=0x7f872c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:22.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 --2- 172.21.15.67:0/3302670288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c1547e0 0x7f872c06c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 -- 172.21.15.67:0/3302670288 >> v2:172.21.15.176:3300/0 conn(0x7f872c06cb80 msgr2=0x7f872c15aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:22.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 --2- 172.21.15.67:0/3302670288 >> v2:172.21.15.176:3300/0 conn(0x7f872c06cb80 0x7f872c15aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.781+0000 7f872bfff640 1 -- 172.21.15.67:0/3302670288 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c0ca100 con 0x7f872c06d460 2024-04-25T15:54:22.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f872bfff640 1 --2- 172.21.15.67:0/3302670288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06d460 0x7f872c0736d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f872c06f9a0 tx=0x7f872401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368f6210a4a18588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f872affd640 1 -- 172.21.15.67:0/3302670288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240a5760 con 0x7f872c06d460 2024-04-25T15:54:22.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f872affd640 1 -- 172.21.15.67:0/3302670288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87240a5900 con 0x7f872c06d460 2024-04-25T15:54:22.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f872affd640 1 -- 172.21.15.67:0/3302670288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240a5cd0 con 0x7f872c06d460 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 -- 172.21.15.67:0/3302670288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06d460 msgr2=0x7f872c0736d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 --2- 172.21.15.67:0/3302670288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06d460 0x7f872c0736d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f872c06f9a0 tx=0x7f872401c9f0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 -- 172.21.15.67:0/3302670288 shutdown_connections 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 --2- 172.21.15.67:0/3302670288 >> v2:172.21.15.176:3300/0 conn(0x7f872c06cb80 0x7f872c15aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 --2- 172.21.15.67:0/3302670288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c1547e0 0x7f872c06c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 --2- 172.21.15.67:0/3302670288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06d460 0x7f872c0736d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 -- 172.21.15.67:0/3302670288 >> 172.21.15.67:0/3302670288 conn(0x7f872c0ce1c0 msgr2=0x7f872c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 -- 172.21.15.67:0/3302670288 shutdown_connections 2024-04-25T15:54:22.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.785+0000 7f87324df640 1 -- 172.21.15.67:0/3302670288 wait complete. 2024-04-25T15:54:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 Processor -- start 2024-04-25T15:54:22.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 -- start start 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06cb80 0x7f872c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c06d460 0x7f872c14ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f872c1547e0 0x7f872c14f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f872c154bf0 con 0x7f872c06cb80 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f872c15b440 con 0x7f872c1547e0 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f87324df640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f872c154d70 con 0x7f872c06d460 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.789+0000 7f872bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06cb80 0x7f872c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06cb80 0x7f872c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54792/0 (socket says 172.21.15.67:54792) 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 -- 172.21.15.67:0/2017163 learned_addr learned my addr 172.21.15.67:0/2017163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872b7fe640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c06d460 0x7f872c14ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8730a7c640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f872c1547e0 0x7f872c14f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c06d460 msgr2=0x7f872c14ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f872c06d460 0x7f872c14ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f872c1547e0 msgr2=0x7f872c14f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f872c1547e0 0x7f872c14f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c0c60e0 con 0x7f872c06cb80 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06cb80 0x7f872c0cd210 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f872401b0a0 tx=0x7f8724019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240b8670 con 0x7f872c06cb80 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87240be070 con 0x7f872c06cb80 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872c0c8650 con 0x7f872c06cb80 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87240b8a70 con 0x7f872c06cb80 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872c15bed0 con 0x7f872c06cb80 2024-04-25T15:54:22.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f872c06a480 con 0x7f872c06cb80 2024-04-25T15:54:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f87240b8c10 con 0x7f872c06cb80 2024-04-25T15:54:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8728ff9640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f871806f400 0x7f8718071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6031+0+0 (secure 0 0 0) 0x7f872412b010 con 0x7f872c06cb80 2024-04-25T15:54:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v104) ==== 54+0+16 (secure 0 0 0) 0x7f87240165e0 con 0x7f872c06cb80 2024-04-25T15:54:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f872b7fe640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f871806f400 0x7f8718071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f872c15c6d0 0x7f872c0cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f872c073450 con 0x7f872c15c6d0 2024-04-25T15:54:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8730a7c640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f872c15c6d0 0x7f872c0cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.793+0000 7f8730a7c640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f872c15c6d0 0x7f872c0cc0e0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.797+0000 7f872b7fe640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f871806f400 0x7f8718071910 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8720006fd0 tx=0x7f8720008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.798 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:22.797+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.797+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c073450 con 0x7f872c15c6d0 2024-04-25T15:54:22.799 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:22.799 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:22.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.809+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c0cdd40 con 0x7f872c15c6d0 2024-04-25T15:54:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.813+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c0cdd40 con 0x7f872c15c6d0 2024-04-25T15:54:22.813 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.813+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f872c1f74d0 0x7f872c1f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.813+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f872c1f7e20 con 0x7f872c1f74d0 2024-04-25T15:54:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.813+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f872c1f74d0 0x7f872c1f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.813+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f872c1f74d0 0x7f872c1f78e0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.813+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c1f7e20 con 0x7f872c1f74d0 2024-04-25T15:54:22.817 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:54:22.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.825+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c14f6b0 con 0x7f872c1f74d0 2024-04-25T15:54:22.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.829+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c14f6b0 con 0x7f872c1f74d0 2024-04-25T15:54:22.830 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:22.831 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:22.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.829+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f872c0635a0 0x7f872c0639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.829+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f872c063ef0 con 0x7f872c0635a0 2024-04-25T15:54:22.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.829+0000 7f872b7fe640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f872c0635a0 0x7f872c0639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.829+0000 7f872b7fe640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f872c0635a0 0x7f872c0639b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.829+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c063ef0 con 0x7f872c0635a0 2024-04-25T15:54:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.845+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c0cd750 con 0x7f872c0635a0 2024-04-25T15:54:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.845+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c0cd750 con 0x7f872c0635a0 2024-04-25T15:54:22.848 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.849+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f872c150e10 0x7f872c151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.849+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f872c0cd750 con 0x7f872c150e10 2024-04-25T15:54:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.849+0000 7f8730a7c640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f872c150e10 0x7f872c151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.849+0000 7f8730a7c640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f872c150e10 0x7f872c151220 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.849+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c0cd750 con 0x7f872c150e10 2024-04-25T15:54:22.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.861+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c0cd750 con 0x7f872c150e10 2024-04-25T15:54:22.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.865+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c0cd750 con 0x7f872c150e10 2024-04-25T15:54:22.865 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:22.866 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:22.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.865+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f872c02cef0 0x7f872c02d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.865+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f872c0cd750 con 0x7f872c02cef0 2024-04-25T15:54:22.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.865+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f872c02cef0 0x7f872c02d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.865+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f872c02cef0 0x7f872c02d300 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.865+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c0cd750 con 0x7f872c02cef0 2024-04-25T15:54:22.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.881+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c0cd750 con 0x7f872c02cef0 2024-04-25T15:54:22.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.881+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c0cd750 con 0x7f872c02cef0 2024-04-25T15:54:22.884 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:22.884 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:22.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.881+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f872c208340 0x7f872c20a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.881+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f872c02dd50 con 0x7f872c208340 2024-04-25T15:54:22.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.885+0000 7f872b7fe640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f872c208340 0x7f872c20a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.885+0000 7f872b7fe640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f872c208340 0x7f872c20a760 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.885+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c02dd50 con 0x7f872c208340 2024-04-25T15:54:22.897 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:22.897+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:22.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.897+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c02dd50 con 0x7f872c208340 2024-04-25T15:54:22.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.901+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c02dd50 con 0x7f872c208340 2024-04-25T15:54:22.904 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:22.905 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:22.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.905+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f872c20b1b0 0x7f872c20d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.905+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f872c2104a0 con 0x7f872c20b1b0 2024-04-25T15:54:22.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.905+0000 7f8730a7c640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f872c20b1b0 0x7f872c20d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.905+0000 7f8730a7c640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f872c20b1b0 0x7f872c20d5d0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.905+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c2104a0 con 0x7f872c20b1b0 2024-04-25T15:54:22.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.921+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c075040 con 0x7f872c20b1b0 2024-04-25T15:54:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.921+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c075040 con 0x7f872c20b1b0 2024-04-25T15:54:22.924 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.925+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f872c20e020 0x7f872c20e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.925+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f872c2138c0 con 0x7f872c20e020 2024-04-25T15:54:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.925+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f872c20e020 0x7f872c20e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:22.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.925+0000 7f872bfff640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f872c20e020 0x7f872c20e430 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:22.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.925+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f872c2138c0 con 0x7f872c20e020 2024-04-25T15:54:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f872c20ea80 con 0x7f872c20e020 2024-04-25T15:54:22.944 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f8728ff9640 1 -- 172.21.15.67:0/2017163 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f872c20ea80 con 0x7f872c20e020 2024-04-25T15:54:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f872c15c6d0 msgr2=0x7f872c0cc0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f872c15c6d0 0x7f872c0cc0e0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f872c1f74d0 msgr2=0x7f872c1f78e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f872c1f74d0 0x7f872c1f78e0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f872c0635a0 msgr2=0x7f872c0639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f872c0635a0 0x7f872c0639b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f872c150e10 msgr2=0x7f872c151220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f8730a7c640 1 -- 172.21.15.67:0/2017163 reap_dead start 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f872c150e10 0x7f872c151220 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f872c02cef0 msgr2=0x7f872c02d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.941+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f872c02cef0 0x7f872c02d300 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f872c208340 msgr2=0x7f872c20a760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f872c208340 0x7f872c20a760 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f872c20b1b0 msgr2=0x7f872c20d5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f872c20b1b0 0x7f872c20d5d0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f872c20e020 msgr2=0x7f872c20e430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f872c20e020 0x7f872c20e430 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f8730a7c640 1 -- 172.21.15.67:0/2017163 reap_dead start 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f871806f400 msgr2=0x7f8718071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f871806f400 0x7f8718071910 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8720006fd0 tx=0x7f8720008040 comp rx=0 tx=0).stop 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06cb80 msgr2=0x7f872c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06cb80 0x7f872c0cd210 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f872401b0a0 tx=0x7f8724019790 comp rx=0 tx=0).stop 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 shutdown_connections 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f871806f400 0x7f8718071910 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 --2- 172.21.15.67:0/2017163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f872c06cb80 0x7f872c0cd210 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 >> 172.21.15.67:0/2017163 conn(0x7f872c0ce1c0 msgr2=0x7f872c153d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 shutdown_connections 2024-04-25T15:54:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:22.945+0000 7f87324df640 1 -- 172.21.15.67:0/2017163 wait complete. 2024-04-25T15:54:22.969 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:22.969 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:22.969 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:22.969 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:22.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:54:23.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:23.005+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:23.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:54:23.099 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:23.097+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:23.132 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:23.132 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:23.132 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:23.132 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:23.132 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:23.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:54:23.199 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:23.198+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.201+0000 7f517b78a640 1 Processor -- start 2024-04-25T15:54:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.201+0000 7f517b78a640 1 -- start start 2024-04-25T15:54:23.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f517b78a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5174071b20 0x7f517415d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f517b78a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f51740c5bb0 0x7f51740c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f517b78a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51740c6890 con 0x7f517415d710 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f517b78a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f517415cd40 con 0x7f5174071b20 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f517b78a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f51740c6b70 con 0x7f51740c5bb0 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179526640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5174071b20 0x7f517415d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5178d25640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f517415d710 0x7f51740c7610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40482/0 (socket says 172.21.15.67:40482) 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179526640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5174071b20 0x7f517415d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54806/0 (socket says 172.21.15.67:54806) 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5178d25640 1 -- 172.21.15.67:0/323648093 learned_addr learned my addr 172.21.15.67:0/323648093 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179d27640 1 --2- 172.21.15.67:0/323648093 >> v2:172.21.15.176:3300/0 conn(0x7f51740c5bb0 0x7f51740c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179526640 1 -- 172.21.15.67:0/323648093 >> v2:172.21.15.176:3300/0 conn(0x7f51740c5bb0 msgr2=0x7f51740c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179526640 1 --2- 172.21.15.67:0/323648093 >> v2:172.21.15.176:3300/0 conn(0x7f51740c5bb0 0x7f51740c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179526640 1 -- 172.21.15.67:0/323648093 >> v1:172.21.15.67:6789/0 conn(0x7f517415d710 legacy=0x7f51740c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179526640 1 -- 172.21.15.67:0/323648093 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51740c6e50 con 0x7f5174071b20 2024-04-25T15:54:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5167fff640 1 -- 172.21.15.67:0/323648093 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3575629259 0 0) 0x7f51740c6890 con 0x7f517415d710 2024-04-25T15:54:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5179526640 1 --2- 172.21.15.67:0/323648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5174071b20 0x7f517415d1d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5174073410 tx=0x7f517001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a431655b3d2792ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5167fff640 1 -- 172.21.15.67:0/323648093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51700b1070 con 0x7f5174071b20 2024-04-25T15:54:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5167fff640 1 -- 172.21.15.67:0/323648093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51700ac5f0 con 0x7f5174071b20 2024-04-25T15:54:23.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.205+0000 7f5167fff640 1 -- 172.21.15.67:0/323648093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51700ac9c0 con 0x7f5174071b20 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 -- 172.21.15.67:0/323648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5174071b20 msgr2=0x7f517415d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 --2- 172.21.15.67:0/323648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5174071b20 0x7f517415d1d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5174073410 tx=0x7f517001c480 comp rx=0 tx=0).stop 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 -- 172.21.15.67:0/323648093 shutdown_connections 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 --2- 172.21.15.67:0/323648093 >> v2:172.21.15.176:3300/0 conn(0x7f51740c5bb0 0x7f51740c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 --2- 172.21.15.67:0/323648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5174071b20 0x7f517415d1d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 -- 172.21.15.67:0/323648093 >> 172.21.15.67:0/323648093 conn(0x7f517406fe70 msgr2=0x7f51740cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 -- 172.21.15.67:0/323648093 shutdown_connections 2024-04-25T15:54:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 -- 172.21.15.67:0/323648093 wait complete. 2024-04-25T15:54:23.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 Processor -- start 2024-04-25T15:54:23.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.209+0000 7f517b78a640 1 -- start start 2024-04-25T15:54:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5174071b20 0x7f517414f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51740c5bb0 0x7f5174162a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517415d710 0x7f51741f98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5174166580 con 0x7f517415d710 2024-04-25T15:54:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f51740c7d20 con 0x7f51740c5bb0 2024-04-25T15:54:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5174166700 con 0x7f5174071b20 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517415d710 0x7f51741f98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517415d710 0x7f51741f98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54810/0 (socket says 172.21.15.67:54810) 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 -- 172.21.15.67:0/3168295890 learned_addr learned my addr 172.21.15.67:0/3168295890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179526640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5174071b20 0x7f517414f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5178d25640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51740c5bb0 0x7f5174162a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 -- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5174071b20 msgr2=0x7f517414f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5174071b20 0x7f517414f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 -- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51740c5bb0 msgr2=0x7f5174162a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51740c5bb0 0x7f5174162a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 -- 172.21.15.67:0/3168295890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517414bda0 con 0x7f517415d710 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179d27640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517415d710 0x7f51741f98e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f516c009cb0 tx=0x7f516c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5165ffb640 1 -- 172.21.15.67:0/3168295890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c018020 con 0x7f517415d710 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5165ffb640 1 -- 172.21.15.67:0/3168295890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516c014480 con 0x7f517415d710 2024-04-25T15:54:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f517414ad50 con 0x7f517415d710 2024-04-25T15:54:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5165ffb640 1 -- 172.21.15.67:0/3168295890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c014870 con 0x7f517415d710 2024-04-25T15:54:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5174161fd0 con 0x7f517415d710 2024-04-25T15:54:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51740c9aa0 con 0x7f517415d710 2024-04-25T15:54:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5165ffb640 1 -- 172.21.15.67:0/3168295890 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f516c01b020 con 0x7f517415d710 2024-04-25T15:54:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5165ffb640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f515806f400 0x7f5158071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5165ffb640 1 -- 172.21.15.67:0/3168295890 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6031+0+0 (secure 0 0 0) 0x7f516c0900d0 con 0x7f517415d710 2024-04-25T15:54:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.213+0000 7f5179526640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f515806f400 0x7f5158071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.217+0000 7f5179526640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f515806f400 0x7f5158071910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f51700ae000 tx=0x7f51700a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.221+0000 7f5165ffb640 1 -- 172.21.15.67:0/3168295890 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f516c022020 con 0x7f517415d710 2024-04-25T15:54:23.301 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:23.301+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:23.307 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:23.307 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:23.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:23.307 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:23.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 244.20509057500001, 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:54:23.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:54:23.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:54:23.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 89.991216042000005, 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.314 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:54:23.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:54:23.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:54:23.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:54:23.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:54:23.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:54:23.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:54:23.320 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.320 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.320 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.321 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:23.321 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:23.321 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 59.720538089999998, 2024-04-25T15:54:23.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:54:23.322 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.322 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.322 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.322 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.322 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.323 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:54:23.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.323 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:23.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:54:23.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:23.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:54:23.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:54:23.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:54:23.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:54:23.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:54:23.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:54:23.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:54:23.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:54:23.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:54:23.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:54:23.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:23.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:54:23.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:54:23.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:54:23.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:54:23.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:54:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:54:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:54:23.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:54:23.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:54:23.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:54:23.335 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.335 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.335 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.336 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2266 3.11 3:89528692:::100000001f8.000004d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:23.336 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:23.336 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.732371374000003, 2024-04-25T15:54:23.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.38717112199999998, 2024-04-25T15:54:23.336 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.337 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.337 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.337 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.337 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2266 2024-04-25T15:54:23.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.338 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:23.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:23.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571186+0000", 2024-04-25T15:54:23.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4569999999999999e-06 2024-04-25T15:54:23.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784888+0000", 2024-04-25T15:54:23.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.213702742 2024-04-25T15:54:23.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785181+0000", 2024-04-25T15:54:23.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029301099999999998 2024-04-25T15:54:23.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785201+0000", 2024-04-25T15:54:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9683000000000001e-05 2024-04-25T15:54:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785333+0000", 2024-04-25T15:54:23.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013224600000000001 2024-04-25T15:54:23.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785779+0000", 2024-04-25T15:54:23.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044548299999999999 2024-04-25T15:54:23.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:23.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.786106+0000", 2024-04-25T15:54:23.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032678999999999998 2024-04-25T15:54:23.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870593+0000", 2024-04-25T15:54:23.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084487524999999994 2024-04-25T15:54:23.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958234+0000", 2024-04-25T15:54:23.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087640933000000004 2024-04-25T15:54:23.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958283+0000", 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8473999999999997e-05 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958354+0000", 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1778000000000003e-05 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2272 3.13 3:ca3e88be:::100000001f8.000004dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.507622726000001, 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13173032000000001, 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.350 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2272 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795936+0000", 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.754e-06 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805167+0000", 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092308890000000008 2024-04-25T15:54:23.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805557+0000", 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039059400000000002 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805588+0000", 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0865999999999999e-05 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805759+0000", 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017085599999999999 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806172+0000", 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000413416 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806552+0000", 2024-04-25T15:54:23.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000379417 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905757+0000", 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099205023000000003 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927568+0000", 2024-04-25T15:54:23.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021810800000000002 2024-04-25T15:54:23.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927616+0000", 2024-04-25T15:54:23.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:54:23.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927662+0000", 2024-04-25T15:54:23.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6535000000000001e-05 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2273 3.39 3:9fcbcc01:::100000001f8.000004dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.498370610999999, 2024-04-25T15:54:23.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15983650499999999, 2024-04-25T15:54:23.359 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.359 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.359 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.359 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.359 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.359 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2273 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805188+0000", 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816847+0000", 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011659119000000001 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817576+0000", 2024-04-25T15:54:23.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00072975000000000004 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817601+0000", 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4450000000000001e-05 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817741+0000", 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014058500000000001 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818074+0000", 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033291000000000001 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818385+0000", 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031034999999999998 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870618+0000", 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052233743999999999 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964814+0000", 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094195216999999998 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964945+0000", 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013086799999999999 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965021+0000", 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6032000000000007e-05 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2275 3.12 3:492b2588:::100000001f8.000004df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.486695232000002, 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14794684399999999, 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2275 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816863+0000", 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4149999999999999e-06 2024-04-25T15:54:23.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829311+0000", 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012448222 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829569+0000", 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025766800000000001 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829598+0000", 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.968e-05 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829786+0000", 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187746 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830129+0000", 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034284099999999998 2024-04-25T15:54:23.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830479+0000", 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034961500000000001 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867146+0000", 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036667733000000001 2024-04-25T15:54:23.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964688+0000", 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097541577000000004 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964748+0000", 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9626000000000001e-05 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964806+0000", 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8721000000000002e-05 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2278 3.2b 3:d79b9024:::100000001f8.000004e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.474229973999996, 2024-04-25T15:54:23.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135647779, 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2278 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:23.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829328+0000", 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7309999999999999e-06 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843821+0000", 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014493012 2024-04-25T15:54:23.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844787+0000", 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096513199999999995 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844876+0000", 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9147999999999994e-05 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844972+0000", 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6566999999999999e-05 2024-04-25T15:54:23.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845195+0000", 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222699 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845422+0000", 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022700699999999999 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870319+0000", 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024897319000000001 2024-04-25T15:54:23.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964892+0000", 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094572233000000006 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964938+0000", 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6646e-05 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964972+0000", 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4285000000000001e-05 2024-04-25T15:54:23.376 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2305 3.2f e90/51 3:f6648653:::100000001f8.000004fd:head v 90'33, mlcod=90'30)", 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.466296184000001, 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12780160600000001, 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837261+0000", 2024-04-25T15:54:23.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1679999999999998e-06 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839587+0000", 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00232607 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839620+0000", 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3503000000000001e-05 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839642+0000", 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1302e-05 2024-04-25T15:54:23.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839754+0000", 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112454 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839798+0000", 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4240999999999999e-05 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965003+0000", 2024-04-25T15:54:23.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12520515600000001 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965060+0000", 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6712000000000002e-05 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3917 3.27 3:e652e8b8:::100000001f8.00000149:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.80548568, 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16896387500000001, 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.381 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3917 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:23.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498071+0000", 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4779999999999998e-06 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509296+0000", 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011224433000000001 2024-04-25T15:54:23.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509572+0000", 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027576500000000003 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509591+0000", 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9466e-05 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509677+0000", 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5607999999999998e-05 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509773+0000", 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6136000000000002e-05 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509942+0000", 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016951999999999999 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.610111+0000", 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.100168231 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666922+0000", 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056811077000000001 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666991+0000", 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9758000000000005e-05 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667033+0000", 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1403000000000003e-05 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.386 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3918 3.11 3:8b260724:::100000001f8.0000014a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.794249467, 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15748047100000001, 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3918 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.387 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509307+0000", 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.12e-06 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515132+0000", 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058250860000000002 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515164+0000", 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1563000000000003e-05 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515179+0000", 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5102e-05 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515262+0000", 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2917999999999995e-05 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515360+0000", 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7532000000000001e-05 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515512+0000", 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152442 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654409+0000", 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13889727499999999 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666644+0000", 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012234584999999999 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666730+0000", 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5793999999999999e-05 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666786+0000", 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6054000000000002e-05 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3922 3.d 3:b3f0d00e:::100000001f8.0000014e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.784887546, 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.150212927, 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.392 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3922 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518670+0000", 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1039999999999998e-06 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529034+0000", 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010364237 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529073+0000", 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8407e-05 2024-04-25T15:54:23.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529092+0000", 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9411000000000001e-05 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529203+0000", 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011113 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529330+0000", 2024-04-25T15:54:23.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012611499999999999 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529517+0000", 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187639 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667032+0000", 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13751481300000001 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668779+0000", 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017469740000000001 2024-04-25T15:54:23.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668835+0000", 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6251000000000001e-05 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668880+0000", 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4845999999999997e-05 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3920 3.26 e97/18 3:6422fb70:::100000001f8.0000014c:head v 97'93, mlcod=96'91)", 2024-04-25T15:54:23.397 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.779864398000001, 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14317563999999999, 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523692+0000", 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8050000000000001e-06 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525897+0000", 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022045290000000002 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525923+0000", 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6225e-05 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525941+0000", 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8304e-05 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526045+0000", 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104158 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526076+0000", 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0454e-05 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666788+0000", 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140712106 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666866+0000", 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8059000000000006e-05 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3926 3.d 3:b20dc027:::100000001f8.00000152:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.765377851, 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131821096, 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3926 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538180+0000", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4690000000000002e-06 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542940+0000", 2024-04-25T15:54:23.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047600250000000002 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542988+0000", 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7633999999999998e-05 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543006+0000", 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8114999999999999e-05 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543103+0000", 2024-04-25T15:54:23.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6723000000000002e-05 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543283+0000", 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018030100000000001 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543476+0000", 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019295200000000001 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668734+0000", 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12525763200000001 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669901+0000", 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011670890000000001 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669955+0000", 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4373999999999998e-05 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669998+0000", 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3971 3.3f 3:febcb5ea:::100000001f8.0000017f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.485854095000001, 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15452237999999999, 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.407 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3971 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:23.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817703+0000", 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822518+0000", 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048147939999999998 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822563+0000", 2024-04-25T15:54:23.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4515000000000003e-05 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822577+0000", 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.432e-05 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822664+0000", 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7440000000000003e-05 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822786+0000", 2024-04-25T15:54:23.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012110800000000001 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822960+0000", 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017468900000000001 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945290+0000", 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122329907 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972057+0000", 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026767102000000001 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972139+0000", 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2070000000000005e-05 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972223+0000", 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3565000000000004e-05 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5059 3.26 e99/18 3:65b3bea5:::100000001f8.000005bf:head v 99'112, mlcod=99'109)", 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.286164923999999, 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131194432, 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.413 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:23.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017392+0000", 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4650000000000001e-06 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021054+0000", 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036613240000000001 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021076+0000", 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2067000000000001e-05 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021092+0000", 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6209999999999999e-05 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021178+0000", 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6260000000000004e-05 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021208+0000", 2024-04-25T15:54:23.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0046999999999999e-05 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148491+0000", 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12728336500000001 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148584+0000", 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2694000000000004e-05 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5063 3.13 3:c9096c28:::100000001f8.000005c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 13.274612768000001, 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13114490600000001, 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5063 2024-04-25T15:54:23.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028946+0000", 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0520000000000003e-06 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035431+0000", 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064851509999999998 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035461+0000", 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9547999999999999e-05 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035474+0000", 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2880999999999999e-05 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035550+0000", 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6027999999999998e-05 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035622+0000", 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2366999999999993e-05 2024-04-25T15:54:23.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035727+0000", 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105561 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.150510+0000", 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11478287099999999 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159903+0000", 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093925880000000003 2024-04-25T15:54:23.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159983+0000", 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0122000000000003e-05 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160087+0000", 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103737 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5730 3.e 3:7357dd18:::100000001f8.0000085e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.5443165489999995, 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14322902300000001, 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5730 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:23.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759240+0000", 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3910000000000001e-06 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765109+0000", 2024-04-25T15:54:23.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058681660000000002 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765139+0000", 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9981e-05 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765149+0000", 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0125e-05 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765215+0000", 2024-04-25T15:54:23.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6700999999999997e-05 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765293+0000", 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7883e-05 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765394+0000", 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100869 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.893428+0000", 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128033758 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902331+0000", 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089030610000000003 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902403+0000", 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1891999999999995e-05 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902467+0000", 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4196000000000007e-05 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5769 3.d 3:b350774d:::100000001f8.00000885:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.3115988040000008, 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12752495899999999, 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.429 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5769 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991958+0000", 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998738+0000", 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067804069999999996 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998782+0000", 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3738e-05 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998800+0000", 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7907e-05 2024-04-25T15:54:23.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998944+0000", 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144089 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999140+0000", 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195465 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999415+0000", 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027557100000000001 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.070872+0000", 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071456602999999994 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119350+0000", 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048478354000000001 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119421+0000", 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0970999999999995e-05 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119481+0000", 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9802000000000001e-05 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5780 3.f 3:f20b812e:::100000001f8.00000890:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 9.2634177260000001, 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13282865699999999, 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.434 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5780 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:23.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040139+0000", 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9570000000000001e-06 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047681+0000", 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075419279999999998 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047720+0000", 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8794999999999997e-05 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047740+0000", 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0091000000000001e-05 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047828+0000", 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8745000000000005e-05 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047922+0000", 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3076000000000001e-05 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048095+0000", 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000173134 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.170485+0000", 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122390033 2024-04-25T15:54:23.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172900+0000", 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002415482 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172933+0000", 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3021000000000001e-05 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172966+0000", 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2394999999999997e-05 2024-04-25T15:54:23.439 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.440 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.440 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.440 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:23.440 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:23.440 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:23.397+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f51740c5990 con 0x7f517415d710 2024-04-25T15:54:23.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:54:23.442 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:23.441+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:23.542 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:23.541+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:23.598 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:23.599 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:23.599 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:23.599 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:23.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:54:23.657 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:23.657+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:23.744 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:23.742+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:23.811 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:23.811 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:23.812 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:23.812 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:23.812 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:23.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:54:23.845 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:23.842+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:23.946 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:23.945+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:23.994 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:668 3.1 e81/76 3:83ba1de1:::100000001f7.00000098:head v 81'14, mlcod=81'12)", 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.122639495000001, 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.045409913000000003, 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:23.995 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872468+0000", 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3659999999999999e-06 2024-04-25T15:54:23.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879062+0000", 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065941039999999999 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879116+0000", 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4372000000000001e-05 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879152+0000", 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6139000000000001e-05 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879318+0000", 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016595499999999999 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879365+0000", 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6897999999999997e-05 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917803+0000", 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038437773000000001 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917875+0000", 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2305999999999995e-05 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:637 3.20 e81/18 3:07263727:::100000001f7.00000079:head v 81'17, mlcod=81'15)", 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.113151004000002, 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040244958999999997, 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:23.999 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881957+0000", 2024-04-25T15:54:24.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3890000000000001e-06 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888368+0000", 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006410682 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888395+0000", 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7477000000000001e-05 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888414+0000", 2024-04-25T15:54:24.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9046e-05 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888512+0000", 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7876999999999998e-05 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888546+0000", 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3204000000000003e-05 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922131+0000", 2024-04-25T15:54:24.002 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033585168999999998 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922199+0000", 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8114999999999994e-05 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1709 3.b e89/21 3:d0c4adc9:::100000001f8.000002a9:head v 89'35, mlcod=89'33)", 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.996175016999999, 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041562524000000003, 2024-04-25T15:54:24.003 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.004 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998932+0000", 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7570000000000001e-06 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000395+0000", 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014629389999999999 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000414+0000", 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9908999999999999e-05 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.005 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000432+0000", 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249999999999999e-05 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000518+0000", 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6501000000000004e-05 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000548+0000", 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9777e-05 2024-04-25T15:54:24.006 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040424+0000", 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039875586999999997 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040492+0000", 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8804000000000001e-05 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.007 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1710 3.b e89/21 3:d079ce42:::100000001f8.000002aa:head v 89'36, mlcod=89'33)", 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 38.9897463, 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040049727, 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.008 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005360+0000", 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9290000000000001e-06 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006358+0000", 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099724699999999993 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.009 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006377+0000", 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9656e-05 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006392+0000", 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4207e-05 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006477+0000", 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4989999999999998e-05 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.010 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006507+0000", 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0786e-05 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045329+0000", 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038821829000000002 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045408+0000", 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9083000000000002e-05 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2273 3.39 e90/18 3:9fcbcc01:::100000001f8.000004dd:head v 90'54, mlcod=90'52)", 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.169597054999997, 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035128116000000001, 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:24.012 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825510+0000", 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837142+0000", 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011632136 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.013 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837175+0000", 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3166000000000001e-05 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837197+0000", 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1495e-05 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837293+0000", 2024-04-25T15:54:24.014 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6353000000000004e-05 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837329+0000", 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5901000000000001e-05 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860257+0000", 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022927804999999999 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.015 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860636+0000", 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000379327 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2392 3.32 e90/40 3:4cdbb504:::100000001f8.00000554:head v 90'45, mlcod=90'43)", 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.585782293999998, 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044535299, 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.016 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409325+0000", 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2220000000000001e-06 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.017 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410846+0000", 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001521454 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410866+0000", 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0129999999999999e-05 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410882+0000", 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5452e-05 2024-04-25T15:54:24.018 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410962+0000", 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9833999999999995e-05 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410991+0000", 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.968e-05 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453793+0000", 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042801621999999998 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453858+0000", 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4905000000000003e-05 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2394 3.6 e90/76 3:63a5c66e:::100000001f8.00000556:head v 90'46, mlcod=90'44)", 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.571087171000002, 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038940880999999997, 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.020 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424019+0000", 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6160000000000001e-06 2024-04-25T15:54:24.021 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428140+0000", 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041208529999999998 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428169+0000", 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8824999999999999e-05 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428190+0000", 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0894999999999999e-05 2024-04-25T15:54:24.022 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428283+0000", 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3277999999999997e-05 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428318+0000", 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5191000000000002e-05 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462890+0000", 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034571283000000001 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.023 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462959+0000", 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000001e-05 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2396 3.3d e90/40 3:bc4ba9bc:::100000001f8.00000558:head v 90'44, mlcod=90'42)", 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 34.561281950000001, 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033872159999999998, 2024-04-25T15:54:24.024 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433825+0000", 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.951e-06 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435239+0000", 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014141729999999999 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435260+0000", 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0775e-05 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.026 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435278+0000", 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8317000000000001e-05 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435363+0000", 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4441e-05 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435395+0000", 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1939e-05 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467549+0000", 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032154061999999997 2024-04-25T15:54:24.027 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467695+0000", 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014650199999999999 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3866 3.1 e97/76 3:81b781a2:::100000001f8.00000116:head v 97'69, mlcod=96'67)", 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.751100012999999, 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035311905999999997, 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244007+0000", 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.029 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249687+0000", 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056798860000000003 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249709+0000", 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2393999999999999e-05 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249729+0000", 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9536000000000001e-05 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.030 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249849+0000", 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120068 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249888+0000", 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9103999999999997e-05 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279251+0000", 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029363087999999999 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.031 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279317+0000", 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5535000000000002e-05 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3865 3.20 e97/18 3:0689c4f5:::100000001f8.00000115:head v 97'72, mlcod=94'70)", 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.748291316, 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032711928000000001, 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.032 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246817+0000", 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.27e-06 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.033 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252975+0000", 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006157872 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252993+0000", 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8070000000000001e-05 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253009+0000", 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6169999999999999e-05 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.034 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253090+0000", 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1312999999999999e-05 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253111+0000", 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279471+0000", 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026359693 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.035 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279525+0000", 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4530999999999997e-05 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4046 3.3a e97/18 3:5c489b63:::100000001f8.000001ca:head v 97'64, mlcod=97'62)", 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.736457377000001, 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037080704999999999, 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.036 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258649+0000", 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.637e-06 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.037 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262631+0000", 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039815079999999999 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262647+0000", 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6745999999999999e-05 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262663+0000", 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6116e-05 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.038 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262779+0000", 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011513700000000001 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262807+0000", 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8200000000000001e-05 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295660+0000", 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032853392000000002 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295728+0000", 2024-04-25T15:54:24.039 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7968999999999999e-05 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4727 3.1b e98/76 3:db35eb1b:::100000001f8.00000473:head v 98'76, mlcod=98'74)", 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.854012162, 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036260694000000003, 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.040 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141095+0000", 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9870000000000002e-06 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.041 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145531+0000", 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044364670000000004 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145560+0000", 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354999999999999e-05 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145578+0000", 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8715e-05 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.042 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145701+0000", 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012246800000000001 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145732+0000", 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1615000000000001e-05 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177281+0000", 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031549063000000002 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.043 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177353+0000", 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2024e-05 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4899 3.2b e98/40 3:d5e979f0:::100000001f8.0000051f:head v 98'72, mlcod=98'70)", 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.814838535, 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034607091, 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.044 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180267+0000", 2024-04-25T15:54:24.045 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6499999999999998e-07 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184620+0000", 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043528660000000004 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184641+0000", 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1126000000000001e-05 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184660+0000", 2024-04-25T15:54:24.046 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9014000000000001e-05 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184747+0000", 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6829999999999994e-05 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184776+0000", 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9473999999999999e-05 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.047 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214781+0000", 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030004797 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214873+0000", 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2219000000000006e-05 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5058 3.17 e99/40 3:eaf2c8b6:::100000001f8.000005be:head v 99'80, mlcod=99'78)", 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.980428976000001, 2024-04-25T15:54:24.048 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035602108, 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014679+0000", 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.095e-06 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021382+0000", 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067033750000000001 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021413+0000", 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0846000000000002e-05 2024-04-25T15:54:24.050 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021435+0000", 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1378000000000001e-05 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021583+0000", 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014862400000000001 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021620+0000", 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6718999999999999e-05 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.051 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050194+0000", 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028573676999999999 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050278+0000", 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4394000000000005e-05 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5611 3.20 e100/18 3:07896b3f:::100000001f8.000007e7:head v 100'100, mlcod=99'98)", 2024-04-25T15:54:24.052 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.911575917, 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034695030000000002, 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:24.053 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083531+0000", 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1179999999999999e-06 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092805+0000", 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.009273524 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.054 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092830+0000", 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5894e-05 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092847+0000", 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6762999999999999e-05 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092944+0000", 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6612000000000003e-05 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.055 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092974+0000", 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0568000000000003e-05 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118174+0000", 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025199813000000001 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118224+0000", 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9737999999999997e-05 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.056 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5680 3.10 e100/18 3:0bf23c77:::100000001f8.0000082c:head v 100'114, mlcod=100'112)", 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.506292876, 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039581641000000001, 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.057 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488821+0000", 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9060000000000008e-06 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494301+0000", 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054798900000000003 2024-04-25T15:54:24.058 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494321+0000", 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0271999999999998e-05 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494338+0000", 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249e-05 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494420+0000", 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2107999999999994e-05 2024-04-25T15:54:24.059 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494443+0000", 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3071e-05 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528334+0000", 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033890497999999998 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528394+0000", 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9647e-05 2024-04-25T15:54:24.060 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5986 3.10 e100/18 3:09f7ffbc:::100000001f8.0000095e:head v 100'117, mlcod=100'115)", 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7720072780000002, 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047845334000000003, 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223099+0000", 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:54:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230025+0000", 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069255319999999999 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230048+0000", 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3135000000000001e-05 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.063 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230062+0000", 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4119e-05 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230143+0000", 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0508999999999998e-05 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230165+0000", 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2314999999999999e-05 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.064 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270851+0000", 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040685763999999999 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270943+0000", 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2326999999999999e-05 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5978 3.b e100/21 3:d3280d2b:::100000001f8.00000956:head v 100'98, mlcod=100'96)", 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7640471259999995, 2024-04-25T15:54:24.065 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051283404999999997, 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.066 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231060+0000", 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1169999999999998e-06 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233300+0000", 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022402749999999999 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233324+0000", 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4261e-05 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.067 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233347+0000", 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.298e-05 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233488+0000", 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000140359 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233523+0000", 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5040000000000003e-05 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.068 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282285+0000", 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048762358999999998 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282341+0000", 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6014000000000003e-05 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5988 3.3d e100/40 3:bdc62163:::100000001f8.00000960:head v 100'102, mlcod=100'100)", 2024-04-25T15:54:24.069 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7550506779999999, 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043901969999999998, 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.070 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240056+0000", 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.674e-06 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242229+0000", 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021732269999999998 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242254+0000", 2024-04-25T15:54:24.071 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4661999999999999e-05 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242271+0000", 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.681e-05 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242362+0000", 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1411999999999998e-05 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242396+0000", 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3983000000000003e-05 2024-04-25T15:54:24.072 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283805+0000", 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041409488000000001 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283956+0000", 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015071400000000001 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.073 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5987 3.2b e100/40 3:d716bd7b:::100000001f8.0000095f:head v 100'88, mlcod=100'86)", 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 8.7509922650000007, 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038273439999999999, 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.074 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244115+0000", 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0020000000000001e-06 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248434+0000", 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043194920000000003 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.075 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248458+0000", 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3941e-05 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248477+0000", 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8995999999999999e-05 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248566+0000", 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9331000000000002e-05 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.076 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248596+0000", 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9332e-05 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282347+0000", 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033750952000000001 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282386+0000", 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9394000000000003e-05 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.077 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.078 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.078 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:24.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:54:24.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.029+0000 7f5165ffb640 1 -- 172.21.15.67:0/3168295890 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v105) ==== 91+0+0 (secure 0 0 0) 0x7f516c010040 con 0x7f517415d710 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f515806f400 msgr2=0x7f5158071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f515806f400 0x7f5158071910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f51700ae000 tx=0x7f51700a7000 comp rx=0 tx=0).stop 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517415d710 msgr2=0x7f51741f98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517415d710 0x7f51741f98e0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f516c009cb0 tx=0x7f516c00c490 comp rx=0 tx=0).stop 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 shutdown_connections 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f515806f400 0x7f5158071910 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f517415d710 0x7f51741f98e0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51740c5bb0 0x7f5174162a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 --2- 172.21.15.67:0/3168295890 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5174071b20 0x7f517414f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 >> 172.21.15.67:0/3168295890 conn(0x7f517406fe70 msgr2=0x7f5174073e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 shutdown_connections 2024-04-25T15:54:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:24.033+0000 7f517b78a640 1 -- 172.21.15.67:0/3168295890 wait complete. 2024-04-25T15:54:24.082 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:24.078+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:24.179 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:24.177+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:24.280 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:24.281+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:24.282 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:24.283 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:24.283 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:24.283 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:24.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:54:24.381 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:24.378+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:24.482 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:24.478+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:24.497 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:24.497 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:24.497 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:24.497 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:24.497 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:24.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:54:24.583 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:24.578+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 222.41699460999999, 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:54:24.665 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:54:24.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:676 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.734956279000002, 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053839007000000001, 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.671 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:24.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929776+0000", 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.743e-06 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931715+0000", 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019394130000000001 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931754+0000", 2024-04-25T15:54:24.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8779000000000003e-05 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931769+0000", 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5231999999999999e-05 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931867+0000", 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7485000000000006e-05 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932043+0000", 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017616100000000001 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932181+0000", 2024-04-25T15:54:24.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013769100000000001 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.942677+0000", 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010495931 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983477+0000", 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040800621000000002 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983546+0000", 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9138000000000001e-05 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983611+0000", 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4813000000000004e-05 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:733 3.22 3:44586a4d:::100000001f7.000000d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:24.677 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:24.678 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 60.356490231000002, 2024-04-25T15:54:24.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037465947999999999, 2024-04-25T15:54:24.678 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.678 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.678 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.678 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.678 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.679 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.679 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 733 2024-04-25T15:54:24.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.679 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308242+0000", 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.399e-06 2024-04-25T15:54:24.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309826+0000", 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015840349999999999 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309863+0000", 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7122000000000001e-05 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309880+0000", 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6888000000000002e-05 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309986+0000", 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105979 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310169+0000", 2024-04-25T15:54:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018349699999999999 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310353+0000", 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018415799999999999 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.342377+0000", 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032023169999999997 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345569+0000", 2024-04-25T15:54:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031928680000000002 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345644+0000", 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4140999999999994e-05 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345704+0000", 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0690999999999999e-05 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1589 3.1a e88/76 3:590c0ff7:::100000001f8.00000231:head v 88'30, mlcod=88'28)", 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.417905230000002, 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032177728000000003, 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246826+0000", 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1900000000000002e-06 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045171680000000002 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251364+0000", 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1444999999999999e-05 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251375+0000", 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.115e-05 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251458+0000", 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3190999999999997e-05 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251485+0000", 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6661000000000001e-05 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.278940+0000", 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027455222000000001 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.279001+0000", 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0701e-05 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1595 3.18 3:1adf79c2:::100000001f8.00000237:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.382544193000001, 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038955741000000002, 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1595 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282187+0000", 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1569999999999998e-06 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284089+0000", 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001902672 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284127+0000", 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7663000000000002e-05 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284141+0000", 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284236+0000", 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4895999999999994e-05 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284417+0000", 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018094500000000001 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284602+0000", 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018422800000000001 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.293831+0000", 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092292650000000004 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321052+0000", 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027220978 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321098+0000", 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6520000000000002e-05 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321140+0000", 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1850000000000001e-05 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2389 3.18 3:1bdc9869:::100000001f8.00000551:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.275492991, 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030905229999999999, 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2389 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389238+0000", 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7099999999999999e-06 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391140+0000", 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001901359 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391174+0000", 2024-04-25T15:54:24.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4165000000000002e-05 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391187+0000", 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3188e-05 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391277+0000", 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0267999999999997e-05 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391455+0000", 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017728700000000001 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391618+0000", 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163308 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.419720+0000", 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028102374999999999 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420001+0000", 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028060900000000002 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420083+0000", 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2010999999999998e-05 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420141+0000", 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7949999999999999e-05 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2392 3.32 3:4cdbb504:::100000001f8.00000554:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.258794233000003, 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048266518000000001, 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.701 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2392 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405937+0000", 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6759999999999999e-06 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408696+0000", 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027593589999999999 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.703 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408735+0000", 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8516000000000002e-05 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408749+0000", 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4117000000000001e-05 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408844+0000", 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4891999999999999e-05 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.704 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409024+0000", 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018000199999999999 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409200+0000", 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017591800000000001 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.427538+0000", 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018337728000000001 2024-04-25T15:54:24.705 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454069+0000", 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026531451000000001 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454141+0000", 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2316000000000004e-05 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.706 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454201+0000", 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9543000000000002e-05 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2393 3.22 3:45fb7032:::100000001f8.00000555:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.252401661999997, 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052582106000000003, 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.707 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2393 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.708 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412330+0000", 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3119999999999998e-06 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413928+0000", 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597521 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413971+0000", 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3086e-05 2024-04-25T15:54:24.709 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413988+0000", 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7336000000000001e-05 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414091+0000", 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103099 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414245+0000", 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015350300000000001 2024-04-25T15:54:24.710 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414359+0000", 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011410500000000001 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.446661+0000", 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032302571000000002 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464781+0000", 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018119283999999999 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464856+0000", 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5153000000000004e-05 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464909+0000", 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3136000000000001e-05 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.712 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2395 3.18 3:183ba26c:::100000001f8.00000557:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.240001554000003, 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029801055999999999, 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2395 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.713 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424735+0000", 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1780000000000003e-06 2024-04-25T15:54:24.714 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426303+0000", 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015679170000000001 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426349+0000", 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6343e-05 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.715 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426370+0000", 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329000000000001e-05 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426458+0000", 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.831e-05 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426609+0000", 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150629 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426735+0000", 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012625900000000001 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.441627+0000", 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014891824999999999 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454417+0000", 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012789994000000001 2024-04-25T15:54:24.717 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454476+0000", 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8752000000000003e-05 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454528+0000", 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2519999999999999e-05 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.718 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3145 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.230042820000001, 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030601595999999998, 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3145 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.719 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434689+0000", 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.005e-06 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.720 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436062+0000", 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001373682 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436096+0000", 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3797000000000002e-05 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436107+0000", 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.721 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436177+0000", 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0069999999999998e-05 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436346+0000", 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169168 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.722 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436510+0000", 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163807 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.443699+0000", 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007189394 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465141+0000", 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021441386999999999 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.723 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465223+0000", 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1920000000000002e-05 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465287+0000", 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4692999999999999e-05 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3841 3.18 3:1a4016cc:::100000001f8.000000fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.579045986000001, 2024-04-25T15:54:24.724 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043270667999999998, 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3841 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085686+0000", 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4510000000000001e-06 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088301+0000", 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002614987 2024-04-25T15:54:24.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088348+0000", 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7057e-05 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088362+0000", 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4352000000000001e-05 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088457+0000", 2024-04-25T15:54:24.727 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4794000000000001e-05 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088583+0000", 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012605400000000001 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088776+0000", 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019268700000000001 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.107828+0000", 2024-04-25T15:54:24.728 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019051702 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128788+0000", 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020960433000000001 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128888+0000", 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9493000000000004e-05 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128953+0000", 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5658000000000001e-05 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3977 3.18 3:1b05c35d:::100000001f8.00000185:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:24.730 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.822507404, 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13853472, 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3977 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:24.731 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842224+0000", 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1130000000000001e-06 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844135+0000", 2024-04-25T15:54:24.732 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019102920000000001 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844181+0000", 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6616000000000002e-05 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844202+0000", 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0542000000000001e-05 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844358+0000", 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156379 2024-04-25T15:54:24.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844555+0000", 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000196651 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844759+0000", 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020436799999999999 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.853781+0000", 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090217130000000006 2024-04-25T15:54:24.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980620+0000", 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.126839061 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980690+0000", 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9596000000000002e-05 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980756+0000", 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:54:24.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4099 3.18 3:181e4796:::100000001f8.000001ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.129435197999999, 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030713761999999999, 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4099 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.737 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535297+0000", 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2049999999999998e-06 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536942+0000", 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00164504 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536980+0000", 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8736999999999999e-05 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.738 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536997+0000", 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6653000000000001e-05 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537091+0000", 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3786e-05 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537191+0000", 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9840000000000006e-05 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.739 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537331+0000", 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014034900000000001 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.556229+0000", 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018898278000000001 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565921+0000", 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096916850000000002 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565961+0000", 2024-04-25T15:54:24.740 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0290999999999998e-05 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.566007+0000", 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5898e-05 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4812 3.18 3:194fa8d8:::100000001f8.000004c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.981020983000001, 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038646409, 2024-04-25T15:54:24.741 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4812 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.742 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683711+0000", 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7170000000000002e-06 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685822+0000", 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105619999999999 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685876+0000", 2024-04-25T15:54:24.743 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3798999999999998e-05 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685910+0000", 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4727999999999997e-05 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686022+0000", 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111549 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.744 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686150+0000", 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826800000000001 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686332+0000", 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181538 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.693171+0000", 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068389519999999997 2024-04-25T15:54:24.745 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722249+0000", 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029078171 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722294+0000", 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4756000000000003e-05 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722354+0000", 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0368999999999997e-05 2024-04-25T15:54:24.746 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4967 3.22 3:478cb4f4:::100000001f8.00000563:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.123087269999999, 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035932619999999998, 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4967 2024-04-25T15:54:24.747 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541672+0000", 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0978000000000001e-05 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.748 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543306+0000", 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016338310000000001 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543342+0000", 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5994000000000001e-05 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543360+0000", 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.749 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543470+0000", 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110319 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543624+0000", 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153549 2024-04-25T15:54:24.750 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543799+0000", 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017463299999999999 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.555562+0000", 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011763177999999999 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.751 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577476+0000", 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021913818000000002 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577529+0000", 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3711000000000001e-05 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577574+0000", 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4678999999999997e-05 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.752 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5705 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 11.040363806, 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080110828999999995, 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5705 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.753 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.754 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624368+0000", 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.117e-06 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626198+0000", 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001829968 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626234+0000", 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6220000000000002e-05 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626249+0000", 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4742999999999999e-05 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626341+0000", 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2477000000000002e-05 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626462+0000", 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121191 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626636+0000", 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017351000000000001 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.635055+0000", 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084194100000000004 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704341+0000", 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.069285804000000006 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.757 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704412+0000", 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0674999999999997e-05 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704476+0000", 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3713999999999993e-05 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5751 3.18 3:1b3fa6ee:::100000001f8.00000873:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:24.758 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.788297924, 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037922083000000002, 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5751 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:24.759 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876434+0000", 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878480+0000", 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020457800000000001 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878539+0000", 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8640000000000001e-05 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878587+0000", 2024-04-25T15:54:24.761 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7746e-05 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878724+0000", 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137572 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878868+0000", 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014331899999999999 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.762 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.879018+0000", 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015069000000000001 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.886482+0000", 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007464048 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914229+0000", 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027746804 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.763 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914300+0000", 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0952999999999996e-05 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914353+0000", 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.274e-05 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5755 3.32 3:4c40e2d2:::100000001f8.00000877:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:24.764 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 10.769261769, 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033152545999999998, 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5755 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895470+0000", 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7839999999999999e-06 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898117+0000", 2024-04-25T15:54:24.766 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026470959999999998 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898163+0000", 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6406000000000003e-05 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898183+0000", 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0092999999999999e-05 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.767 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898295+0000", 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011160800000000001 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898422+0000", 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012762099999999999 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898601+0000", 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017858500000000001 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914502+0000", 2024-04-25T15:54:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015900724000000001 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928454+0000", 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013952585999999999 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928552+0000", 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7299000000000005e-05 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928619+0000", 2024-04-25T15:54:24.769 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7743999999999994e-05 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5966 3.18 3:1b2b62aa:::100000001f8.0000094a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.5646183760000003, 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043485786999999998, 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.770 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5966 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:24.771 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100114+0000", 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.315e-06 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.101977+0000", 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018630109999999999 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.772 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102022+0000", 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5636000000000001e-05 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102040+0000", 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8e-05 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102140+0000", 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9720000000000001e-05 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102270+0000", 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130409 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102415+0000", 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144492 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.774 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.118110+0000", 2024-04-25T15:54:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015694725999999999 2024-04-25T15:54:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.775 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143459+0000", 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025349846999999998 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143534+0000", 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4839999999999995e-05 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.776 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143596+0000", 2024-04-25T15:54:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1791000000000005e-05 2024-04-25T15:54:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.777 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:6009 3.18 3:1b7f6e54:::100000001f8.00000975:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 9.2989485349999992, 2024-04-25T15:54:24.778 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030441923999999999, 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 6009 2024-04-25T15:54:24.779 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.780 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.781 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365782+0000", 2024-04-25T15:54:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3889999999999999e-06 2024-04-25T15:54:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367595+0000", 2024-04-25T15:54:24.782 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001812976 2024-04-25T15:54:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367633+0000", 2024-04-25T15:54:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7185999999999999e-05 2024-04-25T15:54:24.783 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367647+0000", 2024-04-25T15:54:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-04-25T15:54:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.784 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367735+0000", 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7474999999999999e-05 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367814+0000", 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9326000000000006e-05 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.785 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367936+0000", 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122077 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.375787+0000", 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078512010000000004 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.786 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396095+0000", 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02030732 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396164+0000", 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9549999999999996e-05 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396222+0000", 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7741999999999997e-05 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.787 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:24.788 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:24.788 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:24.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:54:24.790 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:24.786+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:24.890 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:24.889+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:24.933 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:24.933 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:24.933 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:24.933 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:24.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:54:24.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:54:24.997 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:24.997+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:25.092 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:25.090+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb03ddf0640 1 Processor -- start 2024-04-25T15:54:25.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb03ddf0640 1 -- start start 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb03ddf0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb0380736d0 0x7fb03806c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb03ddf0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0380caa60 con 0x7fb03806d460 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb03ddf0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb03815a5f0 con 0x7fb03806cb80 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb03ddf0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb038068590 con 0x7fb0380736d0 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0377fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb03806d460 0x7fb0381547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40496/0 (socket says 172.21.15.67:40496) 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0377fe640 1 -- 172.21.15.67:0/2241302951 learned_addr learned my addr 172.21.15.67:0/2241302951 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb036ffd640 1 --2- 172.21.15.67:0/2241302951 >> v2:172.21.15.176:3300/0 conn(0x7fb0380736d0 0x7fb03806c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2210511475 0 0) 0x7fb0380caa60 con 0x7fb03806d460 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb020003530 con 0x7fb03806d460 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1421866097 0 0) 0x7fb020003530 con 0x7fb03806d460 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 >> v2:172.21.15.176:3300/0 conn(0x7fb0380736d0 msgr2=0x7fb03806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 --2- 172.21.15.67:0/2241302951 >> v2:172.21.15.176:3300/0 conn(0x7fb0380736d0 0x7fb03806c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 >> v1:172.21.15.176:6789/0 conn(0x7fb03806cb80 legacy=0x7fb03815aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0380ca100 con 0x7fb03806d460 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fb028001680 con 0x7fb03806d460 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 >> v1:172.21.15.67:6789/0 conn(0x7fb03806d460 legacy=0x7fb0381547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0200039c0 0x7fb020003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb020004810 0x7fb020004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0200051d0 0x7fb02000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb038068590 con 0x7fb0200051d0 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb02000c170 con 0x7fb020004810 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb02000c370 con 0x7fb0200039c0 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb037fff640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0200051d0 0x7fb02000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb036ffd640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb020004810 0x7fb020004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb0377fe640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0200039c0 0x7fb020003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb037fff640 1 -- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0200039c0 msgr2=0x7fb020003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb037fff640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0200039c0 0x7fb020003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb037fff640 1 -- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb020004810 msgr2=0x7fb020004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb037fff640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb020004810 0x7fb020004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.137+0000 7fb037fff640 1 -- 172.21.15.67:0/2241302951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb020003530 con 0x7fb0200051d0 2024-04-25T15:54:25.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb037fff640 1 -- 172.21.15.67:0/2241302951 reap_dead start 2024-04-25T15:54:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb037fff640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0200051d0 0x7fb02000b9d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb038072870 tx=0x7fb03001c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f408d65b104253c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030001aa0 con 0x7fb0200051d0 2024-04-25T15:54:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb030003b80 con 0x7fb0200051d0 2024-04-25T15:54:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb0367fc640 1 -- 172.21.15.67:0/2241302951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0300050d0 con 0x7fb0200051d0 2024-04-25T15:54:25.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb03ddf0640 1 -- 172.21.15.67:0/2241302951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0200051d0 msgr2=0x7fb02000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0200051d0 0x7fb02000b9d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb038072870 tx=0x7fb03001c660 comp rx=0 tx=0).stop 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb03ddf0640 1 -- 172.21.15.67:0/2241302951 shutdown_connections 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/2241302951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0200051d0 0x7fb02000b9d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb03ddf0640 1 -- 172.21.15.67:0/2241302951 >> 172.21.15.67:0/2241302951 conn(0x7fb0380ce1c0 msgr2=0x7fb0380ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb03ddf0640 1 -- 172.21.15.67:0/2241302951 shutdown_connections 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.141+0000 7fb03ddf0640 1 -- 172.21.15.67:0/2241302951 wait complete. 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:25.144 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:25.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 Processor -- start 2024-04-25T15:54:25.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 -- start start 2024-04-25T15:54:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0200051d0 0x7fb0381f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038152bb0 0x7fb0381e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb038150970 0x7fb038150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0380ca340 con 0x7fb038152bb0 2024-04-25T15:54:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb038068ac0 con 0x7fb0200051d0 2024-04-25T15:54:25.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb03806c640 con 0x7fb038150970 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038152bb0 0x7fb0381e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038152bb0 0x7fb0381e3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54826/0 (socket says 172.21.15.67:54826) 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 -- 172.21.15.67:0/3969750794 learned_addr learned my addr 172.21.15.67:0/3969750794 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0200051d0 0x7fb0381f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb038150970 0x7fb038150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb038150970 msgr2=0x7fb038150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb038150970 0x7fb038150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0200051d0 msgr2=0x7fb0381f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0200051d0 0x7fb0381f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0380cba20 con 0x7fb038152bb0 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb038150970 0x7fb038150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb036ffd640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038152bb0 0x7fb0381e3a10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb02c011590 tx=0x7fb02c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c015870 con 0x7fb038152bb0 2024-04-25T15:54:25.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb02c015a10 con 0x7fb038152bb0 2024-04-25T15:54:25.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0380b2bd0 con 0x7fb038152bb0 2024-04-25T15:54:25.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.145+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c015dc0 con 0x7fb038152bb0 2024-04-25T15:54:25.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0380b3170 con 0x7fb038152bb0 2024-04-25T15:54:25.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb0381f2880 con 0x7fb038152bb0 2024-04-25T15:54:25.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb02c00f460 con 0x7fb038152bb0 2024-04-25T15:54:25.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb01ffff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb02406f430 0x7fb024071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6031+0+0 (secure 0 0 0) 0x7fb02c08fc80 con 0x7fb038152bb0 2024-04-25T15:54:25.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v105) ==== 54+0+16 (secure 0 0 0) 0x7fb0381f2880 con 0x7fb038152bb0 2024-04-25T15:54:25.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb02406f430 0x7fb024071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb0380b3450 0x7fb0381ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb03806a480 con 0x7fb0380b3450 2024-04-25T15:54:25.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb0380b3450 0x7fb0381ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.149+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb0380b3450 0x7fb0381ee3c0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.153+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb02406f430 0x7fb024071940 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb028000f70 tx=0x7fb028009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.153+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb03806a480 con 0x7fb0380b3450 2024-04-25T15:54:25.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:54:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.169+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb03815a3d0 con 0x7fb0380b3450 2024-04-25T15:54:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.169+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb03815a3d0 con 0x7fb0380b3450 2024-04-25T15:54:25.171 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:25.172 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:25.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.169+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb0381522d0 0x7fb0381eee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.169+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb038154980 con 0x7fb0381522d0 2024-04-25T15:54:25.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.169+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb0381522d0 0x7fb0381eee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.169+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb0381522d0 0x7fb0381eee10 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.173+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb038154980 con 0x7fb0381522d0 2024-04-25T15:54:25.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.189+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb038072d10 con 0x7fb0381522d0 2024-04-25T15:54:25.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.189+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb038072d10 con 0x7fb0381522d0 2024-04-25T15:54:25.191 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:25.191 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:25.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.189+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb0380635a0 0x7fb0380639b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.189+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb038153130 con 0x7fb0380635a0 2024-04-25T15:54:25.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.189+0000 7fb036ffd640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb0380635a0 0x7fb0380639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.189+0000 7fb036ffd640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb0380635a0 0x7fb0380639b0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.193+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb038153130 con 0x7fb0380635a0 2024-04-25T15:54:25.193 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:25.190+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:25.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.205+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb0380cac50 con 0x7fb0380635a0 2024-04-25T15:54:25.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.209+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb0380cac50 con 0x7fb0380635a0 2024-04-25T15:54:25.209 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:25.210 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:25.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.209+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb0380ccb90 0x7fb0380ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.209+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb0380cd4e0 con 0x7fb0380ccb90 2024-04-25T15:54:25.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.209+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb0380ccb90 0x7fb0380ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.209+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb0380ccb90 0x7fb0380ccfa0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.209+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0380cd4e0 con 0x7fb0380ccb90 2024-04-25T15:54:25.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.225+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb03802ccb0 con 0x7fb0380ccb90 2024-04-25T15:54:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.225+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb03802ccb0 con 0x7fb0380ccb90 2024-04-25T15:54:25.228 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.225+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb03802d3d0 0x7fb03802d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.225+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb03802dd20 con 0x7fb03802d3d0 2024-04-25T15:54:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.229+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb03802d3d0 0x7fb03802d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.229+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb03802d3d0 0x7fb03802d7e0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.229+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb03802dd20 con 0x7fb03802d3d0 2024-04-25T15:54:25.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.257+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb03802df60 con 0x7fb03802d3d0 2024-04-25T15:54:25.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.265+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb028025430 con 0x7fb03802d3d0 2024-04-25T15:54:25.265 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.265+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb038203200 0x7fb038205620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.265+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb0382085f0 con 0x7fb038203200 2024-04-25T15:54:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.265+0000 7fb036ffd640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb038203200 0x7fb038205620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.265+0000 7fb036ffd640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb038203200 0x7fb038205620 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.269+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0382085f0 con 0x7fb038203200 2024-04-25T15:54:25.294 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:25.293+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.297+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb0382072f0 con 0x7fb038203200 2024-04-25T15:54:25.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.301+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb0382072f0 con 0x7fb038203200 2024-04-25T15:54:25.303 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.301+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb038207a10 0x7fb038207e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.301+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb03820b9c0 con 0x7fb038207a10 2024-04-25T15:54:25.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.301+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb038207a10 0x7fb038207e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.301+0000 7fb037fff640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb038207a10 0x7fb038207e20 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.305+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb03820b9c0 con 0x7fb038207a10 2024-04-25T15:54:25.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.333+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb03820a840 con 0x7fb038207a10 2024-04-25T15:54:25.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.337+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb03820a840 con 0x7fb038207a10 2024-04-25T15:54:25.339 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:25.339 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:25.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.337+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb03820aec0 0x7fb03820b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:25.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.337+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb03820ed30 con 0x7fb03820aec0 2024-04-25T15:54:25.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.337+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb03820aec0 0x7fb03820b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:25.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.337+0000 7fb0377fe640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb03820aec0 0x7fb03820b2d0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:25.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.341+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb03820ed30 con 0x7fb03820aec0 2024-04-25T15:54:25.350 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:25.350 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:25.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.369+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb038205bb0 con 0x7fb03820aec0 2024-04-25T15:54:25.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb01ffff640 1 -- 172.21.15.67:0/3969750794 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb038205bb0 con 0x7fb03820aec0 2024-04-25T15:54:25.373 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb0380b3450 msgr2=0x7fb0381ee3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fb0380b3450 0x7fb0381ee3c0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb0381522d0 msgr2=0x7fb0381eee10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fb0381522d0 0x7fb0381eee10 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb0380635a0 msgr2=0x7fb0380639b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fb0380635a0 0x7fb0380639b0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb0380ccb90 msgr2=0x7fb0380ccfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fb0380ccb90 0x7fb0380ccfa0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb03802d3d0 msgr2=0x7fb03802d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb03802d3d0 0x7fb03802d7e0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb038203200 msgr2=0x7fb038205620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fb038203200 0x7fb038205620 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb038207a10 msgr2=0x7fb038207e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb038207a10 0x7fb038207e20 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb037fff640 1 -- 172.21.15.67:0/3969750794 reap_dead start 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb03820aec0 msgr2=0x7fb03820b2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb03820aec0 0x7fb03820b2d0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb037fff640 1 -- 172.21.15.67:0/3969750794 reap_dead start 2024-04-25T15:54:25.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb037fff640 1 -- 172.21.15.67:0/3969750794 reap_dead start 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb037fff640 1 -- 172.21.15.67:0/3969750794 reap_dead start 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb02406f430 msgr2=0x7fb024071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb02406f430 0x7fb024071940 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb028000f70 tx=0x7fb028009040 comp rx=0 tx=0).stop 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038152bb0 msgr2=0x7fb0381e3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038152bb0 0x7fb0381e3a10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb02c011590 tx=0x7fb02c011a50 comp rx=0 tx=0).stop 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 shutdown_connections 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fb03820aec0 0x7fb03820b2d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb02406f430 0x7fb024071940 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 --2- 172.21.15.67:0/3969750794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038152bb0 0x7fb0381e3a10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 >> 172.21.15.67:0/3969750794 conn(0x7fb0380ce1c0 msgr2=0x7fb038158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 shutdown_connections 2024-04-25T15:54:25.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:25.373+0000 7fb03ddf0640 1 -- 172.21.15.67:0/3969750794 wait complete. 2024-04-25T15:54:25.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:54:25.395 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:25.394+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:25.496 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:25.497+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:25.500 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:54:25.500 INFO:tasks.workunit.client.0.smithi067.stdout: Children see throughput for 1 re-readers = 6394247.50 kB/sec 2024-04-25T15:54:25.500 INFO:tasks.workunit.client.0.smithi067.stdout: Parent sees throughput for 1 re-readers = 6392180.74 kB/sec 2024-04-25T15:54:25.500 INFO:tasks.workunit.client.0.smithi067.stdout: Min throughput per process = 6394247.50 kB/sec 2024-04-25T15:54:25.500 INFO:tasks.workunit.client.0.smithi067.stdout: Max throughput per process = 6394247.50 kB/sec 2024-04-25T15:54:25.500 INFO:tasks.workunit.client.0.smithi067.stdout: Avg throughput per process = 6394247.50 kB/sec 2024-04-25T15:54:25.500 INFO:tasks.workunit.client.0.smithi067.stdout: Min xfer = 10485760.00 kB 2024-04-25T15:54:25.576 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:25.576 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:25.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:54:25.598 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:25.594+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:25.698 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:25.694+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:25.775 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:25.775 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:25.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:54:25.799 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:25.798+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:25.900 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:25.898+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:25.934 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:25.934 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:25.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:54:26.001 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:25.998+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:26.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:54:26.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:54:26.028 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-25T15:54:26.029 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-25T15:54:26.029 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-25T15:54:26.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-04-25T15:54:26.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:54:26.112 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:26.112 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:26.128 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:26.129+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:26.132 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:54:26.180 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:26.180 INFO:teuthology.orchestra.run.smithi176.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-25T15:54:26.180 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:26.202 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:26.201+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.237+0000 7fd616fcb640 1 Processor -- start 2024-04-25T15:54:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.237+0000 7fd616fcb640 1 -- start start 2024-04-25T15:54:26.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd616fcb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6100ca6d0 0x7fd61006e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:26.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd616fcb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6100ccef0 con 0x7fd6100ca2c0 2024-04-25T15:54:26.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd616fcb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6100cafa0 con 0x7fd61006e610 2024-04-25T15:54:26.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd616fcb640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fd61015ca20 con 0x7fd6100ca6d0 2024-04-25T15:54:26.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd60ffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6100ca6d0 0x7fd61006e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:26.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd614d67640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd6100ca2c0 0x7fd61015ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40498/0 (socket says 172.21.15.67:40498) 2024-04-25T15:54:26.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd614d67640 1 -- 172.21.15.67:0/976673168 learned_addr learned my addr 172.21.15.67:0/976673168 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:26.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd60ffff640 1 -- 172.21.15.67:0/976673168 >> v1:172.21.15.176:6789/0 conn(0x7fd61006e610 legacy=0x7fd610167040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:26.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd60ffff640 1 -- 172.21.15.67:0/976673168 >> v1:172.21.15.67:6789/0 conn(0x7fd6100ca2c0 legacy=0x7fd61015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:26.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd60ffff640 1 -- 172.21.15.67:0/976673168 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6100cb280 con 0x7fd6100ca6d0 2024-04-25T15:54:26.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd60f7fe640 1 -- 172.21.15.67:0/976673168 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 930792223 0 0) 0x7fd6100ccef0 con 0x7fd6100ca2c0 2024-04-25T15:54:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.241+0000 7fd60ffff640 1 --2- 172.21.15.67:0/976673168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6100ca6d0 0x7fd61006e0d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd6040097c0 tx=0x7fd60401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d471fd9e3dbed5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd60f7fe640 1 -- 172.21.15.67:0/976673168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6040b5070 con 0x7fd6100ca6d0 2024-04-25T15:54:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd60f7fe640 1 -- 172.21.15.67:0/976673168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6040a8990 con 0x7fd6100ca6d0 2024-04-25T15:54:26.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd60f7fe640 1 -- 172.21.15.67:0/976673168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6040a8d60 con 0x7fd6100ca6d0 2024-04-25T15:54:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 -- 172.21.15.67:0/976673168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6100ca6d0 msgr2=0x7fd61006e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 --2- 172.21.15.67:0/976673168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6100ca6d0 0x7fd61006e0d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd6040097c0 tx=0x7fd60401ec90 comp rx=0 tx=0).stop 2024-04-25T15:54:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 -- 172.21.15.67:0/976673168 shutdown_connections 2024-04-25T15:54:26.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 --2- 172.21.15.67:0/976673168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6100ca6d0 0x7fd61006e0d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 -- 172.21.15.67:0/976673168 >> 172.21.15.67:0/976673168 conn(0x7fd610073c50 msgr2=0x7fd610074060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 -- 172.21.15.67:0/976673168 shutdown_connections 2024-04-25T15:54:26.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 -- 172.21.15.67:0/976673168 wait complete. 2024-04-25T15:54:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 Processor -- start 2024-04-25T15:54:26.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.245+0000 7fd616fcb640 1 -- start start 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61006e610 0x7fd61015e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6100ca2c0 0x7fd610152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6100ca6d0 0x7fd610153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd61015d5c0 con 0x7fd61006e610 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd6100cb840 con 0x7fd6100ca2c0 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd61015d740 con 0x7fd6100ca6d0 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61006e610 0x7fd61015e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61006e610 0x7fd61015e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54838/0 (socket says 172.21.15.67:54838) 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 -- 172.21.15.67:0/541706807 learned_addr learned my addr 172.21.15.67:0/541706807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd60ffff640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6100ca2c0 0x7fd610152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd615568640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6100ca6d0 0x7fd610153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 -- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6100ca6d0 msgr2=0x7fd610153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:26.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6100ca6d0 0x7fd610153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 -- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6100ca2c0 msgr2=0x7fd610152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6100ca2c0 0x7fd610152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 -- 172.21.15.67:0/541706807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61015ee40 con 0x7fd61006e610 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd615568640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6100ca6d0 0x7fd610153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd614d67640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61006e610 0x7fd61015e900 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd60000c440 tx=0x7fd60000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd60d7fa640 1 -- 172.21.15.67:0/541706807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600018070 con 0x7fd61006e610 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd60d7fa640 1 -- 172.21.15.67:0/541706807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6000074e0 con 0x7fd61006e610 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61015e220 con 0x7fd61006e610 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd60d7fa640 1 -- 172.21.15.67:0/541706807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600004a10 con 0x7fd61006e610 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6101e59e0 con 0x7fd61006e610 2024-04-25T15:54:26.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd61006a480 con 0x7fd61006e610 2024-04-25T15:54:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd60d7fa640 1 -- 172.21.15.67:0/541706807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd60001b070 con 0x7fd61006e610 2024-04-25T15:54:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.249+0000 7fd60d7fa640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd5fc06f400 0x7fd5fc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.253+0000 7fd60d7fa640 1 -- 172.21.15.67:0/541706807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 6031+0+0 (secure 0 0 0) 0x7fd60008e8d0 con 0x7fd61006e610 2024-04-25T15:54:26.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.253+0000 7fd60ffff640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd5fc06f400 0x7fd5fc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:26.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.253+0000 7fd60ffff640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd5fc06f400 0x7fd5fc071910 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd6040b2000 tx=0x7fd604008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:26.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.257+0000 7fd60d7fa640 1 -- 172.21.15.67:0/541706807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd60005b110 con 0x7fd61006e610 2024-04-25T15:54:26.279 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:26.279 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:26.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:54:26.343 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:26.342+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:26.403 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:26.401+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:26.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:26.421+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fd61006dca0 con 0x7fd61006e610 2024-04-25T15:54:26.448 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:26.448 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:26.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:54:26.504 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:26.501+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:26.605 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:26.605+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:26.656 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:26.656 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:26.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:54:26.706 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:26.705+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:26.807 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:26.805+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:26.816 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:26.817 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:26.835 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:54:26.908 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:26.905+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:26.986 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:26.987 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:27.004 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:54:27.008 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:27.009+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:27.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.045+0000 7fd60d7fa640 1 -- 172.21.15.67:0/541706807 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v106) ==== 101+0+0 (secure 0 0 0) 0x7fd60005f410 con 0x7fd61006e610 2024-04-25T15:54:27.047 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:54:27.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd5fc06f400 msgr2=0x7fd5fc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd5fc06f400 0x7fd5fc071910 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd6040b2000 tx=0x7fd604008040 comp rx=0 tx=0).stop 2024-04-25T15:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61006e610 msgr2=0x7fd61015e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61006e610 0x7fd61015e900 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd60000c440 tx=0x7fd60000c900 comp rx=0 tx=0).stop 2024-04-25T15:54:27.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 shutdown_connections 2024-04-25T15:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd5fc06f400 0x7fd5fc071910 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6100ca6d0 0x7fd610153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6100ca2c0 0x7fd610152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 --2- 172.21.15.67:0/541706807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd61006e610 0x7fd61015e900 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 >> 172.21.15.67:0/541706807 conn(0x7fd610073c50 msgr2=0x7fd61015cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 shutdown_connections 2024-04-25T15:54:27.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.049+0000 7fd616fcb640 1 -- 172.21.15.67:0/541706807 wait complete. 2024-04-25T15:54:27.116 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:27.117+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:27.156 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:27.156 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:27.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:54:27.219 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:27.218+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:27.311 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:27.306+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:27.325 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:27.325 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:27.343 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:54:27.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:54:27.449 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:27.449+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:27.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:27.513+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:27.533 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:27.534 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:27.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:54:27.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b9a75640 1 Processor -- start 2024-04-25T15:54:27.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b9a75640 1 -- start start 2024-04-25T15:54:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b9a75640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f89b406c640 0x7f89b4154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b9a75640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4164f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b9a75640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89b40749d0 con 0x7f89b415e4a0 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b9a75640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f89b406e510 con 0x7f89b406c640 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b9a75640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89b415dca0 con 0x7f89b415e8b0 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b3fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4164f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b2ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f89b415e4a0 0x7f89b40c7fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:39062/0 (socket says 172.21.15.67:39062) 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b2ffd640 1 -- 172.21.15.67:0/2071666920 learned_addr learned my addr 172.21.15.67:0/2071666920 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.561+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2071666920 >> v2:172.21.15.176:3300/0 conn(0x7f89b406c640 0x7f89b4154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b27fc640 1 -- 172.21.15.67:0/2071666920 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648562724 0 0) 0x7f89b40749d0 con 0x7f89b415e4a0 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b27fc640 1 -- 172.21.15.67:0/2071666920 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89a0003530 con 0x7f89b415e4a0 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b3fff640 1 -- 172.21.15.67:0/2071666920 >> v2:172.21.15.176:3300/0 conn(0x7f89b406c640 msgr2=0x7f89b4154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2071666920 >> v2:172.21.15.176:3300/0 conn(0x7f89b406c640 0x7f89b4154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b3fff640 1 -- 172.21.15.67:0/2071666920 >> v1:172.21.15.176:6789/0 conn(0x7f89b415e4a0 legacy=0x7f89b40c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b3fff640 1 -- 172.21.15.67:0/2071666920 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b415df80 con 0x7f89b415e8b0 2024-04-25T15:54:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2071666920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4164f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89ac0097c0 tx=0x7f89ac01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3c5788cc3c86979 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b27fc640 1 -- 172.21.15.67:0/2071666920 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0b2070 con 0x7f89b415e8b0 2024-04-25T15:54:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b27fc640 1 -- 172.21.15.67:0/2071666920 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89ac0a9070 con 0x7f89b415e8b0 2024-04-25T15:54:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b27fc640 1 -- 172.21.15.67:0/2071666920 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0ad880 con 0x7f89b415e8b0 2024-04-25T15:54:27.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 -- 172.21.15.67:0/2071666920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 msgr2=0x7f89b4164f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2071666920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4164f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f89ac0097c0 tx=0x7f89ac01c410 comp rx=0 tx=0).stop 2024-04-25T15:54:27.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 -- 172.21.15.67:0/2071666920 shutdown_connections 2024-04-25T15:54:27.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2071666920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4164f90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2071666920 >> v2:172.21.15.176:3300/0 conn(0x7f89b406c640 0x7f89b4154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 -- 172.21.15.67:0/2071666920 >> 172.21.15.67:0/2071666920 conn(0x7f89b4070db0 msgr2=0x7f89b40ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:27.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 -- 172.21.15.67:0/2071666920 shutdown_connections 2024-04-25T15:54:27.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.565+0000 7f89b9a75640 1 -- 172.21.15.67:0/2071666920 wait complete. 2024-04-25T15:54:27.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.569+0000 7f89b9a75640 1 Processor -- start 2024-04-25T15:54:27.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.569+0000 7f89b9a75640 1 -- start start 2024-04-25T15:54:27.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.569+0000 7f89b9a75640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b406c640 0x7f89b4150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.569+0000 7f89b9a75640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415e4a0 0x7f89b4148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89b40cad60 con 0x7f89b415e4a0 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89b40c86d0 con 0x7f89b406c640 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89b4147970 con 0x7f89b415e8b0 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415e4a0 0x7f89b4148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b37fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b406c640 0x7f89b4150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415e4a0 0x7f89b4148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54846/0 (socket says 172.21.15.67:54846) 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 -- 172.21.15.67:0/2219983123 learned_addr learned my addr 172.21.15.67:0/2219983123 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 msgr2=0x7f89b4148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89b415e8b0 0x7f89b4148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b406c640 msgr2=0x7f89b4150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89b406c640 0x7f89b4150750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b41490d0 con 0x7f89b415e4a0 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b2ffd640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415e4a0 0x7f89b4148020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f899c00c390 tx=0x7f899c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899c017070 con 0x7f89b415e4a0 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f899c005d10 con 0x7f89b415e4a0 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b40ceb20 con 0x7f89b415e4a0 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899c007310 con 0x7f89b415e4a0 2024-04-25T15:54:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b4149670 con 0x7f89b415e4a0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f89b406a480 con 0x7f89b415e4a0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f899c01a070 con 0x7f89b415e4a0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89abfff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f430 0x7f8994071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6031+0+0 (secure 0 0 0) 0x7f899c08ee40 con 0x7f89b415e4a0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v106) ==== 54+0+16 (secure 0 0 0) 0x7f899c021070 con 0x7f89b415e4a0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f430 0x7f8994071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b415b350 0x7f89b40ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89b4151340 con 0x7f89b415b350 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.573+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b415b350 0x7f89b40ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.577+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f430 0x7f8994071940 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f89a4011340 tx=0x7f89a400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.577+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b415b350 0x7f89b40ca010 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.577+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b4151340 con 0x7f89b415b350 2024-04-25T15:54:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.589+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b4153fe0 con 0x7f89b415b350 2024-04-25T15:54:27.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.593+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b4153fe0 con 0x7f89b415b350 2024-04-25T15:54:27.594 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.593+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b4151a10 0x7f89b4142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.593+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f89b40c8340 con 0x7f89b4151a10 2024-04-25T15:54:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.593+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b4151a10 0x7f89b4142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.593+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b4151a10 0x7f89b4142350 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.593+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b40c8340 con 0x7f89b4151a10 2024-04-25T15:54:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.609+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b40008e0 con 0x7f89b4151a10 2024-04-25T15:54:27.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.613+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b40008e0 con 0x7f89b4151a10 2024-04-25T15:54:27.613 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.613+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.613+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f89b4151580 con 0x7f89b4063090 2024-04-25T15:54:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.613+0000 7f89b2ffd640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.613+0000 7f89b2ffd640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.615 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:27.610+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.613+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b4151580 con 0x7f89b4063090 2024-04-25T15:54:27.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.629+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b4150c90 con 0x7f89b4063090 2024-04-25T15:54:27.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.629+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b4150c90 con 0x7f89b4063090 2024-04-25T15:54:27.631 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.629+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b415b9e0 0x7f89b415bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.629+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f89b4063ef0 con 0x7f89b415b9e0 2024-04-25T15:54:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.629+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b415b9e0 0x7f89b415bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.629+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b415b9e0 0x7f89b415bdf0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.633+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b4063ef0 con 0x7f89b415b9e0 2024-04-25T15:54:27.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.645+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b415c600 con 0x7f89b415b9e0 2024-04-25T15:54:27.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.649+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b415c600 con 0x7f89b415b9e0 2024-04-25T15:54:27.650 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:27.651 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:27.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.649+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402cef0 0x7f89b402d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.649+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f89b415c580 con 0x7f89b402cef0 2024-04-25T15:54:27.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.649+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402cef0 0x7f89b402d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.649+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402cef0 0x7f89b402d300 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.653+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b415c580 con 0x7f89b402cef0 2024-04-25T15:54:27.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.665+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b415c330 con 0x7f89b402cef0 2024-04-25T15:54:27.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.669+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b415c330 con 0x7f89b402cef0 2024-04-25T15:54:27.671 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.669+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b4208030 0x7f89b420a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.669+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f89b415c540 con 0x7f89b4208030 2024-04-25T15:54:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.669+0000 7f89b2ffd640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b4208030 0x7f89b420a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.669+0000 7f89b2ffd640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b4208030 0x7f89b420a450 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.673+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b415c540 con 0x7f89b4208030 2024-04-25T15:54:27.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.689+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b415c540 con 0x7f89b4208030 2024-04-25T15:54:27.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.689+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b415c540 con 0x7f89b4208030 2024-04-25T15:54:27.692 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.689+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420aea0 0x7f89b420d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.689+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f89b402de90 con 0x7f89b420aea0 2024-04-25T15:54:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.693+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420aea0 0x7f89b420d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.693+0000 7f89b3fff640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420aea0 0x7f89b420d2c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.693+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b402de90 con 0x7f89b420aea0 2024-04-25T15:54:27.703 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:27.704 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:27.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.709+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b420d800 con 0x7f89b420aea0 2024-04-25T15:54:27.715 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:27.713+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:27.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.713+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b420d800 con 0x7f89b420aea0 2024-04-25T15:54:27.715 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:27.716 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:27.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.713+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b420dd10 0x7f89b4210130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.713+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f89b4213160 con 0x7f89b420dd10 2024-04-25T15:54:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.713+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b420dd10 0x7f89b4210130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.717+0000 7f89b37fe640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b420dd10 0x7f89b4210130 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:27.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.717+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89b4213160 con 0x7f89b420dd10 2024-04-25T15:54:27.722 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:54:27.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f89b4213130 con 0x7f89b420dd10 2024-04-25T15:54:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89abfff640 1 -- 172.21.15.67:0/2219983123 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f89b4213130 con 0x7f89b420dd10 2024-04-25T15:54:27.735 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b415b350 msgr2=0x7f89b40ca010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f89b415b350 0x7f89b40ca010 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b4151a10 msgr2=0x7f89b4142350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f89b4151a10 0x7f89b4142350 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 msgr2=0x7f89b40634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f89b4063090 0x7f89b40634a0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b415b9e0 msgr2=0x7f89b415bdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f89b415b9e0 0x7f89b415bdf0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b3fff640 1 -- 172.21.15.67:0/2219983123 reap_dead start 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402cef0 msgr2=0x7f89b402d300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f89b402cef0 0x7f89b402d300 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b3fff640 1 -- 172.21.15.67:0/2219983123 reap_dead start 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b4208030 msgr2=0x7f89b420a450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f89b4208030 0x7f89b420a450 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420aea0 msgr2=0x7f89b420d2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f89b420aea0 0x7f89b420d2c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b420dd10 msgr2=0x7f89b4210130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f89b420dd10 0x7f89b4210130 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f430 msgr2=0x7f8994071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f899406f430 0x7f8994071940 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f89a4011340 tx=0x7f89a400b040 comp rx=0 tx=0).stop 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415e4a0 msgr2=0x7f89b4148020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b9a75640 1 --2- 172.21.15.67:0/2219983123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b415e4a0 0x7f89b4148020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f899c00c390 tx=0x7f899c00c850 comp rx=0 tx=0).stop 2024-04-25T15:54:27.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.733+0000 7f89b3fff640 1 -- 172.21.15.67:0/2219983123 reap_dead start 2024-04-25T15:54:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.737+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 shutdown_connections 2024-04-25T15:54:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.737+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 >> 172.21.15.67:0/2219983123 conn(0x7f89b4070db0 msgr2=0x7f89b414fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.737+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 shutdown_connections 2024-04-25T15:54:27.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:27.737+0000 7f89b9a75640 1 -- 172.21.15.67:0/2219983123 wait complete. 2024-04-25T15:54:27.816 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:27.813+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:27.911 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:27.911 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:27.911 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:27.911 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:27.911 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:27.917 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:27.914+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:27.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:54:28.019 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:28.018+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:28.118 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:28.117+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:28.124 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 249.024034435, 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:54:28.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 94.810159901999995, 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.131 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.539481949999995, 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.137 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:54:28.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:54:28.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:54:28.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2266 3.11 3:89528692:::100000001f8.000004d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.551315234, 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.38717112199999998, 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2266 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.143 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571186+0000", 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4569999999999999e-06 2024-04-25T15:54:28.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784888+0000", 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.213702742 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785181+0000", 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029301099999999998 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785201+0000", 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9683000000000001e-05 2024-04-25T15:54:28.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785333+0000", 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013224600000000001 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785779+0000", 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044548299999999999 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.786106+0000", 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032678999999999998 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870593+0000", 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084487524999999994 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958234+0000", 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087640933000000004 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958283+0000", 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8473999999999997e-05 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958354+0000", 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1778000000000003e-05 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2272 3.13 3:ca3e88be:::100000001f8.000004dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:28.148 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.326566585999998, 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13173032000000001, 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2272 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795936+0000", 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.754e-06 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805167+0000", 2024-04-25T15:54:28.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092308890000000008 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805557+0000", 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039059400000000002 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805588+0000", 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0865999999999999e-05 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805759+0000", 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017085599999999999 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806172+0000", 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000413416 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806552+0000", 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000379417 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905757+0000", 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099205023000000003 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927568+0000", 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021810800000000002 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927616+0000", 2024-04-25T15:54:28.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927662+0000", 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6535000000000001e-05 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2273 3.39 3:9fcbcc01:::100000001f8.000004dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:28.155 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.317314471000003, 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15983650499999999, 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2273 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805188+0000", 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816847+0000", 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011659119000000001 2024-04-25T15:54:28.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817576+0000", 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00072975000000000004 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817601+0000", 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4450000000000001e-05 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817741+0000", 2024-04-25T15:54:28.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014058500000000001 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818074+0000", 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033291000000000001 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818385+0000", 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031034999999999998 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870618+0000", 2024-04-25T15:54:28.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052233743999999999 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964814+0000", 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094195216999999998 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964945+0000", 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013086799999999999 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965021+0000", 2024-04-25T15:54:28.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6032000000000007e-05 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2275 3.12 3:492b2588:::100000001f8.000004df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.305639092, 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14794684399999999, 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2275 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816863+0000", 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4149999999999999e-06 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829311+0000", 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012448222 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829569+0000", 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025766800000000001 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829598+0000", 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.968e-05 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829786+0000", 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187746 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830129+0000", 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034284099999999998 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830479+0000", 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034961500000000001 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867146+0000", 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036667733000000001 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964688+0000", 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097541577000000004 2024-04-25T15:54:28.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964748+0000", 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9626000000000001e-05 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964806+0000", 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8721000000000002e-05 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2278 3.2b 3:d79b9024:::100000001f8.000004e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.293173834000001, 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135647779, 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2278 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829328+0000", 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7309999999999999e-06 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843821+0000", 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014493012 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844787+0000", 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096513199999999995 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844876+0000", 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9147999999999994e-05 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844972+0000", 2024-04-25T15:54:28.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6566999999999999e-05 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845195+0000", 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222699 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845422+0000", 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022700699999999999 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870319+0000", 2024-04-25T15:54:28.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024897319000000001 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964892+0000", 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094572233000000006 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964938+0000", 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6646e-05 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964972+0000", 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4285000000000001e-05 2024-04-25T15:54:28.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2305 3.2f e90/51 3:f6648653:::100000001f8.000004fd:head v 90'33, mlcod=90'30)", 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 39.285240043999998, 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12780160600000001, 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837261+0000", 2024-04-25T15:54:28.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1679999999999998e-06 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839587+0000", 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00232607 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839620+0000", 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3503000000000001e-05 2024-04-25T15:54:28.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839642+0000", 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1302e-05 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839754+0000", 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112454 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839798+0000", 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4240999999999999e-05 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965003+0000", 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12520515600000001 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965060+0000", 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6712000000000002e-05 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3917 3.27 3:e652e8b8:::100000001f8.00000149:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.624429540000001, 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16896387500000001, 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.177 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3917 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498071+0000", 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4779999999999998e-06 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509296+0000", 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011224433000000001 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509572+0000", 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027576500000000003 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509591+0000", 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9466e-05 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509677+0000", 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5607999999999998e-05 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509773+0000", 2024-04-25T15:54:28.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6136000000000002e-05 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509942+0000", 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016951999999999999 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.610111+0000", 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.100168231 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666922+0000", 2024-04-25T15:54:28.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056811077000000001 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666991+0000", 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9758000000000005e-05 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667033+0000", 2024-04-25T15:54:28.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1403000000000003e-05 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3918 3.11 3:8b260724:::100000001f8.0000014a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.613193327000001, 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15748047100000001, 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.183 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3918 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509307+0000", 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.12e-06 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515132+0000", 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058250860000000002 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515164+0000", 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1563000000000003e-05 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515179+0000", 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5102e-05 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515262+0000", 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2917999999999995e-05 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515360+0000", 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7532000000000001e-05 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515512+0000", 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152442 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654409+0000", 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13889727499999999 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666644+0000", 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012234584999999999 2024-04-25T15:54:28.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666730+0000", 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5793999999999999e-05 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666786+0000", 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6054000000000002e-05 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3922 3.d 3:b3f0d00e:::100000001f8.0000014e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.603831406000001, 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.150212927, 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3922 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518670+0000", 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1039999999999998e-06 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529034+0000", 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010364237 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529073+0000", 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8407e-05 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529092+0000", 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9411000000000001e-05 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529203+0000", 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011113 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529330+0000", 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012611499999999999 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529517+0000", 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187639 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667032+0000", 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13751481300000001 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668779+0000", 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017469740000000001 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668835+0000", 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6251000000000001e-05 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668880+0000", 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4845999999999997e-05 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3920 3.26 e97/18 3:6422fb70:::100000001f8.0000014c:head v 97'93, mlcod=96'91)", 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.598808257999998, 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14317563999999999, 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.194 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523692+0000", 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8050000000000001e-06 2024-04-25T15:54:28.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525897+0000", 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022045290000000002 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525923+0000", 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6225e-05 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525941+0000", 2024-04-25T15:54:28.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8304e-05 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526045+0000", 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104158 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526076+0000", 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0454e-05 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666788+0000", 2024-04-25T15:54:28.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140712106 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666866+0000", 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8059000000000006e-05 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3926 3.d 3:b20dc027:::100000001f8.00000152:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.584321711000001, 2024-04-25T15:54:28.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131821096, 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3926 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538180+0000", 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4690000000000002e-06 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542940+0000", 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047600250000000002 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542988+0000", 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7633999999999998e-05 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543006+0000", 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8114999999999999e-05 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543103+0000", 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6723000000000002e-05 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543283+0000", 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018030100000000001 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543476+0000", 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019295200000000001 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668734+0000", 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12525763200000001 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669901+0000", 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011670890000000001 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669955+0000", 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4373999999999998e-05 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669998+0000", 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.203 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3971 3.3f 3:febcb5ea:::100000001f8.0000017f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.304797955000002, 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15452237999999999, 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3971 2024-04-25T15:54:28.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817703+0000", 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:54:28.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822518+0000", 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048147939999999998 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822563+0000", 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4515000000000003e-05 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822577+0000", 2024-04-25T15:54:28.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.432e-05 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822664+0000", 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7440000000000003e-05 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822786+0000", 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012110800000000001 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822960+0000", 2024-04-25T15:54:28.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017468900000000001 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945290+0000", 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122329907 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972057+0000", 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026767102000000001 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972139+0000", 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2070000000000005e-05 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972223+0000", 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3565000000000004e-05 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5059 3.26 e99/18 3:65b3bea5:::100000001f8.000005bf:head v 99'112, mlcod=99'109)", 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.105108783999999, 2024-04-25T15:54:28.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131194432, 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017392+0000", 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4650000000000001e-06 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021054+0000", 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036613240000000001 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021076+0000", 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2067000000000001e-05 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021092+0000", 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6209999999999999e-05 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021178+0000", 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6260000000000004e-05 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021208+0000", 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0046999999999999e-05 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148491+0000", 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12728336500000001 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148584+0000", 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2694000000000004e-05 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5063 3.13 3:c9096c28:::100000001f8.000005c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.093556628000002, 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13114490600000001, 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5063 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028946+0000", 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0520000000000003e-06 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035431+0000", 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064851509999999998 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035461+0000", 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9547999999999999e-05 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035474+0000", 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2880999999999999e-05 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035550+0000", 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6027999999999998e-05 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035622+0000", 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2366999999999993e-05 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035727+0000", 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105561 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.150510+0000", 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11478287099999999 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159903+0000", 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093925880000000003 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159983+0000", 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0122000000000003e-05 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160087+0000", 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103737 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5730 3.e 3:7357dd18:::100000001f8.0000085e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.363260409, 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14322902300000001, 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.219 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5730 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759240+0000", 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3910000000000001e-06 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765109+0000", 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058681660000000002 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765139+0000", 2024-04-25T15:54:28.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9981e-05 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765149+0000", 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0125e-05 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765215+0000", 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6700999999999997e-05 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765293+0000", 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7883e-05 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765394+0000", 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100869 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.893428+0000", 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128033758 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902331+0000", 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089030610000000003 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902403+0000", 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1891999999999995e-05 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902467+0000", 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4196000000000007e-05 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.224 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5769 3.d 3:b350774d:::100000001f8.00000885:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.130542664, 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12752495899999999, 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.225 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5769 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991958+0000", 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998738+0000", 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067804069999999996 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998782+0000", 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3738e-05 2024-04-25T15:54:28.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998800+0000", 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7907e-05 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998944+0000", 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144089 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999140+0000", 2024-04-25T15:54:28.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195465 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999415+0000", 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027557100000000001 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.070872+0000", 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071456602999999994 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119350+0000", 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048478354000000001 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119421+0000", 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0970999999999995e-05 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119481+0000", 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9802000000000001e-05 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.230 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5780 3.f 3:f20b812e:::100000001f8.00000890:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.082361585999999, 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13282865699999999, 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5780 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.231 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040139+0000", 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9570000000000001e-06 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047681+0000", 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075419279999999998 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047720+0000", 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8794999999999997e-05 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047740+0000", 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0091000000000001e-05 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047828+0000", 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8745000000000005e-05 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047922+0000", 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3076000000000001e-05 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:28.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048095+0000", 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000173134 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.170485+0000", 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122390033 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172900+0000", 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002415482 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172933+0000", 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3021000000000001e-05 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172966+0000", 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2394999999999997e-05 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:28.236 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:28.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:54:28.239 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:28.237+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:28.339 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:28.338+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:28.403 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:28.403 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:28.403 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:28.404 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:28.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:54:28.440 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:28.438+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:28.541 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:28.541+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:28.568 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:28.568 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:28.568 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:28.568 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:28.569 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:28.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:54:28.676 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:28.677+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:28.742 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:28.738+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:28.787 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:28.787 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:28.787 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:28.787 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:28.787 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.787 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:668 3.1 e81/76 3:83ba1de1:::100000001f7.00000098:head v 81'14, mlcod=81'12)", 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.913993684000005, 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.045409913000000003, 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.788 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872468+0000", 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3659999999999999e-06 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879062+0000", 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065941039999999999 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.789 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879116+0000", 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4372000000000001e-05 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879152+0000", 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6139000000000001e-05 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879318+0000", 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016595499999999999 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879365+0000", 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6897999999999997e-05 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917803+0000", 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038437773000000001 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917875+0000", 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2305999999999995e-05 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.791 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:637 3.20 e81/18 3:07263727:::100000001f7.00000079:head v 81'17, mlcod=81'15)", 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 64.904505193000006, 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040244958999999997, 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:28.792 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881957+0000", 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3890000000000001e-06 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888368+0000", 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006410682 2024-04-25T15:54:28.793 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888395+0000", 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7477000000000001e-05 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888414+0000", 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9046e-05 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888512+0000", 2024-04-25T15:54:28.794 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7876999999999998e-05 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888546+0000", 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3204000000000003e-05 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922131+0000", 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033585168999999998 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922199+0000", 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8114999999999994e-05 2024-04-25T15:54:28.795 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1709 3.b e89/21 3:d0c4adc9:::100000001f8.000002a9:head v 89'35, mlcod=89'33)", 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.787529206000002, 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041562524000000003, 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.796 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998932+0000", 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7570000000000001e-06 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.797 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000395+0000", 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014629389999999999 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000414+0000", 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9908999999999999e-05 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000432+0000", 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249999999999999e-05 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.798 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000518+0000", 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6501000000000004e-05 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000548+0000", 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9777e-05 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040424+0000", 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039875586999999997 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.799 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040492+0000", 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8804000000000001e-05 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1710 3.b e89/21 3:d079ce42:::100000001f8.000002aa:head v 89'36, mlcod=89'33)", 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 43.781100488999996, 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040049727, 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.801 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005360+0000", 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9290000000000001e-06 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006358+0000", 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099724699999999993 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006377+0000", 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9656e-05 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.802 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006392+0000", 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4207e-05 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006477+0000", 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4989999999999998e-05 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006507+0000", 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0786e-05 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.803 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045329+0000", 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038821829000000002 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045408+0000", 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9083000000000002e-05 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.804 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2273 3.39 e90/18 3:9fcbcc01:::100000001f8.000004dd:head v 90'54, mlcod=90'52)", 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.960951244, 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035128116000000001, 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.805 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825510+0000", 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837142+0000", 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011632136 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.806 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837175+0000", 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3166000000000001e-05 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837197+0000", 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1495e-05 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837293+0000", 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6353000000000004e-05 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837329+0000", 2024-04-25T15:54:28.807 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5901000000000001e-05 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860257+0000", 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022927804999999999 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860636+0000", 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000379327 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.808 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2392 3.32 e90/40 3:4cdbb504:::100000001f8.00000554:head v 90'45, mlcod=90'43)", 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.377136483000001, 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044535299, 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.809 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409325+0000", 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2220000000000001e-06 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410846+0000", 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001521454 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.810 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410866+0000", 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0129999999999999e-05 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410882+0000", 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5452e-05 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410962+0000", 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9833999999999995e-05 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.811 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410991+0000", 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.968e-05 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453793+0000", 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042801621999999998 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453858+0000", 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4905000000000003e-05 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2394 3.6 e90/76 3:63a5c66e:::100000001f8.00000556:head v 90'46, mlcod=90'44)", 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.362441359999998, 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038940880999999997, 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.813 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424019+0000", 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6160000000000001e-06 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428140+0000", 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041208529999999998 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.814 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428169+0000", 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8824999999999999e-05 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428190+0000", 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0894999999999999e-05 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428283+0000", 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3277999999999997e-05 2024-04-25T15:54:28.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428318+0000", 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5191000000000002e-05 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462890+0000", 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034571283000000001 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462959+0000", 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000001e-05 2024-04-25T15:54:28.816 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2396 3.3d e90/40 3:bc4ba9bc:::100000001f8.00000558:head v 90'44, mlcod=90'42)", 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 39.352636138999998, 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033872159999999998, 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.817 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433825+0000", 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.951e-06 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.818 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435239+0000", 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014141729999999999 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435260+0000", 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0775e-05 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435278+0000", 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8317000000000001e-05 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.819 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435363+0000", 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4441e-05 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435395+0000", 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1939e-05 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467549+0000", 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032154061999999997 2024-04-25T15:54:28.820 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467695+0000", 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014650199999999999 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3866 3.1 e97/76 3:81b781a2:::100000001f8.00000116:head v 97'69, mlcod=96'67)", 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.542454202000002, 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035311905999999997, 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.821 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.822 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244007+0000", 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249687+0000", 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056798860000000003 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249709+0000", 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2393999999999999e-05 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.823 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249729+0000", 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9536000000000001e-05 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249849+0000", 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120068 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249888+0000", 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9103999999999997e-05 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.824 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279251+0000", 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029363087999999999 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279317+0000", 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5535000000000002e-05 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3865 3.20 e97/18 3:0689c4f5:::100000001f8.00000115:head v 97'72, mlcod=94'70)", 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:28.825 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.539645504999999, 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032711928000000001, 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.826 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246817+0000", 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.27e-06 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252975+0000", 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006157872 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.827 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252993+0000", 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8070000000000001e-05 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253009+0000", 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6169999999999999e-05 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253090+0000", 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1312999999999999e-05 2024-04-25T15:54:28.828 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253111+0000", 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279471+0000", 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026359693 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279525+0000", 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4530999999999997e-05 2024-04-25T15:54:28.829 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4046 3.3a e97/18 3:5c489b63:::100000001f8.000001ca:head v 97'64, mlcod=97'62)", 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.527811566, 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037080704999999999, 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.830 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258649+0000", 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.637e-06 2024-04-25T15:54:28.831 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262631+0000", 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039815079999999999 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262647+0000", 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6745999999999999e-05 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262663+0000", 2024-04-25T15:54:28.832 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6116e-05 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262779+0000", 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011513700000000001 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262807+0000", 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8200000000000001e-05 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.833 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295660+0000", 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032853392000000002 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295728+0000", 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7968999999999999e-05 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.834 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4727 3.1b e98/76 3:db35eb1b:::100000001f8.00000473:head v 98'76, mlcod=98'74)", 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.645366351, 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036260694000000003, 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.835 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141095+0000", 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9870000000000002e-06 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145531+0000", 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044364670000000004 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.836 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145560+0000", 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354999999999999e-05 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145578+0000", 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8715e-05 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145701+0000", 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012246800000000001 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145732+0000", 2024-04-25T15:54:28.837 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1615000000000001e-05 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177281+0000", 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031549063000000002 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177353+0000", 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2024e-05 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.838 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4899 3.2b e98/40 3:d5e979f0:::100000001f8.0000051f:head v 98'72, mlcod=98'70)", 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.606192724, 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034607091, 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.839 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180267+0000", 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6499999999999998e-07 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.840 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184620+0000", 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043528660000000004 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184641+0000", 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1126000000000001e-05 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184660+0000", 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9014000000000001e-05 2024-04-25T15:54:28.841 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184747+0000", 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6829999999999994e-05 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184776+0000", 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9473999999999999e-05 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214781+0000", 2024-04-25T15:54:28.842 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030004797 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214873+0000", 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2219000000000006e-05 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5058 3.17 e99/40 3:eaf2c8b6:::100000001f8.000005be:head v 99'80, mlcod=99'78)", 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.771783164999999, 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035602108, 2024-04-25T15:54:28.843 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.844 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014679+0000", 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.095e-06 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021382+0000", 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067033750000000001 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021413+0000", 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0846000000000002e-05 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.845 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021435+0000", 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1378000000000001e-05 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021583+0000", 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014862400000000001 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021620+0000", 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6718999999999999e-05 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.846 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050194+0000", 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028573676999999999 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050278+0000", 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4394000000000005e-05 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5611 3.20 e100/18 3:07896b3f:::100000001f8.000007e7:head v 100'100, mlcod=99'98)", 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:28.847 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.702930106, 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034695030000000002, 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083531+0000", 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1179999999999999e-06 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092805+0000", 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.009273524 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092830+0000", 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5894e-05 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092847+0000", 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6762999999999999e-05 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092944+0000", 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6612000000000003e-05 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092974+0000", 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0568000000000003e-05 2024-04-25T15:54:28.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118174+0000", 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025199813000000001 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118224+0000", 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9737999999999997e-05 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5680 3.10 e100/18 3:0bf23c77:::100000001f8.0000082c:head v 100'114, mlcod=100'112)", 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.297647065, 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039581641000000001, 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.852 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488821+0000", 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9060000000000008e-06 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494301+0000", 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054798900000000003 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.853 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494321+0000", 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0271999999999998e-05 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494338+0000", 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249e-05 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.854 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494420+0000", 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2107999999999994e-05 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494443+0000", 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3071e-05 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528334+0000", 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033890497999999998 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.855 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528394+0000", 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9647e-05 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5986 3.10 e100/18 3:09f7ffbc:::100000001f8.0000095e:head v 100'117, mlcod=100'115)", 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.563361467, 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047845334000000003, 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.856 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223099+0000", 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230025+0000", 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069255319999999999 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230048+0000", 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3135000000000001e-05 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230062+0000", 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4119e-05 2024-04-25T15:54:28.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230143+0000", 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0508999999999998e-05 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230165+0000", 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2314999999999999e-05 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270851+0000", 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040685763999999999 2024-04-25T15:54:28.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270943+0000", 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2326999999999999e-05 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5978 3.b e100/21 3:d3280d2b:::100000001f8.00000956:head v 100'98, mlcod=100'96)", 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.555401314999999, 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051283404999999997, 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.860 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.861 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231060+0000", 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1169999999999998e-06 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233300+0000", 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022402749999999999 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233324+0000", 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4261e-05 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.862 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233347+0000", 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.298e-05 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233488+0000", 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000140359 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233523+0000", 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5040000000000003e-05 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.863 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282285+0000", 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048762358999999998 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282341+0000", 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6014000000000003e-05 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5988 3.3d e100/40 3:bdc62163:::100000001f8.00000960:head v 100'102, mlcod=100'100)", 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:28.864 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.546404867, 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043901969999999998, 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240056+0000", 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.674e-06 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242229+0000", 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021732269999999998 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242254+0000", 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4661999999999999e-05 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.866 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242271+0000", 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.681e-05 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242362+0000", 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1411999999999998e-05 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242396+0000", 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3983000000000003e-05 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.867 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283805+0000", 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041409488000000001 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283956+0000", 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015071400000000001 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5987 3.2b e100/40 3:d716bd7b:::100000001f8.0000095f:head v 100'88, mlcod=100'86)", 2024-04-25T15:54:28.868 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 13.542346454, 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038273439999999999, 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:28.869 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244115+0000", 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0020000000000001e-06 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248434+0000", 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043194920000000003 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248458+0000", 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3941e-05 2024-04-25T15:54:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248477+0000", 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8995999999999999e-05 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248566+0000", 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9331000000000002e-05 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248596+0000", 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9332e-05 2024-04-25T15:54:28.871 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282347+0000", 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033750952000000001 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282386+0000", 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9394000000000003e-05 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:28.872 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:28.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:54:28.874 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:28.873+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:28.974 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:28.973+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:29.074 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:29.075 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:29.075 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:29.075 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:29.075 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:29.073+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:29.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-25T15:54:29.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:54:29.188 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:29.189+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff92e3c7640 1 Processor -- start 2024-04-25T15:54:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff92e3c7640 1 -- start start 2024-04-25T15:54:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff92e3c7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff928160af0 0x7ff9280c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff92e3c7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9280cb800 con 0x7ff928156d30 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff92e3c7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9280cbae0 con 0x7ff9280c5720 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff92e3c7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff9280cb5b0 con 0x7ff928160af0 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff927fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff928156d30 0x7ff9280ca540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40504/0 (socket says 172.21.15.67:40504) 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff927fff640 1 -- 172.21.15.67:0/2344165285 learned_addr learned my addr 172.21.15.67:0/2344165285 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff92c964640 1 --2- 172.21.15.67:0/2344165285 >> v2:172.21.15.176:3300/0 conn(0x7ff928160af0 0x7ff9280c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2349047248 0 0) 0x7ff9280cb800 con 0x7ff928156d30 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff914003530 con 0x7ff928156d30 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2871457208 0 0) 0x7ff914003530 con 0x7ff928156d30 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 >> v2:172.21.15.176:3300/0 conn(0x7ff928160af0 msgr2=0x7ff9280c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 --2- 172.21.15.67:0/2344165285 >> v2:172.21.15.176:3300/0 conn(0x7ff928160af0 0x7ff9280c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 >> v1:172.21.15.176:6789/0 conn(0x7ff9280c5720 legacy=0x7ff9281603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9280cbdc0 con 0x7ff928156d30 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7ff918001680 con 0x7ff928156d30 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 >> v1:172.21.15.67:6789/0 conn(0x7ff928156d30 legacy=0x7ff9280ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9140039c0 0x7ff914003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff914004810 0x7ff914004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff91400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280cb5b0 con 0x7ff914005280 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff91400c190 con 0x7ff914004810 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.229+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff91400c390 con 0x7ff9140039c0 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff91400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff91f7fe640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff914004810 0x7ff914004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff927fff640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9140039c0 0x7ff914003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 -- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9140039c0 msgr2=0x7ff914003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9140039c0 0x7ff914003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 -- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff914004810 msgr2=0x7ff914004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff914004810 0x7ff914004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 -- 172.21.15.67:0/2344165285 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff914003530 con 0x7ff914005280 2024-04-25T15:54:29.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 -- 172.21.15.67:0/2344165285 reap_dead start 2024-04-25T15:54:29.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92c964640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff91400b9d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff920009f90 tx=0x7ff92001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bf4fc44f1f04ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920009c10 con 0x7ff914005280 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9200a5c90 con 0x7ff914005280 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff9277fe640 1 -- 172.21.15.67:0/2344165285 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920003be0 con 0x7ff914005280 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2344165285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 msgr2=0x7ff91400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff91400b9d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff920009f90 tx=0x7ff92001cc80 comp rx=0 tx=0).stop 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2344165285 shutdown_connections 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2344165285 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff91400b9d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2344165285 >> 172.21.15.67:0/2344165285 conn(0x7ff9280700b0 msgr2=0x7ff9280704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2344165285 shutdown_connections 2024-04-25T15:54:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.233+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2344165285 wait complete. 2024-04-25T15:54:29.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 Processor -- start 2024-04-25T15:54:29.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 -- start start 2024-04-25T15:54:29.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff9281e5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9280d00f0 0x7ff9281f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9280d0500 0x7ff9281f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff9280cc380 con 0x7ff914005280 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff92815ff50 con 0x7ff9280d00f0 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff92e3c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff9280c5b30 con 0x7ff9280d0500 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff927fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff9281e5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff927fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff9281e5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54870/0 (socket says 172.21.15.67:54870) 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.237+0000 7ff927fff640 1 -- 172.21.15.67:0/2398217530 learned_addr learned my addr 172.21.15.67:0/2398217530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff91f7fe640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9280d00f0 0x7ff9281f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff92c964640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9280d0500 0x7ff9281f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff927fff640 1 -- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9280d0500 msgr2=0x7ff9281f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff927fff640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9280d0500 0x7ff9281f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff927fff640 1 -- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9280d00f0 msgr2=0x7ff9281f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff927fff640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9280d00f0 0x7ff9281f6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff927fff640 1 -- 172.21.15.67:0/2398217530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281e61c0 con 0x7ff914005280 2024-04-25T15:54:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff91f7fe640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9280d00f0 0x7ff9281f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff927fff640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff9281e5c80 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff91800e800 tx=0x7ff91800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff9257fa640 1 -- 172.21.15.67:0/2398217530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91800c6c0 con 0x7ff914005280 2024-04-25T15:54:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff9257fa640 1 -- 172.21.15.67:0/2398217530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff918018070 con 0x7ff914005280 2024-04-25T15:54:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff9257fa640 1 -- 172.21.15.67:0/2398217530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91800caa0 con 0x7ff914005280 2024-04-25T15:54:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92815edd0 con 0x7ff914005280 2024-04-25T15:54:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92815f070 con 0x7ff914005280 2024-04-25T15:54:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92806a480 con 0x7ff914005280 2024-04-25T15:54:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff9257fa640 1 -- 172.21.15.67:0/2398217530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff91800cc40 con 0x7ff914005280 2024-04-25T15:54:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff9257fa640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff90806f430 0x7ff908071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff9257fa640 1 -- 172.21.15.67:0/2398217530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6031+0+0 (secure 0 0 0) 0x7ff91808ed30 con 0x7ff914005280 2024-04-25T15:54:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.241+0000 7ff91f7fe640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff90806f430 0x7ff908071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.245+0000 7ff91f7fe640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff90806f430 0x7ff908071940 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff910000c10 tx=0x7ff910009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.249+0000 7ff9257fa640 1 -- 172.21.15.67:0/2398217530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff91805bc00 con 0x7ff914005280 2024-04-25T15:54:29.276 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:29.274+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:29.285 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:29.285 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:29.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:54:29.377 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:29.374+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:29.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.417+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7ff928071e00 con 0x7ff914005280 2024-04-25T15:54:29.478 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:29.477+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 227.249241622, 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:54:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.500 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:54:29.501 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.502 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:54:29.503 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.504 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:676 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.567203290999998, 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053839007000000001, 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:29.505 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929776+0000", 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.743e-06 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931715+0000", 2024-04-25T15:54:29.506 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019394130000000001 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931754+0000", 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8779000000000003e-05 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931769+0000", 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5231999999999999e-05 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931867+0000", 2024-04-25T15:54:29.507 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7485000000000006e-05 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932043+0000", 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017616100000000001 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932181+0000", 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013769100000000001 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.942677+0000", 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010495931 2024-04-25T15:54:29.508 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983477+0000", 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040800621000000002 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983546+0000", 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9138000000000001e-05 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983611+0000", 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4813000000000004e-05 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.509 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:733 3.22 3:44586a4d:::100000001f7.000000d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 65.188737243000006, 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037465947999999999, 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 733 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.510 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308242+0000", 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.399e-06 2024-04-25T15:54:29.511 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309826+0000", 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015840349999999999 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309863+0000", 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7122000000000001e-05 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309880+0000", 2024-04-25T15:54:29.512 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6888000000000002e-05 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309986+0000", 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105979 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310169+0000", 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018349699999999999 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310353+0000", 2024-04-25T15:54:29.513 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018415799999999999 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.342377+0000", 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032023169999999997 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345569+0000", 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031928680000000002 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345644+0000", 2024-04-25T15:54:29.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4140999999999994e-05 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345704+0000", 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0690999999999999e-05 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1589 3.1a e88/76 3:590c0ff7:::100000001f8.00000231:head v 88'30, mlcod=88'28)", 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.250152241999999, 2024-04-25T15:54:29.515 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032177728000000003, 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.516 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246826+0000", 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1900000000000002e-06 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045171680000000002 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251364+0000", 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1444999999999999e-05 2024-04-25T15:54:29.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251375+0000", 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.115e-05 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251458+0000", 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3190999999999997e-05 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251485+0000", 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6661000000000001e-05 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.278940+0000", 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027455222000000001 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.279001+0000", 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0701e-05 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1595 3.18 3:1adf79c2:::100000001f8.00000237:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.214791204999997, 2024-04-25T15:54:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038955741000000002, 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1595 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282187+0000", 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1569999999999998e-06 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284089+0000", 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001902672 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284127+0000", 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7663000000000002e-05 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284141+0000", 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284236+0000", 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4895999999999994e-05 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284417+0000", 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018094500000000001 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284602+0000", 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018422800000000001 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.293831+0000", 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092292650000000004 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321052+0000", 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027220978 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321098+0000", 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6520000000000002e-05 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321140+0000", 2024-04-25T15:54:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1850000000000001e-05 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2389 3.18 3:1bdc9869:::100000001f8.00000551:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.107740003000004, 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030905229999999999, 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2389 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389238+0000", 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7099999999999999e-06 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391140+0000", 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001901359 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391174+0000", 2024-04-25T15:54:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4165000000000002e-05 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391187+0000", 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3188e-05 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391277+0000", 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0267999999999997e-05 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391455+0000", 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017728700000000001 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391618+0000", 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163308 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.419720+0000", 2024-04-25T15:54:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028102374999999999 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420001+0000", 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028060900000000002 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420083+0000", 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2010999999999998e-05 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420141+0000", 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7949999999999999e-05 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2392 3.32 3:4cdbb504:::100000001f8.00000554:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.091041245, 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048266518000000001, 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2392 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405937+0000", 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6759999999999999e-06 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408696+0000", 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027593589999999999 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408735+0000", 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8516000000000002e-05 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408749+0000", 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4117000000000001e-05 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408844+0000", 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4891999999999999e-05 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409024+0000", 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018000199999999999 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409200+0000", 2024-04-25T15:54:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017591800000000001 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.427538+0000", 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018337728000000001 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454069+0000", 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026531451000000001 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454141+0000", 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2316000000000004e-05 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454201+0000", 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9543000000000002e-05 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2393 3.22 3:45fb7032:::100000001f8.00000555:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.084648674, 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052582106000000003, 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2393 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412330+0000", 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3119999999999998e-06 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413928+0000", 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597521 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413971+0000", 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3086e-05 2024-04-25T15:54:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413988+0000", 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7336000000000001e-05 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414091+0000", 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103099 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414245+0000", 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015350300000000001 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414359+0000", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011410500000000001 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.446661+0000", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032302571000000002 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464781+0000", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018119283999999999 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464856+0000", 2024-04-25T15:54:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5153000000000004e-05 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464909+0000", 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3136000000000001e-05 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2395 3.18 3:183ba26c:::100000001f8.00000557:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.072248565999999, 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029801055999999999, 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2395 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424735+0000", 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1780000000000003e-06 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426303+0000", 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015679170000000001 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426349+0000", 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6343e-05 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426370+0000", 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329000000000001e-05 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426458+0000", 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.831e-05 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426609+0000", 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150629 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426735+0000", 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012625900000000001 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.441627+0000", 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014891824999999999 2024-04-25T15:54:29.546 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454417+0000", 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012789994000000001 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454476+0000", 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8752000000000003e-05 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.547 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454528+0000", 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2519999999999999e-05 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3145 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 35.062289831999998, 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030601595999999998, 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.548 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3145 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.549 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434689+0000", 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.005e-06 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436062+0000", 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001373682 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436096+0000", 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3797000000000002e-05 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.550 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436107+0000", 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436177+0000", 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0069999999999998e-05 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436346+0000", 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169168 2024-04-25T15:54:29.551 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436510+0000", 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163807 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.443699+0000", 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007189394 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.552 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465141+0000", 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021441386999999999 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465223+0000", 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1920000000000002e-05 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465287+0000", 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4692999999999999e-05 2024-04-25T15:54:29.553 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3841 3.18 3:1a4016cc:::100000001f8.000000fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.411292998, 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043270667999999998, 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.554 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3841 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085686+0000", 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4510000000000001e-06 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088301+0000", 2024-04-25T15:54:29.556 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002614987 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088348+0000", 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7057e-05 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088362+0000", 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4352000000000001e-05 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.557 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088457+0000", 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4794000000000001e-05 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088583+0000", 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012605400000000001 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088776+0000", 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019268700000000001 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.107828+0000", 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019051702 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128788+0000", 2024-04-25T15:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020960433000000001 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128888+0000", 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9493000000000004e-05 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128953+0000", 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5658000000000001e-05 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.560 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3977 3.18 3:1b05c35d:::100000001f8.00000185:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.654754415999999, 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13853472, 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3977 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.561 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842224+0000", 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1130000000000001e-06 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.562 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844135+0000", 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019102920000000001 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844181+0000", 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6616000000000002e-05 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.563 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844202+0000", 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0542000000000001e-05 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844358+0000", 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156379 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844555+0000", 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000196651 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.564 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844759+0000", 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020436799999999999 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.853781+0000", 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090217130000000006 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980620+0000", 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.126839061 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.565 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980690+0000", 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9596000000000002e-05 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980756+0000", 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.566 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4099 3.18 3:181e4796:::100000001f8.000001ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.961682209999999, 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030713761999999999, 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4099 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.568 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535297+0000", 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2049999999999998e-06 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536942+0000", 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00164504 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536980+0000", 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8736999999999999e-05 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536997+0000", 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6653000000000001e-05 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537091+0000", 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3786e-05 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537191+0000", 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9840000000000006e-05 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.570 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537331+0000", 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014034900000000001 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.556229+0000", 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018898278000000001 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565921+0000", 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096916850000000002 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565961+0000", 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0290999999999998e-05 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.566007+0000", 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5898e-05 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.572 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4812 3.18 3:194fa8d8:::100000001f8.000004c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.813267995, 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038646409, 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4812 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.573 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.574 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683711+0000", 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7170000000000002e-06 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685822+0000", 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105619999999999 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685876+0000", 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3798999999999998e-05 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.575 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685910+0000", 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4727999999999997e-05 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686022+0000", 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111549 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686150+0000", 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826800000000001 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.576 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686332+0000", 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181538 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.693171+0000", 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068389519999999997 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722249+0000", 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029078171 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.577 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722294+0000", 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4756000000000003e-05 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722354+0000", 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0368999999999997e-05 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.578 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4967 3.22 3:478cb4f4:::100000001f8.00000563:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.955334281999999, 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035932619999999998, 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4967 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.579 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.580 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541672+0000", 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0978000000000001e-05 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543306+0000", 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016338310000000001 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543342+0000", 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5994000000000001e-05 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543360+0000", 2024-04-25T15:54:29.581 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543470+0000", 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110319 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543624+0000", 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153549 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543799+0000", 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017463299999999999 2024-04-25T15:54:29.582 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.555562+0000", 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011763177999999999 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577476+0000", 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021913818000000002 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577529+0000", 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3711000000000001e-05 2024-04-25T15:54:29.583 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577574+0000", 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4678999999999997e-05 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5705 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.872610818, 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080110828999999995, 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.584 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5705 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.585 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624368+0000", 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.117e-06 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626198+0000", 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001829968 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626234+0000", 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6220000000000002e-05 2024-04-25T15:54:29.586 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626249+0000", 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4742999999999999e-05 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626341+0000", 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2477000000000002e-05 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626462+0000", 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121191 2024-04-25T15:54:29.587 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626636+0000", 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017351000000000001 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.635055+0000", 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084194100000000004 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704341+0000", 2024-04-25T15:54:29.588 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.069285804000000006 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704412+0000", 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0674999999999997e-05 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704476+0000", 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3713999999999993e-05 2024-04-25T15:54:29.589 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5751 3.18 3:1b3fa6ee:::100000001f8.00000873:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.620544936, 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037922083000000002, 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.590 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5751 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876434+0000", 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878480+0000", 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020457800000000001 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878539+0000", 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8640000000000001e-05 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878587+0000", 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7746e-05 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878724+0000", 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137572 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878868+0000", 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014331899999999999 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.879018+0000", 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015069000000000001 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.886482+0000", 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007464048 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914229+0000", 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027746804 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914300+0000", 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0952999999999996e-05 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914353+0000", 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.274e-05 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5755 3.32 3:4c40e2d2:::100000001f8.00000877:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 15.601508781, 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033152545999999998, 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5755 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895470+0000", 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7839999999999999e-06 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898117+0000", 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026470959999999998 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898163+0000", 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6406000000000003e-05 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898183+0000", 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0092999999999999e-05 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898295+0000", 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011160800000000001 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898422+0000", 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012762099999999999 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898601+0000", 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017858500000000001 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914502+0000", 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015900724000000001 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928454+0000", 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013952585999999999 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928552+0000", 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7299000000000005e-05 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928619+0000", 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7743999999999994e-05 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5966 3.18 3:1b2b62aa:::100000001f8.0000094a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.396865388, 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043485786999999998, 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.601 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5966 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100114+0000", 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.315e-06 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.101977+0000", 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018630109999999999 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102022+0000", 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5636000000000001e-05 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102040+0000", 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8e-05 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102140+0000", 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9720000000000001e-05 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102270+0000", 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130409 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102415+0000", 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144492 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.118110+0000", 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015694725999999999 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143459+0000", 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025349846999999998 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143534+0000", 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4839999999999995e-05 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143596+0000", 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1791000000000005e-05 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:6009 3.18 3:1b7f6e54:::100000001f8.00000975:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:29.606 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 14.131195547000001, 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030441923999999999, 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 6009 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:29.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365782+0000", 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3889999999999999e-06 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367595+0000", 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001812976 2024-04-25T15:54:29.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367633+0000", 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7185999999999999e-05 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367647+0000", 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367735+0000", 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7474999999999999e-05 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367814+0000", 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9326000000000006e-05 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367936+0000", 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122077 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.375787+0000", 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078512010000000004 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396095+0000", 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02030732 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396164+0000", 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9549999999999996e-05 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396222+0000", 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7741999999999997e-05 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.611 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.612 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:29.612 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:29.612 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:29.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:54:29.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:54:29.614 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:29.613+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:29.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:54:29.747 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:29.746+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:29.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-25T15:54:29.811 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:29.811 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:29.811 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:29.811 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:29.815 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:29.814+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:29.830 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:54:29.916 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:29.917+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.917+0000 7fbcc0c7e640 1 Processor -- start 2024-04-25T15:54:29.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.917+0000 7fbcc0c7e640 1 -- start start 2024-04-25T15:54:29.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcc0c7e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbcbc0caf60 0x7fbcbc15d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcc0c7e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcbc073bf0 0x7fbcbc163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcc0c7e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcbc0c77a0 con 0x7fbcbc15d790 2024-04-25T15:54:29.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcc0c7e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fbcbc0cab60 con 0x7fbcbc073bf0 2024-04-25T15:54:29.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcc0c7e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbcbc159620 con 0x7fbcbc0caf60 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcbad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcbc073bf0 0x7fbcbc163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcbad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcbc073bf0 0x7fbcbc163ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54876/0 (socket says 172.21.15.67:54876) 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcbad9d640 1 -- 172.21.15.67:0/3001461483 learned_addr learned my addr 172.21.15.67:0/3001461483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcba59c640 1 --2- 172.21.15.67:0/3001461483 >> v2:172.21.15.176:3300/0 conn(0x7fbcbc0caf60 0x7fbcbc15d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcbad9d640 1 -- 172.21.15.67:0/3001461483 >> v2:172.21.15.176:3300/0 conn(0x7fbcbc0caf60 msgr2=0x7fbcbc15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/3001461483 >> v2:172.21.15.176:3300/0 conn(0x7fbcbc0caf60 0x7fbcbc15d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcbad9d640 1 -- 172.21.15.67:0/3001461483 >> v1:172.21.15.176:6789/0 conn(0x7fbcbc15d790 legacy=0x7fbcbc06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcbad9d640 1 -- 172.21.15.67:0/3001461483 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcbc0c7a60 con 0x7fbcbc073bf0 2024-04-25T15:54:29.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.921+0000 7fbcba59c640 1 --2- 172.21.15.67:0/3001461483 >> v2:172.21.15.176:3300/0 conn(0x7fbcbc0caf60 0x7fbcbc15d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/3001461483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc073bf0 0x7fbcbc163ec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0720b0 tx=0x7fbcb401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00205dda9997061 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcb959a640 1 -- 172.21.15.67:0/3001461483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40a5670 con 0x7fbcbc073bf0 2024-04-25T15:54:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcb959a640 1 -- 172.21.15.67:0/3001461483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb40a5810 con 0x7fbcbc073bf0 2024-04-25T15:54:29.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcb959a640 1 -- 172.21.15.67:0/3001461483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40a5be0 con 0x7fbcbc073bf0 2024-04-25T15:54:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/3001461483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc073bf0 msgr2=0x7fbcbc163ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/3001461483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc073bf0 0x7fbcbc163ec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc0720b0 tx=0x7fbcb401c9b0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/3001461483 shutdown_connections 2024-04-25T15:54:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/3001461483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc073bf0 0x7fbcbc163ec0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/3001461483 >> v2:172.21.15.176:3300/0 conn(0x7fbcbc0caf60 0x7fbcbc15d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/3001461483 >> 172.21.15.67:0/3001461483 conn(0x7fbcbc070db0 msgr2=0x7fbcbc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:29.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/3001461483 shutdown_connections 2024-04-25T15:54:29.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.925+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/3001461483 wait complete. 2024-04-25T15:54:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 Processor -- start 2024-04-25T15:54:29.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 -- start start 2024-04-25T15:54:29.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcbc073bf0 0x7fbcbc14ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcbc0caf60 0x7fbcbc153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc15d790 0x7fbcbc162180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbcbc15a570 con 0x7fbcbc15d790 2024-04-25T15:54:29.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbcbc15a3f0 con 0x7fbcbc073bf0 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbcbc06c640 con 0x7fbcbc0caf60 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc15d790 0x7fbcbc162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc15d790 0x7fbcbc162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54886/0 (socket says 172.21.15.67:54886) 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 learned_addr learned my addr 172.21.15.67:0/1254382605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcbc073bf0 0x7fbcbc14ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcb9d9b640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcbc0caf60 0x7fbcbc153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcbc0caf60 msgr2=0x7fbcbc153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcbc0caf60 0x7fbcbc153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcbc073bf0 msgr2=0x7fbcbc14ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcbc073bf0 0x7fbcbc14ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcbc14f010 con 0x7fbcbc15d790 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc15d790 0x7fbcbc162180 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40af040 tx=0x7fbcb4019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40b8670 con 0x7fbcbc15d790 2024-04-25T15:54:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcbc14d150 con 0x7fbcbc15d790 2024-04-25T15:54:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb40be070 con 0x7fbcbc15d790 2024-04-25T15:54:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb40b8a90 con 0x7fbcbc15d790 2024-04-25T15:54:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.929+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcbc14d710 con 0x7fbcbc15d790 2024-04-25T15:54:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fbcbc06a480 con 0x7fbcbc15d790 2024-04-25T15:54:29.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbcb40b8c30 con 0x7fbcbc15d790 2024-04-25T15:54:29.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcaaffd640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9c06f400 0x7fbc9c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 6031+0+0 (secure 0 0 0) 0x7fbcb412afb0 con 0x7fbcbc15d790 2024-04-25T15:54:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v106) ==== 54+0+16 (secure 0 0 0) 0x7fbcb40165a0 con 0x7fbcbc15d790 2024-04-25T15:54:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9c06f400 0x7fbc9c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbcbc163fd0 0x7fbcbc14a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc0c6510 con 0x7fbcbc163fd0 2024-04-25T15:54:29.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcb9d9b640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbcbc163fd0 0x7fbcbc14a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.933+0000 7fbcb9d9b640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbcbc163fd0 0x7fbcbc14a110 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.937+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9c06f400 0x7fbc9c071910 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbcac006fd0 tx=0x7fbcac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.937+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc0c6510 con 0x7fbcbc163fd0 2024-04-25T15:54:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.949+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc0c8a40 con 0x7fbcbc163fd0 2024-04-25T15:54:29.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.953+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbcbc0c8a40 con 0x7fbcbc163fd0 2024-04-25T15:54:29.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.953+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbcbc14f1e0 0x7fbcbc1f9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.953+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc14ab60 con 0x7fbcbc14f1e0 2024-04-25T15:54:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.953+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbcbc14f1e0 0x7fbcbc1f9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.953+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbcbc14f1e0 0x7fbcbc1f9870 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.953+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc14ab60 con 0x7fbcbc14f1e0 2024-04-25T15:54:29.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.969+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc153c10 con 0x7fbcbc14f1e0 2024-04-25T15:54:29.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.969+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbcbc153c10 con 0x7fbcbc14f1e0 2024-04-25T15:54:29.971 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.969+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbcbc15a780 0x7fbcbc1fa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.969+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc159880 con 0x7fbcbc15a780 2024-04-25T15:54:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.969+0000 7fbcb9d9b640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbcbc15a780 0x7fbcbc1fa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.969+0000 7fbcb9d9b640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbcbc15a780 0x7fbcbc1fa2c0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.972 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:29.972 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:29.972 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:29.972 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:29.972 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:29.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.969+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc159880 con 0x7fbcbc15a780 2024-04-25T15:54:29.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.985+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc0c9ae0 con 0x7fbcbc15a780 2024-04-25T15:54:29.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.989+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbcbc0c9ae0 con 0x7fbcbc15a780 2024-04-25T15:54:29.989 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.989+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbcbc063960 0x7fbcbc0cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:29.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.989+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc0c9ae0 con 0x7fbcbc063960 2024-04-25T15:54:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.989+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbcbc063960 0x7fbcbc0cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:29.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.989+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbcbc063960 0x7fbcbc0cc830 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:29.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:54:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:29.989+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc0c9ae0 con 0x7fbcbc063960 2024-04-25T15:54:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.001+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc0c9ae0 con 0x7fbcbc063960 2024-04-25T15:54:30.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.005+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbcbc0c9ae0 con 0x7fbcbc063960 2024-04-25T15:54:30.005 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:30.006 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:30.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.005+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 0x7fbcbc02cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:30.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.005+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc0c9ae0 con 0x7fbcbc02c9e0 2024-04-25T15:54:30.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.005+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 0x7fbcbc02cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:30.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.005+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 0x7fbcbc02cdf0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.009+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc0c9ae0 con 0x7fbcbc02c9e0 2024-04-25T15:54:30.017 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:30.017+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:30.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.021+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc0c9ae0 con 0x7fbcbc02c9e0 2024-04-25T15:54:30.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcba59c640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 msgr2=0x7fbcbc02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 25 2024-04-25T15:54:30.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcba59c640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 msgr2=0x7fbcbc02cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 0x7fbcbc02cdf0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 0x7fbcbc02cdf0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 msgr2=0x7fbcbc02cdf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcaaffd640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbcbc02c9e0 0x7fbcbc02cdf0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcaaffd640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc9c079a30 0x7fbc9c07be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc159880 con 0x7fbc9c079a30 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=107}) -- 0x7fbc9c07c3b0 con 0x7fbcbc15d790 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc9c079a30 0x7fbc9c07be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:30.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.025+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc9c079a30 0x7fbc9c07be50 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:30.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.029+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 10: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fbcbc159880 con 0x7fbc9c079a30 2024-04-25T15:54:30.032 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:30.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.033+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbcbc02d300 0x7fbcbc02d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:30.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.033+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc0cd7b0 con 0x7fbcbc02d300 2024-04-25T15:54:30.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.033+0000 7fbcb9d9b640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbcbc02d300 0x7fbcbc02d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:30.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.033+0000 7fbcb9d9b640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbcbc02d300 0x7fbcbc02d710 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:30.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.033+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc0cd7b0 con 0x7fbcbc02d300 2024-04-25T15:54:30.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.049+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc02dcb0 con 0x7fbcbc02d300 2024-04-25T15:54:30.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.053+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbcbc02dcb0 con 0x7fbcbc02d300 2024-04-25T15:54:30.054 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:30.054 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:30.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.053+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbcbc02dc50 0x7fbcbc20d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.053+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc2101b0 con 0x7fbcbc02dc50 2024-04-25T15:54:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.053+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbcbc02dc50 0x7fbcbc20d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:30.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.053+0000 7fbcbad9d640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbcbc02dc50 0x7fbcbc20d110 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:30.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.057+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc2101b0 con 0x7fbcbc02dc50 2024-04-25T15:54:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.057+0000 7ff9257fa640 1 -- 172.21.15.67:0/2398217530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v107) ==== 87+0+0 (secure 0 0 0) 0x7ff91805b500 con 0x7ff914005280 2024-04-25T15:54:30.060 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-04-25T15:54:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff90806f430 msgr2=0x7ff908071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff90806f430 0x7ff908071940 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff910000c10 tx=0x7ff910009040 comp rx=0 tx=0).stop 2024-04-25T15:54:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 msgr2=0x7ff9281e5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff9281e5c80 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff91800e800 tx=0x7ff91800ecc0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 shutdown_connections 2024-04-25T15:54:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff90806f430 0x7ff908071940 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9280d0500 0x7ff9281f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9280d00f0 0x7ff9281f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 --2- 172.21.15.67:0/2398217530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff914005280 0x7ff9281e5c80 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 >> 172.21.15.67:0/2398217530 conn(0x7ff9280700b0 msgr2=0x7ff928151fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== mon.0 v2:172.21.15.67:3300/0 7 ==== osd_map(107..107 src has 1..107) ==== 288+0+0 (secure 0 0 0) 0x7fbcb41294b0 con 0x7fbcbc15d790 2024-04-25T15:54:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 shutdown_connections 2024-04-25T15:54:30.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.061+0000 7ff92e3c7640 1 -- 172.21.15.67:0/2398217530 wait complete. 2024-04-25T15:54:30.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.073+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc210100 con 0x7fbcbc02dc50 2024-04-25T15:54:30.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.077+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbcbc210100 con 0x7fbcbc02dc50 2024-04-25T15:54:30.078 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.077+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbcbc20f3e0 0x7fbcbc20f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.077+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fbcbc20fe40 con 0x7fbcbc20f3e0 2024-04-25T15:54:30.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.077+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbcbc20f3e0 0x7fbcbc20f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:30.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.077+0000 7fbcba59c640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbcbc20f3e0 0x7fbcbc20f7f0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:30.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.081+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcbc20fe40 con 0x7fbcbc20f3e0 2024-04-25T15:54:30.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.093+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fbcbc20fe40 con 0x7fbcbc20f3e0 2024-04-25T15:54:30.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.097+0000 7fbcaaffd640 1 -- 172.21.15.67:0/1254382605 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fbcbc20fe40 con 0x7fbcbc20f3e0 2024-04-25T15:54:30.100 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbcbc163fd0 msgr2=0x7fbcbc14a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7fbcbc163fd0 0x7fbcbc14a110 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbcbc14f1e0 msgr2=0x7fbcbc1f9870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fbcbc14f1e0 0x7fbcbc1f9870 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbcbc15a780 msgr2=0x7fbcbc1fa2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7fbcbc15a780 0x7fbcbc1fa2c0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbcbc063960 msgr2=0x7fbcbc0cc830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7fbcbc063960 0x7fbcbc0cc830 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc9c079a30 msgr2=0x7fbc9c07be50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fbc9c079a30 0x7fbc9c07be50 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbcbc02d300 msgr2=0x7fbcbc02d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fbcbc02d300 0x7fbcbc02d710 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbcbc02dc50 msgr2=0x7fbcbc20d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fbcbc02dc50 0x7fbcbc20d110 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbcbc20f3e0 msgr2=0x7fbcbc20f7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7fbcbc20f3e0 0x7fbcbc20f7f0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 reap_dead start 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 reap_dead start 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 reap_dead start 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 reap_dead start 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcbad9d640 1 -- 172.21.15.67:0/1254382605 reap_dead start 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9c06f400 msgr2=0x7fbc9c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9c06f400 0x7fbc9c071910 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbcac006fd0 tx=0x7fbcac008040 comp rx=0 tx=0).stop 2024-04-25T15:54:30.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc15d790 msgr2=0x7fbcbc162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc15d790 0x7fbcbc162180 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fbcb40af040 tx=0x7fbcb4019750 comp rx=0 tx=0).stop 2024-04-25T15:54:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 shutdown_connections 2024-04-25T15:54:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fbc9c06f400 0x7fbc9c071910 secure :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbcac006fd0 tx=0x7fbcac008040 comp rx=0 tx=0).stop 2024-04-25T15:54:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 --2- 172.21.15.67:0/1254382605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcbc15d790 0x7fbcbc162180 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 >> 172.21.15.67:0/1254382605 conn(0x7fbcbc070db0 msgr2=0x7fbcbc0c9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 shutdown_connections 2024-04-25T15:54:30.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:30.101+0000 7fbcc0c7e640 1 -- 172.21.15.67:0/1254382605 wait complete. 2024-04-25T15:54:30.117 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:30.117+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:30.141 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:30.141 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:30.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:54:30.244 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:30.245+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:30.319 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:30.317+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:30.341 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:30.342 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:30.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:54:30.448 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:30.449+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:30.521 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:30.518+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:30.551 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:30.551 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:30.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:54:30.660 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:30.661+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:30.722 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:30.721+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:30.767 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:30.768 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:30.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:54:30.832 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:30.833+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:30.924 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:30.922+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:30.943 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:30.944 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:30.962 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:54:31.047 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:31.046+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:31.127 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:31.125+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:31.149 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:31.149 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:31.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:54:31.199 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 5, 6, 7, 1, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-25T15:54:31.199 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-25T15:54:31.199 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.42190496287163026 2024-04-25T15:54:31.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 0.42190496287163026 2024-04-25T15:54:31.244 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:31.245+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:31.329 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:31.326+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:31.362 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:31.363 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c0d64e640 1 Processor -- start 2024-04-25T15:54:31.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c0d64e640 1 -- start start 2024-04-25T15:54:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c0d64e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c08071bc0 0x7f0c080cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:31.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c0d64e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0c080ca540 0x7f0c080ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c0d64e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c0815de10 con 0x7f0c080cc100 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c0d64e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0c0815ebb0 con 0x7f0c08071bc0 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c0d64e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0c0815e720 con 0x7f0c080ca540 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c06ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c08071bc0 0x7f0c080cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0bfffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0c080cc100 0x7f0c0815a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40514/0 (socket says 172.21.15.67:40514) 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0bfffff640 1 -- 172.21.15.67:0/1500053756 learned_addr learned my addr 172.21.15.67:0/1500053756 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c077fe640 1 --2- 172.21.15.67:0/1500053756 >> v2:172.21.15.176:3300/0 conn(0x7f0c080ca540 0x7f0c080ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c06ffd640 1 -- 172.21.15.67:0/1500053756 >> v2:172.21.15.176:3300/0 conn(0x7f0c080ca540 msgr2=0x7f0c080ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c06ffd640 1 --2- 172.21.15.67:0/1500053756 >> v2:172.21.15.176:3300/0 conn(0x7f0c080ca540 0x7f0c080ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c06ffd640 1 -- 172.21.15.67:0/1500053756 >> v1:172.21.15.67:6789/0 conn(0x7f0c080cc100 legacy=0x7f0c0815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:31.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.369+0000 7f0c06ffd640 1 -- 172.21.15.67:0/1500053756 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c08072390 con 0x7f0c08071bc0 2024-04-25T15:54:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c06ffd640 1 --2- 172.21.15.67:0/1500053756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c08071bc0 0x7f0c080cbbc0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0c0806d180 tx=0x7f0bf801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55ba55df220421a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c067fc640 1 -- 172.21.15.67:0/1500053756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80b1070 con 0x7f0c08071bc0 2024-04-25T15:54:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c067fc640 1 -- 172.21.15.67:0/1500053756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf80ac5f0 con 0x7f0c08071bc0 2024-04-25T15:54:31.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c067fc640 1 -- 172.21.15.67:0/1500053756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80ac9c0 con 0x7f0c08071bc0 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 -- 172.21.15.67:0/1500053756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c08071bc0 msgr2=0x7f0c080cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/1500053756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c08071bc0 0x7f0c080cbbc0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0c0806d180 tx=0x7f0bf801c480 comp rx=0 tx=0).stop 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 -- 172.21.15.67:0/1500053756 shutdown_connections 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/1500053756 >> v2:172.21.15.176:3300/0 conn(0x7f0c080ca540 0x7f0c080ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/1500053756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c08071bc0 0x7f0c080cbbc0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 -- 172.21.15.67:0/1500053756 >> 172.21.15.67:0/1500053756 conn(0x7f0c080700b0 msgr2=0x7f0c080704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 -- 172.21.15.67:0/1500053756 shutdown_connections 2024-04-25T15:54:31.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 -- 172.21.15.67:0/1500053756 wait complete. 2024-04-25T15:54:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 Processor -- start 2024-04-25T15:54:31.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.373+0000 7f0c0d64e640 1 -- start start 2024-04-25T15:54:31.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c08071bc0 0x7f0c081f6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:31.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c080ca540 0x7f0c0814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:31.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c080cc100 0x7f0c0815b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0c08072150 con 0x7f0c080cc100 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0c08071fd0 con 0x7f0c08071bc0 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0c0815ae30 con 0x7f0c080ca540 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c080cc100 0x7f0c0815b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c080cc100 0x7f0c0815b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54908/0 (socket says 172.21.15.67:54908) 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 -- 172.21.15.67:0/2364463500 learned_addr learned my addr 172.21.15.67:0/2364463500 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0bfffff640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c080ca540 0x7f0c0814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c06ffd640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c08071bc0 0x7f0c081f6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 -- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c080ca540 msgr2=0x7f0c0814d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c080ca540 0x7f0c0814d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 -- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c08071bc0 msgr2=0x7f0c081f6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c08071bc0 0x7f0c081f6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 -- 172.21.15.67:0/2364463500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c081f6b80 con 0x7f0c080cc100 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c06ffd640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c08071bc0 0x7f0c081f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c077fe640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c080cc100 0x7f0c0815b880 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0c00007e60 tx=0x7f0c0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0bff7fe640 1 -- 172.21.15.67:0/2364463500 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00014420 con 0x7f0c080cc100 2024-04-25T15:54:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0bff7fe640 1 -- 172.21.15.67:0/2364463500 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c00019070 con 0x7f0c080cc100 2024-04-25T15:54:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c08152f70 con 0x7f0c080cc100 2024-04-25T15:54:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0bff7fe640 1 -- 172.21.15.67:0/2364463500 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00014820 con 0x7f0c080cc100 2024-04-25T15:54:31.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c08153530 con 0x7f0c080cc100 2024-04-25T15:54:31.381 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:54:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c081f6d50 con 0x7f0c080cc100 2024-04-25T15:54:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0bff7fe640 1 -- 172.21.15.67:0/2364463500 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0c000149c0 con 0x7f0c080cc100 2024-04-25T15:54:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0bff7fe640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0be806f430 0x7f0be8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.377+0000 7f0bff7fe640 1 -- 172.21.15.67:0/2364463500 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 6031+0+0 (secure 0 0 0) 0x7f0c0008fc10 con 0x7f0c080cc100 2024-04-25T15:54:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.381+0000 7f0c06ffd640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0be806f430 0x7f0be8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:31.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.381+0000 7f0c06ffd640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0be806f430 0x7f0be8071940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80ae000 tx=0x7f0bf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.385+0000 7f0bff7fe640 1 -- 172.21.15.67:0/2364463500 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0c081f6d50 con 0x7f0c080cc100 2024-04-25T15:54:31.467 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:31.466+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:31.531 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:31.529+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:31.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:31.557+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 0.42190496287163026} v 0) -- 0x7f0c080c8220 con 0x7f0c080cc100 2024-04-25T15:54:31.568 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:31.568 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:31.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:54:31.632 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:31.629+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:31.733 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:31.733+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:31.779 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:31.780 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:31.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:54:31.843 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:31.842+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:31.935 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:31.930+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:31.942 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:31.943 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:31.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:54:32.047 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:32.046+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:32.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.065+0000 7f0bff7fe640 1 -- 172.21.15.67:0/2364463500 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 0.42190496287163026}]=0 reweighted osd.6 to 0.421905 (6c01) v108) ==== 135+0+0 (secure 0 0 0) 0x7f0c0005ca50 con 0x7f0c080cc100 2024-04-25T15:54:32.066 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 0.421905 (6c01) 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0be806f430 msgr2=0x7f0be8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0be806f430 0x7f0be8071940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80ae000 tx=0x7f0bf8008040 comp rx=0 tx=0).stop 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c080cc100 msgr2=0x7f0c0815b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c080cc100 0x7f0c0815b880 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0c00007e60 tx=0x7f0c0000e4b0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 shutdown_connections 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0be806f430 0x7f0be8071940 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c080cc100 0x7f0c0815b880 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c080ca540 0x7f0c0814d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 --2- 172.21.15.67:0/2364463500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c08071bc0 0x7f0c081f6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 >> 172.21.15.67:0/2364463500 conn(0x7f0c080700b0 msgr2=0x7f0c080c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 shutdown_connections 2024-04-25T15:54:32.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.069+0000 7f0c0d64e640 1 -- 172.21.15.67:0/2364463500 wait complete. 2024-04-25T15:54:32.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-25T15:54:32.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-25T15:54:32.137 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-04-25T15:54:32.134+0000 7f077acff640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27898) UID: 0 2024-04-25T15:54:32.155 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:32.155 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:32.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:54:32.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3e5412640 1 Processor -- start 2024-04-25T15:54:32.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3e5412640 1 -- start start 2024-04-25T15:54:32.239 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-25T15:54:32.237+0000 7f4991ad6640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27699) UID: 0 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3e5412640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff3e00cbea0 0x7ff3e0071af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3e5412640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e0167230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3e5412640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3e00c57b0 con 0x7ff3e00caff0 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3e5412640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff3e00c5a80 con 0x7ff3e00cbea0 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3e5412640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e00c5d60 con 0x7ff3e0072030 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3deffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff3e00caff0 0x7ff3e01608c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40530/0 (socket says 172.21.15.67:40530) 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3deffd640 1 -- 172.21.15.67:0/945393542 learned_addr learned my addr 172.21.15.67:0/945393542 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3de7fc640 1 --2- 172.21.15.67:0/945393542 >> v2:172.21.15.176:3300/0 conn(0x7ff3e00cbea0 0x7ff3e0071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.237+0000 7ff3df7fe640 1 --2- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e0167230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3ddffb640 1 -- 172.21.15.67:0/945393542 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2012473456 0 0) 0x7ff3e00c57b0 con 0x7ff3e00caff0 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3ddffb640 1 -- 172.21.15.67:0/945393542 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3c8003530 con 0x7ff3e00caff0 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3de7fc640 1 -- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 msgr2=0x7ff3e0167230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3de7fc640 1 --2- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e0167230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3de7fc640 1 -- 172.21.15.67:0/945393542 >> v1:172.21.15.67:6789/0 conn(0x7ff3e00caff0 legacy=0x7ff3e01608c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3de7fc640 1 -- 172.21.15.67:0/945393542 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e015cb10 con 0x7ff3e00cbea0 2024-04-25T15:54:32.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3df7fe640 1 --2- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e0167230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3de7fc640 1 --2- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00cbea0 0x7ff3e0071af0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097c0 tx=0x7ff3d401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f79f41829292875b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3ddffb640 1 -- 172.21.15.67:0/945393542 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d40b0070 con 0x7ff3e00cbea0 2024-04-25T15:54:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3ddffb640 1 -- 172.21.15.67:0/945393542 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d40ab600 con 0x7ff3e00cbea0 2024-04-25T15:54:32.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3ddffb640 1 -- 172.21.15.67:0/945393542 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d40ab9b0 con 0x7ff3e00cbea0 2024-04-25T15:54:32.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 -- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00cbea0 msgr2=0x7ff3e0071af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 --2- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00cbea0 0x7ff3e0071af0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40097c0 tx=0x7ff3d401c4a0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 -- 172.21.15.67:0/945393542 shutdown_connections 2024-04-25T15:54:32.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 --2- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e0167230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 --2- 172.21.15.67:0/945393542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00cbea0 0x7ff3e0071af0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 -- 172.21.15.67:0/945393542 >> 172.21.15.67:0/945393542 conn(0x7ff3e00700b0 msgr2=0x7ff3e00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:32.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 -- 172.21.15.67:0/945393542 shutdown_connections 2024-04-25T15:54:32.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.241+0000 7ff3e5412640 1 -- 172.21.15.67:0/945393542 wait complete. 2024-04-25T15:54:32.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 Processor -- start 2024-04-25T15:54:32.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 -- start start 2024-04-25T15:54:32.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e014ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00caff0 0x7ff3e0150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e00cbea0 0x7ff3e015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e00c6040 con 0x7ff3e00cbea0 2024-04-25T15:54:32.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e0157110 con 0x7ff3e00caff0 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e00cc2b0 con 0x7ff3e0072030 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e00cbea0 0x7ff3e015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e00cbea0 0x7ff3e015bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54914/0 (socket says 172.21.15.67:54914) 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 -- 172.21.15.67:0/2991021747 learned_addr learned my addr 172.21.15.67:0/2991021747 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3deffd640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e014ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3de7fc640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00caff0 0x7ff3e0150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 -- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 msgr2=0x7ff3e014ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e014ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 -- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00caff0 msgr2=0x7ff3e0150f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00caff0 0x7ff3e0150f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 -- 172.21.15.67:0/2991021747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e00d0180 con 0x7ff3e00cbea0 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3df7fe640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e00cbea0 0x7ff3e015bf10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800efc0 tx=0x7ff3d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3c77fe640 1 -- 172.21.15.67:0/2991021747 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8018030 con 0x7ff3e00cbea0 2024-04-25T15:54:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3c77fe640 1 -- 172.21.15.67:0/2991021747 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d800cdf0 con 0x7ff3e00cbea0 2024-04-25T15:54:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e00d0430 con 0x7ff3e00cbea0 2024-04-25T15:54:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3c77fe640 1 -- 172.21.15.67:0/2991021747 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8003bc0 con 0x7ff3e00cbea0 2024-04-25T15:54:32.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.245+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e00d0710 con 0x7ff3e00cbea0 2024-04-25T15:54:32.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.249+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e006a480 con 0x7ff3e00cbea0 2024-04-25T15:54:32.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.249+0000 7ff3c77fe640 1 -- 172.21.15.67:0/2991021747 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff3d801b070 con 0x7ff3e00cbea0 2024-04-25T15:54:32.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.249+0000 7ff3c77fe640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3cc06f400 0x7ff3cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.249+0000 7ff3c77fe640 1 -- 172.21.15.67:0/2991021747 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6031+0+0 (secure 0 0 0) 0x7ff3d8090440 con 0x7ff3e00cbea0 2024-04-25T15:54:32.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.249+0000 7ff3deffd640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3cc06f400 0x7ff3cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.249+0000 7ff3deffd640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3cc06f400 0x7ff3cc071910 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0007fd0 tx=0x7ff3d0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.257+0000 7ff3c77fe640 1 -- 172.21.15.67:0/2991021747 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff3d8022070 con 0x7ff3e00cbea0 2024-04-25T15:54:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.289+0000 7f1c2c40f640 1 Processor -- start 2024-04-25T15:54:32.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.289+0000 7f1c2c40f640 1 -- start start 2024-04-25T15:54:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2c40f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 0x7f1c2406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2c40f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c2406d780 con 0x7f1c2406ca60 2024-04-25T15:54:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2c40f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c24160920 con 0x7f1c240b5d70 2024-04-25T15:54:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2c40f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2415d510 con 0x7f1c24160510 2024-04-25T15:54:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1c2406ca60 0x7f1c240b5660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40538/0 (socket says 172.21.15.67:40538) 2024-04-25T15:54:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 -- 172.21.15.67:0/793546970 learned_addr learned my addr 172.21.15.67:0/793546970 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 0x7f1c2406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797310182 0 0) 0x7f1c2406d780 con 0x7f1c2406ca60 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c10003530 con 0x7f1c2406ca60 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2987161426 0 0) 0x7f1c10003530 con 0x7f1c2406ca60 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 msgr2=0x7f1c2406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 0x7f1c2406ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 >> v1:172.21.15.176:6789/0 conn(0x7f1c240b5d70 legacy=0x7f1c2415fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c24160be0 con 0x7f1c2406ca60 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f1c200016a0 con 0x7f1c2406ca60 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 >> v1:172.21.15.67:6789/0 conn(0x7f1c2406ca60 legacy=0x7f1c240b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c100039f0 0x7f1c10003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c10004810 0x7f1c10004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 0x7f1c10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2415d510 con 0x7f1c100039f0 2024-04-25T15:54:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1000bab0 con 0x7f1c10004810 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1000bcb0 con 0x7f1c24160510 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c100039f0 0x7f1c10003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 0x7f1c10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c299aa640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c10004810 0x7f1c10004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 -- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 msgr2=0x7f1c10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c24160510 0x7f1c10005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 -- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c10004810 msgr2=0x7f1c10004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c10004810 0x7f1c10004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.293+0000 7f1c2a1ab640 1 -- 172.21.15.67:0/793546970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c10003530 con 0x7f1c100039f0 2024-04-25T15:54:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c100039f0 0x7f1c10003e20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c2406d950 tx=0x7f1c2001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91db94a4068b4490 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20007d40 con 0x7f1c100039f0 2024-04-25T15:54:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c200a9020 con 0x7f1c100039f0 2024-04-25T15:54:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c291a9640 1 -- 172.21.15.67:0/793546970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200a5b20 con 0x7f1c100039f0 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 -- 172.21.15.67:0/793546970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c100039f0 msgr2=0x7f1c10003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/793546970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c100039f0 0x7f1c10003e20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c2406d950 tx=0x7f1c2001cc20 comp rx=0 tx=0).stop 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/793546970 reap_dead start 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 -- 172.21.15.67:0/793546970 shutdown_connections 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 -- 172.21.15.67:0/793546970 >> 172.21.15.67:0/793546970 conn(0x7f1c2406e420 msgr2=0x7f1c2406e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 -- 172.21.15.67:0/793546970 shutdown_connections 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 -- 172.21.15.67:0/793546970 wait complete. 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 Processor -- start 2024-04-25T15:54:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.297+0000 7f1c2c40f640 1 -- start start 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c100039f0 0x7f1c241f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c240c9350 0x7f1c241f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c2415b100 0x7f1c24151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2415d1a0 con 0x7f1c240c9350 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2415cec0 con 0x7f1c100039f0 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2415c890 con 0x7f1c2415b100 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c240c9350 0x7f1c241f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c240c9350 0x7f1c241f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54932/0 (socket says 172.21.15.67:54932) 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 -- 172.21.15.67:0/3601172036 learned_addr learned my addr 172.21.15.67:0/3601172036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c100039f0 0x7f1c241f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c2415b100 0x7f1c24151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c2415b100 msgr2=0x7f1c24151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c2415b100 0x7f1c24151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c100039f0 msgr2=0x7f1c241f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c100039f0 0x7f1c241f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c240b7db0 con 0x7f1c240c9350 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c100039f0 0x7f1c241f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c299aa640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c240c9350 0x7f1c241f3200 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1c14010f40 tx=0x7f1c1400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14018020 con 0x7f1c240c9350 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1400ee50 con 0x7f1c240c9350 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c240b51b0 con 0x7f1c240c9350 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14003600 con 0x7f1c240c9350 2024-04-25T15:54:32.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c241f2a20 con 0x7f1c240c9350 2024-04-25T15:54:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1c2406a480 con 0x7f1c240c9350 2024-04-25T15:54:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.301+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1c1401b020 con 0x7f1c240c9350 2024-04-25T15:54:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c1affd640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c0806f400 0x7f1c08071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 6031+0+0 (secure 0 0 0) 0x7f1c14092890 con 0x7f1c240c9350 2024-04-25T15:54:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v108) ==== 54+0+16 (secure 0 0 0) 0x7f1c14022030 con 0x7f1c240c9350 2024-04-25T15:54:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c0806f400 0x7f1c08071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1c2415b510 0x7f1c24159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c240cd380 con 0x7f1c2415b510 2024-04-25T15:54:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1c2415b510 0x7f1c24159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c0806f400 0x7f1c08071910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c200aa000 tx=0x7f1c240b53d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1c2415b510 0x7f1c24159da0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.305+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c240cd380 con 0x7f1c2415b510 2024-04-25T15:54:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.333+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c24159870 con 0x7f1c2415b510 2024-04-25T15:54:32.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.337+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c24159870 con 0x7f1c2415b510 2024-04-25T15:54:32.337 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-04-25T15:54:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-25T15:54:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.337+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1c24158230 0x7f1c24158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.337+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1c241f3950 con 0x7f1c24158230 2024-04-25T15:54:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.337+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1c24158230 0x7f1c24158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.337+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1c24158230 0x7f1c24158640 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.337+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c241f3950 con 0x7f1c24158230 2024-04-25T15:54:32.339 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:32.338+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:32.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.361+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c24159ac0 con 0x7f1c24158230 2024-04-25T15:54:32.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.365+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c24159ac0 con 0x7f1c24158230 2024-04-25T15:54:32.364 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-04-25T15:54:32.365 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-25T15:54:32.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.365+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1c24063090 0x7f1c240634a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.365+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1c240639e0 con 0x7f1c24063090 2024-04-25T15:54:32.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.365+0000 7f1c299aa640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1c24063090 0x7f1c240634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.365+0000 7f1c299aa640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1c24063090 0x7f1c240634a0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.365+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c240639e0 con 0x7f1c24063090 2024-04-25T15:54:32.366 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:32.366 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.377+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c24063e40 con 0x7f1c24063090 2024-04-25T15:54:32.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c24063e40 con 0x7f1c24063090 2024-04-25T15:54:32.381 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 0x7f1c2402c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1c24064050 con 0x7f1c2415ed70 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 0x7f1c2402c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 msgr2=0x7f1c2402c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 24 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 msgr2=0x7f1c2402c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 0x7f1c2402c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:54:32.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.381+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 0x7f1c2402c9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:54:32.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:54:32.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.457+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7ff3e00c5380 con 0x7ff3e00cbea0 2024-04-25T15:54:32.471 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:32.470+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:32.541 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-25T15:54:32.541+0000 7f7be088f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27697) UID: 0 2024-04-25T15:54:32.575 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-04-25T15:54:32.575 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-25T15:54:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.585+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 0x7f1c2402c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.585+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 0x7f1c2402c9e0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.585+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c1c01b420 con 0x7f1c2415ed70 2024-04-25T15:54:32.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:54:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.601+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c2415f450 con 0x7f1c2415ed70 2024-04-25T15:54:32.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.605+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c2415f450 con 0x7f1c2415ed70 2024-04-25T15:54:32.605 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-04-25T15:54:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-25T15:54:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.605+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1c2402d430 0x7f1c2402d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.605+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1c2402dd80 con 0x7f1c2402d430 2024-04-25T15:54:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.605+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1c2402d430 0x7f1c2402d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.605+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1c2402d430 0x7f1c2402d840 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.605+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c2402dd80 con 0x7f1c2402d430 2024-04-25T15:54:32.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.629+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c2402dd80 con 0x7f1c2402d430 2024-04-25T15:54:32.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.633+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c2402dd80 con 0x7f1c2402d430 2024-04-25T15:54:32.636 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-04-25T15:54:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-25T15:54:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.637+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1c24207820 0x7f1c24209c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.637+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1c2402def0 con 0x7f1c24207820 2024-04-25T15:54:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.637+0000 7f1c299aa640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1c24207820 0x7f1c24209c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.637+0000 7f1c299aa640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1c24207820 0x7f1c24209c40 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.637+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c2402def0 con 0x7f1c24207820 2024-04-25T15:54:32.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.661+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c2402def0 con 0x7f1c24207820 2024-04-25T15:54:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.665+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c2402def0 con 0x7f1c24207820 2024-04-25T15:54:32.665 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-04-25T15:54:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-25T15:54:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.665+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1c2420a690 0x7f1c2420cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.665+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1c2420f9b0 con 0x7f1c2420a690 2024-04-25T15:54:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.665+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1c2420a690 0x7f1c2420cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.665+0000 7f1c2a9ac640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1c2420a690 0x7f1c2420cab0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.665+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c2420f9b0 con 0x7f1c2420a690 2024-04-25T15:54:32.682 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:32.678+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:32.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.693+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c24000960 con 0x7f1c2420a690 2024-04-25T15:54:32.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.693+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c24000960 con 0x7f1c2420a690 2024-04-25T15:54:32.696 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-04-25T15:54:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-25T15:54:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.697+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1c2420d500 0x7f1c2420d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.697+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1c24212d80 con 0x7f1c2420d500 2024-04-25T15:54:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.697+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1c2420d500 0x7f1c2420d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.697+0000 7f1c2a1ab640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1c2420d500 0x7f1c2420d910 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:32.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.697+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1c24212d80 con 0x7f1c2420d500 2024-04-25T15:54:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.721+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1c2420df60 con 0x7f1c2420d500 2024-04-25T15:54:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c1affd640 1 -- 172.21.15.67:0/3601172036 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1c2420df60 con 0x7f1c2420d500 2024-04-25T15:54:32.725 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1c2415b510 msgr2=0x7f1c24159da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7f1c2415b510 0x7f1c24159da0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1c24158230 msgr2=0x7f1c24158640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7f1c24158230 0x7f1c24158640 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1c24063090 msgr2=0x7f1c240634a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f1c24063090 0x7f1c240634a0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 msgr2=0x7f1c2402c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f1c2415ed70 0x7f1c2402c9e0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1c2402d430 msgr2=0x7f1c2402d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f1c2402d430 0x7f1c2402d840 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1c24207820 msgr2=0x7f1c24209c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7f1c24207820 0x7f1c24209c40 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1c2420a690 msgr2=0x7f1c2420cab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f1c2420a690 0x7f1c2420cab0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1c2420d500 msgr2=0x7f1c2420d910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f1c2420d500 0x7f1c2420d910 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/3601172036 reap_dead start 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/3601172036 reap_dead start 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/3601172036 reap_dead start 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/3601172036 reap_dead start 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2a9ac640 1 -- 172.21.15.67:0/3601172036 reap_dead start 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c0806f400 msgr2=0x7f1c08071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c0806f400 0x7f1c08071910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c200aa000 tx=0x7f1c240b53d0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c240c9350 msgr2=0x7f1c241f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c240c9350 0x7f1c241f3200 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1c14010f40 tx=0x7f1c1400e530 comp rx=0 tx=0).stop 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 shutdown_connections 2024-04-25T15:54:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c0806f400 0x7f1c08071910 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 --2- 172.21.15.67:0/3601172036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c240c9350 0x7f1c241f3200 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 >> 172.21.15.67:0/3601172036 conn(0x7f1c2406e420 msgr2=0x7f1c2406e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 shutdown_connections 2024-04-25T15:54:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:32.725+0000 7f1c2c40f640 1 -- 172.21.15.67:0/3601172036 wait complete. 2024-04-25T15:54:32.742 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:32.741+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:32.786 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:32.786 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:32.787 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:32.787 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:32.794 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:54:32.794 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:54:32.794 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-25T15:54:32.794 INFO:tasks.workunit.client.0.smithi067.stdout:iozone test complete. 2024-04-25T15:54:32.796 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T15:54:32.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T15:54:32.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:54:32.847 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-04-25T15:54:32.846+0000 7fd2ff8f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27900) UID: 0 2024-04-25T15:54:32.944 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:32.945+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:32.997 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:32.998 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:32.998 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:32.998 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:32.998 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:33.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:54:33.059 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-04-25T15:54:33.058+0000 7f7b7c4eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27893) UID: 0 2024-04-25T15:54:33.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3c77fe640 1 -- 172.21.15.67:0/2991021747 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v109) ==== 97+0+0 (secure 0 0 0) 0x7ff3d8010040 con 0x7ff3e00cbea0 2024-04-25T15:54:33.069 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-04-25T15:54:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3cc06f400 msgr2=0x7ff3cc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3cc06f400 0x7ff3cc071910 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0007fd0 tx=0x7ff3d0009040 comp rx=0 tx=0).stop 2024-04-25T15:54:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e00cbea0 msgr2=0x7ff3e015bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e00cbea0 0x7ff3e015bf10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800efc0 tx=0x7ff3d800c490 comp rx=0 tx=0).stop 2024-04-25T15:54:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 shutdown_connections 2024-04-25T15:54:33.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff3cc06f400 0x7ff3cc071910 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3e00cbea0 0x7ff3e015bf10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3e00caff0 0x7ff3e0150f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 --2- 172.21.15.67:0/2991021747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3e0072030 0x7ff3e014ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 >> 172.21.15.67:0/2991021747 conn(0x7ff3e00700b0 msgr2=0x7ff3e00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 shutdown_connections 2024-04-25T15:54:33.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:33.069+0000 7ff3e5412640 1 -- 172.21.15.67:0/2991021747 wait complete. 2024-04-25T15:54:33.109 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-04-25T15:54:33.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T15:54:33.153 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:33.153+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:33.182 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 254.08203342100001, 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:54:33.183 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:54:33.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:54:33.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:54:33.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 99.868158887999996, 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.191 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:54:33.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:54:33.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:54:33.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:54:33.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:54:33.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:54:33.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 69.597480935999997, 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:54:33.199 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:33.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:54:33.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:54:33.205 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2266 3.11 3:89528692:::100000001f8.000004d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.609314220000002, 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.38717112199999998, 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.206 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2266 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571186+0000", 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4569999999999999e-06 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784888+0000", 2024-04-25T15:54:33.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.213702742 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785181+0000", 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029301099999999998 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785201+0000", 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9683000000000001e-05 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785333+0000", 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013224600000000001 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785779+0000", 2024-04-25T15:54:33.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044548299999999999 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.786106+0000", 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032678999999999998 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870593+0000", 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084487524999999994 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958234+0000", 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087640933000000004 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958283+0000", 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8473999999999997e-05 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958354+0000", 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1778000000000003e-05 2024-04-25T15:54:33.212 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2272 3.13 3:ca3e88be:::100000001f8.000004dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.384565572, 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13173032000000001, 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.213 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2272 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795936+0000", 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.754e-06 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805167+0000", 2024-04-25T15:54:33.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092308890000000008 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805557+0000", 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039059400000000002 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805588+0000", 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0865999999999999e-05 2024-04-25T15:54:33.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805759+0000", 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017085599999999999 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806172+0000", 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000413416 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806552+0000", 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000379417 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905757+0000", 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099205023000000003 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927568+0000", 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021810800000000002 2024-04-25T15:54:33.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927616+0000", 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927662+0000", 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6535000000000001e-05 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.219 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2273 3.39 3:9fcbcc01:::100000001f8.000004dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.375313456999997, 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15983650499999999, 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.220 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2273 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805188+0000", 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816847+0000", 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011659119000000001 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817576+0000", 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00072975000000000004 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817601+0000", 2024-04-25T15:54:33.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4450000000000001e-05 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817741+0000", 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014058500000000001 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818074+0000", 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033291000000000001 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818385+0000", 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031034999999999998 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870618+0000", 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052233743999999999 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964814+0000", 2024-04-25T15:54:33.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094195216999999998 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964945+0000", 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013086799999999999 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965021+0000", 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6032000000000007e-05 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2275 3.12 3:492b2588:::100000001f8.000004df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.363638078000001, 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14794684399999999, 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.227 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2275 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816863+0000", 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4149999999999999e-06 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829311+0000", 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012448222 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829569+0000", 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025766800000000001 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829598+0000", 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.968e-05 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829786+0000", 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187746 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830129+0000", 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034284099999999998 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:33.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830479+0000", 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034961500000000001 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867146+0000", 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036667733000000001 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964688+0000", 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097541577000000004 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964748+0000", 2024-04-25T15:54:33.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9626000000000001e-05 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964806+0000", 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8721000000000002e-05 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2278 3.2b 3:d79b9024:::100000001f8.000004e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.351172820000002, 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135647779, 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.232 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2278 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829328+0000", 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7309999999999999e-06 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843821+0000", 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014493012 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844787+0000", 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096513199999999995 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844876+0000", 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9147999999999994e-05 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844972+0000", 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6566999999999999e-05 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845195+0000", 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222699 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845422+0000", 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022700699999999999 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870319+0000", 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024897319000000001 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964892+0000", 2024-04-25T15:54:33.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094572233000000006 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964938+0000", 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6646e-05 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964972+0000", 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4285000000000001e-05 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.237 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2305 3.2f e90/51 3:f6648653:::100000001f8.000004fd:head v 90'33, mlcod=90'30)", 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 44.343239029999999, 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12780160600000001, 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837261+0000", 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1679999999999998e-06 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839587+0000", 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00232607 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839620+0000", 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3503000000000001e-05 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839642+0000", 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1302e-05 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839754+0000", 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112454 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839798+0000", 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4240999999999999e-05 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965003+0000", 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12520515600000001 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965060+0000", 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6712000000000002e-05 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3917 3.27 3:e652e8b8:::100000001f8.00000149:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.682428525999999, 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16896387500000001, 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3917 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498071+0000", 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4779999999999998e-06 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509296+0000", 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011224433000000001 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509572+0000", 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027576500000000003 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509591+0000", 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9466e-05 2024-04-25T15:54:33.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509677+0000", 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5607999999999998e-05 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509773+0000", 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6136000000000002e-05 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509942+0000", 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016951999999999999 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.610111+0000", 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.100168231 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666922+0000", 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056811077000000001 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666991+0000", 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9758000000000005e-05 2024-04-25T15:54:33.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667033+0000", 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1403000000000003e-05 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3918 3.11 3:8b260724:::100000001f8.0000014a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.671192312999999, 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15748047100000001, 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.247 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3918 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:33.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509307+0000", 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.12e-06 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515132+0000", 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058250860000000002 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515164+0000", 2024-04-25T15:54:33.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1563000000000003e-05 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515179+0000", 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5102e-05 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515262+0000", 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2917999999999995e-05 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515360+0000", 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7532000000000001e-05 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515512+0000", 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152442 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654409+0000", 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13889727499999999 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666644+0000", 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012234584999999999 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666730+0000", 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5793999999999999e-05 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666786+0000", 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6054000000000002e-05 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3922 3.d 3:b3f0d00e:::100000001f8.0000014e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.661830391999999, 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.150212927, 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3922 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518670+0000", 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1039999999999998e-06 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529034+0000", 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010364237 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529073+0000", 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8407e-05 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529092+0000", 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9411000000000001e-05 2024-04-25T15:54:33.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529203+0000", 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011113 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529330+0000", 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012611499999999999 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529517+0000", 2024-04-25T15:54:33.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187639 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667032+0000", 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13751481300000001 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668779+0000", 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017469740000000001 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668835+0000", 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6251000000000001e-05 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668880+0000", 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4845999999999997e-05 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3920 3.26 e97/18 3:6422fb70:::100000001f8.0000014c:head v 97'93, mlcod=96'91)", 2024-04-25T15:54:33.258 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.656807243999999, 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14317563999999999, 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523692+0000", 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8050000000000001e-06 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525897+0000", 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022045290000000002 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525923+0000", 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6225e-05 2024-04-25T15:54:33.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525941+0000", 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8304e-05 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526045+0000", 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104158 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526076+0000", 2024-04-25T15:54:33.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0454e-05 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666788+0000", 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140712106 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666866+0000", 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8059000000000006e-05 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.262 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3926 3.d 3:b20dc027:::100000001f8.00000152:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.642320696999999, 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131821096, 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3926 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:33.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538180+0000", 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4690000000000002e-06 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542940+0000", 2024-04-25T15:54:33.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047600250000000002 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542988+0000", 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7633999999999998e-05 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543006+0000", 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8114999999999999e-05 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543103+0000", 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6723000000000002e-05 2024-04-25T15:54:33.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543283+0000", 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018030100000000001 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543476+0000", 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019295200000000001 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668734+0000", 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12525763200000001 2024-04-25T15:54:33.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669901+0000", 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011670890000000001 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669955+0000", 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4373999999999998e-05 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669998+0000", 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:54:33.267 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3971 3.3f 3:febcb5ea:::100000001f8.0000017f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 29.362796940999999, 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15452237999999999, 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.268 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3971 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817703+0000", 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822518+0000", 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048147939999999998 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822563+0000", 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4515000000000003e-05 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822577+0000", 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.432e-05 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822664+0000", 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7440000000000003e-05 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822786+0000", 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012110800000000001 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822960+0000", 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017468900000000001 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945290+0000", 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122329907 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972057+0000", 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026767102000000001 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972139+0000", 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2070000000000005e-05 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972223+0000", 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3565000000000004e-05 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5059 3.26 e99/18 3:65b3bea5:::100000001f8.000005bf:head v 99'112, mlcod=99'109)", 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.16310777, 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131194432, 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:33.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017392+0000", 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4650000000000001e-06 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021054+0000", 2024-04-25T15:54:33.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036613240000000001 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021076+0000", 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2067000000000001e-05 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021092+0000", 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6209999999999999e-05 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021178+0000", 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6260000000000004e-05 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021208+0000", 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0046999999999999e-05 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148491+0000", 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12728336500000001 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148584+0000", 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2694000000000004e-05 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5063 3.13 3:c9096c28:::100000001f8.000005c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.151555613999999, 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13114490600000001, 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5063 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028946+0000", 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0520000000000003e-06 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035431+0000", 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064851509999999998 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035461+0000", 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9547999999999999e-05 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035474+0000", 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2880999999999999e-05 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035550+0000", 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6027999999999998e-05 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035622+0000", 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2366999999999993e-05 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035727+0000", 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105561 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.150510+0000", 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11478287099999999 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159903+0000", 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093925880000000003 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159983+0000", 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0122000000000003e-05 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160087+0000", 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103737 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5730 3.e 3:7357dd18:::100000001f8.0000085e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:33.283 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.421259395, 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14322902300000001, 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5730 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759240+0000", 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3910000000000001e-06 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765109+0000", 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058681660000000002 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765139+0000", 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9981e-05 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765149+0000", 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0125e-05 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765215+0000", 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6700999999999997e-05 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765293+0000", 2024-04-25T15:54:33.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7883e-05 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765394+0000", 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100869 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.893428+0000", 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128033758 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902331+0000", 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089030610000000003 2024-04-25T15:54:33.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902403+0000", 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1891999999999995e-05 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902467+0000", 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4196000000000007e-05 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5769 3.d 3:b350774d:::100000001f8.00000885:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:33.288 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.188541650000001, 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12752495899999999, 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5769 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991958+0000", 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998738+0000", 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067804069999999996 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998782+0000", 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3738e-05 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998800+0000", 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7907e-05 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998944+0000", 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144089 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999140+0000", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195465 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999415+0000", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027557100000000001 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.070872+0000", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071456602999999994 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119350+0000", 2024-04-25T15:54:33.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048478354000000001 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119421+0000", 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0970999999999995e-05 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119481+0000", 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9802000000000001e-05 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5780 3.f 3:f20b812e:::100000001f8.00000890:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:33.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 19.140360571999999, 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13282865699999999, 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5780 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:33.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040139+0000", 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9570000000000001e-06 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047681+0000", 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075419279999999998 2024-04-25T15:54:33.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047720+0000", 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8794999999999997e-05 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047740+0000", 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0091000000000001e-05 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047828+0000", 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8745000000000005e-05 2024-04-25T15:54:33.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047922+0000", 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3076000000000001e-05 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048095+0000", 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000173134 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.170485+0000", 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122390033 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172900+0000", 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002415482 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172933+0000", 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3021000000000001e-05 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172966+0000", 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2394999999999997e-05 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.298 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.299 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.299 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:33.299 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:33.299 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:33.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T15:54:33.301 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-25T15:54:33.297+0000 7fee78b39640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27693) UID: 0 2024-04-25T15:54:33.401 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-04-25T15:54:33.398+0000 7fb8fb299640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27896) UID: 0 2024-04-25T15:54:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPush(3.0 109/108 [PushOp(3:034274dd:::100000001f8.000004c6:head, version: 98'70, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:034274dd:::100000001f8.000004c6:head@98'70, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-25T15:54:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:33.460409+0000", 2024-04-25T15:54:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.001862669, 2024-04-25T15:54:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018912689999999999, 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "started", 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.460409+0000", 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.460409+0000", 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.460411+0000", 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9999999999999999e-06 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.460906+0000", 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00049503999999999998 2024-04-25T15:54:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.460932+0000", 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5367e-05 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.460951+0000", 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.98e-05 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.461074+0000", 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122376 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.461092+0000", 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7978e-05 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:33.462301+0000", 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001208708 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 1 2024-04-25T15:54:33.464 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:33.479 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:33.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T15:54:33.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-25T15:54:33.513 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T15:54:33.513 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-25T15:54:33.523 INFO:tasks.thrashosds:joining thrashosds 2024-04-25T15:54:33.523 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-25T15:54:33.513+0000 7f3afc0e0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27695) UID: 0 2024-04-25T15:54:33.718 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:33.718 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:33.718 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:33.718 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:33.719 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:33.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:668 3.1 e81/76 3:83ba1de1:::100000001f7.00000098:head v 81'14, mlcod=81'12)", 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.043439331000002, 2024-04-25T15:54:33.917 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.045409913000000003, 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872465+0000", 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.918 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.872468+0000", 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3659999999999999e-06 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879062+0000", 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0065941039999999999 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879116+0000", 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4372000000000001e-05 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.919 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879152+0000", 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6139000000000001e-05 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879318+0000", 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016595499999999999 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.879365+0000", 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6897999999999997e-05 2024-04-25T15:54:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917803+0000", 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038437773000000001 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.917875+0000", 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2305999999999995e-05 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:637 3.20 e81/18 3:07263727:::100000001f7.00000079:head v 81'17, mlcod=81'15)", 2024-04-25T15:54:33.921 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.033950840000003, 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040244958999999997, 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.922 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881954+0000", 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.881957+0000", 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3890000000000001e-06 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888368+0000", 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006410682 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.923 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888395+0000", 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7477000000000001e-05 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888414+0000", 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9046e-05 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888512+0000", 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7876999999999998e-05 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.924 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.888546+0000", 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3204000000000003e-05 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922131+0000", 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033585168999999998 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.922199+0000", 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8114999999999994e-05 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.925 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1709 3.b e89/21 3:d0c4adc9:::100000001f8.000002a9:head v 89'35, mlcod=89'33)", 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.916974852999999, 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041562524000000003, 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.926 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998930+0000", 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.998932+0000", 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7570000000000001e-06 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000395+0000", 2024-04-25T15:54:33.927 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014629389999999999 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000414+0000", 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9908999999999999e-05 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000432+0000", 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249999999999999e-05 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000518+0000", 2024-04-25T15:54:33.928 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6501000000000004e-05 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.000548+0000", 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9777e-05 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040424+0000", 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039875586999999997 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.929 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.040492+0000", 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8804000000000001e-05 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1710 3.b e89/21 3:d079ce42:::100000001f8.000002aa:head v 89'36, mlcod=89'33)", 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 48.910546136000001, 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040049727, 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.930 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005359+0000", 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.005360+0000", 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9290000000000001e-06 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.931 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006358+0000", 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00099724699999999993 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006377+0000", 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9656e-05 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006392+0000", 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4207e-05 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.932 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006477+0000", 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4989999999999998e-05 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.006507+0000", 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0786e-05 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045329+0000", 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038821829000000002 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:45.045408+0000", 2024-04-25T15:54:33.933 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9083000000000002e-05 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2273 3.39 e90/18 3:9fcbcc01:::100000001f8.000004dd:head v 90'54, mlcod=90'52)", 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.090396890999997, 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035128116000000001, 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.934 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825508+0000", 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.825510+0000", 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.933e-06 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.935 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837142+0000", 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011632136 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837175+0000", 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3166000000000001e-05 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837197+0000", 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1495e-05 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837293+0000", 2024-04-25T15:54:33.936 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6353000000000004e-05 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.837329+0000", 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5901000000000001e-05 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860257+0000", 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.022927804999999999 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.860636+0000", 2024-04-25T15:54:33.937 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000379327 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2392 3.32 e90/40 3:4cdbb504:::100000001f8.00000554:head v 90'45, mlcod=90'43)", 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.506582129999998, 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.044535299, 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.938 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409323+0000", 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409325+0000", 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2220000000000001e-06 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410846+0000", 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001521454 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410866+0000", 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0129999999999999e-05 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410882+0000", 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5452e-05 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.940 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410962+0000", 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9833999999999995e-05 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.410991+0000", 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.968e-05 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453793+0000", 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042801621999999998 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.941 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.453858+0000", 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4905000000000003e-05 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2394 3.6 e90/76 3:63a5c66e:::100000001f8.00000556:head v 90'46, mlcod=90'44)", 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.491887007000003, 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038940880999999997, 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.942 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424018+0000", 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424019+0000", 2024-04-25T15:54:33.943 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6160000000000001e-06 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428140+0000", 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0041208529999999998 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428169+0000", 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8824999999999999e-05 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428190+0000", 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0894999999999999e-05 2024-04-25T15:54:33.944 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428283+0000", 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3277999999999997e-05 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.428318+0000", 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5191000000000002e-05 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462890+0000", 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034571283000000001 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.945 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.462959+0000", 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8940000000000001e-05 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:2396 3.3d e90/40 3:bc4ba9bc:::100000001f8.00000558:head v 90'44, mlcod=90'42)", 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 44.482081786000002, 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033872159999999998, 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.946 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433823+0000", 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.433825+0000", 2024-04-25T15:54:33.947 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.951e-06 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435239+0000", 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014141729999999999 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435260+0000", 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0775e-05 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435278+0000", 2024-04-25T15:54:33.948 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8317000000000001e-05 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435363+0000", 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4441e-05 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.435395+0000", 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1939e-05 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467549+0000", 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032154061999999997 2024-04-25T15:54:33.949 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.467695+0000", 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014650199999999999 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3866 3.1 e97/76 3:81b781a2:::100000001f8.00000116:head v 97'69, mlcod=96'67)", 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.671899848999999, 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035311905999999997, 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244005+0000", 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.951 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.244007+0000", 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.295e-06 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249687+0000", 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056798860000000003 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249709+0000", 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2393999999999999e-05 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.952 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249729+0000", 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9536000000000001e-05 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249849+0000", 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000120068 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.249888+0000", 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9103999999999997e-05 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.953 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279251+0000", 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029363087999999999 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279317+0000", 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5535000000000002e-05 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:3865 3.20 e97/18 3:0689c4f5:::100000001f8.00000115:head v 97'72, mlcod=94'70)", 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.669091152, 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032711928000000001, 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.955 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246814+0000", 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.246817+0000", 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.27e-06 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252975+0000", 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006157872 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.956 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.252993+0000", 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8070000000000001e-05 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253009+0000", 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6169999999999999e-05 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253090+0000", 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1312999999999999e-05 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.957 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.253111+0000", 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1009000000000002e-05 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279471+0000", 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026359693 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.279525+0000", 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4530999999999997e-05 2024-04-25T15:54:33.958 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4046 3.3a e97/18 3:5c489b63:::100000001f8.000001ca:head v 97'64, mlcod=97'62)", 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 29.657257213000001, 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037080704999999999, 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.959 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258647+0000", 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.960 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.258649+0000", 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.637e-06 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262631+0000", 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0039815079999999999 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262647+0000", 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6745999999999999e-05 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.961 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262663+0000", 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6116e-05 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262779+0000", 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011513700000000001 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.262807+0000", 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8200000000000001e-05 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.962 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295660+0000", 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032853392000000002 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.295728+0000", 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7968999999999999e-05 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.963 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4727 3.1b e98/76 3:db35eb1b:::100000001f8.00000473:head v 98'76, mlcod=98'74)", 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.774811998000001, 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.036260694000000003, 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.964 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141093+0000", 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.141095+0000", 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9870000000000002e-06 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145531+0000", 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0044364670000000004 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.965 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145560+0000", 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8354999999999999e-05 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145578+0000", 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8715e-05 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145701+0000", 2024-04-25T15:54:33.966 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012246800000000001 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.145732+0000", 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1615000000000001e-05 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177281+0000", 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.031549063000000002 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.967 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.177353+0000", 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2024e-05 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:4899 3.2b e98/40 3:d5e979f0:::100000001f8.0000051f:head v 98'72, mlcod=98'70)", 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 24.735638371, 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034607091, 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.968 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180266+0000", 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.180267+0000", 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6499999999999998e-07 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.969 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184620+0000", 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043528660000000004 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184641+0000", 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1126000000000001e-05 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184660+0000", 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9014000000000001e-05 2024-04-25T15:54:33.970 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184747+0000", 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6829999999999994e-05 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.184776+0000", 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9473999999999999e-05 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214781+0000", 2024-04-25T15:54:33.971 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030004797 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.214873+0000", 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2219000000000006e-05 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5058 3.17 e99/40 3:eaf2c8b6:::100000001f8.000005be:head v 99'80, mlcod=99'78)", 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 23.901228811999999, 2024-04-25T15:54:33.972 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035602108, 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014676+0000", 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.973 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.014679+0000", 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.095e-06 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021382+0000", 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0067033750000000001 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.974 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021413+0000", 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0846000000000002e-05 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021435+0000", 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1378000000000001e-05 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021583+0000", 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014862400000000001 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.975 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021620+0000", 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6718999999999999e-05 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050194+0000", 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028573676999999999 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.050278+0000", 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4394000000000005e-05 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.976 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5611 3.20 e100/18 3:07896b3f:::100000001f8.000007e7:head v 100'100, mlcod=99'98)", 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.832375753000001, 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.034695030000000002, 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.977 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083529+0000", 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.083531+0000", 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1179999999999999e-06 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092805+0000", 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.009273524 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092830+0000", 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5894e-05 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092847+0000", 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6762999999999999e-05 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092944+0000", 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.6612000000000003e-05 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.092974+0000", 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0568000000000003e-05 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118174+0000", 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025199813000000001 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.118224+0000", 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9737999999999997e-05 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5680 3.10 e100/18 3:0bf23c77:::100000001f8.0000082c:head v 100'114, mlcod=100'112)", 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.427092712, 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.039581641000000001, 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:33.981 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488812+0000", 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.488821+0000", 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9060000000000008e-06 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.982 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494301+0000", 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054798900000000003 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494321+0000", 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0271999999999998e-05 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494338+0000", 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7249e-05 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494420+0000", 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2107999999999994e-05 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.494443+0000", 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3071e-05 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528334+0000", 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033890497999999998 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.528394+0000", 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9647e-05 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5986 3.10 e100/18 3:09f7ffbc:::100000001f8.0000095e:head v 100'117, mlcod=100'115)", 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.692807114000001, 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.047845334000000003, 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223098+0000", 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.986 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.223099+0000", 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6330000000000001e-06 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230025+0000", 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0069255319999999999 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230048+0000", 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3135000000000001e-05 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230062+0000", 2024-04-25T15:54:33.987 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4119e-05 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230143+0000", 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0508999999999998e-05 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230165+0000", 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2314999999999999e-05 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270851+0000", 2024-04-25T15:54:33.988 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040685763999999999 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.270943+0000", 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2326999999999999e-05 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5978 3.b e100/21 3:d3280d2b:::100000001f8.00000956:head v 100'98, mlcod=100'96)", 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.684846962000002, 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.051283404999999997, 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.989 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231058+0000", 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.231060+0000", 2024-04-25T15:54:33.990 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1169999999999998e-06 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233300+0000", 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022402749999999999 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233324+0000", 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4261e-05 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233347+0000", 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.298e-05 2024-04-25T15:54:33.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233488+0000", 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000140359 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.233523+0000", 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5040000000000003e-05 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282285+0000", 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048762358999999998 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.992 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282341+0000", 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.6014000000000003e-05 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5988 3.3d e100/40 3:bdc62163:::100000001f8.00000960:head v 100'102, mlcod=100'100)", 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.675850514, 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043901969999999998, 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240054+0000", 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.240056+0000", 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.674e-06 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242229+0000", 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021732269999999998 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242254+0000", 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4661999999999999e-05 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242271+0000", 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.681e-05 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242362+0000", 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1411999999999998e-05 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.242396+0000", 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3983000000000003e-05 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283805+0000", 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.041409488000000001 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.283956+0000", 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015071400000000001 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:5987 3.2b e100/40 3:d716bd7b:::100000001f8.0000095f:head v 100'88, mlcod=100'86)", 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 18.671792101000001, 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038273439999999999, 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:33.997 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244113+0000", 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:33.998 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.244115+0000", 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0020000000000001e-06 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248434+0000", 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0043194920000000003 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248458+0000", 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3941e-05 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:33.999 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248477+0000", 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8995999999999999e-05 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248566+0000", 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9331000000000002e-05 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.248596+0000", 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9332e-05 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282347+0000", 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033750952000000001 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282386+0000", 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.9394000000000003e-05 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.001 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:34.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T15:54:34.179 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:34.179 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:34.179 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:34.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-25T15:54:34.375 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:34.376 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:34.376 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:34.376 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:34.376 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:34.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-25T15:54:34.589 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:34.589 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(mds.1.10:27 2.9 2:91bd965c:::501.00000000:head [writefull 0~90] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e20)", 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 232.34141510000001, 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93975624899999999, 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247734+0000", 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247735+0000", 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.296e-06 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:42.247743+0000", 2024-04-25T15:54:34.591 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.413e-06 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183513+0000", 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.93577073200000005 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183539+0000", 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.5378999999999999e-05 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.183732+0000", 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000193129 2024-04-25T15:54:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184166+0000", 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00043362499999999998 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.184583+0000", 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00041794 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.593 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.186344+0000", 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017603460000000001 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187164+0000", 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00081991599999999996 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187351+0000", 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018709799999999999 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.594 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:50:43.187490+0000", 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013937500000000001 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:676 3.18 3:191fe66d:::100000001f7.000000a0:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.659376769000005, 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.053839007000000001, 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.595 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 676 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929772+0000", 2024-04-25T15:54:34.596 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.929776+0000", 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.743e-06 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931715+0000", 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019394130000000001 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.597 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931754+0000", 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8779000000000003e-05 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931769+0000", 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5231999999999999e-05 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.931867+0000", 2024-04-25T15:54:34.598 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7485000000000006e-05 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932043+0000", 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017616100000000001 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.932181+0000", 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013769100000000001 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.599 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.942677+0000", 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010495931 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983477+0000", 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040800621000000002 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983546+0000", 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9138000000000001e-05 2024-04-25T15:54:34.600 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.983611+0000", 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4813000000000004e-05 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:733 3.22 3:44586a4d:::100000001f7.000000d9:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 70.280910720999998, 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037465947999999999, 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.601 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 733 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.602 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308238+0000", 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.308242+0000", 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.399e-06 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309826+0000", 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015840349999999999 2024-04-25T15:54:34.603 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309863+0000", 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7122000000000001e-05 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309880+0000", 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6888000000000002e-05 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.604 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.309986+0000", 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105979 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310169+0000", 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018349699999999999 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.310353+0000", 2024-04-25T15:54:34.605 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018415799999999999 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.342377+0000", 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032023169999999997 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345569+0000", 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0031928680000000002 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.606 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345644+0000", 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4140999999999994e-05 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:24.345704+0000", 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0690999999999999e-05 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_repop(client.4815.0:1589 3.1a e88/76 3:590c0ff7:::100000001f8.00000231:head v 88'30, mlcod=88'28)", 2024-04-25T15:54:34.607 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.342325719999998, 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032177728000000003, 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246823+0000", 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.608 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.246826+0000", 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1900000000000002e-06 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251343+0000", 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0045171680000000002 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251364+0000", 2024-04-25T15:54:34.609 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1444999999999999e-05 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251375+0000", 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.115e-05 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251458+0000", 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3190999999999997e-05 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.610 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.251485+0000", 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6661000000000001e-05 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.278940+0000", 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027455222000000001 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.279001+0000", 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0701e-05 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.611 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1595 3.18 3:1adf79c2:::100000001f8.00000237:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 50.306964682999997, 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038955741000000002, 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1595 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.612 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282184+0000", 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.613 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.282187+0000", 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1569999999999998e-06 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284089+0000", 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001902672 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284127+0000", 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7663000000000002e-05 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.614 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284141+0000", 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4567e-05 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284236+0000", 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4895999999999994e-05 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284417+0000", 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018094500000000001 2024-04-25T15:54:34.615 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.284602+0000", 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018422800000000001 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.293831+0000", 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0092292650000000004 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321052+0000", 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027220978 2024-04-25T15:54:34.616 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321098+0000", 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6520000000000002e-05 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.321140+0000", 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.1850000000000001e-05 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.617 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2389 3.18 3:1bdc9869:::100000001f8.00000551:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.199913481000003, 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030905229999999999, 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2389 2024-04-25T15:54:34.618 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389236+0000", 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.619 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.389238+0000", 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7099999999999999e-06 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391140+0000", 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001901359 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391174+0000", 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4165000000000002e-05 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391187+0000", 2024-04-25T15:54:34.620 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3188e-05 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391277+0000", 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0267999999999997e-05 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391455+0000", 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017728700000000001 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.391618+0000", 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163308 2024-04-25T15:54:34.621 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.419720+0000", 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.028102374999999999 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420001+0000", 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00028060900000000002 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420083+0000", 2024-04-25T15:54:34.622 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2010999999999998e-05 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.420141+0000", 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7949999999999999e-05 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2392 3.32 3:4cdbb504:::100000001f8.00000554:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.183214722999999, 2024-04-25T15:54:34.623 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.048266518000000001, 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2392 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.624 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405934+0000", 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.405937+0000", 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6759999999999999e-06 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408696+0000", 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027593589999999999 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.625 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408735+0000", 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8516000000000002e-05 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408749+0000", 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4117000000000001e-05 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.408844+0000", 2024-04-25T15:54:34.626 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4891999999999999e-05 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409024+0000", 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018000199999999999 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.409200+0000", 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017591800000000001 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.427538+0000", 2024-04-25T15:54:34.627 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018337728000000001 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454069+0000", 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.026531451000000001 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454141+0000", 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2316000000000004e-05 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454201+0000", 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9543000000000002e-05 2024-04-25T15:54:34.628 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2393 3.22 3:45fb7032:::100000001f8.00000555:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.176822152, 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052582106000000003, 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.629 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2393 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412327+0000", 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.630 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.412330+0000", 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3119999999999998e-06 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413928+0000", 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597521 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413971+0000", 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3086e-05 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.631 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.413988+0000", 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7336000000000001e-05 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414091+0000", 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103099 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414245+0000", 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015350300000000001 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.632 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.414359+0000", 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011410500000000001 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.446661+0000", 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032302571000000002 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464781+0000", 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018119283999999999 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.633 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464856+0000", 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5153000000000004e-05 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.464909+0000", 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3136000000000001e-05 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.634 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2395 3.18 3:183ba26c:::100000001f8.00000557:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 45.164422043999998, 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029801055999999999, 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2395 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.635 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424727+0000", 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.424735+0000", 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1780000000000003e-06 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.636 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426303+0000", 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015679170000000001 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426349+0000", 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6343e-05 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426370+0000", 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0329000000000001e-05 2024-04-25T15:54:34.637 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426458+0000", 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.831e-05 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426609+0000", 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000150629 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.638 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.426735+0000", 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012625900000000001 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.441627+0000", 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014891824999999999 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454417+0000", 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012789994000000001 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.639 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454476+0000", 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8752000000000003e-05 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:49.454528+0000", 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.2519999999999999e-05 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.640 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3145 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 40.154463309999997, 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030601595999999998, 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3145 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.641 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434686+0000", 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.434689+0000", 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.005e-06 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.642 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436062+0000", 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001373682 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436096+0000", 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3797000000000002e-05 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436107+0000", 2024-04-25T15:54:34.643 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0672999999999999e-05 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436177+0000", 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0069999999999998e-05 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436346+0000", 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000169168 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.644 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.436510+0000", 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000163807 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.443699+0000", 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007189394 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465141+0000", 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021441386999999999 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.645 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465223+0000", 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1920000000000002e-05 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:54.465287+0000", 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4692999999999999e-05 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3841 3.18 3:1a4016cc:::100000001f8.000000fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-04-25T15:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 31.503466476, 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043270667999999998, 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3841 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085683+0000", 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.085686+0000", 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4510000000000001e-06 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088301+0000", 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002614987 2024-04-25T15:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088348+0000", 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7057e-05 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088362+0000", 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4352000000000001e-05 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088457+0000", 2024-04-25T15:54:34.649 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4794000000000001e-05 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088583+0000", 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012605400000000001 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.088776+0000", 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019268700000000001 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.650 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.107828+0000", 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019051702 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128788+0000", 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.020960433000000001 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128888+0000", 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9493000000000004e-05 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.128953+0000", 2024-04-25T15:54:34.651 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5658000000000001e-05 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3977 3.18 3:1b05c35d:::100000001f8.00000185:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.746927893999999, 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13853472, 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.652 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3977 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842221+0000", 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.842224+0000", 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1130000000000001e-06 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844135+0000", 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019102920000000001 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844181+0000", 2024-04-25T15:54:34.654 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6616000000000002e-05 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844202+0000", 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0542000000000001e-05 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844358+0000", 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000156379 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.655 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844555+0000", 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000196651 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.844759+0000", 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020436799999999999 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.853781+0000", 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0090217130000000006 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.656 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980620+0000", 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.126839061 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980690+0000", 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9596000000000002e-05 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980756+0000", 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.6389000000000004e-05 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4099 3.18 3:181e4796:::100000001f8.000001ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 30.053855687999999, 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030713761999999999, 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4099 2024-04-25T15:54:34.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535293+0000", 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.535297+0000", 2024-04-25T15:54:34.659 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2049999999999998e-06 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536942+0000", 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00164504 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536980+0000", 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8736999999999999e-05 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.660 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.536997+0000", 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6653000000000001e-05 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537091+0000", 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.3786e-05 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537191+0000", 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9840000000000006e-05 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.661 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.537331+0000", 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014034900000000001 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.556229+0000", 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.018898278000000001 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565921+0000", 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0096916850000000002 2024-04-25T15:54:34.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.565961+0000", 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0290999999999998e-05 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.566007+0000", 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5898e-05 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.663 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4812 3.18 3:194fa8d8:::100000001f8.000004c8:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.905441473, 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038646409, 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4812 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.664 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683708+0000", 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.683711+0000", 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7170000000000002e-06 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685822+0000", 2024-04-25T15:54:34.665 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0021105619999999999 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685876+0000", 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3798999999999998e-05 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.685910+0000", 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.4727999999999997e-05 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.666 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686022+0000", 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111549 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686150+0000", 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012826800000000001 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.686332+0000", 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000181538 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.667 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.693171+0000", 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068389519999999997 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722249+0000", 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.029078171 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722294+0000", 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4756000000000003e-05 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.668 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:08.722354+0000", 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0368999999999997e-05 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4967 3.22 3:478cb4f4:::100000001f8.00000563:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.047507759999998, 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035932619999999998, 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.669 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4967 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.670 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541641+0000", 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.541672+0000", 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0978000000000001e-05 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543306+0000", 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016338310000000001 2024-04-25T15:54:34.671 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543342+0000", 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5994000000000001e-05 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543360+0000", 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7929999999999999e-05 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543470+0000", 2024-04-25T15:54:34.672 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110319 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543624+0000", 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000153549 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.543799+0000", 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017463299999999999 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.673 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.555562+0000", 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011763177999999999 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577476+0000", 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021913818000000002 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577529+0000", 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3711000000000001e-05 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.674 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.577574+0000", 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4678999999999997e-05 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5705 3.18 3:1bfd1a0f:::100000001f8.00000845:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.964784296000001, 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080110828999999995, 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5705 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.676 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624365+0000", 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.624368+0000", 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.117e-06 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626198+0000", 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001829968 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.677 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626234+0000", 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6220000000000002e-05 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626249+0000", 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4742999999999999e-05 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626341+0000", 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2477000000000002e-05 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.678 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626462+0000", 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000121191 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.626636+0000", 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017351000000000001 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.635055+0000", 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084194100000000004 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.679 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704341+0000", 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.069285804000000006 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704412+0000", 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0674999999999997e-05 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.704476+0000", 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3713999999999993e-05 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5751 3.18 3:1b3fa6ee:::100000001f8.00000873:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.712718414000001, 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.037922083000000002, 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5751 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.681 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876431+0000", 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.876434+0000", 2024-04-25T15:54:34.682 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7910000000000002e-06 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878480+0000", 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020457800000000001 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878539+0000", 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8640000000000001e-05 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.683 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878587+0000", 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.7746e-05 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878724+0000", 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000137572 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.878868+0000", 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014331899999999999 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.684 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.879018+0000", 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015069000000000001 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.886482+0000", 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.007464048 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914229+0000", 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.027746804 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914300+0000", 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0952999999999996e-05 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914353+0000", 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.274e-05 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.686 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5755 3.32 3:4c40e2d2:::100000001f8.00000877:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.693682258999999, 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033152545999999998, 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5755 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.687 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895467+0000", 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.895470+0000", 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7839999999999999e-06 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.688 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898117+0000", 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0026470959999999998 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898163+0000", 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6406000000000003e-05 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898183+0000", 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0092999999999999e-05 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.689 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898295+0000", 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011160800000000001 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898422+0000", 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012762099999999999 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.898601+0000", 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017858500000000001 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.914502+0000", 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015900724000000001 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928454+0000", 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013952585999999999 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928552+0000", 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7299000000000005e-05 2024-04-25T15:54:34.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.928619+0000", 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7743999999999994e-05 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5966 3.18 3:1b2b62aa:::100000001f8.0000094a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.489038866000001, 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.043485786999999998, 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.692 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5966 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.693 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100110+0000", 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.100114+0000", 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.315e-06 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.101977+0000", 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018630109999999999 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.694 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102022+0000", 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.5636000000000001e-05 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102040+0000", 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8e-05 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102140+0000", 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9720000000000001e-05 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102270+0000", 2024-04-25T15:54:34.695 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130409 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.102415+0000", 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000144492 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.118110+0000", 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.015694725999999999 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143459+0000", 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.025349846999999998 2024-04-25T15:54:34.696 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143534+0000", 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4839999999999995e-05 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.143596+0000", 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.1791000000000005e-05 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:6009 3.18 3:1b7f6e54:::100000001f8.00000975:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:34.697 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 19.223369025, 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030441923999999999, 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 6009 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.698 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365780+0000", 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.365782+0000", 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3889999999999999e-06 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367595+0000", 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001812976 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.699 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367633+0000", 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7185999999999999e-05 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367647+0000", 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4681999999999999e-05 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367735+0000", 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7474999999999999e-05 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.700 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367814+0000", 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9326000000000006e-05 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.367936+0000", 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000122077 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.375787+0000", 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0078512010000000004 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.701 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396095+0000", 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.02030732 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396164+0000", 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9549999999999996e-05 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.396222+0000", 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7741999999999997e-05 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.702 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.703 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.703 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:34.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-25T15:54:34.901 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:34.901 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:34.901 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.901 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "MOSDPGPushReply(3.2e 110/109 [PushReplyOp(3:77f5c19e:::100000001f8.00000583:head)])", 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:34.901627+0000", 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 0.0015492909999999999, 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015793599999999999, 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "reached pg", 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:34.901627+0000", 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.902 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:34.901627+0000", 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:34.901629+0000", 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4619999999999999e-06 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:34.901642+0000", 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2682e-05 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.903 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:34.901670+0000", 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8257999999999999e-05 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:34.901690+0000", 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0438e-05 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:34.901805+0000", 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011444 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.904 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:34.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.905 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:34.905 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2024-04-25T15:54:34.905 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 1 2024-04-25T15:54:34.905 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:34.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-25T15:54:35.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-25T15:54:35.112 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:35.112 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:35.112 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:35.112 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:35.112 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:35.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-25T15:54:35.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.293+0000 7f666f23d640 1 Processor -- start 2024-04-25T15:54:35.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.293+0000 7f666f23d640 1 -- start start 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666f23d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f666815dd00 0x7f6668068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666f23d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6668160990 con 0x7f66681569a0 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666f23d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6668160c70 con 0x7f6668156db0 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666f23d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f666815e110 con 0x7f666815dd00 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f6667fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66681569a0 0x7f666806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50028/0 (socket says 172.21.15.67:50028) 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f6667fff640 1 -- 172.21.15.67:0/2645624280 learned_addr learned my addr 172.21.15.67:0/2645624280 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666cfd9640 1 --2- 172.21.15.67:0/2645624280 >> v2:172.21.15.176:3300/0 conn(0x7f666815dd00 0x7f6668068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886496314 0 0) 0x7f6668160990 con 0x7f66681569a0 2024-04-25T15:54:35.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6658003530 con 0x7f66681569a0 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 172021391 0 0) 0x7f6658003530 con 0x7f66681569a0 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 >> v2:172.21.15.176:3300/0 conn(0x7f666815dd00 msgr2=0x7f6668068830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 --2- 172.21.15.67:0/2645624280 >> v2:172.21.15.176:3300/0 conn(0x7f666815dd00 0x7f6668068830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 >> v1:172.21.15.176:6789/0 conn(0x7f6668156db0 legacy=0x7f6668167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66680c3bf0 con 0x7f66681569a0 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f6654001800 con 0x7f66681569a0 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666cfd9640 1 --2- 172.21.15.67:0/2645624280 >> v2:172.21.15.176:3300/0 conn(0x7f666815dd00 0x7f6668068830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 >> v1:172.21.15.67:6789/0 conn(0x7f66681569a0 legacy=0x7f666806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580039c0 0x7f6658003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6658004810 0x7f6658004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66580051d0 0x7f665800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f666815e110 con 0x7f66580051d0 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f665800c170 con 0x7f6658004810 2024-04-25T15:54:35.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f665800c370 con 0x7f66580039c0 2024-04-25T15:54:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666d7da640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66580051d0 0x7f665800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666d7da640 1 -- 172.21.15.67:0/2645624280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580039c0 msgr2=0x7f6658003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666d7da640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580039c0 0x7f6658003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666d7da640 1 -- 172.21.15.67:0/2645624280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6658004810 msgr2=0x7f6658004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666d7da640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6658004810 0x7f6658004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:35.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.297+0000 7f666d7da640 1 -- 172.21.15.67:0/2645624280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6658003530 con 0x7f66580051d0 2024-04-25T15:54:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666d7da640 1 -- 172.21.15.67:0/2645624280 reap_dead start 2024-04-25T15:54:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666d7da640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66580051d0 0x7f665800b9d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6660001e40 tx=0x7f666001c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4152226269575fed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660001a90 con 0x7f66580051d0 2024-04-25T15:54:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6660003b80 con 0x7f66580051d0 2024-04-25T15:54:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f66677fe640 1 -- 172.21.15.67:0/2645624280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66600050d0 con 0x7f66580051d0 2024-04-25T15:54:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666f23d640 1 -- 172.21.15.67:0/2645624280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66580051d0 msgr2=0x7f665800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666f23d640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66580051d0 0x7f665800b9d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6660001e40 tx=0x7f666001c6a0 comp rx=0 tx=0).stop 2024-04-25T15:54:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666f23d640 1 -- 172.21.15.67:0/2645624280 shutdown_connections 2024-04-25T15:54:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666f23d640 1 --2- 172.21.15.67:0/2645624280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66580051d0 0x7f665800b9d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666f23d640 1 -- 172.21.15.67:0/2645624280 >> 172.21.15.67:0/2645624280 conn(0x7f66680ce1c0 msgr2=0x7f66680ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666f23d640 1 -- 172.21.15.67:0/2645624280 shutdown_connections 2024-04-25T15:54:35.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.301+0000 7f666f23d640 1 -- 172.21.15.67:0/2645624280 wait complete. 2024-04-25T15:54:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 Processor -- start 2024-04-25T15:54:35.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 -- start start 2024-04-25T15:54:35.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580051d0 0x7f6668072700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6668148420 0x7f6668148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668146530 0x7f6668146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66680c4240 con 0x7f6668146530 2024-04-25T15:54:35.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6668160810 con 0x7f6668148420 2024-04-25T15:54:35.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666f23d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f666806d660 con 0x7f66580051d0 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668146530 0x7f6668146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666cfd9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580051d0 0x7f6668072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668146530 0x7f6668146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46068/0 (socket says 172.21.15.67:46068) 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 -- 172.21.15.67:0/3487559315 learned_addr learned my addr 172.21.15.67:0/3487559315 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f6667fff640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6668148420 0x7f6668148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 -- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580051d0 msgr2=0x7f6668072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580051d0 0x7f6668072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 -- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6668148420 msgr2=0x7f6668148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:35.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6668148420 0x7f6668148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.305+0000 7f666d7da640 1 -- 172.21.15.67:0/3487559315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6668072c40 con 0x7f6668146530 2024-04-25T15:54:35.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f666d7da640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668146530 0x7f6668146940 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f666001cf40 tx=0x7f6658005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f66657fa640 1 -- 172.21.15.67:0/3487559315 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660006d70 con 0x7f6668146530 2024-04-25T15:54:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f66657fa640 1 -- 172.21.15.67:0/3487559315 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66600af070 con 0x7f6668146530 2024-04-25T15:54:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66680c6150 con 0x7f6668146530 2024-04-25T15:54:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f66657fa640 1 -- 172.21.15.67:0/3487559315 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66600056a0 con 0x7f6668146530 2024-04-25T15:54:35.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66680c63f0 con 0x7f6668146530 2024-04-25T15:54:35.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f666806a480 con 0x7f6668146530 2024-04-25T15:54:35.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f66657fa640 1 -- 172.21.15.67:0/3487559315 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f66600b1070 con 0x7f6668146530 2024-04-25T15:54:35.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f66657fa640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f664c06f400 0x7f664c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:35.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.309+0000 7f66657fa640 1 -- 172.21.15.67:0/3487559315 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 6156+0+0 (secure 0 0 0) 0x7f6660128470 con 0x7f6668146530 2024-04-25T15:54:35.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.313+0000 7f666cfd9640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f664c06f400 0x7f664c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:35.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.313+0000 7f666cfd9640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f664c06f400 0x7f664c071910 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f665c0093f0 tx=0x7f665c006a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:35.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.317+0000 7f66657fa640 1 -- 172.21.15.67:0/3487559315 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f66600163b0 con 0x7f6668146530 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:148 3.9 3:91a8729a:::10000000002.00000090:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:35.373 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 106.815127277, 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.432712019, 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 148 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:35.374 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558218+0000", 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.558221+0000", 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.3929999999999998e-06 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.559704+0000", 2024-04-25T15:54:35.375 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001483807 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975277+0000", 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.41557216400000002 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975295+0000", 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8408999999999999e-05 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975372+0000", 2024-04-25T15:54:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7235000000000002e-05 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975484+0000", 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111499 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.975582+0000", 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.8589000000000001e-05 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.980957+0000", 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0053747600000000001 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990855+0000", 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0098981350000000006 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990894+0000", 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8436000000000003e-05 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.378 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:52:48.990930+0000", 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.6591999999999998e-05 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1494 3.37 3:efe547ea:::100000001f8.000001d2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.728942729000003, 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089208711999999996, 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.379 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1494 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644403+0000", 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.644406+0000", 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2229999999999999e-06 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.646356+0000", 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0019497990000000001 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718358+0000", 2024-04-25T15:54:35.381 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072002624000000001 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718376+0000", 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7649999999999999e-05 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718458+0000", 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1984999999999995e-05 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718600+0000", 2024-04-25T15:54:35.382 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000141922 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718731+0000", 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000131488 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.724805+0000", 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.006074032 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733498+0000", 2024-04-25T15:54:35.383 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0086928439999999999 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733549+0000", 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1137000000000002e-05 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.733611+0000", 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2008000000000007e-05 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.384 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1504 3.2a 3:54fb97d1:::100000001f8.000001dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.656649293000001, 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.121933077, 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1504 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.385 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716696+0000", 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.716699+0000", 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4250000000000001e-06 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.386 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.718317+0000", 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016187440000000001 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.815989+0000", 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.097671735999999995 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816008+0000", 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9131e-05 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816090+0000", 2024-04-25T15:54:35.387 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.1717000000000004e-05 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816244+0000", 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015372199999999999 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.816348+0000", 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010470499999999999 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.826448+0000", 2024-04-25T15:54:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010099281999999999 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838528+0000", 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012080206 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838583+0000", 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5105000000000002e-05 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.838629+0000", 2024-04-25T15:54:35.389 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6304000000000003e-05 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1528 3.3e 3:7f4f4afd:::100000001f8.000001f4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.509175446999997, 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.133348521, 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.390 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1528 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864170+0000", 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.391 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.864172+0000", 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.204e-06 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.865554+0000", 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013821899999999999 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986006+0000", 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.12045179 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.392 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986022+0000", 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5999e-05 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986101+0000", 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9251000000000004e-05 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986224+0000", 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012263899999999999 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.986330+0000", 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106164 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.996058+0000", 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0097274319999999994 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997353+0000", 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001294882 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.394 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997450+0000", 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.7446000000000002e-05 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.997519+0000", 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8523999999999996e-05 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1548 3.21 3:86681322:::100000001f8.00000208:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:35.395 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.389709740000001, 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096155170999999998, 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1548 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.396 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983636+0000", 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.983639+0000", 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1750000000000001e-06 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:43.985960+0000", 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023213790000000002 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.397 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058434+0000", 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072473913000000001 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058454+0000", 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9329000000000001e-05 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058541+0000", 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7702999999999997e-05 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.398 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058708+0000", 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00016712300000000001 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.058864+0000", 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015600100000000001 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.065501+0000", 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066362299999999999 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.399 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079696+0000", 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014195228000000001 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079745+0000", 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8980000000000002e-05 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.079791+0000", 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6109999999999997e-05 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:1561 3.2e 3:75253c31:::100000001f8.00000215:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 51.305315210000003, 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099620287000000002, 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 1561 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:35.401 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068030+0000", 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.068033+0000", 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1240000000000001e-06 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.070277+0000", 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002243676 2024-04-25T15:54:35.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143785+0000", 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.073508283999999993 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143804+0000", 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8808e-05 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.143881+0000", 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6379999999999997e-05 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.403 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144063+0000", 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018198500000000001 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.144168+0000", 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105253 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.154374+0000", 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010206374000000001 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.404 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167481+0000", 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013106405999999999 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167580+0000", 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.9329999999999999e-05 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:44.167651+0000", 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.0667000000000006e-05 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2308 3.4 3:2343a40c:::100000001f8.00000500:head [write 0~4194304 in=4194304b] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 46.552260898, 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.129707092, 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2308 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.406 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821085+0000", 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.821088+0000", 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1599999999999998e-06 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823092+0000", 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020043909999999999 2024-04-25T15:54:35.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823143+0000", 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.0791999999999998e-05 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823162+0000", 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9332999999999999e-05 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823254+0000", 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2169000000000005e-05 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.408 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823637+0000", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00038265100000000002 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.823849+0000", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000211879 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.834007+0000", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010158377 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950654+0000", 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.116646229 2024-04-25T15:54:35.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950721+0000", 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7020000000000005e-05 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.950792+0000", 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.1091000000000001e-05 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.410 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4976 3.2a 3:57c1fe5a:::100000001f8.0000056c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.78222461, 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.15973960500000001, 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4976 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591121+0000", 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.591123+0000", 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.306e-06 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.592621+0000", 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014980250000000001 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.412 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703776+0000", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.111154948 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703790+0000", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3915e-05 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703878+0000", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7527000000000004e-05 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.703996+0000", 2024-04-25T15:54:35.413 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118602 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.704166+0000", 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017002799999999999 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.710386+0000", 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0062202239999999999 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750704+0000", 2024-04-25T15:54:35.414 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.040317106999999998 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750795+0000", 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.1880999999999997e-05 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.750860+0000", 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5042000000000005e-05 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.415 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5006 3.2a 3:566ba4fe:::100000001f8.0000058a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.633714550000001, 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094006696000000001, 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5006 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.416 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739631+0000", 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.739634+0000", 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7319999999999998e-06 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.740943+0000", 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0013092780000000001 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.417 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791183+0000", 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050239791999999998 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791197+0000", 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4250000000000001e-05 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791287+0000", 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9627000000000001e-05 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.418 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791404+0000", 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117663 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.791560+0000", 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000155997 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.801092+0000", 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0095317030000000007 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833424+0000", 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032332514999999999 2024-04-25T15:54:35.419 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833527+0000", 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000102997 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.833638+0000", 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000110142 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.420 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5024 3.2a 3:55e8c2c8:::100000001f8.0000059c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.543849660999999, 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14725615, 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5024 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.421 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829496+0000", 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.829499+0000", 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0740000000000001e-06 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.831085+0000", 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001586241 2024-04-25T15:54:35.422 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932577+0000", 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.10149145399999999 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932592+0000", 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4934e-05 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932681+0000", 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9140000000000004e-05 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.423 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932803+0000", 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012215800000000001 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.932954+0000", 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000151328 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.943046+0000", 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010092261999999999 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.424 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976650+0000", 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.033603840000000003 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976710+0000", 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9348999999999997e-05 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.976752+0000", 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2370000000000003e-05 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.425 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5049 3.2e 3:740912a7:::100000001f8.000005b5:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.409941692, 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.162097727, 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5049 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.426 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963404+0000", 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.963406+0000", 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4279999999999999e-06 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.965277+0000", 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0018703210000000001 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.021994+0000", 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056717005000000001 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022018+0000", 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4043e-05 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.428 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022095+0000", 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6959999999999995e-05 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022198+0000", 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103167 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.022356+0000", 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015838899999999999 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.052459+0000", 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030102694999999999 2024-04-25T15:54:35.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125349+0000", 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072890606999999996 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125439+0000", 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9587999999999997e-05 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125502+0000", 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2524e-05 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5064 3.21 3:845d14c1:::100000001f8.000005c4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.338948425000002, 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14591528400000001, 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5064 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034397+0000", 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.034400+0000", 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3079999999999999e-06 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.036710+0000", 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023096000000000002 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075414+0000", 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.038703923000000001 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075434+0000", 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9712e-05 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075522+0000", 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.8472000000000003e-05 2024-04-25T15:54:35.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075609+0000", 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7200000000000005e-05 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075766+0000", 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015715400000000001 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.089149+0000", 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013382338000000001 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180138+0000", 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.090989451999999998 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180200+0000", 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.2098000000000001e-05 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180312+0000", 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000112027 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.435 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5072 3.2e 3:75b854b3:::100000001f8.000005cc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 25.299749088999999, 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.106817279, 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5072 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073596+0000", 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.073599+0000", 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2759999999999999e-06 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.075372+0000", 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001773574 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125855+0000", 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.050482955000000003 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125865+0000", 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.0029e-05 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.438 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.125933+0000", 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8100000000000002e-05 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.126051+0000", 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000117333 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.126222+0000", 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000171609 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.133559+0000", 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0073365649999999998 2024-04-25T15:54:35.439 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180298+0000", 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.046739050999999997 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180358+0000", 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0479000000000002e-05 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.180414+0000", 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5308e-05 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5601 3.2e 3:75930891:::100000001f8.000007dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.359278830000001, 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088347508000000005, 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5601 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014067+0000", 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.014070+0000", 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0869999999999998e-06 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.442 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.015590+0000", 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015200649999999999 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086402+0000", 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.070812293999999998 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086419+0000", 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7088999999999999e-05 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086530+0000", 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011049599999999999 2024-04-25T15:54:35.443 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086686+0000", 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015618899999999999 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.086917+0000", 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000231198 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.095623+0000", 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0087063049999999993 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.444 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102292+0000", 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0066687609999999996 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102356+0000", 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3958e-05 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.102414+0000", 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8066000000000003e-05 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.445 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5729 3.2e 3:77f75ed6:::100000001f8.0000085d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.616993624999999, 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.114341208, 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5729 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.446 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756352+0000", 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.756354+0000", 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.576e-06 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.447 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757773+0000", 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014189560000000001 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757806+0000", 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2428000000000002e-05 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757820+0000", 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4279e-05 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.757910+0000", 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0138999999999999e-05 2024-04-25T15:54:35.448 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.758024+0000", 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114158 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.758197+0000", 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172931 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.769252+0000", 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011054342 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.449 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870574+0000", 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.10132216099999999 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870628+0000", 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3996999999999998e-05 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870693+0000", 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5240999999999994e-05 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.450 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5739 3.2e 3:757c9716:::100000001f8.00000867:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 21.567866725999998, 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099555849000000002, 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5739 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:35.451 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805479+0000", 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.805482+0000", 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9670000000000001e-06 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807186+0000", 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017046520000000001 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807217+0000", 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0892999999999997e-05 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807230+0000", 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2235999999999999e-05 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807314+0000", 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4481999999999995e-05 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.453 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807445+0000", 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130977 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.807554+0000", 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000108805 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.824323+0000", 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016769217999999999 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.454 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904744+0000", 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080421406000000001 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.904929+0000", 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018458600000000001 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.905035+0000", 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000105627 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5870 3.4 3:2197a314:::100000001f8.000008ea:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.549685+0000", 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.823660929999999, 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.14750286300000001, 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5870 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.456 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.549685+0000", 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.549685+0000", 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.549687+0000", 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.1780000000000002e-06 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.551318+0000", 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001631179 2024-04-25T15:54:35.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641743+0000", 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.090425249999999999 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641758+0000", 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5299999999999999e-05 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641875+0000", 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116809 2024-04-25T15:54:35.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.641952+0000", 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7021000000000007e-05 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.642075+0000", 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00012264 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.647613+0000", 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.005538472 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.459 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.697082+0000", 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.049468828999999999 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.697136+0000", 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.3616999999999998e-05 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.697187+0000", 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1567999999999998e-05 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.460 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5915 3.2e 3:7446f81c:::100000001f8.00000917:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.822335+0000", 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.551010288000001, 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.25176186900000003, 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5915 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.822335+0000", 2024-04-25T15:54:35.461 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.822335+0000", 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.822338+0000", 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8329999999999998e-06 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.823856+0000", 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001518092 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035519+0000", 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.21166296100000001 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035533+0000", 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.346e-05 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035609+0000", 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6087999999999994e-05 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.463 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035684+0000", 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5240999999999993e-05 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.035788+0000", 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010430899999999999 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.043303+0000", 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0075152530000000004 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.073955+0000", 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.030651226 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.074031+0000", 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6631000000000005e-05 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.074097+0000", 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.5775e-05 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5965 3.4 3:2356ea06:::100000001f8.00000949:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.092631+0000", 2024-04-25T15:54:35.465 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.280714408000001, 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096216884000000003, 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5965 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.092631+0000", 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.466 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.092631+0000", 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.092634+0000", 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.915e-06 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.094975+0000", 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023405980000000002 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.467 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164137+0000", 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.069162491000000006 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164157+0000", 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9545e-05 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164250+0000", 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.2872000000000001e-05 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.468 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164324+0000", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4405000000000003e-05 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.164469+0000", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014499000000000001 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.174678+0000", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010209197999999999 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.188695+0000", 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014016768000000001 2024-04-25T15:54:35.469 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.188780+0000", 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5573000000000002e-05 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.188848+0000", 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7528999999999997e-05 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.470 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5978 3.b 3:d3280d2b:::100000001f8.00000956:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:15.167843+0000", 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 20.205502766999999, 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.11479445200000001, 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5978 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.167843+0000", 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.471 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.167843+0000", 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.167845+0000", 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6740000000000001e-06 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.169444+0000", 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0015984149999999999 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.472 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230486+0000", 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.061041911999999997 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230500+0000", 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4758e-05 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230585+0000", 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4509e-05 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.473 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230698+0000", 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011307999999999999 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.230897+0000", 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00019934900000000001 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.273195+0000", 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.042297522999999997 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.474 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282533+0000", 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0093378950000000006 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282587+0000", 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.4514e-05 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:15.282637+0000", 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.9823000000000001e-05 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.475 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:35.476 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:35.476 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:35.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-25T15:54:35.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:35.565+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f6668155000 con 0x7f6668146530 2024-04-25T15:54:35.693 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:35.694 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-04-25T15:54:35.694 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:35.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-25T15:54:35.926 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:35.926 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-04-25T15:54:35.926 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 180, 2024-04-25T15:54:35.926 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-04-25T15:54:35.926 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:35.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-25T15:54:36.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.101+0000 7f66657fa640 1 -- 172.21.15.67:0/3487559315 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v111) ==== 91+0+0 (secure 0 0 0) 0x7f66600f4c70 con 0x7f6668146530 2024-04-25T15:54:36.104 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f664c06f400 msgr2=0x7f664c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f664c06f400 0x7f664c071910 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f665c0093f0 tx=0x7f665c006a30 comp rx=0 tx=0).stop 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668146530 msgr2=0x7f6668146940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668146530 0x7f6668146940 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f666001cf40 tx=0x7f6658005840 comp rx=0 tx=0).stop 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 shutdown_connections 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f664c06f400 0x7f664c071910 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6668146530 0x7f6668146940 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6668148420 0x7f6668148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 --2- 172.21.15.67:0/3487559315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f66580051d0 0x7f6668072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 >> 172.21.15.67:0/3487559315 conn(0x7f66680ce1c0 msgr2=0x7f6668155750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 shutdown_connections 2024-04-25T15:54:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.105+0000 7f666f23d640 1 -- 172.21.15.67:0/3487559315 wait complete. 2024-04-25T15:54:36.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-25T15:54:36.160 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:54:36.160 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-04-25T15:54:36.160 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-04-25T15:54:36.160 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [ 2024-04-25T15:54:36.160 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.160 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:343 3.8 3:1115f3ca:::10000000002.00000053:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 93.153367334999999, 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.46089677899999998, 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 343 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006867+0000", 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.006870+0000", 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.9280000000000002e-06 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008363+0000", 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001493288 2024-04-25T15:54:36.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008398+0000", 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.5067000000000002e-05 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008416+0000", 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7782e-05 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008522+0000", 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000106327 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.163 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for peered", 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.008535+0000", 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.2883e-05 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.387477+0000", 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.37894188200000001 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for degraded object", 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.392562+0000", 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0050850870000000003 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.164 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445264+0000", 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.052702037 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445382+0000", 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118433 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.445603+0000", 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00022077500000000001 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.165 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.465250+0000", 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.019646976 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467658+0000", 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024073850000000002 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467695+0000", 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7234999999999998e-05 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.166 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:03.467763+0000", 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8694000000000003e-05 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:670 3.8 3:1323134e:::100000001f7.0000009a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 72.286297172000005, 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094982472999999998, 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.167 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 670 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873937+0000", 2024-04-25T15:54:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.873941+0000", 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0840000000000002e-06 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876488+0000", 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0025470290000000001 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876647+0000", 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015912000000000001 2024-04-25T15:54:36.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876664+0000", 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7215e-05 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.876782+0000", 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000118082 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877222+0000", 2024-04-25T15:54:36.170 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000439596 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.877596+0000", 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00037371700000000003 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.913106+0000", 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.035509813000000001 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968792+0000", 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.055686739999999998 2024-04-25T15:54:36.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968850+0000", 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.8032000000000003e-05 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:23.968919+0000", 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9045e-05 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.172 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:2313 3.1e 3:7a2a8d26:::100000001f8.00000505:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 47.2955848, 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091581081999999994, 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 2313 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:36.173 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864649+0000", 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.864653+0000", 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3699999999999999e-06 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866695+0000", 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.002042744 2024-04-25T15:54:36.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866732+0000", 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7054000000000001e-05 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866765+0000", 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2141999999999998e-05 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866853+0000", 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.7950999999999999e-05 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.175 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.866986+0000", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013319800000000001 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.867189+0000", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020333800000000001 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.883954+0000", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016764729999999999 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956056+0000", 2024-04-25T15:54:36.176 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.072101922999999998 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956116+0000", 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.9815000000000002e-05 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:53:48.956230+0000", 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000114817 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3921 3.29 3:95d78bcd:::100000001f8.0000014d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.177 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.645499323000003, 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.156667944, 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3921 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:36.178 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514735+0000", 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.514737+0000", 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.029e-06 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517027+0000", 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022901739999999999 2024-04-25T15:54:36.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517155+0000", 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127763 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517189+0000", 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3847999999999998e-05 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517337+0000", 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014894999999999999 2024-04-25T15:54:36.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517464+0000", 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000126608 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.517651+0000", 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018649000000000001 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.531319+0000", 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.013667959 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.181 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671222+0000", 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.139903946 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671326+0000", 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000103785 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.671403+0000", 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6391999999999996e-05 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.182 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3935 3.23 3:c6f60763:::100000001f8.0000015b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.574237254000003, 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.091319014000000004, 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3935 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.585997+0000", 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.586000+0000", 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.2789999999999999e-06 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.587431+0000", 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00143063 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.184 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.662977+0000", 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.075546586999999998 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663000+0000", 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.2413999999999999e-05 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663082+0000", 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.2659999999999998e-05 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.185 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663161+0000", 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.8322e-05 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.663262+0000", 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010089099999999999 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.674457+0000", 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011195544999999999 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.186 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677208+0000", 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0027504769999999999 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677272+0000", 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.4862000000000003e-05 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.677316+0000", 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3346999999999997e-05 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.187 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3959 3.7 3:e1bee111:::100000001f8.00000173:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.398105307000002, 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089007748999999997, 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3959 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762129+0000", 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.762131+0000", 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.0430000000000002e-06 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.764378+0000", 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0022474560000000001 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.189 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786050+0000", 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021672166 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786065+0000", 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4652e-05 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786161+0000", 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5873999999999997e-05 2024-04-25T15:54:36.190 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786278+0000", 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116684 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786462+0000", 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000184625 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.794936+0000", 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0084740370000000002 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.191 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851010+0000", 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.056073265999999997 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851079+0000", 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.9556999999999998e-05 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.851136+0000", 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7389000000000002e-05 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.192 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3963 3.19 3:9b5b1331:::100000001f8.00000177:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.379641970999998, 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.167075797, 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3963 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.193 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780592+0000", 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.780596+0000", 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8079999999999998e-06 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.194 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.786013+0000", 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0054168410000000004 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800764+0000", 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.014751306 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800778+0000", 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4171999999999999e-05 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.195 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800924+0000", 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146286 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.801019+0000", 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4456000000000005e-05 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.801147+0000", 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000127603 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.809771+0000", 2024-04-25T15:54:36.196 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.008624095 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947582+0000", 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13781142499999999 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947625+0000", 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2928e-05 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.947668+0000", 2024-04-25T15:54:36.197 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2877000000000003e-05 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3967 3.24 3:260ecc6d:::100000001f8.0000017b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.361204129999997, 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.18196963599999999, 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3967 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799030+0000", 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.799032+0000", 2024-04-25T15:54:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4040000000000002e-06 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.800724+0000", 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001691889 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861343+0000", 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.060618914000000003 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861359+0000", 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.6294999999999999e-05 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.200 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861444+0000", 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4251000000000004e-05 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861517+0000", 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.3098999999999999e-05 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861617+0000", 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000100603 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.201 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.878626+0000", 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.017008836999999999 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980913+0000", 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.102287103 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.980951+0000", 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7718e-05 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.202 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.981000+0000", 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8523000000000003e-05 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3980 3.8 3:13e1c2d8:::100000001f8.00000188:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.300330725999999, 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.111276876, 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.203 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3980 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859903+0000", 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.204 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.859907+0000", 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7950000000000001e-06 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.861306+0000", 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001398505 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893810+0000", 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.032503907999999998 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.205 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893829+0000", 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9860999999999998e-05 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.893916+0000", 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.6890000000000003e-05 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.894052+0000", 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00013542200000000001 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:36.206 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.894230+0000", 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017861600000000001 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.902225+0000", 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079943059999999996 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971087+0000", 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.068862265000000006 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971132+0000", 2024-04-25T15:54:36.207 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.4903e-05 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.971180+0000", 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.8405000000000002e-05 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:3990 3.19 3:9b35cac6:::100000001f8.00000192:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.251974812, 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.093327906000000002, 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.208 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 3990 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.209 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908259+0000", 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.908262+0000", 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.258e-06 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.909909+0000", 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0016468979999999999 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.210 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987236+0000", 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.077326763000000007 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987250+0000", 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4248e-05 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987323+0000", 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.2416000000000006e-05 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.211 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987413+0000", 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.0482999999999994e-05 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.987616+0000", 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00020251899999999999 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:03.999192+0000", 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011576229 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001500+0000", 2024-04-25T15:54:36.212 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0023079099999999998 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001546+0000", 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6508999999999998e-05 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.001587+0000", 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0673000000000002e-05 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.213 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:4024 3.24 3:268a62b6:::100000001f8.000001b4:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 32.030277734000002, 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.101232684, 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 4024 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:36.214 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129956+0000", 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.129960+0000", 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1949999999999999e-06 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.131869+0000", 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001909151 2024-04-25T15:54:36.215 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208635+0000", 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.076766361000000005 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208652+0000", 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7093999999999999e-05 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208741+0000", 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.899e-05 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.216 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208831+0000", 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9621000000000001e-05 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.208947+0000", 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000115788 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.225235+0000", 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016288894000000002 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.217 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.230862+0000", 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0056270290000000004 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.230947+0000", 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5019999999999996e-05 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:04.231189+0000", 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00024154100000000001 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.218 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5056 3.38 3:1d039b80:::100000001f8.000005bc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:09.996489+0000", 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 26.163744831999999, 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.113765959, 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5056 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.996489+0000", 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.996489+0000", 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.996492+0000", 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.7690000000000001e-06 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998555+0000", 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0020628059999999999 2024-04-25T15:54:36.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998596+0000", 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0855000000000001e-05 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998611+0000", 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.5204e-05 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998717+0000", 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00010653 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.221 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998847+0000", 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0001292 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:09.998993+0000", 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000146682 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.010434+0000", 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011441066999999999 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.110173+0000", 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.099738744000000004 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.110217+0000", 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.3467999999999998e-05 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:10.110255+0000", 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8634000000000003e-05 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5722 3.38 3:1deaa7c3:::100000001f8.00000856:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.719324+0000", 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.440910512999999, 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096971450000000001, 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5722 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.224 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.719324+0000", 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.719324+0000", 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.719327+0000", 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.1530000000000001e-06 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.720925+0000", 2024-04-25T15:54:36.225 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001597916 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.720963+0000", 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8380000000000002e-05 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.720977+0000", 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3774000000000001e-05 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.721072+0000", 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.5136999999999994e-05 2024-04-25T15:54:36.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.721202+0000", 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000130068 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.721379+0000", 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00017673 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.732921+0000", 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.011542202999999999 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.227 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.816192+0000", 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.083271077999999998 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.816234+0000", 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.206e-05 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.816295+0000", 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.0951e-05 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.228 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5724 3.2 3:4214d263:::100000001f8.00000858:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.731630+0000", 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.428604230000001, 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.13283815800000001, 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5724 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.229 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.731630+0000", 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.731630+0000", 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.731633+0000", 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.6910000000000002e-06 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.230 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734039+0000", 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0024061310000000002 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734069+0000", 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.0539e-05 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734086+0000", 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.7218999999999999e-05 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734154+0000", 2024-04-25T15:54:36.231 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.7228000000000001e-05 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734237+0000", 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.3102000000000005e-05 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.734348+0000", 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000111759 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.744430+0000", 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.010081639999999999 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.864304+0000", 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.11987374200000001 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.864393+0000", 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.9310999999999999e-05 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.864468+0000", 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4795999999999993e-05 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.233 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5728 3.29 3:94338435:::100000001f8.0000085c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.750848+0000", 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.409386331, 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.107473081, 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5728 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.234 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.750848+0000", 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.750848+0000", 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.750851+0000", 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8329999999999998e-06 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752255+0000", 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0014045430000000001 2024-04-25T15:54:36.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752301+0000", 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.6385999999999999e-05 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752318+0000", 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.643e-05 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752398+0000", 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9745000000000003e-05 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.236 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752477+0000", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9290999999999997e-05 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.752596+0000", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000119268 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.769023+0000", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.016426473 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.858162+0000", 2024-04-25T15:54:36.237 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.089139180999999998 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.858247+0000", 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.4647000000000005e-05 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.858321+0000", 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4283999999999996e-05 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.238 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5731 3.1e 3:7ae5ac58:::100000001f8.0000085f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.765834+0000", 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.394400062999999, 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.105229766, 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5731 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.765834+0000", 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.239 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.765834+0000", 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.765836+0000", 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.99e-06 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767215+0000", 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001378881 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.240 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767254+0000", 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.8868000000000001e-05 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767273+0000", 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.9230000000000001e-05 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767349+0000", 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.6217000000000006e-05 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767424+0000", 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.4597000000000004e-05 2024-04-25T15:54:36.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.767540+0000", 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000116661 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.774440+0000", 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0068993980000000002 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870906+0000", 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096465683999999996 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.242 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.870985+0000", 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9066999999999994e-05 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871064+0000", 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9172999999999996e-05 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5733 3.1e 3:79c5b821:::100000001f8.00000861:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:13.776276+0000", 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.383958423999999, 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.094987504, 2024-04-25T15:54:36.243 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5733 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.776276+0000", 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.244 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.776276+0000", 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.776278+0000", 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8059999999999999e-06 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.777883+0000", 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001604205 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.777926+0000", 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2985999999999998e-05 2024-04-25T15:54:36.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.777940+0000", 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.4107000000000001e-05 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.778025+0000", 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.5066000000000002e-05 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.778105+0000", 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0506000000000005e-05 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.246 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.778219+0000", 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000113596 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.790528+0000", 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.012308723000000001 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871143+0000", 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.080615081000000005 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.247 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871211+0000", 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.8670000000000005e-05 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:13.871263+0000", 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.1758000000000001e-05 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5777 3.2 3:42907a88:::100000001f8.0000088d:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.024509+0000", 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.135725346000001, 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.133883539, 2024-04-25T15:54:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5777 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.024509+0000", 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.024509+0000", 2024-04-25T15:54:36.249 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.024512+0000", 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.027e-06 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026152+0000", 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001640308 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026193+0000", 2024-04-25T15:54:36.250 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.0967999999999998e-05 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026211+0000", 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8139e-05 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026288+0000", 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.7043000000000001e-05 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026368+0000", 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 8.0118999999999996e-05 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.251 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.026512+0000", 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00014386600000000001 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.034500+0000", 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0079875799999999993 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.158295+0000", 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.123794761 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.252 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.158350+0000", 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.5235000000000002e-05 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.158392+0000", 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 4.2493000000000001e-05 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5783 3.1e 3:79270052:::100000001f8.00000893:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.053917+0000", 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.106317377, 2024-04-25T15:54:36.253 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.096657987000000001, 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5783 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.053917+0000", 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.254 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.053917+0000", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.053919+0000", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.4490000000000002e-06 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055598+0000", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.001679304 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055632+0000", 2024-04-25T15:54:36.255 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.3905000000000001e-05 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055646+0000", 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.3720000000000001e-05 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055740+0000", 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 9.4035000000000003e-05 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.055854+0000", 2024-04-25T15:54:36.256 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00011380200000000001 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.056027+0000", 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.000172919 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.061846+0000", 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0058194650000000002 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.150432+0000", 2024-04-25T15:54:36.257 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.088585632999999997 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.150511+0000", 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.9191999999999997e-05 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.150575+0000", 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 6.3563000000000001e-05 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "description": "osd_op(client.4815.0:5787 3.19 3:9bb5c4e2:::100000001f8.00000897:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.258 INFO:teuthology.orchestra.run.smithi067.stdout: "initiated_at": "2024-04-25T15:54:14.070830+0000", 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "age": 22.089403909000001, 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.102205173, 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "continuous": false, 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "type_data": { 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "client_info": { 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "client": "client.4815", 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "tid": 5787 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "events": [ 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "initiated", 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.070830+0000", 2024-04-25T15:54:36.259 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "header_read", 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.070830+0000", 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "throttled", 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.070833+0000", 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 2.8499999999999998e-06 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "all_read", 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072599+0000", 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.0017661770000000001 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.260 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "dispatched", 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072637+0000", 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 3.7438999999999998e-05 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072655+0000", 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 1.8552000000000001e-05 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "reached_pg", 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.072814+0000", 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00015876900000000001 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "started", 2024-04-25T15:54:36.261 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.073065+0000", 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00025143700000000003 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.073253+0000", 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.00018773499999999999 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "op_commit", 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.094308+0000", 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.021054812999999999 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.172902+0000", 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 0.078594302000000005 2024-04-25T15:54:36.262 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "commit_sent", 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.172977+0000", 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 7.5195000000000001e-05 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "event": "done", 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "time": "2024-04-25T15:54:14.173035+0000", 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 5.7904e-05 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-25T15:54:36.263 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:54:36.264 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-25T15:54:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d88e05640 1 Processor -- start 2024-04-25T15:54:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d88e05640 1 -- start start 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d88e05640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d8415aaf0 0x7f3d8415af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d88e05640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d840caa60 con 0x7f3d8406d460 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d88e05640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d8415a5f0 con 0x7f3d840736d0 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d88e05640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3d84068590 con 0x7f3d8415aaf0 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d8259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d8406d460 0x7f3d841547e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50030/0 (socket says 172.21.15.67:50030) 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d8259c640 1 -- 172.21.15.67:0/3114446405 learned_addr learned my addr 172.21.15.67:0/3114446405 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d82d9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d8415aaf0 0x7f3d8415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d81d9b640 1 -- 172.21.15.67:0/3114446405 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1811275819 0 0) 0x7f3d840caa60 con 0x7f3d8406d460 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d81d9b640 1 -- 172.21.15.67:0/3114446405 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d70003530 con 0x7f3d8406d460 2024-04-25T15:54:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d82d9d640 1 -- 172.21.15.67:0/3114446405 >> v1:172.21.15.176:6789/0 conn(0x7f3d840736d0 legacy=0x7f3d8406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d82d9d640 1 -- 172.21.15.67:0/3114446405 >> v1:172.21.15.67:6789/0 conn(0x7f3d8406d460 legacy=0x7f3d841547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:36.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.301+0000 7f3d82d9d640 1 -- 172.21.15.67:0/3114446405 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d840ca100 con 0x7f3d8415aaf0 2024-04-25T15:54:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d82d9d640 1 --2- 172.21.15.67:0/3114446405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8415aaf0 0x7f3d8415af00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3d840b14e0 tx=0x7f3d7c01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4896687572efa385 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d81d9b640 1 -- 172.21.15.67:0/3114446405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c0b2070 con 0x7f3d8415aaf0 2024-04-25T15:54:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d81d9b640 1 -- 172.21.15.67:0/3114446405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d7c0a58a0 con 0x7f3d8415aaf0 2024-04-25T15:54:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d81d9b640 1 -- 172.21.15.67:0/3114446405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c0a5c70 con 0x7f3d8415aaf0 2024-04-25T15:54:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d88e05640 1 -- 172.21.15.67:0/3114446405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8415aaf0 msgr2=0x7f3d8415af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d88e05640 1 --2- 172.21.15.67:0/3114446405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8415aaf0 0x7f3d8415af00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3d840b14e0 tx=0x7f3d7c01c8f0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d88e05640 1 -- 172.21.15.67:0/3114446405 shutdown_connections 2024-04-25T15:54:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d88e05640 1 --2- 172.21.15.67:0/3114446405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d8415aaf0 0x7f3d8415af00 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d88e05640 1 -- 172.21.15.67:0/3114446405 >> 172.21.15.67:0/3114446405 conn(0x7f3d840ce1c0 msgr2=0x7f3d840ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d88e05640 1 -- 172.21.15.67:0/3114446405 shutdown_connections 2024-04-25T15:54:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.305+0000 7f3d88e05640 1 -- 172.21.15.67:0/3114446405 wait complete. 2024-04-25T15:54:36.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 Processor -- start 2024-04-25T15:54:36.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 -- start start 2024-04-25T15:54:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8406d460 0x7f3d840c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d840736d0 0x7f3d84152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d8415aaf0 0x7f3d84157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8406cd50 con 0x7f3d840736d0 2024-04-25T15:54:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8415b440 con 0x7f3d8406d460 2024-04-25T15:54:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3d8406ced0 con 0x7f3d8415aaf0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d840736d0 0x7f3d84152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d840736d0 0x7f3d84152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46080/0 (socket says 172.21.15.67:46080) 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 -- 172.21.15.67:0/2190327438 learned_addr learned my addr 172.21.15.67:0/2190327438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d82d9d640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d8415aaf0 0x7f3d84157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d8259c640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8406d460 0x7f3d840c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 -- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d8415aaf0 msgr2=0x7f3d84157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d8415aaf0 0x7f3d84157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 -- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8406d460 msgr2=0x7f3d840c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8406d460 0x7f3d840c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 -- 172.21.15.67:0/2190327438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d840c6cf0 con 0x7f3d840736d0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7bfff640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d840736d0 0x7f3d84152370 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00b3f0 tx=0x7f3d6c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7affd640 1 -- 172.21.15.67:0/2190327438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c015020 con 0x7f3d840736d0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7affd640 1 -- 172.21.15.67:0/2190327438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d6c011850 con 0x7f3d840736d0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d84153670 con 0x7f3d840736d0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d7affd640 1 -- 172.21.15.67:0/2190327438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c011c40 con 0x7f3d840736d0 2024-04-25T15:54:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.309+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d840c5cd0 con 0x7f3d840736d0 2024-04-25T15:54:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.313+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d840b0d60 con 0x7f3d840736d0 2024-04-25T15:54:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.313+0000 7f3d7affd640 1 -- 172.21.15.67:0/2190327438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3d6c018020 con 0x7f3d840736d0 2024-04-25T15:54:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.313+0000 7f3d7affd640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6406f400 0x7f3d64071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.313+0000 7f3d7affd640 1 -- 172.21.15.67:0/2190327438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 6056+0+0 (secure 0 0 0) 0x7f3d6c08f2e0 con 0x7f3d840736d0 2024-04-25T15:54:36.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.313+0000 7f3d8259c640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6406f400 0x7f3d64071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:36.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.313+0000 7f3d8259c640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6406f400 0x7f3d64071910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3d74007fd0 tx=0x7f3d74009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:36.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.317+0000 7f3d7affd640 1 -- 172.21.15.67:0/2190327438 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3d6c01f020 con 0x7f3d840736d0 2024-04-25T15:54:36.414 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:36.414 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:36.414 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:36.414 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:36.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-25T15:54:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:36.489+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f3d8406a480 con 0x7f3d840736d0 2024-04-25T15:54:36.580 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:36.580 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:36.580 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:36.580 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:36.580 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:36.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:1777 3.1b 3:daa4f757:::100000001f8.000002ed:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:45.401998+0000", 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 51.339139183999997, 2024-04-25T15:54:36.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.116780538, 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1777 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.401998+0000", 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.401998+0000", 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.402025+0000", 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6185000000000001e-05 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.407170+0000", 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051459419999999997 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.496935+0000", 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.089764201000000002 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.496993+0000", 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8739000000000001e-05 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.497146+0000", 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015288500000000001 2024-04-25T15:54:36.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.497267+0000", 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120788 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.497375+0000", 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010806000000000001 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.512753+0000", 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015378249 2024-04-25T15:54:36.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.518595+0000", 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005842023 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.518687+0000", 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1265000000000001e-05 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:45.518779+0000", 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2200999999999994e-05 2024-04-25T15:54:36.750 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3918 3.11 e97/76 3:8b260724:::100000001f8.0000014a:head v 97'96, mlcod=96'94)", 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518716+0000", 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.222421646000001, 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13541348, 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518716+0000", 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518716+0000", 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518717+0000", 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.302e-06 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524601+0000", 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058834430000000004 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524652+0000", 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0924000000000002e-05 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524723+0000", 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1606999999999994e-05 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524827+0000", 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103522 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.524861+0000", 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4541e-05 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654069+0000", 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12920799299999999 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654129+0000", 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0148000000000001e-05 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3919 3.2c e97/76 3:37bce3fe:::100000001f8.0000014b:head v 97'73, mlcod=96'70)", 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.520193+0000", 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.220944672999998, 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14953171500000001, 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520193+0000", 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520193+0000", 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520195+0000", 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.703e-06 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522166+0000", 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019710040000000002 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522206+0000", 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0635e-05 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522229+0000", 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2847999999999999e-05 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522327+0000", 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8157000000000002e-05 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522365+0000", 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7437000000000001e-05 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669684+0000", 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.147319441 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669725+0000", 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.049e-05 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.758 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3921 3.29 e97/76 3:95d78bcd:::100000001f8.0000014d:head v 97'62, mlcod=96'60)", 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.522303+0000", 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.218834973, 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14725027600000001, 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522303+0000", 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522303+0000", 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.522304+0000", 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.031e-06 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531230+0000", 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089262600000000001 2024-04-25T15:54:36.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531250+0000", 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0458e-05 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531271+0000", 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0185000000000001e-05 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531359+0000", 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8338999999999996e-05 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531391+0000", 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2465999999999997e-05 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669508+0000", 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.138116562 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669553+0000", 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4975000000000002e-05 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.762 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3927 3.34 3:2dd1f0a3:::100000001f8.00000153:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.542469+0000", 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.198668670000004, 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12736851099999999, 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3927 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.763 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542469+0000", 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542469+0000", 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542477+0000", 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8120000000000008e-06 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553089+0000", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010612018000000001 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553128+0000", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9477000000000002e-05 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553142+0000", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.376e-05 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553223+0000", 2024-04-25T15:54:36.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1075999999999994e-05 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553324+0000", 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101331 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.553462+0000", 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000137856 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669674+0000", 2024-04-25T15:54:36.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.116212207 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669735+0000", 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0303999999999997e-05 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669789+0000", 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4168e-05 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669837+0000", 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8501999999999997e-05 2024-04-25T15:54:36.767 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3963 3.19 e97/76 3:9b5b1331:::100000001f8.00000177:head v 97'70, mlcod=97'68)", 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.800007+0000", 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.941130516000001, 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14600712099999999, 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.800007+0000", 2024-04-25T15:54:36.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.800007+0000", 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.800010+0000", 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5679999999999998e-06 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804743+0000", 2024-04-25T15:54:36.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047334430000000004 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804764+0000", 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1237e-05 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804797+0000", 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3062999999999998e-05 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804866+0000", 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8603999999999996e-05 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.804886+0000", 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0497999999999999e-05 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945932+0000", 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.141045114 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.946014+0000", 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2594000000000002e-05 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3966 3.2c e97/76 3:36f7a988:::100000001f8.0000017a:head v 97'74, mlcod=97'72)", 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.805041+0000", 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.936096092, 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.124131596, 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.805041+0000", 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.805041+0000", 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.805043+0000", 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0729999999999999e-06 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806829+0000", 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017861470000000001 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806845+0000", 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6404000000000001e-05 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806859+0000", 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3749e-05 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806952+0000", 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3166999999999998e-05 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.806982+0000", 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9612999999999999e-05 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929100+0000", 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122118665 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929173+0000", 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2778e-05 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3972 3.a e97/76 3:506f1d9f:::100000001f8.00000180:head v 97'56, mlcod=96'54)", 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.830816+0000", 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.910321965999998, 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.149891782, 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.776 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830816+0000", 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830816+0000", 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830817+0000", 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1069999999999999e-06 2024-04-25T15:54:36.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834726+0000", 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039096490000000003 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834762+0000", 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5710000000000002e-05 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834785+0000", 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3141000000000001e-05 2024-04-25T15:54:36.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834948+0000", 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016240199999999999 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.834980+0000", 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2116999999999998e-05 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.980644+0000", 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14566418 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.980707+0000", 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3476e-05 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3974 3.2c e97/76 3:35c2bab0:::100000001f8.00000182:head v 97'75, mlcod=97'72)", 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.838921+0000", 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.902216146999997, 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13982625600000001, 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.780 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.838921+0000", 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.838921+0000", 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.838924+0000", 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1440000000000001e-06 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.840999+0000", 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020758159999999999 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841032+0000", 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2503999999999999e-05 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841056+0000", 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3901999999999999e-05 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841152+0000", 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5743999999999997e-05 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841220+0000", 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8163000000000005e-05 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978691+0000", 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13747087999999999 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978748+0000", 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7102999999999999e-05 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3977 3.18 e97/76 3:1b05c35d:::100000001f8.00000185:head v 97'74, mlcod=96'72)", 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.843861+0000", 2024-04-25T15:54:36.784 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.897276265999999, 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13525642099999999, 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.843861+0000", 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.843861+0000", 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.843863+0000", 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.559e-06 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850482+0000", 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066194130000000002 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850499+0000", 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7232e-05 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850512+0000", 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2152e-05 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850619+0000", 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107804 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.850653+0000", 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3905999999999997e-05 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979060+0000", 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128406873 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979118+0000", 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7482000000000003e-05 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3976 3.2c e97/76 3:35666940:::100000001f8.00000184:head v 97'76, mlcod=97'72)", 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.846326+0000", 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.894811494000002, 2024-04-25T15:54:36.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13284864299999999, 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846326+0000", 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846326+0000", 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846327+0000", 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1799999999999999e-06 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.849694+0000", 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003366834 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.849717+0000", 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2884e-05 2024-04-25T15:54:36.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.849737+0000", 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.001e-05 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.855904+0000", 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061673459999999998 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.855939+0000", 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4866000000000001e-05 2024-04-25T15:54:36.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979128+0000", 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12318931800000001 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979175+0000", 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6205000000000002e-05 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.792 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3967 3.24 e97/76 3:260ecc6d:::100000001f8.0000017b:head v 97'65, mlcod=96'63)", 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.860511+0000", 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 32.880626268, 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.118943471, 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.860511+0000", 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.860511+0000", 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.860513+0000", 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5650000000000001e-06 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864390+0000", 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038770699999999998 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864408+0000", 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7723999999999998e-05 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864421+0000", 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3507e-05 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864487+0000", 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5382999999999994e-05 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.864507+0000", 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0160999999999999e-05 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979407+0000", 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.114900351 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979455+0000", 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7710000000000002e-05 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.796 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5063 3.13 e99/76 3:c9096c28:::100000001f8.000005c3:head v 99'94, mlcod=98'92)", 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.035855+0000", 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.705282459999999, 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12388408300000001, 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035855+0000", 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035855+0000", 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035858+0000", 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4619999999999999e-06 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.038295+0000", 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002437715 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.038320+0000", 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5023000000000001e-05 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.038339+0000", 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9004999999999998e-05 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.038433+0000", 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4060000000000004e-05 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.038466+0000", 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2591e-05 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159657+0000", 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.121191412 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159739+0000", 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1815000000000002e-05 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5065 3.13 e99/76 3:c93be02c:::100000001f8.000005c5:head v 99'95, mlcod=98'92)", 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.042071+0000", 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.699066506000001, 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11780579100000001, 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.042071+0000", 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.042071+0000", 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.042073+0000", 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.441e-06 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.044354+0000", 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022805310000000001 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.044377+0000", 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2639000000000001e-05 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.044393+0000", 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6541e-05 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.048811+0000", 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044181259999999997 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.048830+0000", 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8448999999999999e-05 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159825+0000", 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.110995487 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159877+0000", 2024-04-25T15:54:36.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1576999999999998e-05 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5066 3.1b 3:da8118f1:::100000001f8.000005c6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.042509+0000", 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.698628515999999, 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11784069699999999, 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:36.805 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5066 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.042509+0000", 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.042509+0000", 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.042512+0000", 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0479999999999999e-06 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.046572+0000", 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004059478 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.046599+0000", 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7827e-05 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.046613+0000", 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4068e-05 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.048805+0000", 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002191536 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.048921+0000", 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116297 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.049097+0000", 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017534699999999999 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.062432+0000", 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013335509000000001 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160260+0000", 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097827955999999994 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160305+0000", 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4472999999999999e-05 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160350+0000", 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5157999999999997e-05 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.810 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5068 3.e e99/76 3:70f4d9e0:::100000001f8.000005c8:head v 99'82, mlcod=98'80)", 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.060378+0000", 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.680760028999998, 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.116301449, 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.060378+0000", 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.060378+0000", 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.060380+0000", 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9860000000000001e-06 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.063830+0000", 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034506889999999998 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.063865+0000", 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5089999999999998e-05 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.063883+0000", 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7308000000000001e-05 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.064064+0000", 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018170799999999999 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.064095+0000", 2024-04-25T15:54:36.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0501000000000002e-05 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.176618+0000", 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11252345499999999 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.176679+0000", 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0711999999999997e-05 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5141 3.a e99/76 3:51fd756b:::100000001f8.00000611:head v 99'73, mlcod=99'71)", 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.470053+0000", 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 26.271084542000001, 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11776887699999999, 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.470053+0000", 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.470053+0000", 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.470054+0000", 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1289999999999999e-06 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.475937+0000", 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058825420000000002 2024-04-25T15:54:36.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.475992+0000", 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5211999999999999e-05 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.476044+0000", 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1722999999999999e-05 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.476400+0000", 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035656799999999998 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.476441+0000", 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0670999999999997e-05 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.587781+0000", 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.111340061 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.587822+0000", 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0970999999999998e-05 2024-04-25T15:54:36.818 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5730 3.e e100/76 3:7357dd18:::100000001f8.0000085e:head v 100'92, mlcod=100'90)", 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.765507+0000", 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 22.975630068000001, 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12624703800000001, 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765507+0000", 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765507+0000", 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765509+0000", 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6670000000000001e-06 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.768699+0000", 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031895790000000001 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.768776+0000", 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7056000000000002e-05 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.768797+0000", 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1557e-05 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.768915+0000", 2024-04-25T15:54:36.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000117514 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.768938+0000", 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3464000000000001e-05 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.891675+0000", 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122736883 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.891755+0000", 2024-04-25T15:54:36.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9318000000000002e-05 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5732 3.6 3:61d8cf54:::100000001f8.00000860:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.769138+0000", 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 22.971999403000002, 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12252412, 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:36.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5732 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.769138+0000", 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.769138+0000", 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.769141+0000", 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6460000000000002e-06 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.775082+0000", 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059407449999999999 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.775117+0000", 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5295e-05 2024-04-25T15:54:36.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.775132+0000", 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5464999999999998e-05 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.775230+0000", 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7683999999999995e-05 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.775328+0000", 2024-04-25T15:54:36.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7582000000000002e-05 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.775468+0000", 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014009299999999999 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.792637+0000", 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017169803000000001 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:36.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.891454+0000", 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.098816218999999997 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.891564+0000", 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110074 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.891662+0000", 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8513999999999999e-05 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.828 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5780 3.f e100/76 3:f20b812e:::100000001f8.00000890:head v 100'87, mlcod=100'84)", 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.048228+0000", 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 22.692909820000001, 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12193096, 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048228+0000", 2024-04-25T15:54:36.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048228+0000", 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048230+0000", 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.216e-06 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050256+0000", 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020261540000000001 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050287+0000", 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1170000000000001e-05 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050304+0000", 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7037e-05 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050392+0000", 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7635999999999999e-05 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050426+0000", 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3608000000000001e-05 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.170056+0000", 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11963080399999999 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:36.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.170159+0000", 2024-04-25T15:54:36.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102335 2024-04-25T15:54:36.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.833 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.833 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.833 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.833 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:36.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1645 3.20 3.aeeac8e0 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e111)", 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.82967539199999996, 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.82972345199999997, 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for peered", 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6300000000000004e-07 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155198+0000", 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8860000000000003e-06 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155213+0000", 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4858000000000001e-05 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155221+0000", 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3189999999999995e-06 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155279+0000", 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7309999999999998e-05 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155290+0000", 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1807000000000001e-05 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1686 3.36 3.444dddb6 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e111)", 2024-04-25T15:54:36.988 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:36.191317+0000", 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.79355089499999998, 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.79370649400000004, 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for peered", 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191317+0000", 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191317+0000", 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191318+0000", 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4389999999999999e-06 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191325+0000", 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2759999999999997e-06 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191349+0000", 2024-04-25T15:54:36.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4657000000000001e-05 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191361+0000", 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2082999999999999e-05 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191435+0000", 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3330000000000004e-05 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191445+0000", 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0607e-05 2024-04-25T15:54:36.991 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1613 3.36 3.55e90af6 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e111)", 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:36.130817+0000", 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.85405067300000004, 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.85430537500000003, 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for peered", 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:36.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130817+0000", 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130817+0000", 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130819+0000", 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.894e-06 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:36.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130825+0000", 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1829999999999999e-06 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130850+0000", 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4573e-05 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130862+0000", 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1841000000000001e-05 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130931+0000", 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8799000000000003e-05 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130944+0000", 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3389e-05 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 3 2024-04-25T15:54:36.995 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:37.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-25T15:54:37.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-25T15:54:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.105+0000 7f3d7affd640 1 -- 172.21.15.67:0/2190327438 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v112) ==== 101+0+0 (secure 0 0 0) 0x7f3d6c0640f0 con 0x7f3d840736d0 2024-04-25T15:54:37.107 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6406f400 msgr2=0x7f3d64071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6406f400 0x7f3d64071910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3d74007fd0 tx=0x7f3d74009040 comp rx=0 tx=0).stop 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d840736d0 msgr2=0x7f3d84152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d840736d0 0x7f3d84152370 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00b3f0 tx=0x7f3d6c00b8b0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 shutdown_connections 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3d6406f400 0x7f3d64071910 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3d8415aaf0 0x7f3d84157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d840736d0 0x7f3d84152370 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 --2- 172.21.15.67:0/2190327438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8406d460 0x7f3d840c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 >> 172.21.15.67:0/2190327438 conn(0x7f3d840ce1c0 msgr2=0x7f3d84158460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 shutdown_connections 2024-04-25T15:54:37.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.109+0000 7f3d88e05640 1 -- 172.21.15.67:0/2190327438 wait complete. 2024-04-25T15:54:37.176 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:37.176 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:37.176 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:37.176 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:37.176 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:37.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-25T15:54:37.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.285+0000 7ff435b12640 1 Processor -- start 2024-04-25T15:54:37.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.285+0000 7ff435b12640 1 -- start start 2024-04-25T15:54:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.285+0000 7ff435b12640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 0x7ff4300cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.285+0000 7ff435b12640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff43015de00 0x7ff4300cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff43015e210 0x7ff430073c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50040/0 (socket says 172.21.15.67:50040) 2024-04-25T15:54:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42ffff640 1 -- 172.21.15.67:0/157916558 learned_addr learned my addr 172.21.15.67:0/157916558 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42effd640 1 --2- 172.21.15.67:0/157916558 >> v2:172.21.15.67:3300/0 conn(0x7ff43015de00 0x7ff4300cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4300ca8a0 con 0x7ff43015e210 2024-04-25T15:54:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff4300cab80 con 0x7ff43015de00 2024-04-25T15:54:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff43015a920 con 0x7ff430154ac0 2024-04-25T15:54:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42e7fc640 1 -- 172.21.15.67:0/157916558 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219996588 0 0) 0x7ff4300ca8a0 con 0x7ff43015e210 2024-04-25T15:54:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42e7fc640 1 -- 172.21.15.67:0/157916558 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff41800bd40 con 0x7ff43015e210 2024-04-25T15:54:37.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42effd640 1 -- 172.21.15.67:0/157916558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 msgr2=0x7ff4300cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42effd640 1 --2- 172.21.15.67:0/157916558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 0x7ff4300cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42effd640 1 -- 172.21.15.67:0/157916558 >> v1:172.21.15.67:6789/0 conn(0x7ff43015e210 legacy=0x7ff430073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42effd640 1 -- 172.21.15.67:0/157916558 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43015ac00 con 0x7ff43015de00 2024-04-25T15:54:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42effd640 1 --2- 172.21.15.67:0/157916558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015de00 0x7ff4300cc600 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff42400dbe0 tx=0x7ff42401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b88166d38d71a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42e7fc640 1 -- 172.21.15.67:0/157916558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240b0070 con 0x7ff43015de00 2024-04-25T15:54:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42e7fc640 1 -- 172.21.15.67:0/157916558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff42400d0d0 con 0x7ff43015de00 2024-04-25T15:54:37.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.289+0000 7ff42e7fc640 1 -- 172.21.15.67:0/157916558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42400b670 con 0x7ff43015de00 2024-04-25T15:54:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015de00 msgr2=0x7ff4300cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 --2- 172.21.15.67:0/157916558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015de00 0x7ff4300cc600 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff42400dbe0 tx=0x7ff42401c320 comp rx=0 tx=0).stop 2024-04-25T15:54:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 shutdown_connections 2024-04-25T15:54:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 --2- 172.21.15.67:0/157916558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015de00 0x7ff4300cc600 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 --2- 172.21.15.67:0/157916558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 0x7ff4300cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 >> 172.21.15.67:0/157916558 conn(0x7ff430068c70 msgr2=0x7ff430071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 shutdown_connections 2024-04-25T15:54:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.293+0000 7ff435b12640 1 -- 172.21.15.67:0/157916558 wait complete. 2024-04-25T15:54:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 Processor -- start 2024-04-25T15:54:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 -- start start 2024-04-25T15:54:37.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 0x7ff43014af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff43015de00 0x7ff43014f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015e210 0x7ff430148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4300caec0 con 0x7ff43015e210 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff43015b6e0 con 0x7ff43015de00 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4300cb040 con 0x7ff430154ac0 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015e210 0x7ff430148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42effd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff43015de00 0x7ff43014f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42effd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff43015de00 0x7ff43014f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:39590/0 (socket says 172.21.15.67:39590) 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42effd640 1 -- 172.21.15.67:0/2001272937 learned_addr learned my addr 172.21.15.67:0/2001272937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42f7fe640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 0x7ff43014af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42ffff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 msgr2=0x7ff43014af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42ffff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff430154ac0 0x7ff43014af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42ffff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff43015de00 msgr2=0x7ff43014f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42ffff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff43015de00 0x7ff43014f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42ffff640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43014b440 con 0x7ff43015e210 2024-04-25T15:54:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff42ffff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015e210 0x7ff430148b50 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff430073a30 tx=0x7ff42800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428017070 con 0x7ff43015e210 2024-04-25T15:54:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff430149f40 con 0x7ff43015e210 2024-04-25T15:54:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43014a500 con 0x7ff43015e210 2024-04-25T15:54:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff42800aae0 con 0x7ff43015e210 2024-04-25T15:54:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.297+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428021060 con 0x7ff43015e210 2024-04-25T15:54:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.301+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff4100052c0 con 0x7ff43015e210 2024-04-25T15:54:37.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.301+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff42801a070 con 0x7ff43015e210 2024-04-25T15:54:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.301+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff41c06f430 0x7ff41c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.301+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 6031+0+0 (secure 0 0 0) 0x7ff42808eda0 con 0x7ff43015e210 2024-04-25T15:54:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.301+0000 7ff42f7fe640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff41c06f430 0x7ff41c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.301+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+1285 (secure 0 0 0) 0x7ff42805b590 con 0x7ff43015e210 2024-04-25T15:54:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff42f7fe640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff41c06f430 0x7ff41c071940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff4200098f0 tx=0x7ff420006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410002a20 0x7ff4100058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4100050d0 con 0x7ff410002a20 2024-04-25T15:54:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410002a20 0x7ff4100058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff42effd640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7ff4100086a0 con 0x7ff410002a20 2024-04-25T15:54:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410002a20 0x7ff4100058a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff424019f50 tx=0x7ff4240190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240b0040 con 0x7ff410002a20 2024-04-25T15:54:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4240b5440 con 0x7ff410002a20 2024-04-25T15:54:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410002a20 msgr2=0x7ff4100058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410002a20 0x7ff4100058a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff424019f50 tx=0x7ff4240190c0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410002a20 msgr2=0x7ff4100058a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:37.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.305+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410002a20 0x7ff4100058a0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.313+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410008b10 0x7ff410007df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.313+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff4100084c0 con 0x7ff410008b10 2024-04-25T15:54:37.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.313+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410008b10 0x7ff410007df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.313+0000 7ff42effd640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7ff41000a720 con 0x7ff410008b10 2024-04-25T15:54:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.313+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410008b10 0x7ff410007df0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff424000c10 tx=0x7ff4240a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.313+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff424003d60 con 0x7ff410008b10 2024-04-25T15:54:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7ff424003d60 con 0x7ff410008b10 2024-04-25T15:54:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410008b10 msgr2=0x7ff410007df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410008b10 0x7ff410007df0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff424000c10 tx=0x7ff4240a8000 comp rx=0 tx=0).stop 2024-04-25T15:54:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410008b10 msgr2=0x7ff410007df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:37.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff410008b10 0x7ff410007df0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.317 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a: {} 2024-04-25T15:54:37.317 INFO:teuthology.orchestra.run.smithi067.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T15:54:37.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff410001f40 0x7ff410009700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff410009c70 con 0x7ff410001f40 2024-04-25T15:54:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff410001f40 0x7ff410009700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff42effd640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7ff410009e70 con 0x7ff410001f40 2024-04-25T15:54:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff410001f40 0x7ff410009700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff424018f30 tx=0x7ff4240190c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240b0030 con 0x7ff410001f40 2024-04-25T15:54:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4240b5a00 con 0x7ff410001f40 2024-04-25T15:54:37.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff410001f40 msgr2=0x7ff410009700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff410001f40 0x7ff410009700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff424018f30 tx=0x7ff4240190c0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff410001f40 msgr2=0x7ff410009700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff410001f40 0x7ff410009700 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff424018f30 tx=0x7ff4240190c0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.317+0000 7ff42ffff640 1 -- 172.21.15.67:0/2001272937 reap_dead start 2024-04-25T15:54:37.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.325+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41000c970 0x7ff41000ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.325+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff41000a3a0 con 0x7ff41000c970 2024-04-25T15:54:37.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.325+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41000c970 0x7ff41000ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.325+0000 7ff42effd640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7ff4100109e0 con 0x7ff41000c970 2024-04-25T15:54:37.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.325+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41000c970 0x7ff41000ed90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff410009e40 tx=0x7ff4240cb000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.325+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240b0030 con 0x7ff41000c970 2024-04-25T15:54:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7ff4240d5600 con 0x7ff41000c970 2024-04-25T15:54:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41000c970 msgr2=0x7ff41000ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41000c970 0x7ff41000ed90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff410009e40 tx=0x7ff4240cb000 comp rx=0 tx=0).stop 2024-04-25T15:54:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41000c970 msgr2=0x7ff41000ed90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41000c970 0x7ff41000ed90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.329 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b: {} 2024-04-25T15:54:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T15:54:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff41000f2d0 0x7ff41000f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff41000fd60 con 0x7ff41000f2d0 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff41000f2d0 0x7ff41000f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff42effd640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7ff41000ff60 con 0x7ff41000f2d0 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff41000f2d0 0x7ff41000f6e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff424000c10 tx=0x7ff4240c4000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240b0050 con 0x7ff41000f2d0 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.2 v2:172.21.15.176:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4240ea480 con 0x7ff41000f2d0 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff41000f2d0 msgr2=0x7ff41000f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff41000f2d0 0x7ff41000f6e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff424000c10 tx=0x7ff4240c4000 comp rx=0 tx=0).stop 2024-04-25T15:54:37.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff41000f2d0 msgr2=0x7ff41000f6e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:37.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.329+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff41000f2d0 0x7ff41000f6e0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff424000c10 tx=0x7ff4240c4000 comp rx=0 tx=0).stop 2024-04-25T15:54:37.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.337+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff410010170 0x7ff410014c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.337+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4100105b0 con 0x7ff410010170 2024-04-25T15:54:37.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.341+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff410010170 0x7ff410014c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.341+0000 7ff42effd640 1 -- 172.21.15.67:0/2001272937 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7ff4100107b0 con 0x7ff410010170 2024-04-25T15:54:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.341+0000 7ff42effd640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff410010170 0x7ff410014c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff424019f50 tx=0x7ff4240e0000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.341+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240c5050 con 0x7ff410010170 2024-04-25T15:54:37.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 <== mon.2 v2:172.21.15.176:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7ff4240ea630 con 0x7ff410010170 2024-04-25T15:54:37.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff410010170 msgr2=0x7ff410014c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff410010170 0x7ff410014c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff424019f50 tx=0x7ff4240e0000 comp rx=0 tx=0).stop 2024-04-25T15:54:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff417fff640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff410010170 msgr2=0x7ff410014c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T15:54:37.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff417fff640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff410010170 0x7ff410014c40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.345 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c: {} 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff41c06f430 msgr2=0x7ff41c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff41c06f430 0x7ff41c071940 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff4200098f0 tx=0x7ff420006f90 comp rx=0 tx=0).stop 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015e210 msgr2=0x7ff430148b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 --2- 172.21.15.67:0/2001272937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43015e210 0x7ff430148b50 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff430073a30 tx=0x7ff42800ce40 comp rx=0 tx=0).stop 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff42ffff640 1 -- 172.21.15.67:0/2001272937 reap_dead start 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 shutdown_connections 2024-04-25T15:54:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 >> 172.21.15.67:0/2001272937 conn(0x7ff430068c70 msgr2=0x7ff430152f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 shutdown_connections 2024-04-25T15:54:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.345+0000 7ff435b12640 1 -- 172.21.15.67:0/2001272937 wait complete. 2024-04-25T15:54:37.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-25T15:54:37.420 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:37.420 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:37.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:349 3.1c 3:3aa14ed1:::10000000002.00000059:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 94.200016406000003, 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.22837695099999999, 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 349 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219027+0000", 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.219031+0000", 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2529999999999998e-06 2024-04-25T15:54:37.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223646+0000", 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004615358 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223703+0000", 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7126000000000002e-05 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223733+0000", 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9263000000000001e-05 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.423 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223875+0000", 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142492 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.223896+0000", 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1379999999999999e-05 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.382174+0000", 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.158277742 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.389008+0000", 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068335669999999996 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427535+0000", 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.038527529999999997 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.427740+0000", 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020501699999999999 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.428060+0000", 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031972499999999998 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447045+0000", 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018984955000000001 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447084+0000", 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9245000000000001e-05 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.426 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447158+0000", 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3986000000000006e-05 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:03.447404+0000", 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024531200000000003 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2305 3.2f 3:f6648653:::100000001f8.000004fd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:37.427 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.616252768999999, 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16263440000000001, 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2305 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.428 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802790+0000", 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802795+0000", 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7530000000000001e-06 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815251+0000", 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012456312000000001 2024-04-25T15:54:37.429 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815675+0000", 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042352399999999999 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815700+0000", 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5225e-05 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815859+0000", 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015856099999999999 2024-04-25T15:54:37.430 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816239+0000", 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00038042099999999998 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816444+0000", 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020472099999999999 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.906577+0000", 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090133536 2024-04-25T15:54:37.431 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965298+0000", 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058721044999999999 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965364+0000", 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6013000000000006e-05 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965425+0000", 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0288999999999998e-05 2024-04-25T15:54:37.432 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2304 3.36 e90/18 3:6f9fe141:::100000001f8.000004fc:head v 90'48, mlcod=89'46)", 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.614926506000003, 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14476808899999999, 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.433 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804117+0000", 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.804121+0000", 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3950000000000003e-06 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.434 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818809+0000", 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014687881999999999 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818838+0000", 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9689e-05 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818859+0000", 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0353999999999999e-05 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.435 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.819110+0000", 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025134700000000002 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.819181+0000", 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1094999999999996e-05 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.948829+0000", 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12964726300000001 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.436 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.948885+0000", 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6063999999999997e-05 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2306 3.5 3:a0445fe6:::100000001f8.000004fe:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.603780380000003, 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.1339195, 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.437 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2306 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815263+0000", 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.438 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.815265+0000", 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2759999999999999e-06 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830149+0000", 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014884298000000001 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830753+0000", 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060401599999999997 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830772+0000", 2024-04-25T15:54:37.439 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9017000000000001e-05 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.831046+0000", 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027419800000000002 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.832603+0000", 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015568450000000001 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833182+0000", 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057851500000000004 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.440 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.859487+0000", 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026305354 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949024+0000", 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.089536491999999995 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949121+0000", 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7589000000000004e-05 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.441 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949182+0000", 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0900000000000003e-05 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2308 3.4 e90/68 3:2343a40c:::100000001f8.00000500:head v 90'35, mlcod=90'33)", 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 48.595845943, 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12610180200000001, 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.442 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823197+0000", 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.823200+0000", 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4590000000000001e-06 2024-04-25T15:54:37.443 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833167+0000", 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0099672500000000004 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833193+0000", 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6231999999999999e-05 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833214+0000", 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0942e-05 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.444 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833316+0000", 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101772 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.833360+0000", 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3835999999999998e-05 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949250+0000", 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11589087000000001 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.445 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949299+0000", 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8440999999999999e-05 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3916 3.c 3:3043a2b3:::100000001f8.00000148:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.493213+0000", 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.925830324000003, 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15984567799999999, 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.446 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3916 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493213+0000", 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493213+0000", 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.447 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493219+0000", 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0730000000000001e-06 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503369+0000", 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010149867 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503412+0000", 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3680000000000002e-05 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.448 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503429+0000", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6395000000000001e-05 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503520+0000", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1250999999999998e-05 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503645+0000", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125463 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.503826+0000", 2024-04-25T15:54:37.449 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000180543 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.530142+0000", 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026315782999999999 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.652922+0000", 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122779765 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.652979+0000", 2024-04-25T15:54:37.450 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7485999999999998e-05 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653058+0000", 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9371999999999998e-05 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3915 3.35 e97/18 3:ac1713fb:::100000001f8.00000147:head v 97'80, mlcod=96'78)", 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.501267+0000", 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.917776482000001, 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15192006999999999, 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.451 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501267+0000", 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501267+0000", 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501268+0000", 2024-04-25T15:54:37.452 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.674e-06 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504879+0000", 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003611104 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504900+0000", 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0581000000000002e-05 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504920+0000", 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9905999999999999e-05 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.453 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.505199+0000", 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000279399 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.505242+0000", 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3078000000000003e-05 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653126+0000", 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14788367899999999 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.454 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653187+0000", 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0649000000000002e-05 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3922 3.d e97/18 3:b3f0d00e:::100000001f8.0000014e:head v 97'72, mlcod=97'69)", 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.529755+0000", 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.889288110999999, 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13872857699999999, 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.455 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529755+0000", 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529755+0000", 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529757+0000", 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.996e-06 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.456 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.536951+0000", 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00719384 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.536973+0000", 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1857000000000001e-05 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.536990+0000", 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7382999999999999e-05 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.457 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.537138+0000", 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014826999999999999 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.537160+0000", 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1552e-05 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668418+0000", 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13125822300000001 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.458 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668483+0000", 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5456000000000005e-05 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3923 3.35 e97/18 3:ac352473:::100000001f8.0000014f:head v 97'81, mlcod=96'78)", 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.532588+0000", 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.886454962000002, 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13704064699999999, 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.459 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532588+0000", 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532588+0000", 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.460 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532590+0000", 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2979999999999999e-06 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535339+0000", 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027488389999999999 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535368+0000", 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8747000000000001e-05 2024-04-25T15:54:37.461 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535389+0000", 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1192e-05 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535503+0000", 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011338200000000001 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.535538+0000", 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5169000000000002e-05 2024-04-25T15:54:37.462 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669570+0000", 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13403183299999999 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669629+0000", 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9187000000000001e-05 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.463 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3925 3.28 3:143638ba:::100000001f8.00000151:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.534277+0000", 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.884765971, 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13462772200000001, 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3925 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.464 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.534277+0000", 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.534277+0000", 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.534281+0000", 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0010000000000003e-06 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541593+0000", 2024-04-25T15:54:37.465 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073119880000000002 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541641+0000", 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7651999999999997e-05 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541660+0000", 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9785000000000001e-05 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.541838+0000", 2024-04-25T15:54:37.466 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177877 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542057+0000", 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021911399999999999 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3", 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542366+0000", 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030894100000000002 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.467 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.578475+0000", 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036108636999999999 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668789+0000", 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090313702999999995 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668851+0000", 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2022999999999999e-05 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.468 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668905+0000", 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4001e-05 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3926 3.d e97/18 3:b20dc027:::100000001f8.00000152:head v 97'73, mlcod=97'69)", 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.543642+0000", 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.875401441999998, 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12602907699999999, 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.469 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543642+0000", 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543642+0000", 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543643+0000", 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5540000000000001e-06 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.470 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546584+0000", 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002940397 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546605+0000", 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1013e-05 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546619+0000", 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4219000000000001e-05 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.471 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546713+0000", 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4411000000000001e-05 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.546738+0000", 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4433000000000001e-05 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669633+0000", 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122895448 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.472 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669671+0000", 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7601999999999997e-05 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3970 3.16 3:694e1180:::100000001f8.0000017e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.813653+0000", 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.605389842000001, 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16449430000000001, 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.473 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3970 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.813653+0000", 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.813653+0000", 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.474 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.813655+0000", 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2290000000000002e-06 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817783+0000", 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041279630000000001 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817818+0000", 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5043000000000003e-05 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.475 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817832+0000", 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4054e-05 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818594+0000", 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076106100000000005 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818664+0000", 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0543000000000005e-05 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.476 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.818770+0000", 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010637300000000001 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945674+0000", 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12690309999999999 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978006+0000", 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032332847999999997 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.477 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978086+0000", 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9950000000000005e-05 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978147+0000", 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1136000000000006e-05 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3973 3.3a 3:5d7d7b7b:::100000001f8.00000181:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:37.478 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.825096+0000", 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 33.593947395000001, 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.153213244, 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3973 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.825096+0000", 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.479 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.825096+0000", 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.825098+0000", 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4600000000000002e-06 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832630+0000", 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075320489999999999 2024-04-25T15:54:37.480 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832678+0000", 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8359999999999998e-05 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832702+0000", 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3402999999999999e-05 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832810+0000", 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107723 2024-04-25T15:54:37.481 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.832947+0000", 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013714900000000001 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.833088+0000", 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014118099999999999 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846427+0000", 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013338645 2024-04-25T15:54:37.482 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978210+0000", 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13178303999999999 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978263+0000", 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3341000000000003e-05 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978309+0000", 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5893000000000003e-05 2024-04-25T15:54:37.483 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5054 3.3a 3:5eb75885:::100000001f8.000005ba:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:09.986550+0000", 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 27.432492594999999, 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12179451500000001, 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5054 2024-04-25T15:54:37.484 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.986550+0000", 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.986550+0000", 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.986552+0000", 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.809e-06 2024-04-25T15:54:37.485 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991418+0000", 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048656230000000003 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991614+0000", 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019635900000000001 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991630+0000", 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5789000000000001e-05 2024-04-25T15:54:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991727+0000", 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6520000000000004e-05 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.991853+0000", 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000126542 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:09.992031+0000", 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017744700000000001 2024-04-25T15:54:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.009109+0000", 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017078945000000002 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108172+0000", 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099062997 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108255+0000", 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2055e-05 2024-04-25T15:54:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.108345+0000", 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0428999999999998e-05 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5062 3.2f 3:f4e1ea89:::100000001f8.000005c2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.024415+0000", 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 27.394627904, 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12491956899999999, 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5062 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.024415+0000", 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.024415+0000", 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.024417+0000", 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0269999999999998e-06 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028822+0000", 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044052029999999999 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028859+0000", 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6504000000000002e-05 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.491 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028878+0000", 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9363000000000001e-05 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028992+0000", 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011408099999999999 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.029609+0000", 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061717599999999999 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.492 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.029826+0000", 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000216257 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.109119+0000", 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079293001000000002 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149258+0000", 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040138807999999998 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.493 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149300+0000", 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2765999999999997e-05 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149335+0000", 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4382999999999999e-05 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5777 3.2 e100/18 3:42907a88:::100000001f8.0000088d:head v 100'102, mlcod=100'100)", 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.025276+0000", 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.393767273999998, 2024-04-25T15:54:37.494 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13130074799999999, 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.025276+0000", 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.025276+0000", 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.495 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.025278+0000", 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5280000000000002e-06 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.030810+0000", 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055314040000000002 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.030845+0000", 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5522999999999999e-05 2024-04-25T15:54:37.496 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.030868+0000", 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2687000000000001e-05 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.031035+0000", 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016739700000000001 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.031082+0000", 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6999999999999997e-05 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.497 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.156524+0000", 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.125442211 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.156576+0000", 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1997999999999999e-05 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.498 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5781 3.2f 3:f565194a:::100000001f8.00000891:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.043944+0000", 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 23.375099243000001, 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12942135399999999, 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5781 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.499 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.043944+0000", 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.043944+0000", 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.043946+0000", 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0619999999999999e-06 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050351+0000", 2024-04-25T15:54:37.500 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064052220000000003 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050393+0000", 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2271000000000002e-05 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050410+0000", 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6617e-05 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050504+0000", 2024-04-25T15:54:37.501 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4523000000000003e-05 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050621+0000", 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116223 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.050800+0000", 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000179593 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.157212+0000", 2024-04-25T15:54:37.502 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.106412114 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.173250+0000", 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016038101999999999 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.173316+0000", 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5167999999999997e-05 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.173365+0000", 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9459000000000002e-05 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.503 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1613 3.36 3:6f5097aa:::100000001f8.000003b6:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111)", 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:36.130817+0000", 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2882257939999999, 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99535730200000005, 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130817+0000", 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.504 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130817+0000", 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130819+0000", 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.894e-06 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130825+0000", 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1829999999999999e-06 2024-04-25T15:54:37.505 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130850+0000", 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4573e-05 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130862+0000", 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1841000000000001e-05 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130931+0000", 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8799000000000003e-05 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.506 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.130944+0000", 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3389e-05 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.120159+0000", 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98921513100000003 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.120311+0000", 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000151891 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.507 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.120406+0000", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4596e-05 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.125099+0000", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046939570000000003 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.126025+0000", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00092528700000000001 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.126110+0000", 2024-04-25T15:54:37.508 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5401000000000004e-05 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.126175+0000", 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.436e-05 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1645 3.20 3:07135775:::100000001f8.000003d6:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111)", 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.263850513, 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96498462799999996, 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.509 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155193+0000", 2024-04-25T15:54:37.510 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6300000000000004e-07 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155198+0000", 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8860000000000003e-06 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155213+0000", 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4858000000000001e-05 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155221+0000", 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3189999999999995e-06 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.511 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155279+0000", 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7309999999999998e-05 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.155290+0000", 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1807000000000001e-05 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.116458+0000", 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.961167985 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.512 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.116703+0000", 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000244601 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.116814+0000", 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110782 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.118645+0000", 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001831116 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.513 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.119994+0000", 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001349542 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.120112+0000", 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000117858 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.120177+0000", 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4801000000000005e-05 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.514 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:1686 3.36 3:6dbbb222:::100000001f8.000003ff:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111)", 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:36.191317+0000", 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.2277260160000001, 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93649148100000001, 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191317+0000", 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.515 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191317+0000", 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191318+0000", 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4389999999999999e-06 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191325+0000", 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2759999999999997e-06 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:37.516 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191349+0000", 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4657000000000001e-05 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191361+0000", 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2082999999999999e-05 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191435+0000", 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3330000000000004e-05 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:37.517 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:36.191445+0000", 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0607e-05 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.122634+0000", 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93118825999999999 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.122871+0000", 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023782 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1", 2024-04-25T15:54:37.518 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.122991+0000", 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119543 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.126286+0000", 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032945460000000002 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.127657+0000", 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013711769999999999 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.127741+0000", 2024-04-25T15:54:37.519 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4544999999999998e-05 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:37.127808+0000", 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7198000000000003e-05 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:37.520 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:37.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 Processor -- start 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 -- start start 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2c0074790 0x7fb2c015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb2c015e020 0x7fb2c00723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c00c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c0072dc0 con 0x7fb2c0074790 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c015b430 con 0x7fb2c015e020 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c77e2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c015a670 con 0x7fb2c00728e0 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c557e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2c0074790 0x7fb2c015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c557e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2c0074790 0x7fb2c015dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46122/0 (socket says 172.21.15.67:46122) 2024-04-25T15:54:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.517+0000 7fb2c557e640 1 -- 172.21.15.67:0/2210305460 learned_addr learned my addr 172.21.15.67:0/2210305460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c4d7d640 1 --2- 172.21.15.67:0/2210305460 >> v2:172.21.15.176:3300/0 conn(0x7fb2c015e020 0x7fb2c00723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c5d7f640 1 --2- 172.21.15.67:0/2210305460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c00c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c557e640 1 -- 172.21.15.67:0/2210305460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 msgr2=0x7fb2c00c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c557e640 1 --2- 172.21.15.67:0/2210305460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c00c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c557e640 1 -- 172.21.15.67:0/2210305460 >> v2:172.21.15.176:3300/0 conn(0x7fb2c015e020 msgr2=0x7fb2c00723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c557e640 1 --2- 172.21.15.67:0/2210305460 >> v2:172.21.15.176:3300/0 conn(0x7fb2c015e020 0x7fb2c00723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c557e640 1 -- 172.21.15.67:0/2210305460 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c015a950 con 0x7fb2c0074790 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c4d7d640 1 --2- 172.21.15.67:0/2210305460 >> v2:172.21.15.176:3300/0 conn(0x7fb2c015e020 0x7fb2c00723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c5d7f640 1 --2- 172.21.15.67:0/2210305460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c00c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c557e640 1 --2- 172.21.15.67:0/2210305460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c015dae0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0070a50 tx=0x7fb2bc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80afd9d52f7b200 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2b7fff640 1 -- 172.21.15.67:0/2210305460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0b5070 con 0x7fb2c0074790 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2b7fff640 1 -- 172.21.15.67:0/2210305460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2bc0a89d0 con 0x7fb2c0074790 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2b7fff640 1 -- 172.21.15.67:0/2210305460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0a8da0 con 0x7fb2c0074790 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 -- 172.21.15.67:0/2210305460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 msgr2=0x7fb2c015dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/2210305460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c015dae0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0070a50 tx=0x7fb2bc01ed10 comp rx=0 tx=0).stop 2024-04-25T15:54:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 -- 172.21.15.67:0/2210305460 shutdown_connections 2024-04-25T15:54:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/2210305460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c00c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/2210305460 >> v2:172.21.15.176:3300/0 conn(0x7fb2c015e020 0x7fb2c00723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/2210305460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c015dae0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 -- 172.21.15.67:0/2210305460 >> 172.21.15.67:0/2210305460 conn(0x7fb2c006fe70 msgr2=0x7fb2c00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 -- 172.21.15.67:0/2210305460 shutdown_connections 2024-04-25T15:54:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.521+0000 7fb2c77e2640 1 -- 172.21.15.67:0/2210305460 wait complete. 2024-04-25T15:54:37.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 Processor -- start 2024-04-25T15:54:37.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 -- start start 2024-04-25T15:54:37.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c01f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c00cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2c015e020 0x7fb2c014b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c0073080 con 0x7fb2c0074790 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c015af10 con 0x7fb2c015e020 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c0074ba0 con 0x7fb2c00728e0 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c00cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c00cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46130/0 (socket says 172.21.15.67:46130) 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 -- 172.21.15.67:0/3360018372 learned_addr learned my addr 172.21.15.67:0/3360018372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c557e640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c01f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c5d7f640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2c015e020 0x7fb2c014b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 -- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 msgr2=0x7fb2c01f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c01f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 -- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2c015e020 msgr2=0x7fb2c014b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2c015e020 0x7fb2c014b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 -- 172.21.15.67:0/3360018372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c01f17d0 con 0x7fb2c0074790 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c5d7f640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2c015e020 0x7fb2c014b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c4d7d640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c00cc9a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000c820 tx=0x7fb2b000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2b5ffb640 1 -- 172.21.15.67:0/3360018372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0015070 con 0x7fb2c0074790 2024-04-25T15:54:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2b5ffb640 1 -- 172.21.15.67:0/3360018372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b0007a30 con 0x7fb2c0074790 2024-04-25T15:54:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c015d250 con 0x7fb2c0074790 2024-04-25T15:54:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2b5ffb640 1 -- 172.21.15.67:0/3360018372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0007e40 con 0x7fb2c0074790 2024-04-25T15:54:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.525+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c0155260 con 0x7fb2c0074790 2024-04-25T15:54:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.529+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c0155460 con 0x7fb2c0074790 2024-04-25T15:54:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.529+0000 7fb2b5ffb640 1 -- 172.21.15.67:0/3360018372 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb2b0018070 con 0x7fb2c0074790 2024-04-25T15:54:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.529+0000 7fb2b5ffb640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2a406f430 0x7fb2a4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.529+0000 7fb2b5ffb640 1 -- 172.21.15.67:0/3360018372 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 6031+0+0 (secure 0 0 0) 0x7fb2b008f670 con 0x7fb2c0074790 2024-04-25T15:54:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.529+0000 7fb2c557e640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2a406f430 0x7fb2a4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:37.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.529+0000 7fb2c557e640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2a406f430 0x7fb2a4071940 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0b2000 tx=0x7fb2bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:37.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.533+0000 7fb2b5ffb640 1 -- 172.21.15.67:0/3360018372 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb2c0155460 con 0x7fb2c0074790 2024-04-25T15:54:37.677 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:37.677 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:37.677 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:37.678 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:37.695 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-25T15:54:37.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:37.709+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7fb2c006a480 con 0x7fb2c0074790 2024-04-25T15:54:37.840 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:37.840 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:37.841 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:37.841 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:37.841 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:37.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-25T15:54:38.012 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:637 3.20 3:07263727:::100000001f7.00000079:head [write 0~4194304 in=4194304b] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 74.349524974000005, 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.26243838600000002, 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 637 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659364+0000", 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.659366+0000", 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8330000000000001e-06 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.878383+0000", 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.219017295 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880178+0000", 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017948459999999999 2024-04-25T15:54:38.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880293+0000", 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115108 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880471+0000", 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000178126 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880753+0000", 2024-04-25T15:54:38.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000281871 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.880924+0000", 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017151499999999999 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.920373+0000", 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.039448372000000002 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921649+0000", 2024-04-25T15:54:38.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012757210000000001 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921724+0000", 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5549000000000005e-05 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.921802+0000", 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8150000000000002e-05 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(mds.0.4:354 3.10 3:0ba0dc78:::10000000002.000000fd:head [delete] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e81)", 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 73.409572183999998, 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56691576200000005, 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599317+0000", 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599318+0000", 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:24.599326+0000", 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9459999999999998e-06 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161759+0000", 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.56243287799999997 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161783+0000", 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4057e-05 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.161899+0000", 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011615799999999999 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162007+0000", 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107823 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.162110+0000", 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010235500000000001 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.163904+0000", 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017941470000000001 2024-04-25T15:54:38.022 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166138+0000", 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022346190000000002 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166186+0000", 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7902999999999999e-05 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:25.166232+0000", 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6220999999999997e-05 2024-04-25T15:54:38.023 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2304 3.36 3:6f9fe141:::100000001f8.000004fc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.218327283999997, 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15859584900000001, 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.024 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2304 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790562+0000", 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.790566+0000", 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5369999999999999e-06 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802415+0000", 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011848747999999999 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802510+0000", 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5036000000000003e-05 2024-04-25T15:54:38.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802550+0000", 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9807e-05 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.802712+0000", 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162619 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.803017+0000", 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000304433 2024-04-25T15:54:38.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.803308+0000", 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029095100000000001 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.835450+0000", 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.032142776999999997 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949058+0000", 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.113607898 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949100+0000", 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1213e-05 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949157+0000", 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.783e-05 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.029 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2307 3.35 3:ad417e89:::100000001f8.000004ff:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.197757604000003, 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.138515105, 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2307 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811131+0000", 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.811135+0000", 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1629999999999998e-06 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824210+0000", 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013074564 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824274+0000", 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4131e-05 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824302+0000", 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8167e-05 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824464+0000", 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000161605 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.824749+0000", 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028465599999999999 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.825010+0000", 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026149799999999998 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844599+0000", 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.019588795999999999 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949581+0000", 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10498215399999999 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949617+0000", 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6498000000000002e-05 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.949646+0000", 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8873e-05 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2309 3.2c 3:354b7d43:::100000001f8.00000501:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.181900669000001, 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.127916427, 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.035 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2309 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826988+0000", 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.826993+0000", 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.746e-06 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839270+0000", 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012277221 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839339+0000", 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8709999999999998e-05 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839369+0000", 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0590999999999999e-05 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839557+0000", 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018798099999999999 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839975+0000", 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041773500000000002 2024-04-25T15:54:38.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.840214+0000", 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023835300000000001 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.925866+0000", 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.085652939999999997 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954707+0000", 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028840472999999998 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954819+0000", 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011174 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.954905+0000", 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5937000000000001e-05 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.040 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2480 3.35 3:afdde0ae:::100000001f8.000005ac:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.970419102000001, 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14409117900000001, 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2480 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:54:38.041 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038470+0000", 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.038475+0000", 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0000000000000004e-06 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.046847+0000", 2024-04-25T15:54:38.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083724569999999998 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131712+0000", 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084864593000000002 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131735+0000", 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3448e-05 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.131860+0000", 2024-04-25T15:54:38.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124258 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.132129+0000", 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026964399999999998 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.132397+0000", 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026743300000000003 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.167840+0000", 2024-04-25T15:54:38.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.035443241 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182375+0000", 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014535002 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182462+0000", 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6887999999999999e-05 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.182561+0000", 2024-04-25T15:54:38.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9215000000000005e-05 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2499 3.26 3:65b3bea5:::100000001f8.000005bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.838173675999997, 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.32409411900000001, 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.046 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2499 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170715+0000", 2024-04-25T15:54:38.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.170718+0000", 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9699999999999999e-06 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.178145+0000", 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074272360000000003 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457402+0000", 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.27925624300000002 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457419+0000", 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7387000000000001e-05 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457521+0000", 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101615 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457720+0000", 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019950299999999999 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.457907+0000", 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000186973 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.478350+0000", 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.020442846000000001 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494652+0000", 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016301573999999999 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494737+0000", 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5216000000000005e-05 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:50.494809+0000", 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2556000000000001e-05 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3312 3.35 3:af24358c:::100000001f8.000008ec:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-25T15:54:38.051 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.539972482000003, 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14714041, 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3312 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468916+0000", 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.468918+0000", 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9929999999999998e-06 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.473248+0000", 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043301030000000001 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585462+0000", 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.112213177 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585475+0000", 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3438000000000001e-05 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585550+0000", 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4975000000000006e-05 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585692+0000", 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142022 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.585805+0000", 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112736 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.601250+0000", 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015444866999999999 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.615882+0000", 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014632116000000001 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.615956+0000", 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3769000000000005e-05 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:55.616057+0000", 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101214 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3915 3.35 3:ac1713fb:::100000001f8.00000147:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.493809+0000", 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.515079978000003, 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17255479300000001, 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3915 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493809+0000", 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493809+0000", 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.493812+0000", 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3440000000000001e-06 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500643+0000", 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0068309570000000003 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500700+0000", 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7003999999999998e-05 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500724+0000", 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3328000000000001e-05 2024-04-25T15:54:38.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500816+0000", 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2429999999999994e-05 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.500912+0000", 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6422000000000006e-05 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.501075+0000", 2024-04-25T15:54:38.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016272500000000001 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.653383+0000", 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15230775299999999 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666228+0000", 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012845132 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666300+0000", 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2504999999999998e-05 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666364+0000", 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3193000000000003e-05 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.062 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3919 3.2c 3:37bce3fe:::100000001f8.0000014b:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.504543+0000", 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.504346140000003, 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16566847200000001, 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3919 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504543+0000", 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504543+0000", 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.504546+0000", 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0960000000000001e-06 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512225+0000", 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0076787909999999999 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514415+0000", 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021900980000000001 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514429+0000", 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3848e-05 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514537+0000", 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108316 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514672+0000", 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013469099999999999 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515838+0000", 2024-04-25T15:54:38.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001166411 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666543+0000", 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15070541400000001 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670106+0000", 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0035621989999999998 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670163+0000", 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7689000000000003e-05 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670211+0000", 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7919e-05 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3917 3.27 e97/85 3:e652e8b8:::100000001f8.00000149:head v 97'58, mlcod=95'56)", 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.510092+0000", 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.498796448, 2024-04-25T15:54:38.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15662477599999999, 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.510092+0000", 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.510092+0000", 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.510094+0000", 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.455e-06 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514379+0000", 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042855510000000003 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514408+0000", 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8903000000000001e-05 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514429+0000", 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0230000000000001e-05 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514575+0000", 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146596 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.514612+0000", 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7092000000000003e-05 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666650+0000", 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15203760599999999 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666717+0000", 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7342999999999995e-05 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.072 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3920 3.26 3:6422fb70:::100000001f8.0000014c:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.512238+0000", 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.496650582000001, 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15489787399999999, 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3920 2024-04-25T15:54:38.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512238+0000", 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512238+0000", 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.512241+0000", 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8849999999999999e-06 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520461+0000", 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0082202270000000001 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520514+0000", 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2318999999999998e-05 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520535+0000", 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1488000000000002e-05 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520660+0000", 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012459999999999999 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.520822+0000", 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000161862 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.521056+0000", 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000234384 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666410+0000", 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14535368400000001 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667015+0000", 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00060488299999999996 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667082+0000", 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7322000000000004e-05 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667136+0000", 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4219999999999999e-05 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3923 3.35 3:ac352473:::100000001f8.0000014f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.078 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523946+0000", 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.484943254999997, 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14618500700000001, 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3923 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523946+0000", 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523946+0000", 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523949+0000", 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9299999999999999e-06 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531931+0000", 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007982701 2024-04-25T15:54:38.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531977+0000", 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5793e-05 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.531998+0000", 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1097000000000001e-05 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532113+0000", 2024-04-25T15:54:38.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114997 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532237+0000", 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012414699999999999 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.532445+0000", 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020746499999999999 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669235+0000", 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.136789986 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670026+0000", 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00079101099999999999 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670086+0000", 2024-04-25T15:54:38.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0399999999999998e-05 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.670131+0000", 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4480000000000001e-05 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3966 3.2c 3:36f7a988:::100000001f8.0000017a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.793699+0000", 2024-04-25T15:54:38.084 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.215189678999998, 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13581676300000001, 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3966 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.793699+0000", 2024-04-25T15:54:38.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.793699+0000", 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.793703+0000", 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6770000000000001e-06 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801135+0000", 2024-04-25T15:54:38.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007432564 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801198+0000", 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.232e-05 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801213+0000", 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5705e-05 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801321+0000", 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010709000000000001 2024-04-25T15:54:38.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801404+0000", 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3844000000000006e-05 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.801525+0000", 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121007 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.852854+0000", 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051328560000000002 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929426+0000", 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.076572232000000004 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929470+0000", 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3918000000000002e-05 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.929516+0000", 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5846000000000001e-05 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3972 3.a 3:506f1d9f:::100000001f8.00000180:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.821706+0000", 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.187183179000002, 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15946833799999999, 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3972 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821706+0000", 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821706+0000", 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.821708+0000", 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7480000000000002e-06 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829901+0000", 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0081928009999999996 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830176+0000", 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000275006 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830207+0000", 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1019999999999998e-05 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830305+0000", 2024-04-25T15:54:38.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8097999999999995e-05 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830484+0000", 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017865099999999999 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.830670+0000", 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018631799999999999 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971559+0000", 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14088915499999999 2024-04-25T15:54:38.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981057+0000", 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0094979709999999992 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981123+0000", 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5902999999999996e-05 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981174+0000", 2024-04-25T15:54:38.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0667000000000001e-05 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3971 3.3f e97/18 3:febcb5ea:::100000001f8.0000017f:head v 97'71, mlcod=97'68)", 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.827561+0000", 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.181328231999998, 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14429470999999999, 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.827561+0000", 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.827561+0000", 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.827562+0000", 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.849e-06 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.828970+0000", 2024-04-25T15:54:38.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001407438 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829000+0000", 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0320000000000001e-05 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829020+0000", 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9624999999999999e-05 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829144+0000", 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012414199999999999 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829169+0000", 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4834999999999999e-05 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971813+0000", 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14264386100000001 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971855+0000", 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2639999999999998e-05 2024-04-25T15:54:38.098 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3974 3.2c 3:35c2bab0:::100000001f8.00000182:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.829920+0000", 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.178969035000001, 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14917461800000001, 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.099 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3974 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829920+0000", 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829920+0000", 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.829922+0000", 2024-04-25T15:54:38.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3879999999999998e-06 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835844+0000", 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059220439999999996 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836262+0000", 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00041762199999999998 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836280+0000", 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8255999999999999e-05 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836402+0000", 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012145400000000001 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836576+0000", 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000174184 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.836798+0000", 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022228499999999999 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971962+0000", 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135163916 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.978979+0000", 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070174729999999998 2024-04-25T15:54:38.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979046+0000", 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6931e-05 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.979094+0000", 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8065000000000001e-05 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3975 3.20 3:06072bd3:::100000001f8.00000183:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.104 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.835855+0000", 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.173033982, 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135919927, 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3975 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835855+0000", 2024-04-25T15:54:38.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835855+0000", 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.835857+0000", 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3980000000000002e-06 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841292+0000", 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054348349999999998 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841593+0000", 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030063000000000003 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841605+0000", 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2245e-05 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841688+0000", 2024-04-25T15:54:38.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3443999999999996e-05 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841801+0000", 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011285600000000001 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841952+0000", 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000150717 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.856301+0000", 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014349259 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971673+0000", 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.115372129 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971720+0000", 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6700999999999999e-05 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.971775+0000", 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4713000000000003e-05 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3976 3.2c 3:35666940:::100000001f8.00000184:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.841300+0000", 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 34.167588533999997, 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13985472299999999, 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.110 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3976 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841300+0000", 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841300+0000", 2024-04-25T15:54:38.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.841302+0000", 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6190000000000001e-06 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845593+0000", 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042905900000000004 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845639+0000", 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6851999999999998e-05 2024-04-25T15:54:38.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845664+0000", 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4301e-05 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845780+0000", 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011649799999999999 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.845927+0000", 2024-04-25T15:54:38.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014731 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.846114+0000", 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018610400000000001 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972147+0000", 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12603325100000001 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981025+0000", 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088782259999999995 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981104+0000", 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8702000000000006e-05 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.981155+0000", 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1270000000000002e-05 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5059 3.26 3:65b3bea5:::100000001f8.000005bf:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.009444+0000", 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 27.999444953000001, 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13971987, 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5059 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.116 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.009444+0000", 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.009444+0000", 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.009458+0000", 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3752e-05 2024-04-25T15:54:38.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.016728+0000", 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072703159999999998 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.016763+0000", 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4585000000000002e-05 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.016782+0000", 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9771000000000001e-05 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.016904+0000", 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121635 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017029+0000", 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125046 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 7", 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017201+0000", 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017233099999999999 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.123177+0000", 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.105976025 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149025+0000", 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.025847972 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149099+0000", 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3444999999999999e-05 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.149164+0000", 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4992000000000004e-05 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.121 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:38.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.109+0000 7fb2b5ffb640 1 -- 172.21.15.67:0/3360018372 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v113) ==== 113+0+0 (secure 0 0 0) 0x7fb2b005c500 con 0x7fb2c0074790 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2a406f430 msgr2=0x7fb2a4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2a406f430 0x7fb2a4071940 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0b2000 tx=0x7fb2bc008040 comp rx=0 tx=0).stop 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 msgr2=0x7fb2c00cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c00cc9a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000c820 tx=0x7fb2b000cce0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 shutdown_connections 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb2a406f430 0x7fb2a4071940 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2c015e020 0x7fb2c014b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2c0074790 0x7fb2c00cc9a0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 --2- 172.21.15.67:0/3360018372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2c00728e0 0x7fb2c01f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 >> 172.21.15.67:0/3360018372 conn(0x7fb2c006fe70 msgr2=0x7fb2c00c6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 shutdown_connections 2024-04-25T15:54:38.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.113+0000 7fb2c77e2640 1 -- 172.21.15.67:0/3360018372 wait complete. 2024-04-25T15:54:38.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-25T15:54:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.341+0000 7fa27de02640 1 Processor -- start 2024-04-25T15:54:38.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.341+0000 7fa27de02640 1 -- start start 2024-04-25T15:54:38.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa27de02640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa278160c20 0x7fa278161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa27de02640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa27806c640 0x7fa27806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa27de02640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa278164ff0 con 0x7fa27815e180 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa277fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa27806c640 0x7fa27806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa277fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa27806c640 0x7fa27806ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46146/0 (socket says 172.21.15.67:46146) 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa277fff640 1 -- 172.21.15.67:0/938594404 learned_addr learned my addr 172.21.15.67:0/938594404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa27de02640 1 -- 172.21.15.67:0/938594404 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fa278164b60 con 0x7fa27806c640 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa27de02640 1 -- 172.21.15.67:0/938594404 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa2780c5d90 con 0x7fa278160c20 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa277fff640 1 -- 172.21.15.67:0/938594404 >> v2:172.21.15.176:3300/0 conn(0x7fa278160c20 msgr2=0x7fa278161030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:54:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa277fff640 1 --2- 172.21.15.67:0/938594404 >> v2:172.21.15.176:3300/0 conn(0x7fa278160c20 0x7fa278161030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa277fff640 1 -- 172.21.15.67:0/938594404 >> v1:172.21.15.67:6789/0 conn(0x7fa27815e180 legacy=0x7fa2780c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.345+0000 7fa277fff640 1 -- 172.21.15.67:0/938594404 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa278164230 con 0x7fa27806c640 2024-04-25T15:54:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa277fff640 1 --2- 172.21.15.67:0/938594404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27806c640 0x7fa27806ca50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa27806d2d0 tx=0x7fa27001f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ef0e57a374a5ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa2767fc640 1 -- 172.21.15.67:0/938594404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2700b4070 con 0x7fa27806c640 2024-04-25T15:54:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa2767fc640 1 -- 172.21.15.67:0/938594404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2700af400 con 0x7fa27806c640 2024-04-25T15:54:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa2767fc640 1 -- 172.21.15.67:0/938594404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2700af7d0 con 0x7fa27806c640 2024-04-25T15:54:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa27de02640 1 -- 172.21.15.67:0/938594404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27806c640 msgr2=0x7fa27806ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa27de02640 1 --2- 172.21.15.67:0/938594404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27806c640 0x7fa27806ca50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa27806d2d0 tx=0x7fa27001f200 comp rx=0 tx=0).stop 2024-04-25T15:54:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa27de02640 1 -- 172.21.15.67:0/938594404 shutdown_connections 2024-04-25T15:54:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa27de02640 1 --2- 172.21.15.67:0/938594404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27806c640 0x7fa27806ca50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa27de02640 1 --2- 172.21.15.67:0/938594404 >> v2:172.21.15.176:3300/0 conn(0x7fa278160c20 0x7fa278161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.349+0000 7fa27de02640 1 -- 172.21.15.67:0/938594404 >> 172.21.15.67:0/938594404 conn(0x7fa27806fe70 msgr2=0x7fa2780cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.353+0000 7fa27de02640 1 -- 172.21.15.67:0/938594404 shutdown_connections 2024-04-25T15:54:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.353+0000 7fa27de02640 1 -- 172.21.15.67:0/938594404 wait complete. 2024-04-25T15:54:38.352 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:38.353 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-04-25T15:54:38.353 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.353+0000 7fa27de02640 1 Processor -- start 2024-04-25T15:54:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.353+0000 7fa27de02640 1 -- start start 2024-04-25T15:54:38.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa27806c640 0x7fa2781fb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:38.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27815e180 0x7fa278072c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa278160c20 0x7fa278156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2780c5a40 con 0x7fa27815e180 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa2781647f0 con 0x7fa278160c20 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2780c6070 con 0x7fa27806c640 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27815e180 0x7fa278072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27815e180 0x7fa278072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46154/0 (socket says 172.21.15.67:46154) 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 -- 172.21.15.67:0/1574570753 learned_addr learned my addr 172.21.15.67:0/1574570753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa2777fe640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa27806c640 0x7fa2781fb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa277fff640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa278160c20 0x7fa278156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 -- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa27806c640 msgr2=0x7fa2781fb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa27806c640 0x7fa2781fb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 -- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa278160c20 msgr2=0x7fa278156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa278160c20 0x7fa278156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 -- 172.21.15.67:0/1574570753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27815c550 con 0x7fa27815e180 2024-04-25T15:54:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa2777fe640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa27806c640 0x7fa2781fb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa276ffd640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27815e180 0x7fa278072c80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa26800b410 tx=0x7fa26800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa25ffff640 1 -- 172.21.15.67:0/1574570753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268015020 con 0x7fa27815e180 2024-04-25T15:54:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa25ffff640 1 -- 172.21.15.67:0/1574570753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa268011830 con 0x7fa27815e180 2024-04-25T15:54:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa27815c830 con 0x7fa27815e180 2024-04-25T15:54:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa25ffff640 1 -- 172.21.15.67:0/1574570753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268011c20 con 0x7fa27815e180 2024-04-25T15:54:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.357+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2781f75a0 con 0x7fa27815e180 2024-04-25T15:54:38.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.361+0000 7fa25ffff640 1 -- 172.21.15.67:0/1574570753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa268018020 con 0x7fa27815e180 2024-04-25T15:54:38.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.361+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa27806a480 con 0x7fa27815e180 2024-04-25T15:54:38.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.361+0000 7fa25ffff640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa26406f430 0x7fa264071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:38.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.361+0000 7fa25ffff640 1 -- 172.21.15.67:0/1574570753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 6031+0+0 (secure 0 0 0) 0x7fa26808ec40 con 0x7fa27815e180 2024-04-25T15:54:38.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.361+0000 7fa2777fe640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa26406f430 0x7fa264071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:38.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.361+0000 7fa2777fe640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa26406f430 0x7fa264071940 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa26c001670 tx=0x7fa26c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:38.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.373+0000 7fa25ffff640 1 -- 172.21.15.67:0/1574570753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa26801f070 con 0x7fa27815e180 2024-04-25T15:54:38.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-25T15:54:38.597 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-04-25T15:54:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 180, 2024-04-25T15:54:38.597 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-04-25T15:54:38.598 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:38.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-25T15:54:38.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:38.641+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) -- 0x7fa2780c9e50 con 0x7fa27815e180 2024-04-25T15:54:38.828 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-25T15:54:38.828 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 259.727476567, 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.42299859899999998, 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4201", 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "172.21.15.176:0/18783227", 2024-04-25T15:54:38.829 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098464+0000", 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098467+0000", 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8200000000000001e-06 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098484+0000", 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6966e-05 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098536+0000", 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1969999999999999e-05 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098562+0000", 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6483e-05 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098694+0000", 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013152199999999999 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.098714+0000", 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9942999999999999e-05 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517404+0000", 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.41869070400000002 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.517685+0000", 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028070200000000002 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.518106+0000", 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000420822 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521083+0000", 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029773170000000002 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521239+0000", 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000155585 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521334+0000", 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5556000000000005e-05 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:50:19.521463+0000", 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012820900000000001 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:254 3.13 3:cb13e0c9:::10000000002.000000fa:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 105.513602034, 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.19133792999999999, 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.835 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312339+0000", 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.312342+0000", 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.484e-06 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.318298+0000", 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005955452 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492074+0000", 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17377627400000001 2024-04-25T15:54:38.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492089+0000", 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5325e-05 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492211+0000", 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121501 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492389+0000", 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000177937 2024-04-25T15:54:38.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.492523+0000", 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134442 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.499440+0000", 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006917065 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503564+0000", 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041241999999999997 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503625+0000", 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0426e-05 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:52:53.503677+0000", 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1823999999999997e-05 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:627 3.2d 3:b469fc05:::100000001f7.0000006f:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 75.242924082000002, 2024-04-25T15:54:38.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.23636167899999999, 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 627 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583017+0000", 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.583018+0000", 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.655e-06 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800453+0000", 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.21743459700000001 2024-04-25T15:54:38.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800610+0000", 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015757499999999999 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800631+0000", 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0319999999999999e-05 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800755+0000", 2024-04-25T15:54:38.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124268 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.800988+0000", 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023269100000000001 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2", 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.801215+0000", 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227285 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.815427+0000", 2024-04-25T15:54:38.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014212373 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819216+0000", 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037888869999999999 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819303+0000", 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7213000000000007e-05 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:23.819378+0000", 2024-04-25T15:54:38.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4814999999999994e-05 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2266 3.11 3:89528692:::100000001f8.000004d6:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.254757366, 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.38717112199999998, 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.846 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2266 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571183+0000", 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.571186+0000", 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4569999999999999e-06 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.784888+0000", 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.213702742 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785181+0000", 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029301099999999998 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785201+0000", 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9683000000000001e-05 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785333+0000", 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013224600000000001 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.785779+0000", 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044548299999999999 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.786106+0000", 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032678999999999998 2024-04-25T15:54:38.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870593+0000", 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.084487524999999994 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958234+0000", 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.087640933000000004 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958283+0000", 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8473999999999997e-05 2024-04-25T15:54:38.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.958354+0000", 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1778000000000003e-05 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2272 3.13 3:ca3e88be:::100000001f8.000004dc:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.030008717999998, 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13173032000000001, 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.851 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2272 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795932+0000", 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.795936+0000", 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.754e-06 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805167+0000", 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092308890000000008 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805557+0000", 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039059400000000002 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805588+0000", 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0865999999999999e-05 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805759+0000", 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017085599999999999 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806172+0000", 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000413416 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.806552+0000", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000379417 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.905757+0000", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.099205023000000003 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927568+0000", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021810800000000002 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927616+0000", 2024-04-25T15:54:38.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8170000000000001e-05 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.927662+0000", 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6535000000000001e-05 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2273 3.39 3:9fcbcc01:::100000001f8.000004dd:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.020756603000002, 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15983650499999999, 2024-04-25T15:54:38.856 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2273 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805184+0000", 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.805188+0000", 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4800000000000001e-06 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816847+0000", 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011659119000000001 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817576+0000", 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00072975000000000004 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817601+0000", 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4450000000000001e-05 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.817741+0000", 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014058500000000001 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818074+0000", 2024-04-25T15:54:38.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00033291000000000001 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.818385+0000", 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031034999999999998 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870618+0000", 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052233743999999999 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964814+0000", 2024-04-25T15:54:38.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094195216999999998 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964945+0000", 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013086799999999999 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965021+0000", 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6032000000000007e-05 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.861 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2275 3.12 3:492b2588:::100000001f8.000004df:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 50.009081223999999, 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14794684399999999, 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2275 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816859+0000", 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.816863+0000", 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4149999999999999e-06 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829311+0000", 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012448222 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829569+0000", 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025766800000000001 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829598+0000", 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.968e-05 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829786+0000", 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187746 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830129+0000", 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034284099999999998 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.830479+0000", 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034961500000000001 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.867146+0000", 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036667733000000001 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964688+0000", 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.097541577000000004 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964748+0000", 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9626000000000001e-05 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964806+0000", 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8721000000000002e-05 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:2278 3.2b 3:d79b9024:::100000001f8.000004e2:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.996615966, 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135647779, 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.867 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 2278 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829325+0000", 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.829328+0000", 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7309999999999999e-06 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.843821+0000", 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014493012 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844787+0000", 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096513199999999995 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844876+0000", 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9147999999999994e-05 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.844972+0000", 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6566999999999999e-05 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845195+0000", 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000222699 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0", 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.845422+0000", 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022700699999999999 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.870319+0000", 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.024897319000000001 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964892+0000", 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.094572233000000006 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964938+0000", 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6646e-05 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.964972+0000", 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4285000000000001e-05 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:2305 3.2f e90/51 3:f6648653:::100000001f8.000004fd:head v 90'33, mlcod=90'30)", 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 49.988682175999998, 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12780160600000001, 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837258+0000", 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.837261+0000", 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1679999999999998e-06 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839587+0000", 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00232607 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839620+0000", 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3503000000000001e-05 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839642+0000", 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1302e-05 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839754+0000", 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112454 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.839798+0000", 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4240999999999999e-05 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965003+0000", 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12520515600000001 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:53:48.965060+0000", 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6712000000000002e-05 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.876 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3917 3.27 3:e652e8b8:::100000001f8.00000149:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.327871672000001, 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.16896387500000001, 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3917 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498069+0000", 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.498071+0000", 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4779999999999998e-06 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509296+0000", 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011224433000000001 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509572+0000", 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027576500000000003 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509591+0000", 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9466e-05 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509677+0000", 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5607999999999998e-05 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509773+0000", 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6136000000000002e-05 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509942+0000", 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016951999999999999 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.610111+0000", 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.100168231 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666922+0000", 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056811077000000001 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666991+0000", 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9758000000000005e-05 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667033+0000", 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1403000000000003e-05 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3918 3.11 3:8b260724:::100000001f8.0000014a:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.882 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.316635458999997, 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15748047100000001, 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3918 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:38.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509305+0000", 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.509307+0000", 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.12e-06 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515132+0000", 2024-04-25T15:54:38.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058250860000000002 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515164+0000", 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1563000000000003e-05 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515179+0000", 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5102e-05 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515262+0000", 2024-04-25T15:54:38.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2917999999999995e-05 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515360+0000", 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7532000000000001e-05 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.515512+0000", 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000152442 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.654409+0000", 2024-04-25T15:54:38.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13889727499999999 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666644+0000", 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012234584999999999 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666730+0000", 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5793999999999999e-05 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666786+0000", 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6054000000000002e-05 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3922 3.d 3:b3f0d00e:::100000001f8.0000014e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.307273537999997, 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.150212927, 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.888 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3922 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518667+0000", 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.518670+0000", 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1039999999999998e-06 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529034+0000", 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010364237 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529073+0000", 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8407e-05 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529092+0000", 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9411000000000001e-05 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529203+0000", 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011113 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529330+0000", 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012611499999999999 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.529517+0000", 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000187639 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.667032+0000", 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13751481300000001 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668779+0000", 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017469740000000001 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668835+0000", 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6251000000000001e-05 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668880+0000", 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4845999999999997e-05 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:3920 3.26 e97/18 3:6422fb70:::100000001f8.0000014c:head v 97'93, mlcod=96'91)", 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.302250389999998, 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14317563999999999, 2024-04-25T15:54:38.893 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523690+0000", 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.523692+0000", 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8050000000000001e-06 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525897+0000", 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022045290000000002 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525923+0000", 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6225e-05 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.525941+0000", 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8304e-05 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526045+0000", 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104158 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.526076+0000", 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0454e-05 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666788+0000", 2024-04-25T15:54:38.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140712106 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.666866+0000", 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8059000000000006e-05 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3926 3.d 3:b20dc027:::100000001f8.00000152:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.287763843, 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131821096, 2024-04-25T15:54:38.897 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3926 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538177+0000", 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.538180+0000", 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4690000000000002e-06 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542940+0000", 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0047600250000000002 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.542988+0000", 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7633999999999998e-05 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543006+0000", 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8114999999999999e-05 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543103+0000", 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6723000000000002e-05 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543283+0000", 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018030100000000001 2024-04-25T15:54:38.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.543476+0000", 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019295200000000001 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.668734+0000", 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12525763200000001 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669901+0000", 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011670890000000001 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669955+0000", 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4373999999999998e-05 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.669998+0000", 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2781999999999998e-05 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:3971 3.3f 3:febcb5ea:::100000001f8.0000017f:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-04-25T15:54:38.902 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 35.008240086999997, 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15452237999999999, 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 3971 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817701+0000", 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.817703+0000", 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8700000000000001e-06 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822518+0000", 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048147939999999998 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822563+0000", 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4515000000000003e-05 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822577+0000", 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.432e-05 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822664+0000", 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7440000000000003e-05 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822786+0000", 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012110800000000001 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 6", 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.822960+0000", 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017468900000000001 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.945290+0000", 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122329907 2024-04-25T15:54:38.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972057+0000", 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.026767102000000001 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972139+0000", 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2070000000000005e-05 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:03.972223+0000", 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3565000000000004e-05 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.907 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.4815.0:5059 3.26 e99/18 3:65b3bea5:::100000001f8.000005bf:head v 99'112, mlcod=99'109)", 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 28.808550916000002, 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.131194432, 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:38.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017390+0000", 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.017392+0000", 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4650000000000001e-06 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021054+0000", 2024-04-25T15:54:38.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036613240000000001 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021076+0000", 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2067000000000001e-05 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021092+0000", 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6209999999999999e-05 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021178+0000", 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6260000000000004e-05 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.021208+0000", 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0046999999999999e-05 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148491+0000", 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12728336500000001 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.148584+0000", 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2694000000000004e-05 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5063 3.13 3:c9096c28:::100000001f8.000005c3:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 28.796998760000001, 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13114490600000001, 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.912 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5063 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028942+0000", 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.028946+0000", 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0520000000000003e-06 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035431+0000", 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0064851509999999998 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035461+0000", 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9547999999999999e-05 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035474+0000", 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2880999999999999e-05 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035550+0000", 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6027999999999998e-05 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035622+0000", 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2366999999999993e-05 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.035727+0000", 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105561 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.150510+0000", 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11478287099999999 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159903+0000", 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0093925880000000003 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.159983+0000", 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0122000000000003e-05 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:10.160087+0000", 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000103737 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5730 3.e 3:7357dd18:::100000001f8.0000085e:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 25.066702541000002, 2024-04-25T15:54:38.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14322902300000001, 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5730 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759238+0000", 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.759240+0000", 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3910000000000001e-06 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765109+0000", 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058681660000000002 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765139+0000", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9981e-05 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765149+0000", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0125e-05 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765215+0000", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6700999999999997e-05 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765293+0000", 2024-04-25T15:54:38.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7883e-05 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.765394+0000", 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100869 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.893428+0000", 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128033758 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.921 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902331+0000", 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089030610000000003 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902403+0000", 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1891999999999995e-05 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.902467+0000", 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4196000000000007e-05 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5769 3.d 3:b350774d:::100000001f8.00000885:head [write 0~4194304] snapc 1=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.833984795999999, 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12752495899999999, 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5769 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991956+0000", 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.991958+0000", 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998738+0000", 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067804069999999996 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998782+0000", 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3738e-05 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998800+0000", 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7907e-05 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.998944+0000", 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000144089 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999140+0000", 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000195465 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5", 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:13.999415+0000", 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027557100000000001 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.070872+0000", 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071456602999999994 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119350+0000", 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.048478354000000001 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119421+0000", 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0970999999999995e-05 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.927 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.119481+0000", 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9802000000000001e-05 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.4815.0:5780 3.f 3:f20b812e:::100000001f8.00000890:head [write 0~4194304] snapc 1=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.785803718, 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13282865699999999, 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "continuous": false, 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-04-25T15:54:38.928 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.4815", 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "192.168.0.1:0/2848005286", 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 5780 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040137+0000", 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-04-25T15:54:38.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.040139+0000", 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9570000000000001e-06 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047681+0000", 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075419279999999998 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047720+0000", 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8794999999999997e-05 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-04-25T15:54:38.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047740+0000", 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0091000000000001e-05 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047828+0000", 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8745000000000005e-05 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.047922+0000", 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3076000000000001e-05 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 4", 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.048095+0000", 2024-04-25T15:54:38.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000173134 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.170485+0000", 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.122390033 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172900+0000", 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002415482 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172933+0000", 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3021000000000001e-05 2024-04-25T15:54:38.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-04-25T15:54:14.172966+0000", 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2394999999999997e-05 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-25T15:54:38.933 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-25T15:54:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.125+0000 7fa25ffff640 1 -- 172.21.15.67:0/1574570753 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v114) ==== 133+0+0 (secure 0 0 0) 0x7fa26805bb10 con 0x7fa27815e180 2024-04-25T15:54:39.127 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-25T15:54:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa26406f430 msgr2=0x7fa264071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa26406f430 0x7fa264071940 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa26c001670 tx=0x7fa26c009040 comp rx=0 tx=0).stop 2024-04-25T15:54:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27815e180 msgr2=0x7fa278072c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27815e180 0x7fa278072c80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa26800b410 tx=0x7fa26800b8d0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 shutdown_connections 2024-04-25T15:54:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fa26406f430 0x7fa264071940 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa278160c20 0x7fa278156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa27815e180 0x7fa278072c80 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 --2- 172.21.15.67:0/1574570753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa27806c640 0x7fa2781fb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 >> 172.21.15.67:0/1574570753 conn(0x7fa27806fe70 msgr2=0x7fa2780cabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 shutdown_connections 2024-04-25T15:54:39.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.129+0000 7fa27de02640 1 -- 172.21.15.67:0/1574570753 wait complete. 2024-04-25T15:54:39.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-25T15:54:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d5d2b1640 1 Processor -- start 2024-04-25T15:54:39.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d5d2b1640 1 -- start start 2024-04-25T15:54:39.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d5d2b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 0x7f8d580c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d5d2b1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d580749d0 con 0x7f8d580c8500 2024-04-25T15:54:39.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d5d2b1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d5806e510 con 0x7f8d5806c640 2024-04-25T15:54:39.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d5d2b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d5815ebb0 con 0x7f8d58154a60 2024-04-25T15:54:39.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8d580c8500 0x7f8d58164f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50064/0 (socket says 172.21.15.67:50064) 2024-04-25T15:54:39.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 -- 172.21.15.67:0/4184148620 learned_addr learned my addr 172.21.15.67:0/4184148620 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d567fc640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 0x7f8d580c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2265568110 0 0) 0x7f8d580749d0 con 0x7f8d580c8500 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d40003530 con 0x7f8d580c8500 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 491564194 0 0) 0x7f8d40003530 con 0x7f8d580c8500 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 msgr2=0x7f8d580c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 0x7f8d580c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 >> v1:172.21.15.176:6789/0 conn(0x7f8d5806c640 legacy=0x7f8d5815e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5815dca0 con 0x7f8d580c8500 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8d50001a00 con 0x7f8d580c8500 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 >> v1:172.21.15.67:6789/0 conn(0x7f8d580c8500 legacy=0x7f8d58164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 0x7f8d400039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d40003f70 0x7f8d40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d40004d50 0x7f8d40005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5815ebb0 con 0x7f8d40004d50 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4000bb60 con 0x7f8d40003f70 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d4000bd60 con 0x7f8d58154a60 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d40004d50 0x7f8d40005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d567fc640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d40003f70 0x7f8d40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d56ffd640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 0x7f8d400039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 -- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 msgr2=0x7f8d400039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58154a60 0x7f8d400039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 -- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d40003f70 msgr2=0x7f8d40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d40003f70 0x7f8d40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d577fe640 1 -- 172.21.15.67:0/4184148620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d40003530 con 0x7f8d40004d50 2024-04-25T15:54:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.325+0000 7f8d567fc640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d40003f70 0x7f8d40004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:39.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d577fe640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d40004d50 0x7f8d40005160 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8d58074900 tx=0x7f8d5001d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb0adcfaa07826f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000a480 con 0x7f8d40004d50 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d500ad630 con 0x7f8d40004d50 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d55ffb640 1 -- 172.21.15.67:0/4184148620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d500ada60 con 0x7f8d40004d50 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/4184148620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d40004d50 msgr2=0x7f8d40005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d5d2b1640 1 --2- 172.21.15.67:0/4184148620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d40004d50 0x7f8d40005160 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8d58074900 tx=0x7f8d5001d320 comp rx=0 tx=0).stop 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d577fe640 1 -- 172.21.15.67:0/4184148620 reap_dead start 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/4184148620 shutdown_connections 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/4184148620 >> 172.21.15.67:0/4184148620 conn(0x7f8d58070db0 msgr2=0x7f8d580ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/4184148620 shutdown_connections 2024-04-25T15:54:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.329+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/4184148620 wait complete. 2024-04-25T15:54:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 Processor -- start 2024-04-25T15:54:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 -- start start 2024-04-25T15:54:39.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58072c50 0x7f8d581f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d580cda80 0x7f8d581451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d580cde90 0x7f8d58145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5806e7f0 con 0x7f8d580cde90 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5806e210 con 0x7f8d580cda80 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d580c92d0 con 0x7f8d58072c50 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d580cde90 0x7f8d58145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d580cde90 0x7f8d58145720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46160/0 (socket says 172.21.15.67:46160) 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 -- 172.21.15.67:0/2805907514 learned_addr learned my addr 172.21.15.67:0/2805907514 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d56ffd640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58072c50 0x7f8d581f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d567fc640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d580cda80 0x7f8d581451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 -- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58072c50 msgr2=0x7f8d581f12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58072c50 0x7f8d581f12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 -- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d580cda80 msgr2=0x7f8d581451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:39.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d580cda80 0x7f8d581451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 -- 172.21.15.67:0/2805907514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d581500f0 con 0x7f8d580cde90 2024-04-25T15:54:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d577fe640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d580cde90 0x7f8d58145720 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8d5001d150 tx=0x7f8d40005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d3f7fe640 1 -- 172.21.15.67:0/2805907514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000a480 con 0x7f8d580cde90 2024-04-25T15:54:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d3f7fe640 1 -- 172.21.15.67:0/2805907514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d500ad430 con 0x7f8d580cde90 2024-04-25T15:54:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d58150390 con 0x7f8d580cde90 2024-04-25T15:54:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d3f7fe640 1 -- 172.21.15.67:0/2805907514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d500011d0 con 0x7f8d580cde90 2024-04-25T15:54:39.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.333+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d58150950 con 0x7f8d580cde90 2024-04-25T15:54:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.337+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d5806a480 con 0x7f8d580cde90 2024-04-25T15:54:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.337+0000 7f8d3f7fe640 1 -- 172.21.15.67:0/2805907514 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8d500bd030 con 0x7f8d580cde90 2024-04-25T15:54:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.337+0000 7f8d3f7fe640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d4406f430 0x7f8d44071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.337+0000 7f8d3f7fe640 1 -- 172.21.15.67:0/2805907514 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6031+0+0 (secure 0 0 0) 0x7f8d50129c40 con 0x7f8d580cde90 2024-04-25T15:54:39.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.337+0000 7f8d56ffd640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d4406f430 0x7f8d44071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:39.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.337+0000 7f8d56ffd640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d4406f430 0x7f8d44071940 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8d581e3af0 tx=0x7f8d48009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:39.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.341+0000 7f8d3f7fe640 1 -- 172.21.15.67:0/2805907514 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8d500c4020 con 0x7f8d580cde90 2024-04-25T15:54:39.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:39.517+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f8d580cb070 con 0x7f8d580cde90 2024-04-25T15:54:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.137+0000 7f8d3f7fe640 1 -- 172.21.15.67:0/2805907514 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v115) ==== 113+0+0 (secure 0 0 0) 0x7f8d50017030 con 0x7f8d580cde90 2024-04-25T15:54:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-04-25T15:54:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d4406f430 msgr2=0x7f8d44071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d4406f430 0x7f8d44071940 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8d581e3af0 tx=0x7f8d48009040 comp rx=0 tx=0).stop 2024-04-25T15:54:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d580cde90 msgr2=0x7f8d58145720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d580cde90 0x7f8d58145720 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8d5001d150 tx=0x7f8d40005840 comp rx=0 tx=0).stop 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 shutdown_connections 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8d4406f430 0x7f8d44071940 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d580cde90 0x7f8d58145720 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d580cda80 0x7f8d581451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 --2- 172.21.15.67:0/2805907514 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d58072c50 0x7f8d581f12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 >> 172.21.15.67:0/2805907514 conn(0x7f8d58070db0 msgr2=0x7f8d58147480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 shutdown_connections 2024-04-25T15:54:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.141+0000 7f8d5d2b1640 1 -- 172.21.15.67:0/2805907514 wait complete. 2024-04-25T15:54:40.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-25T15:54:40.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230f694640 1 Processor -- start 2024-04-25T15:54:40.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230f694640 1 -- start start 2024-04-25T15:54:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230f694640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2308154a60 0x7f23080c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230f694640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23080749d0 con 0x7f23080c8500 2024-04-25T15:54:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230f694640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f230806e510 con 0x7f230806c640 2024-04-25T15:54:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230f694640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f230815ebb0 con 0x7f2308154a60 2024-04-25T15:54:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230cc2f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2308154a60 0x7f23080c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230dc31640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f23080c8500 0x7f2308164f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50080/0 (socket says 172.21.15.67:50080) 2024-04-25T15:54:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230dc31640 1 -- 172.21.15.67:0/936736100 learned_addr learned my addr 172.21.15.67:0/936736100 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f22fffff640 1 -- 172.21.15.67:0/936736100 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2226034678 0 0) 0x7f23080749d0 con 0x7f23080c8500 2024-04-25T15:54:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f22fffff640 1 -- 172.21.15.67:0/936736100 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22f4003530 con 0x7f23080c8500 2024-04-25T15:54:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230cc2f640 1 -- 172.21.15.67:0/936736100 >> v1:172.21.15.176:6789/0 conn(0x7f230806c640 legacy=0x7f230815e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230cc2f640 1 -- 172.21.15.67:0/936736100 >> v1:172.21.15.67:6789/0 conn(0x7f23080c8500 legacy=0x7f2308164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.345+0000 7f230cc2f640 1 -- 172.21.15.67:0/936736100 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230815dca0 con 0x7f2308154a60 2024-04-25T15:54:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230cc2f640 1 --2- 172.21.15.67:0/936736100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f23080c7fc0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f22f80097c0 tx=0x7f22f801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d9b19478ccf979 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f22fffff640 1 -- 172.21.15.67:0/936736100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80b0070 con 0x7f2308154a60 2024-04-25T15:54:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f22fffff640 1 -- 172.21.15.67:0/936736100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f80ab600 con 0x7f2308154a60 2024-04-25T15:54:40.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f22fffff640 1 -- 172.21.15.67:0/936736100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80ab9b0 con 0x7f2308154a60 2024-04-25T15:54:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230f694640 1 -- 172.21.15.67:0/936736100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 msgr2=0x7f23080c7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230f694640 1 --2- 172.21.15.67:0/936736100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f23080c7fc0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f22f80097c0 tx=0x7f22f801c4a0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230f694640 1 -- 172.21.15.67:0/936736100 shutdown_connections 2024-04-25T15:54:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230f694640 1 --2- 172.21.15.67:0/936736100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f23080c7fc0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230f694640 1 -- 172.21.15.67:0/936736100 >> 172.21.15.67:0/936736100 conn(0x7f2308070db0 msgr2=0x7f23080ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:40.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230f694640 1 -- 172.21.15.67:0/936736100 shutdown_connections 2024-04-25T15:54:40.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.349+0000 7f230f694640 1 -- 172.21.15.67:0/936736100 wait complete. 2024-04-25T15:54:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 Processor -- start 2024-04-25T15:54:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 -- start start 2024-04-25T15:54:40.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230806c640 0x7f2308157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23080c8500 0x7f230815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f2308155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f23080cad60 con 0x7f2308154a60 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f23080c92d0 con 0x7f230806c640 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f230815ba90 con 0x7f23080c8500 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f2308155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f2308155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46188/0 (socket says 172.21.15.67:46188) 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 -- 172.21.15.67:0/548199757 learned_addr learned my addr 172.21.15.67:0/548199757 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230d430640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230806c640 0x7f2308157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230cc2f640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23080c8500 0x7f230815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 -- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23080c8500 msgr2=0x7f230815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:40.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23080c8500 0x7f230815ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 -- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230806c640 msgr2=0x7f2308157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230806c640 0x7f2308157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 -- 172.21.15.67:0/548199757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23081553a0 con 0x7f2308154a60 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230dc31640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f2308155a20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f230000cb00 tx=0x7f230000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f22fdffb640 1 -- 172.21.15.67:0/548199757 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300017070 con 0x7f2308154a60 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f22fdffb640 1 -- 172.21.15.67:0/548199757 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230000abe0 con 0x7f2308154a60 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f22fdffb640 1 -- 172.21.15.67:0/548199757 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300012400 con 0x7f2308154a60 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2308158020 con 0x7f2308154a60 2024-04-25T15:54:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.353+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2308152ce0 con 0x7f2308154a60 2024-04-25T15:54:40.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.357+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f230806cbe0 con 0x7f2308154a60 2024-04-25T15:54:40.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.357+0000 7f22fdffb640 1 -- 172.21.15.67:0/548199757 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f230001a070 con 0x7f2308154a60 2024-04-25T15:54:40.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.357+0000 7f22fdffb640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22ec06f400 0x7f22ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:40.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.357+0000 7f22fdffb640 1 -- 172.21.15.67:0/548199757 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 6031+0+0 (secure 0 0 0) 0x7f230008f420 con 0x7f2308154a60 2024-04-25T15:54:40.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.357+0000 7f230d430640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22ec06f400 0x7f22ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.357+0000 7f230d430640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22ec06f400 0x7f22ec071910 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2304000f70 tx=0x7f2304009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.361+0000 7f22fdffb640 1 -- 172.21.15.67:0/548199757 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2300021020 con 0x7f2308154a60 2024-04-25T15:54:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:40.537+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7f230806a480 con 0x7f2308154a60 2024-04-25T15:54:41.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.141+0000 7f22fdffb640 1 -- 172.21.15.67:0/548199757 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v116) ==== 133+0+0 (secure 0 0 0) 0x7f23081553a0 con 0x7f2308154a60 2024-04-25T15:54:41.143 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22ec06f400 msgr2=0x7f22ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22ec06f400 0x7f22ec071910 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2304000f70 tx=0x7f2304009040 comp rx=0 tx=0).stop 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 msgr2=0x7f2308155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f2308155a20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f230000cb00 tx=0x7f230000a060 comp rx=0 tx=0).stop 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 shutdown_connections 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f22ec06f400 0x7f22ec071910 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2308154a60 0x7f2308155a20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23080c8500 0x7f230815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 --2- 172.21.15.67:0/548199757 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230806c640 0x7f2308157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.145+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 >> 172.21.15.67:0/548199757 conn(0x7f2308070db0 msgr2=0x7f2308153100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.149+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 shutdown_connections 2024-04-25T15:54:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.149+0000 7f230f694640 1 -- 172.21.15.67:0/548199757 wait complete. 2024-04-25T15:54:41.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-25T15:54:41.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4934536640 1 Processor -- start 2024-04-25T15:54:41.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4934536640 1 -- start start 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4934536640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f492c074790 0x7f492c15dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4934536640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f492c0c6fb0 0x7f492c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4934536640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f492c072dc0 con 0x7f492c15e020 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4934536640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f492c15b430 con 0x7f492c074790 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4934536640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f492c15a670 con 0x7f492c0c6fb0 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f49322d2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f492c074790 0x7f492c15dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4931ad1640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f492c15e020 0x7f492c0723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50086/0 (socket says 172.21.15.67:50086) 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4931ad1640 1 -- 172.21.15.67:0/3966340628 learned_addr learned my addr 172.21.15.67:0/3966340628 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f4932ad3640 1 --2- 172.21.15.67:0/3966340628 >> v2:172.21.15.176:3300/0 conn(0x7f492c0c6fb0 0x7f492c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f49322d2640 1 -- 172.21.15.67:0/3966340628 >> v2:172.21.15.176:3300/0 conn(0x7f492c0c6fb0 msgr2=0x7f492c0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f49322d2640 1 --2- 172.21.15.67:0/3966340628 >> v2:172.21.15.176:3300/0 conn(0x7f492c0c6fb0 0x7f492c0c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f49322d2640 1 -- 172.21.15.67:0/3966340628 >> v1:172.21.15.67:6789/0 conn(0x7f492c15e020 legacy=0x7f492c0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:41.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.345+0000 7f49322d2640 1 -- 172.21.15.67:0/3966340628 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c15a950 con 0x7f492c074790 2024-04-25T15:54:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f49322d2640 1 --2- 172.21.15.67:0/3966340628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c074790 0x7f492c15dae0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f492c0cb730 tx=0x7f492801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f6939ff2d4f629 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f49312d0640 1 -- 172.21.15.67:0/3966340628 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49280b5070 con 0x7f492c074790 2024-04-25T15:54:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f49312d0640 1 -- 172.21.15.67:0/3966340628 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49280a8970 con 0x7f492c074790 2024-04-25T15:54:41.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f49312d0640 1 -- 172.21.15.67:0/3966340628 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49280a8d40 con 0x7f492c074790 2024-04-25T15:54:41.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 -- 172.21.15.67:0/3966340628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c074790 msgr2=0x7f492c15dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:41.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 --2- 172.21.15.67:0/3966340628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c074790 0x7f492c15dae0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f492c0cb730 tx=0x7f492801ec70 comp rx=0 tx=0).stop 2024-04-25T15:54:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 -- 172.21.15.67:0/3966340628 shutdown_connections 2024-04-25T15:54:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 --2- 172.21.15.67:0/3966340628 >> v2:172.21.15.176:3300/0 conn(0x7f492c0c6fb0 0x7f492c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 --2- 172.21.15.67:0/3966340628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c074790 0x7f492c15dae0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 -- 172.21.15.67:0/3966340628 >> 172.21.15.67:0/3966340628 conn(0x7f492c06fe70 msgr2=0x7f492c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 -- 172.21.15.67:0/3966340628 shutdown_connections 2024-04-25T15:54:41.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 -- 172.21.15.67:0/3966340628 wait complete. 2024-04-25T15:54:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.349+0000 7f4934536640 1 Processor -- start 2024-04-25T15:54:41.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 -- start start 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f492c074790 0x7f492c1e34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f492c0c6fb0 0x7f492c1e5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c15e020 0x7f492c149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f492c073080 con 0x7f492c15e020 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f492c15af10 con 0x7f492c0c6fb0 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f492c072ab0 con 0x7f492c074790 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c15e020 0x7f492c149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c15e020 0x7f492c149d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46202/0 (socket says 172.21.15.67:46202) 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 -- 172.21.15.67:0/1859620286 learned_addr learned my addr 172.21.15.67:0/1859620286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4931ad1640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f492c0c6fb0 0x7f492c1e5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f49322d2640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f492c074790 0x7f492c1e34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 -- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f492c074790 msgr2=0x7f492c1e34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f492c074790 0x7f492c1e34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 -- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f492c0c6fb0 msgr2=0x7f492c1e5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f492c0c6fb0 0x7f492c1e5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 -- 172.21.15.67:0/1859620286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c1f6ba0 con 0x7f492c15e020 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4932ad3640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c15e020 0x7f492c149d50 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4924007e90 tx=0x7f492400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4922ffd640 1 -- 172.21.15.67:0/1859620286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4924014420 con 0x7f492c15e020 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4922ffd640 1 -- 172.21.15.67:0/1859620286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4924019070 con 0x7f492c15e020 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492c1e3a00 con 0x7f492c15e020 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4922ffd640 1 -- 172.21.15.67:0/1859620286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4924014800 con 0x7f492c15e020 2024-04-25T15:54:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492c1e4df0 con 0x7f492c15e020 2024-04-25T15:54:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.353+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f492c06a480 con 0x7f492c15e020 2024-04-25T15:54:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.357+0000 7f4922ffd640 1 -- 172.21.15.67:0/1859620286 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f49240149a0 con 0x7f492c15e020 2024-04-25T15:54:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.357+0000 7f4922ffd640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f491006f400 0x7f4910071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.357+0000 7f4922ffd640 1 -- 172.21.15.67:0/1859620286 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 6264+0+0 (secure 0 0 0) 0x7f4924090800 con 0x7f492c15e020 2024-04-25T15:54:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.357+0000 7f49322d2640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f491006f400 0x7f4910071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:41.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.357+0000 7f49322d2640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f491006f400 0x7f4910071910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f49280b2000 tx=0x7f4928008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.361+0000 7f4922ffd640 1 -- 172.21.15.67:0/1859620286 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f492405d5a0 con 0x7f492c15e020 2024-04-25T15:54:41.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:41.537+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7f492c0ca010 con 0x7f492c15e020 2024-04-25T15:54:42.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.145+0000 7f4922ffd640 1 -- 172.21.15.67:0/1859620286 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v117) ==== 113+0+0 (secure 0 0 0) 0x7f492405cea0 con 0x7f492c15e020 2024-04-25T15:54:42.146 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-04-25T15:54:42.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f491006f400 msgr2=0x7f4910071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:42.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f491006f400 0x7f4910071910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f49280b2000 tx=0x7f4928008040 comp rx=0 tx=0).stop 2024-04-25T15:54:42.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c15e020 msgr2=0x7f492c149d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:42.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c15e020 0x7f492c149d50 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4924007e90 tx=0x7f492400e4b0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 shutdown_connections 2024-04-25T15:54:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f491006f400 0x7f4910071910 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f492c15e020 0x7f492c149d50 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f492c0c6fb0 0x7f492c1e5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 --2- 172.21.15.67:0/1859620286 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f492c074790 0x7f492c1e34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 >> 172.21.15.67:0/1859620286 conn(0x7f492c06fe70 msgr2=0x7f492c14e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 shutdown_connections 2024-04-25T15:54:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.149+0000 7f4934536640 1 -- 172.21.15.67:0/1859620286 wait complete. 2024-04-25T15:54:42.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-25T15:54:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.361+0000 7f1bc03d1640 1 Processor -- start 2024-04-25T15:54:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.361+0000 7f1bc03d1640 1 -- start start 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bc03d1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 0x7f1bb80c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bc03d1640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb80c3bc0 con 0x7f1bb80725c0 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bc03d1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb80c53a0 con 0x7f1bb815a0b0 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bc03d1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb8153e10 con 0x7f1bb8153a00 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1bb80725c0 0x7f1bb8073f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50088/0 (socket says 172.21.15.67:50088) 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 -- 172.21.15.67:0/4056002813 learned_addr learned my addr 172.21.15.67:0/4056002813 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbe16d640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 0x7f1bb80c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4235399386 0 0) 0x7f1bb80c3bc0 con 0x7f1bb80725c0 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ba4003530 con 0x7f1bb80725c0 2024-04-25T15:54:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1778896933 0 0) 0x7f1ba4003530 con 0x7f1bb80725c0 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 msgr2=0x7f1bb80c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 0x7f1bb80c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 >> v1:172.21.15.176:6789/0 conn(0x7f1bb815a0b0 legacy=0x7f1bb80746a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb80686c0 con 0x7f1bb80725c0 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f1ba8001800 con 0x7f1bb80725c0 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 >> v1:172.21.15.67:6789/0 conn(0x7f1bb80725c0 legacy=0x7f1bb8073f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 0x7f1ba40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1ba4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4004d50 0x7f1ba4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba400bb60 con 0x7f1ba4003f50 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba400bd10 con 0x7f1ba4004d50 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba400bf10 con 0x7f1bb8153a00 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1ba4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbe16d640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 0x7f1ba40039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbe96e640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4004d50 0x7f1ba4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 -- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 msgr2=0x7f1ba40039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb8153a00 0x7f1ba40039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 -- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4004d50 msgr2=0x7f1ba4005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1ba4004d50 0x7f1ba4005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.365+0000 7f1bbd96c640 1 -- 172.21.15.67:0/4056002813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba4003530 con 0x7f1ba4003f50 2024-04-25T15:54:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1ba4004810 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8007cf0 tx=0x7f1ba80210c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c05ea89c0f6158 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8007d30 con 0x7f1ba4003f50 2024-04-25T15:54:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba8007270 con 0x7f1ba4003f50 2024-04-25T15:54:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bbd16b640 1 -- 172.21.15.67:0/4056002813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80057b0 con 0x7f1ba4003f50 2024-04-25T15:54:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bc03d1640 1 -- 172.21.15.67:0/4056002813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 msgr2=0x7f1ba4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bc03d1640 1 --2- 172.21.15.67:0/4056002813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1ba4004810 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8007cf0 tx=0x7f1ba80210c0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bbe96e640 1 -- 172.21.15.67:0/4056002813 reap_dead start 2024-04-25T15:54:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bc03d1640 1 -- 172.21.15.67:0/4056002813 shutdown_connections 2024-04-25T15:54:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bc03d1640 1 -- 172.21.15.67:0/4056002813 >> 172.21.15.67:0/4056002813 conn(0x7f1bb80ce1c0 msgr2=0x7f1bb80ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bc03d1640 1 -- 172.21.15.67:0/4056002813 shutdown_connections 2024-04-25T15:54:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.369+0000 7f1bc03d1640 1 -- 172.21.15.67:0/4056002813 wait complete. 2024-04-25T15:54:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 Processor -- start 2024-04-25T15:54:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 -- start start 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1bb814d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bb81f2050 0x7f1bb80cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb814e0f0 0x7f1bb814fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80c3600 con 0x7f1ba4003f50 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb8068c40 con 0x7f1bb81f2050 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bc03d1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb80c38e0 con 0x7f1bb814e0f0 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1bb814d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1bb814d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46214/0 (socket says 172.21.15.67:46214) 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 -- 172.21.15.67:0/635619788 learned_addr learned my addr 172.21.15.67:0/635619788 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bb81f2050 0x7f1bb80cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe96e640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb814e0f0 0x7f1bb814fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 -- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb814e0f0 msgr2=0x7f1bb814fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb814e0f0 0x7f1bb814fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 -- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bb81f2050 msgr2=0x7f1bb80cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bb81f2050 0x7f1bb80cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.373+0000 7f1bbe16d640 1 -- 172.21.15.67:0/635619788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb814d5d0 con 0x7f1ba4003f50 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bb81f2050 0x7f1bb80cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1bbe96e640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb814e0f0 0x7f1bb814fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1bbe16d640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1bb814d090 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4011610 tx=0x7f1bb4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1baeffd640 1 -- 172.21.15.67:0/635619788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb400f8f0 con 0x7f1ba4003f50 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1baeffd640 1 -- 172.21.15.67:0/635619788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb400fa90 con 0x7f1ba4003f50 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb80cd320 con 0x7f1ba4003f50 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1baeffd640 1 -- 172.21.15.67:0/635619788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb400fe40 con 0x7f1ba4003f50 2024-04-25T15:54:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb815d040 con 0x7f1ba4003f50 2024-04-25T15:54:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb806a480 con 0x7f1ba4003f50 2024-04-25T15:54:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1baeffd640 1 -- 172.21.15.67:0/635619788 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1bb4015460 con 0x7f1ba4003f50 2024-04-25T15:54:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1baeffd640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9c06f400 0x7f1b9c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1baeffd640 1 -- 172.21.15.67:0/635619788 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 6264+0+0 (secure 0 0 0) 0x7f1bb408fe60 con 0x7f1ba4003f50 2024-04-25T15:54:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.377+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9c06f400 0x7f1b9c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.381+0000 7f1bbd96c640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9c06f400 0x7f1b9c071910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ba80b0000 tx=0x7f1ba80ab000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.385+0000 7f1baeffd640 1 -- 172.21.15.67:0/635619788 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1bb4019030 con 0x7f1ba4003f50 2024-04-25T15:54:42.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:42.633+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) -- 0x7f1bb80cc150 con 0x7f1ba4003f50 2024-04-25T15:54:43.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.185+0000 7f1baeffd640 1 -- 172.21.15.67:0/635619788 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v118) ==== 133+0+0 (secure 0 0 0) 0x7f1bb405cbb0 con 0x7f1ba4003f50 2024-04-25T15:54:43.187 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-25T15:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9c06f400 msgr2=0x7f1b9c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9c06f400 0x7f1b9c071910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ba80b0000 tx=0x7f1ba80ab000 comp rx=0 tx=0).stop 2024-04-25T15:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 msgr2=0x7f1bb814d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1bb814d090 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4011610 tx=0x7f1bb4011ad0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 shutdown_connections 2024-04-25T15:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1b9c06f400 0x7f1b9c071910 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bb814e0f0 0x7f1bb814fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bb81f2050 0x7f1bb80cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 --2- 172.21.15.67:0/635619788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ba4003f50 0x7f1bb814d090 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 >> 172.21.15.67:0/635619788 conn(0x7f1bb80ce1c0 msgr2=0x7f1bb80c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 shutdown_connections 2024-04-25T15:54:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.193+0000 7f1bc03d1640 1 -- 172.21.15.67:0/635619788 wait complete. 2024-04-25T15:54:43.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-25T15:54:43.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1187a1c640 1 Processor -- start 2024-04-25T15:54:43.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1187a1c640 1 -- start start 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1187a1c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f11800736d0 0x7f118006c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1187a1c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11800caa60 con 0x7f118006cb80 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1187a1c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f118015a5f0 con 0x7f118006d460 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1187a1c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1180068590 con 0x7f11800736d0 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f118006cb80 0x7f118015aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60268/0 (socket says 172.21.15.67:60268) 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 -- 172.21.15.67:0/4145025290 learned_addr learned my addr 172.21.15.67:0/4145025290 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1184fb7640 1 --2- 172.21.15.67:0/4145025290 >> v2:172.21.15.176:3300/0 conn(0x7f11800736d0 0x7f118006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090194062 0 0) 0x7f11800caa60 con 0x7f118006cb80 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f116c003530 con 0x7f118006cb80 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3090943220 0 0) 0x7f116c003530 con 0x7f118006cb80 2024-04-25T15:54:43.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 >> v2:172.21.15.176:3300/0 conn(0x7f11800736d0 msgr2=0x7f118006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 --2- 172.21.15.67:0/4145025290 >> v2:172.21.15.176:3300/0 conn(0x7f11800736d0 0x7f118006c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 >> v1:172.21.15.176:6789/0 conn(0x7f118006d460 legacy=0x7f11801547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11800ca100 con 0x7f118006cb80 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f1178001a00 con 0x7f118006cb80 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 >> v1:172.21.15.67:6789/0 conn(0x7f118006cb80 legacy=0x7f118015aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f116c0039c0 0x7f116c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c004810 0x7f116c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0051d0 0x7f116c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1180068590 con 0x7f116c0051d0 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f116c00c170 con 0x7f116c004810 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f116c00c370 con 0x7f116c0039c0 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0051d0 0x7f116c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f11857b8640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f116c0039c0 0x7f116c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1184fb7640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c004810 0x7f116c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 -- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f116c0039c0 msgr2=0x7f116c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:43.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f116c0039c0 0x7f116c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 -- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c004810 msgr2=0x7f116c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:43.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c004810 0x7f116c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.381+0000 7f1185fb9640 1 -- 172.21.15.67:0/4145025290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116c003530 con 0x7f116c0051d0 2024-04-25T15:54:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1185fb9640 1 -- 172.21.15.67:0/4145025290 reap_dead start 2024-04-25T15:54:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1185fb9640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0051d0 0x7f116c00b9d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f117800a830 tx=0x7f117801d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748351f760e917cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117800a480 con 0x7f116c0051d0 2024-04-25T15:54:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11780ad630 con 0x7f116c0051d0 2024-04-25T15:54:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1173fff640 1 -- 172.21.15.67:0/4145025290 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11780ada20 con 0x7f116c0051d0 2024-04-25T15:54:43.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1187a1c640 1 -- 172.21.15.67:0/4145025290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0051d0 msgr2=0x7f116c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:43.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1187a1c640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0051d0 0x7f116c00b9d0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f117800a830 tx=0x7f117801d2a0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1187a1c640 1 -- 172.21.15.67:0/4145025290 shutdown_connections 2024-04-25T15:54:43.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1187a1c640 1 --2- 172.21.15.67:0/4145025290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0051d0 0x7f116c00b9d0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1187a1c640 1 -- 172.21.15.67:0/4145025290 >> 172.21.15.67:0/4145025290 conn(0x7f11800ce1c0 msgr2=0x7f11800ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:43.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1187a1c640 1 -- 172.21.15.67:0/4145025290 shutdown_connections 2024-04-25T15:54:43.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.385+0000 7f1187a1c640 1 -- 172.21.15.67:0/4145025290 wait complete. 2024-04-25T15:54:43.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 Processor -- start 2024-04-25T15:54:43.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 -- start start 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c0051d0 0x7f11800c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11800c5e50 0x7f11801557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11800cd590 0x7f11800c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f11800ca340 con 0x7f11800c5e50 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1180068ac0 con 0x7f116c0051d0 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f118006c640 con 0x7f11800cd590 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11800c5e50 0x7f11801557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11800c5e50 0x7f11801557b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54922/0 (socket says 172.21.15.67:54922) 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 -- 172.21.15.67:0/3055721499 learned_addr learned my addr 172.21.15.67:0/3055721499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f11857b8640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c0051d0 0x7f11800c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1185fb9640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11800cd590 0x7f11800c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 -- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11800cd590 msgr2=0x7f11800c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11800cd590 0x7f11800c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 -- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c0051d0 msgr2=0x7f11800c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c0051d0 0x7f11800c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 -- 172.21.15.67:0/3055721499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1180157070 con 0x7f11800c5e50 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1184fb7640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11800c5e50 0x7f11801557b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f11740115c0 tx=0x7f1174011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1171ffb640 1 -- 172.21.15.67:0/3055721499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174015870 con 0x7f11800c5e50 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1171ffb640 1 -- 172.21.15.67:0/3055721499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1174015a10 con 0x7f11800c5e50 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1180156220 con 0x7f11800c5e50 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1171ffb640 1 -- 172.21.15.67:0/3055721499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174015dc0 con 0x7f11800c5e50 2024-04-25T15:54:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.389+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11800c78f0 con 0x7f11800c5e50 2024-04-25T15:54:43.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.393+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f118006a480 con 0x7f11800c5e50 2024-04-25T15:54:43.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.393+0000 7f1171ffb640 1 -- 172.21.15.67:0/3055721499 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f117400f480 con 0x7f11800c5e50 2024-04-25T15:54:43.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.393+0000 7f1171ffb640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f116406f400 0x7f1164071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:43.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.393+0000 7f1171ffb640 1 -- 172.21.15.67:0/3055721499 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 6031+0+0 (secure 0 0 0) 0x7f117408faf0 con 0x7f11800c5e50 2024-04-25T15:54:43.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.393+0000 7f11857b8640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f116406f400 0x7f1164071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:43.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.393+0000 7f11857b8640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f116406f400 0x7f1164071910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1180074190 tx=0x7f117c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:43.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.397+0000 7f1171ffb640 1 -- 172.21.15.67:0/3055721499 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1174019030 con 0x7f11800c5e50 2024-04-25T15:54:43.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:43.569+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f11800cb7d0 con 0x7f11800c5e50 2024-04-25T15:54:44.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.189+0000 7f1171ffb640 1 -- 172.21.15.67:0/3055721499 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v119) ==== 113+0+0 (secure 0 0 0) 0x7f117405c2e0 con 0x7f11800c5e50 2024-04-25T15:54:44.190 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f116406f400 msgr2=0x7f1164071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f116406f400 0x7f1164071910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1180074190 tx=0x7f117c009040 comp rx=0 tx=0).stop 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11800c5e50 msgr2=0x7f11801557b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11800c5e50 0x7f11801557b0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f11740115c0 tx=0x7f1174011a80 comp rx=0 tx=0).stop 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 shutdown_connections 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f116406f400 0x7f1164071910 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11800cd590 0x7f11800c6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11800c5e50 0x7f11801557b0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 --2- 172.21.15.67:0/3055721499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f116c0051d0 0x7f11800c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 >> 172.21.15.67:0/3055721499 conn(0x7f11800ce1c0 msgr2=0x7f11800ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:44.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 shutdown_connections 2024-04-25T15:54:44.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.193+0000 7f1187a1c640 1 -- 172.21.15.67:0/3055721499 wait complete. 2024-04-25T15:54:44.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-25T15:54:44.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6fc2f640 1 Processor -- start 2024-04-25T15:54:44.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6fc2f640 1 -- start start 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6fc2f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1c68074790 0x7f1c6815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6fc2f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c68072dc0 con 0x7f1c6815e020 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6fc2f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c6815b430 con 0x7f1c680c6fb0 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6fc2f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1c6815a670 con 0x7f1c68074790 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6d1ca640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1c6815e020 0x7f1c680723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60276/0 (socket says 172.21.15.67:60276) 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6d1ca640 1 -- 172.21.15.67:0/1017489787 learned_addr learned my addr 172.21.15.67:0/1017489787 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6d9cb640 1 --2- 172.21.15.67:0/1017489787 >> v2:172.21.15.176:3300/0 conn(0x7f1c68074790 0x7f1c6815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 439169096 0 0) 0x7f1c68072dc0 con 0x7f1c6815e020 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c54003530 con 0x7f1c6815e020 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2205992921 0 0) 0x7f1c54003530 con 0x7f1c6815e020 2024-04-25T15:54:44.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 >> v2:172.21.15.176:3300/0 conn(0x7f1c68074790 msgr2=0x7f1c6815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 --2- 172.21.15.67:0/1017489787 >> v2:172.21.15.176:3300/0 conn(0x7f1c68074790 0x7f1c6815dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 >> v1:172.21.15.176:6789/0 conn(0x7f1c680c6fb0 legacy=0x7f1c68164e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6815a950 con 0x7f1c6815e020 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f1c58001680 con 0x7f1c6815e020 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 >> v1:172.21.15.67:6789/0 conn(0x7f1c6815e020 legacy=0x7f1c680723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540039c0 0x7f1c54003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c54004810 0x7f1c54004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c540051d0 0x7f1c5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6815a670 con 0x7f1c540051d0 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5400c160 con 0x7f1c54004810 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c5400c360 con 0x7f1c540039c0 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6e1cc640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c540051d0 0x7f1c5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6d1ca640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c54004810 0x7f1c54004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.393+0000 7f1c6d9cb640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540039c0 0x7f1c54003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540039c0 msgr2=0x7f1c54003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6e1cc640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540039c0 0x7f1c54003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c54004810 msgr2=0x7f1c54004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6e1cc640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c54004810 0x7f1c54004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/1017489787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c54003530 con 0x7f1c540051d0 2024-04-25T15:54:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/1017489787 reap_dead start 2024-04-25T15:54:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6e1cc640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c540051d0 0x7f1c5400b9d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1c60001e70 tx=0x7f1c6001c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842d19a8492cb5af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60001a90 con 0x7f1c540051d0 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c60003bc0 con 0x7f1c540051d0 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6c9c9640 1 -- 172.21.15.67:0/1017489787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c600050d0 con 0x7f1c540051d0 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/1017489787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c540051d0 msgr2=0x7f1c5400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c540051d0 0x7f1c5400b9d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f1c60001e70 tx=0x7f1c6001c6e0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/1017489787 shutdown_connections 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/1017489787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c540051d0 0x7f1c5400b9d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/1017489787 >> 172.21.15.67:0/1017489787 conn(0x7f1c6806fe70 msgr2=0x7f1c680cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/1017489787 shutdown_connections 2024-04-25T15:54:44.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.397+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/1017489787 wait complete. 2024-04-25T15:54:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 Processor -- start 2024-04-25T15:54:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 -- start start 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540051d0 0x7f1c68152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c68153ff0 0x7f1c681f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6806c810 0x7f1c68148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c68074ba0 con 0x7f1c6806c810 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6815af10 con 0x7f1c68153ff0 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c68072fe0 con 0x7f1c540051d0 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6806c810 0x7f1c68148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6806c810 0x7f1c68148ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54930/0 (socket says 172.21.15.67:54930) 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/2296679570 learned_addr learned my addr 172.21.15.67:0/2296679570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6d9cb640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540051d0 0x7f1c68152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6d1ca640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c68153ff0 0x7f1c681f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540051d0 msgr2=0x7f1c68152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540051d0 0x7f1c68152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c68153ff0 msgr2=0x7f1c681f1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c68153ff0 0x7f1c681f1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 -- 172.21.15.67:0/2296679570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c681f0980 con 0x7f1c6806c810 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6e1cc640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6806c810 0x7f1c68148ce0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1c6001cbc0 tx=0x7f1c54005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c5e7fc640 1 -- 172.21.15.67:0/2296679570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60001260 con 0x7f1c6806c810 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c5e7fc640 1 -- 172.21.15.67:0/2296679570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c600af070 con 0x7f1c6806c810 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c68151bc0 con 0x7f1c6806c810 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c5e7fc640 1 -- 172.21.15.67:0/2296679570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c600a8870 con 0x7f1c6806c810 2024-04-25T15:54:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.401+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c681f0c40 con 0x7f1c6806c810 2024-04-25T15:54:44.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.405+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c6806a480 con 0x7f1c6806c810 2024-04-25T15:54:44.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.405+0000 7f1c5e7fc640 1 -- 172.21.15.67:0/2296679570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1c600a8a10 con 0x7f1c6806c810 2024-04-25T15:54:44.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.405+0000 7f1c5e7fc640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c4c06f5d0 0x7f1c4c071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.405+0000 7f1c5e7fc640 1 -- 172.21.15.67:0/2296679570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 6031+0+0 (secure 0 0 0) 0x7f1c601292a0 con 0x7f1c6806c810 2024-04-25T15:54:44.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.405+0000 7f1c6d9cb640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c4c06f5d0 0x7f1c4c071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.405+0000 7f1c6d9cb640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c4c06f5d0 0x7f1c4c071ae0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009990 tx=0x7f1c64008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:44.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.409+0000 7f1c5e7fc640 1 -- 172.21.15.67:0/2296679570 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1c600c3080 con 0x7f1c6806c810 2024-04-25T15:54:44.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:44.585+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) -- 0x7f1c680c8e90 con 0x7f1c6806c810 2024-04-25T15:54:45.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.205+0000 7f1c5e7fc640 1 -- 172.21.15.67:0/2296679570 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v120) ==== 133+0+0 (secure 0 0 0) 0x7f1c600163f0 con 0x7f1c6806c810 2024-04-25T15:54:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-25T15:54:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c4c06f5d0 msgr2=0x7f1c4c071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c4c06f5d0 0x7f1c4c071ae0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1c64009990 tx=0x7f1c64008040 comp rx=0 tx=0).stop 2024-04-25T15:54:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6806c810 msgr2=0x7f1c68148ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6806c810 0x7f1c68148ce0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f1c6001cbc0 tx=0x7f1c54005840 comp rx=0 tx=0).stop 2024-04-25T15:54:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 shutdown_connections 2024-04-25T15:54:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f1c4c06f5d0 0x7f1c4c071ae0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6806c810 0x7f1c68148ce0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c68153ff0 0x7f1c681f1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 --2- 172.21.15.67:0/2296679570 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c540051d0 0x7f1c68152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 >> 172.21.15.67:0/2296679570 conn(0x7f1c6806fe70 msgr2=0x7f1c68072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:45.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 shutdown_connections 2024-04-25T15:54:45.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.209+0000 7f1c6fc2f640 1 -- 172.21.15.67:0/2296679570 wait complete. 2024-04-25T15:54:45.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-25T15:54:45.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.397+0000 7f9163f83640 1 Processor -- start 2024-04-25T15:54:45.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.397+0000 7f9163f83640 1 -- start start 2024-04-25T15:54:45.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9163f83640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f915c074790 0x7f915c15dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:45.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9163f83640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f915c0c6fb0 0x7f915c0c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:45.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9163f83640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915c072dc0 con 0x7f915c15e020 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9163f83640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f915c15b430 con 0x7f915c074790 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9163f83640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f915c15a670 con 0x7f915c0c6fb0 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f915c074790 0x7f915c15dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f915c074790 0x7f915c15dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54938/0 (socket says 172.21.15.67:54938) 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 -- 172.21.15.67:0/1115916171 learned_addr learned my addr 172.21.15.67:0/1115916171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 -- 172.21.15.67:0/1115916171 >> v2:172.21.15.176:3300/0 conn(0x7f915c0c6fb0 msgr2=0x7f915c0c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 --2- 172.21.15.67:0/1115916171 >> v2:172.21.15.176:3300/0 conn(0x7f915c0c6fb0 0x7f915c0c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 -- 172.21.15.67:0/1115916171 >> v1:172.21.15.176:6789/0 conn(0x7f915c15e020 legacy=0x7f915c0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:45.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 -- 172.21.15.67:0/1115916171 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915c15a950 con 0x7f915c074790 2024-04-25T15:54:45.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9161d1f640 1 --2- 172.21.15.67:0/1115916171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c074790 0x7f915c15dae0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f915c0cb730 tx=0x7f915801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac231cb1b9eeb05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:45.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9160d1d640 1 -- 172.21.15.67:0/1115916171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91580b5070 con 0x7f915c074790 2024-04-25T15:54:45.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9160d1d640 1 -- 172.21.15.67:0/1115916171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91580a89d0 con 0x7f915c074790 2024-04-25T15:54:45.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.401+0000 7f9160d1d640 1 -- 172.21.15.67:0/1115916171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91580a8da0 con 0x7f915c074790 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 -- 172.21.15.67:0/1115916171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c074790 msgr2=0x7f915c15dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 --2- 172.21.15.67:0/1115916171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c074790 0x7f915c15dae0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f915c0cb730 tx=0x7f915801ecd0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 -- 172.21.15.67:0/1115916171 shutdown_connections 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 --2- 172.21.15.67:0/1115916171 >> v2:172.21.15.176:3300/0 conn(0x7f915c0c6fb0 0x7f915c0c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 --2- 172.21.15.67:0/1115916171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c074790 0x7f915c15dae0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 -- 172.21.15.67:0/1115916171 >> 172.21.15.67:0/1115916171 conn(0x7f915c06fe70 msgr2=0x7f915c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 -- 172.21.15.67:0/1115916171 shutdown_connections 2024-04-25T15:54:45.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 -- 172.21.15.67:0/1115916171 wait complete. 2024-04-25T15:54:45.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 Processor -- start 2024-04-25T15:54:45.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.405+0000 7f9163f83640 1 -- start start 2024-04-25T15:54:45.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f915c074790 0x7f915c1f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:45.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f915c0c6fb0 0x7f915c155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:45.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15e020 0x7f915c0c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:45.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f915c073080 con 0x7f915c15e020 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f915c15af10 con 0x7f915c074790 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f915c072ab0 con 0x7f915c0c6fb0 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15e020 0x7f915c0c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15e020 0x7f915c0c6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54940/0 (socket says 172.21.15.67:54940) 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 -- 172.21.15.67:0/3574028753 learned_addr learned my addr 172.21.15.67:0/3574028753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f916151e640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f915c0c6fb0 0x7f915c155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9161d1f640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f915c074790 0x7f915c1f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 -- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f915c0c6fb0 msgr2=0x7f915c155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f915c0c6fb0 0x7f915c155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 -- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f915c074790 msgr2=0x7f915c1f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f915c074790 0x7f915c1f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 -- 172.21.15.67:0/3574028753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915c1f64f0 con 0x7f915c15e020 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9161d1f640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f915c074790 0x7f915c1f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:45.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9162520640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15e020 0x7f915c0c6770 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f915400b3c0 tx=0x7f915400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:45.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f914e7fc640 1 -- 172.21.15.67:0/3574028753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154011830 con 0x7f915c15e020 2024-04-25T15:54:45.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f914e7fc640 1 -- 172.21.15.67:0/3574028753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9154016070 con 0x7f915c15e020 2024-04-25T15:54:45.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f914e7fc640 1 -- 172.21.15.67:0/3574028753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154011c10 con 0x7f915c15e020 2024-04-25T15:54:45.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915c1f67d0 con 0x7f915c15e020 2024-04-25T15:54:45.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915c155cc0 con 0x7f915c15e020 2024-04-25T15:54:45.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.409+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f915c076730 con 0x7f915c15e020 2024-04-25T15:54:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.413+0000 7f914e7fc640 1 -- 172.21.15.67:0/3574028753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f915401b460 con 0x7f915c15e020 2024-04-25T15:54:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.413+0000 7f914e7fc640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f914006f400 0x7f9140071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:45.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.413+0000 7f914e7fc640 1 -- 172.21.15.67:0/3574028753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 6031+0+0 (secure 0 0 0) 0x7f915408fac0 con 0x7f915c15e020 2024-04-25T15:54:45.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.413+0000 7f9161d1f640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f914006f400 0x7f9140071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:45.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.413+0000 7f9161d1f640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f914006f400 0x7f9140071910 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f91580b2000 tx=0x7f9158008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.417+0000 7f914e7fc640 1 -- 172.21.15.67:0/3574028753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f915405c9e0 con 0x7f915c15e020 2024-04-25T15:54:45.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:45.597+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f915c14f200 con 0x7f915c15e020 2024-04-25T15:54:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.221+0000 7f914e7fc640 1 -- 172.21.15.67:0/3574028753 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v121) ==== 113+0+0 (secure 0 0 0) 0x7f915405c2e0 con 0x7f915c15e020 2024-04-25T15:54:46.224 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-04-25T15:54:46.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f914006f400 msgr2=0x7f9140071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f914006f400 0x7f9140071910 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f91580b2000 tx=0x7f9158008040 comp rx=0 tx=0).stop 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15e020 msgr2=0x7f915c0c6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15e020 0x7f915c0c6770 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f915400b3c0 tx=0x7f915400b880 comp rx=0 tx=0).stop 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 shutdown_connections 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f914006f400 0x7f9140071910 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f915c15e020 0x7f915c0c6770 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f915c0c6fb0 0x7f915c155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 --2- 172.21.15.67:0/3574028753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f915c074790 0x7f915c1f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 >> 172.21.15.67:0/3574028753 conn(0x7f915c06fe70 msgr2=0x7f915c152190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 shutdown_connections 2024-04-25T15:54:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.225+0000 7f9163f83640 1 -- 172.21.15.67:0/3574028753 wait complete. 2024-04-25T15:54:46.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-25T15:54:46.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2eb30640 1 Processor -- start 2024-04-25T15:54:46.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2eb30640 1 -- start start 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2eb30640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd28159520 0x7fdd280c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2eb30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd280c4ec0 0x7fdd280c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2eb30640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd280c7340 con 0x7fdd28073bd0 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2eb30640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdd280c5810 con 0x7fdd28159520 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2eb30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd280c67c0 con 0x7fdd280c4ec0 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2c8cc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd28159520 0x7fdd280c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2c8cc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd28159520 0x7fdd280c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54950/0 (socket says 172.21.15.67:54950) 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2c8cc640 1 -- 172.21.15.67:0/2292165362 learned_addr learned my addr 172.21.15.67:0/2292165362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2d0cd640 1 --2- 172.21.15.67:0/2292165362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd280c4ec0 0x7fdd280c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2c8cc640 1 -- 172.21.15.67:0/2292165362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd280c4ec0 msgr2=0x7fdd280c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2c8cc640 1 --2- 172.21.15.67:0/2292165362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd280c4ec0 0x7fdd280c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2c8cc640 1 -- 172.21.15.67:0/2292165362 >> v1:172.21.15.176:6789/0 conn(0x7fdd28073bd0 legacy=0x7fdd280c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2c8cc640 1 -- 172.21.15.67:0/2292165362 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd280c6aa0 con 0x7fdd28159520 2024-04-25T15:54:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.425+0000 7fdd2d0cd640 1 --2- 172.21.15.67:0/2292165362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd280c4ec0 0x7fdd280c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2c8cc640 1 --2- 172.21.15.67:0/2292165362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd28159520 0x7fdd280c4760 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009870 tx=0x7fdd2401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfc846d4ec4bbb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd1f7fe640 1 -- 172.21.15.67:0/2292165362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd240b5070 con 0x7fdd28159520 2024-04-25T15:54:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd1f7fe640 1 -- 172.21.15.67:0/2292165362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd240a89d0 con 0x7fdd28159520 2024-04-25T15:54:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd1f7fe640 1 -- 172.21.15.67:0/2292165362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd240a8da0 con 0x7fdd28159520 2024-04-25T15:54:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2292165362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd28159520 msgr2=0x7fdd280c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2292165362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd28159520 0x7fdd280c4760 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009870 tx=0x7fdd2401ed10 comp rx=0 tx=0).stop 2024-04-25T15:54:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2292165362 shutdown_connections 2024-04-25T15:54:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2292165362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd280c4ec0 0x7fdd280c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2292165362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd28159520 0x7fdd280c4760 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2292165362 >> 172.21.15.67:0/2292165362 conn(0x7fdd2806fe70 msgr2=0x7fdd280cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2292165362 shutdown_connections 2024-04-25T15:54:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2292165362 wait complete. 2024-04-25T15:54:46.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 Processor -- start 2024-04-25T15:54:46.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.429+0000 7fdd2eb30640 1 -- start start 2024-04-25T15:54:46.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd28073bd0 0x7fdd2814d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd280c4ec0 0x7fdd28149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd28159520 0x7fdd28149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd280736b0 con 0x7fdd280c4ec0 2024-04-25T15:54:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd280c7060 con 0x7fdd28159520 2024-04-25T15:54:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd280c6310 con 0x7fdd28073bd0 2024-04-25T15:54:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd280c4ec0 0x7fdd28149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd280c4ec0 0x7fdd28149220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54966/0 (socket says 172.21.15.67:54966) 2024-04-25T15:54:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 -- 172.21.15.67:0/2887663375 learned_addr learned my addr 172.21.15.67:0/2887663375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2d0cd640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd28159520 0x7fdd28149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2c8cc640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd28073bd0 0x7fdd2814d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 -- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd28073bd0 msgr2=0x7fdd2814d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd28073bd0 0x7fdd2814d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 -- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd28159520 msgr2=0x7fdd28149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd28159520 0x7fdd28149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 -- 172.21.15.67:0/2887663375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd281513d0 con 0x7fdd280c4ec0 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2d0cd640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd28159520 0x7fdd28149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1ffff640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd280c4ec0 0x7fdd28149220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b440 tx=0x7fdd1800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1d7fa640 1 -- 172.21.15.67:0/2887663375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18015020 con 0x7fdd280c4ec0 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1d7fa640 1 -- 172.21.15.67:0/2887663375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd18011890 con 0x7fdd280c4ec0 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1d7fa640 1 -- 172.21.15.67:0/2887663375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18011c80 con 0x7fdd280c4ec0 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd281e4430 con 0x7fdd280c4ec0 2024-04-25T15:54:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd281f40e0 con 0x7fdd280c4ec0 2024-04-25T15:54:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2806a480 con 0x7fdd280c4ec0 2024-04-25T15:54:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.433+0000 7fdd1d7fa640 1 -- 172.21.15.67:0/2887663375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdd18018020 con 0x7fdd280c4ec0 2024-04-25T15:54:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.437+0000 7fdd1d7fa640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdd1406f400 0x7fdd14071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.437+0000 7fdd1d7fa640 1 -- 172.21.15.67:0/2887663375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 6031+0+0 (secure 0 0 0) 0x7fdd1808f170 con 0x7fdd280c4ec0 2024-04-25T15:54:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.437+0000 7fdd2c8cc640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdd1406f400 0x7fdd14071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.437+0000 7fdd2c8cc640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdd1406f400 0x7fdd14071910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd240b2000 tx=0x7fdd240aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.441+0000 7fdd1d7fa640 1 -- 172.21.15.67:0/2887663375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdd1801f030 con 0x7fdd280c4ec0 2024-04-25T15:54:46.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:46.617+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) -- 0x7fdd280c59e0 con 0x7fdd280c4ec0 2024-04-25T15:54:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.245+0000 7fdd1d7fa640 1 -- 172.21.15.67:0/2887663375 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v122) ==== 133+0+0 (secure 0 0 0) 0x7fdd1800d040 con 0x7fdd280c4ec0 2024-04-25T15:54:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-25T15:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdd1406f400 msgr2=0x7fdd14071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdd1406f400 0x7fdd14071910 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd240b2000 tx=0x7fdd240aa070 comp rx=0 tx=0).stop 2024-04-25T15:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd280c4ec0 msgr2=0x7fdd28149220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd280c4ec0 0x7fdd28149220 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b440 tx=0x7fdd1800b900 comp rx=0 tx=0).stop 2024-04-25T15:54:47.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 shutdown_connections 2024-04-25T15:54:47.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdd1406f400 0x7fdd14071910 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:47.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd28159520 0x7fdd28149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:47.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd280c4ec0 0x7fdd28149220 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:47.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 --2- 172.21.15.67:0/2887663375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd28073bd0 0x7fdd2814d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:47.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 >> 172.21.15.67:0/2887663375 conn(0x7fdd2806fe70 msgr2=0x7fdd280c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:47.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 shutdown_connections 2024-04-25T15:54:47.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.249+0000 7fdd2eb30640 1 -- 172.21.15.67:0/2887663375 wait complete. 2024-04-25T15:54:47.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-25T15:54:47.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.433+0000 7f82fc08e640 1 Processor -- start 2024-04-25T15:54:47.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82fc08e640 1 -- start start 2024-04-25T15:54:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82fc08e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82f406ca60 0x7f82f406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82fc08e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f406d780 con 0x7f82f415fe00 2024-04-25T15:54:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82fc08e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f4160920 con 0x7f82f40b5660 2024-04-25T15:54:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82fc08e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f82f415d510 con 0x7f82f406ca60 2024-04-25T15:54:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82f9e2a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82f406ca60 0x7f82f406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82fa62b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82f415fe00 0x7f82f4160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60286/0 (socket says 172.21.15.67:60286) 2024-04-25T15:54:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82fa62b640 1 -- 172.21.15.67:0/1689656130 learned_addr learned my addr 172.21.15.67:0/1689656130 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:47.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82f9e2a640 1 -- 172.21.15.67:0/1689656130 >> v1:172.21.15.176:6789/0 conn(0x7f82f40b5660 legacy=0x7f82f40b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:47.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82f9e2a640 1 -- 172.21.15.67:0/1689656130 >> v1:172.21.15.67:6789/0 conn(0x7f82f415fe00 legacy=0x7f82f4160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:47.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.437+0000 7f82f9e2a640 1 -- 172.21.15.67:0/1689656130 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f4160be0 con 0x7f82f406ca60 2024-04-25T15:54:47.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82f9e2a640 1 --2- 172.21.15.67:0/1689656130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406ca60 0x7f82f406ce70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f82f00097f0 tx=0x7f82f001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39eae517f14cd15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:47.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82f8e28640 1 -- 172.21.15.67:0/1689656130 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00b0070 con 0x7f82f406ca60 2024-04-25T15:54:47.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82f8e28640 1 -- 172.21.15.67:0/1689656130 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f00ab600 con 0x7f82f406ca60 2024-04-25T15:54:47.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82f8e28640 1 -- 172.21.15.67:0/1689656130 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00ab9d0 con 0x7f82f406ca60 2024-04-25T15:54:47.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82fc08e640 1 -- 172.21.15.67:0/1689656130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406ca60 msgr2=0x7f82f406ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:47.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82fc08e640 1 --2- 172.21.15.67:0/1689656130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406ca60 0x7f82f406ce70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f82f00097f0 tx=0x7f82f001c330 comp rx=0 tx=0).stop 2024-04-25T15:54:47.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82fc08e640 1 -- 172.21.15.67:0/1689656130 shutdown_connections 2024-04-25T15:54:47.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82fc08e640 1 --2- 172.21.15.67:0/1689656130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f406ca60 0x7f82f406ce70 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:47.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82fc08e640 1 -- 172.21.15.67:0/1689656130 >> 172.21.15.67:0/1689656130 conn(0x7f82f406e420 msgr2=0x7f82f406e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:47.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82fc08e640 1 -- 172.21.15.67:0/1689656130 shutdown_connections 2024-04-25T15:54:47.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.441+0000 7f82fc08e640 1 -- 172.21.15.67:0/1689656130 wait complete. 2024-04-25T15:54:47.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 Processor -- start 2024-04-25T15:54:47.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 -- start start 2024-04-25T15:54:47.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82f406ca60 0x7f82f4153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:47.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f40b5660 0x7f82f4155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:47.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82f415fe00 0x7f82f41513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:47.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f82f40b6e90 con 0x7f82f40b5660 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f82f415c890 con 0x7f82f415fe00 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f82f40b7010 con 0x7f82f406ca60 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f40b5660 0x7f82f4155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f40b5660 0x7f82f4155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54980/0 (socket says 172.21.15.67:54980) 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 -- 172.21.15.67:0/2139435532 learned_addr learned my addr 172.21.15.67:0/2139435532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fa62b640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82f415fe00 0x7f82f41513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9e2a640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82f406ca60 0x7f82f4153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 -- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82f406ca60 msgr2=0x7f82f4153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82f406ca60 0x7f82f4153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 -- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82f415fe00 msgr2=0x7f82f41513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82f415fe00 0x7f82f41513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 -- 172.21.15.67:0/2139435532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f41539d0 con 0x7f82f40b5660 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fa62b640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82f415fe00 0x7f82f41513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:47.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82f9629640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f40b5660 0x7f82f4155940 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f82e400c340 tx=0x7f82e400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82ea7fc640 1 -- 172.21.15.67:0/2139435532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e40052d0 con 0x7f82f40b5660 2024-04-25T15:54:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82ea7fc640 1 -- 172.21.15.67:0/2139435532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e4005470 con 0x7f82f40b5660 2024-04-25T15:54:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82ea7fc640 1 -- 172.21.15.67:0/2139435532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4007260 con 0x7f82f40b5660 2024-04-25T15:54:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f4158790 con 0x7f82f40b5660 2024-04-25T15:54:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f4157e60 con 0x7f82f40b5660 2024-04-25T15:54:47.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.445+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f4157bf0 con 0x7f82f40b5660 2024-04-25T15:54:47.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.449+0000 7f82ea7fc640 1 -- 172.21.15.67:0/2139435532 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f82e4007400 con 0x7f82f40b5660 2024-04-25T15:54:47.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.449+0000 7f82ea7fc640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82d806f400 0x7f82d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:47.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.449+0000 7f82ea7fc640 1 -- 172.21.15.67:0/2139435532 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 6031+0+0 (secure 0 0 0) 0x7f82e408e860 con 0x7f82f40b5660 2024-04-25T15:54:47.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.449+0000 7f82f9e2a640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82d806f400 0x7f82d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:47.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.449+0000 7f82f9e2a640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82d806f400 0x7f82d8071910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ad000 tx=0x7f82f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.453+0000 7f82ea7fc640 1 -- 172.21.15.67:0/2139435532 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f82e405b690 con 0x7f82f40b5660 2024-04-25T15:54:47.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:47.629+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f82f406a480 con 0x7f82f40b5660 2024-04-25T15:54:48.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.245+0000 7f82ea7fc640 1 -- 172.21.15.67:0/2139435532 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v123) ==== 113+0+0 (secure 0 0 0) 0x7f82e405af90 con 0x7f82f40b5660 2024-04-25T15:54:48.249 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-04-25T15:54:48.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82d806f400 msgr2=0x7f82d8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:48.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82d806f400 0x7f82d8071910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f82f00ad000 tx=0x7f82f0008040 comp rx=0 tx=0).stop 2024-04-25T15:54:48.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f40b5660 msgr2=0x7f82f4155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:48.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f40b5660 0x7f82f4155940 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f82e400c340 tx=0x7f82e400c800 comp rx=0 tx=0).stop 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 shutdown_connections 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f82d806f400 0x7f82d8071910 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82f415fe00 0x7f82f41513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82f40b5660 0x7f82f4155940 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 --2- 172.21.15.67:0/2139435532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82f406ca60 0x7f82f4153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 >> 172.21.15.67:0/2139435532 conn(0x7f82f406e420 msgr2=0x7f82f406e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 shutdown_connections 2024-04-25T15:54:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.249+0000 7f82fc08e640 1 -- 172.21.15.67:0/2139435532 wait complete. 2024-04-25T15:54:48.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-25T15:54:48.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bccaef640 1 Processor -- start 2024-04-25T15:54:48.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bccaef640 1 -- start start 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bccaef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc806ca60 0x7f3bc806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bccaef640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3bc815fe00 0x7f3bc8160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bccaef640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc806d780 con 0x7f3bc80b5660 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bccaef640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc815d510 con 0x7f3bc815fe00 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bccaef640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc8160be0 con 0x7f3bc806ca60 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc659c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc806ca60 0x7f3bc806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc6d9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3bc815fe00 0x7f3bc8160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc6d9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3bc815fe00 0x7f3bc8160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:38036/0 (socket says 172.21.15.67:38036) 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc6d9d640 1 -- 172.21.15.67:0/3931296789 learned_addr learned my addr 172.21.15.67:0/3931296789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:48.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc5d9b640 1 --1- 172.21.15.67:0/3931296789 >> v1:172.21.15.176:6789/0 conn(0x7f3bc80b5660 0x7f3bc80b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:40938/0 (socket says 172.21.15.67:40938) 2024-04-25T15:54:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc6d9d640 1 -- 172.21.15.67:0/3931296789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc806ca60 msgr2=0x7f3bc806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc6d9d640 1 --2- 172.21.15.67:0/3931296789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc806ca60 0x7f3bc806ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc6d9d640 1 -- 172.21.15.67:0/3931296789 >> v1:172.21.15.176:6789/0 conn(0x7f3bc80b5660 legacy=0x7f3bc80b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:48.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.445+0000 7f3bc6d9d640 1 -- 172.21.15.67:0/3931296789 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc815cbe0 con 0x7f3bc815fe00 2024-04-25T15:54:48.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bc6d9d640 1 --2- 172.21.15.67:0/3931296789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc815fe00 0x7f3bc8160210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009e50 tx=0x7f3bc001f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89c36aba521c609e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bc559a640 1 -- 172.21.15.67:0/3931296789 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00b4070 con 0x7f3bc815fe00 2024-04-25T15:54:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bc559a640 1 -- 172.21.15.67:0/3931296789 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc00af400 con 0x7f3bc815fe00 2024-04-25T15:54:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bc559a640 1 -- 172.21.15.67:0/3931296789 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00af7d0 con 0x7f3bc815fe00 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 -- 172.21.15.67:0/3931296789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc815fe00 msgr2=0x7f3bc8160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3931296789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc815fe00 0x7f3bc8160210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009e50 tx=0x7f3bc001f210 comp rx=0 tx=0).stop 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 -- 172.21.15.67:0/3931296789 shutdown_connections 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3931296789 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc815fe00 0x7f3bc8160210 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3931296789 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc806ca60 0x7f3bc806ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 -- 172.21.15.67:0/3931296789 >> 172.21.15.67:0/3931296789 conn(0x7f3bc806e420 msgr2=0x7f3bc806e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 -- 172.21.15.67:0/3931296789 shutdown_connections 2024-04-25T15:54:48.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.449+0000 7f3bccaef640 1 -- 172.21.15.67:0/3931296789 wait complete. 2024-04-25T15:54:48.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 Processor -- start 2024-04-25T15:54:48.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 -- start start 2024-04-25T15:54:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc806ca60 0x7f3bc81e6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5660 0x7f3bc81e7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc815fe00 0x7f3bc814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc815c890 con 0x7f3bc80b5660 2024-04-25T15:54:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc815d1a0 con 0x7f3bc806ca60 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc80b6e90 con 0x7f3bc815fe00 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5660 0x7f3bc81e7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5660 0x7f3bc81e7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54984/0 (socket says 172.21.15.67:54984) 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 -- 172.21.15.67:0/3451880912 learned_addr learned my addr 172.21.15.67:0/3451880912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc6d9d640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc815fe00 0x7f3bc814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc659c640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc806ca60 0x7f3bc81e6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 -- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc815fe00 msgr2=0x7f3bc814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc815fe00 0x7f3bc814cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 -- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc806ca60 msgr2=0x7f3bc81e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc806ca60 0x7f3bc81e6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 -- 172.21.15.67:0/3451880912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc81e66d0 con 0x7f3bc80b5660 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc659c640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc806ca60 0x7f3bc81e6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bc5d9b640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5660 0x7f3bc81e7330 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00c370 tx=0x7f3bbc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:48.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bb6ffd640 1 -- 172.21.15.67:0/3451880912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc0074e0 con 0x7f3bc80b5660 2024-04-25T15:54:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bb6ffd640 1 -- 172.21.15.67:0/3451880912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bbc0076a0 con 0x7f3bc80b5660 2024-04-25T15:54:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bb6ffd640 1 -- 172.21.15.67:0/3451880912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc004a10 con 0x7f3bc80b5660 2024-04-25T15:54:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc81f3ea0 con 0x7f3bc80b5660 2024-04-25T15:54:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.453+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc81f4160 con 0x7f3bc80b5660 2024-04-25T15:54:48.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.457+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc806a480 con 0x7f3bc80b5660 2024-04-25T15:54:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.457+0000 7f3bb6ffd640 1 -- 172.21.15.67:0/3451880912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3bbc01a020 con 0x7f3bc80b5660 2024-04-25T15:54:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.457+0000 7f3bb6ffd640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ba806f430 0x7f3ba8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.457+0000 7f3bb6ffd640 1 -- 172.21.15.67:0/3451880912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6031+0+0 (secure 0 0 0) 0x7f3bbc08eae0 con 0x7f3bc80b5660 2024-04-25T15:54:48.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.457+0000 7f3bc659c640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ba806f430 0x7f3ba8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:48.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.457+0000 7f3bc659c640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ba806f430 0x7f3ba8071940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80cd350 tx=0x7f3bb8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:48.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.461+0000 7f3bb6ffd640 1 -- 172.21.15.67:0/3451880912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3bbc05b920 con 0x7f3bc80b5660 2024-04-25T15:54:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:48.633+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7f3bc80cd380 con 0x7f3bc80b5660 2024-04-25T15:54:49.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.261+0000 7f3bb6ffd640 1 -- 172.21.15.67:0/3451880912 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v124) ==== 133+0+0 (secure 0 0 0) 0x7f3bbc05b220 con 0x7f3bc80b5660 2024-04-25T15:54:49.262 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-25T15:54:49.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ba806f430 msgr2=0x7f3ba8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:49.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ba806f430 0x7f3ba8071940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80cd350 tx=0x7f3bb8008040 comp rx=0 tx=0).stop 2024-04-25T15:54:49.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5660 msgr2=0x7f3bc81e7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5660 0x7f3bc81e7330 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00c370 tx=0x7f3bbc00c830 comp rx=0 tx=0).stop 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 shutdown_connections 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ba806f430 0x7f3ba8071940 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc815fe00 0x7f3bc814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bc80b5660 0x7f3bc81e7330 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 --2- 172.21.15.67:0/3451880912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc806ca60 0x7f3bc81e6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 >> 172.21.15.67:0/3451880912 conn(0x7f3bc806e420 msgr2=0x7f3bc80c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 shutdown_connections 2024-04-25T15:54:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.265+0000 7f3bccaef640 1 -- 172.21.15.67:0/3451880912 wait complete. 2024-04-25T15:54:49.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-25T15:54:49.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.457+0000 7f042be04640 1 Processor -- start 2024-04-25T15:54:49.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.457+0000 7f042be04640 1 -- start start 2024-04-25T15:54:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042be04640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f04240c7610 0x7f04240c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042be04640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c5bb0 0x7f04240c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:49.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042be04640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04240c6890 con 0x7f0424071b20 2024-04-25T15:54:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042be04640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f042415cd40 con 0x7f04240c7610 2024-04-25T15:54:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042be04640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f04240c6b70 con 0x7f04240c5bb0 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042939f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f04240c7610 0x7f04240c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042a3a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c5bb0 0x7f04240c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042939f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f04240c7610 0x7f04240c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:38056/0 (socket says 172.21.15.67:38056) 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042939f640 1 -- 172.21.15.67:0/1980987098 learned_addr learned my addr 172.21.15.67:0/1980987098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f0429ba0640 1 --1- 172.21.15.67:0/1980987098 >> v1:172.21.15.176:6789/0 conn(0x7f0424071b20 0x7f042415d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:40954/0 (socket says 172.21.15.67:40954) 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042a3a1640 1 -- 172.21.15.67:0/1980987098 >> v2:172.21.15.176:3300/0 conn(0x7f04240c7610 msgr2=0x7f04240c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042a3a1640 1 --2- 172.21.15.67:0/1980987098 >> v2:172.21.15.176:3300/0 conn(0x7f04240c7610 0x7f04240c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042a3a1640 1 -- 172.21.15.67:0/1980987098 >> v1:172.21.15.176:6789/0 conn(0x7f0424071b20 legacy=0x7f042415d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042a3a1640 1 -- 172.21.15.67:0/1980987098 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04240c6e50 con 0x7f04240c5bb0 2024-04-25T15:54:49.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.461+0000 7f042939f640 1 --2- 172.21.15.67:0/1980987098 >> v2:172.21.15.176:3300/0 conn(0x7f04240c7610 0x7f04240c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:49.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042a3a1640 1 --2- 172.21.15.67:0/1980987098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c5bb0 0x7f04240c5fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f041c0097c0 tx=0x7f041c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dc07646b62937d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:49.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f0428b9e640 1 -- 172.21.15.67:0/1980987098 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c0b1070 con 0x7f04240c5bb0 2024-04-25T15:54:49.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f0428b9e640 1 -- 172.21.15.67:0/1980987098 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f041c0ac410 con 0x7f04240c5bb0 2024-04-25T15:54:49.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f0428b9e640 1 -- 172.21.15.67:0/1980987098 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c0ac7e0 con 0x7f04240c5bb0 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 -- 172.21.15.67:0/1980987098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c5bb0 msgr2=0x7f04240c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 --2- 172.21.15.67:0/1980987098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c5bb0 0x7f04240c5fc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f041c0097c0 tx=0x7f041c01c4a0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 -- 172.21.15.67:0/1980987098 shutdown_connections 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 --2- 172.21.15.67:0/1980987098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c5bb0 0x7f04240c5fc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 --2- 172.21.15.67:0/1980987098 >> v2:172.21.15.176:3300/0 conn(0x7f04240c7610 0x7f04240c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 -- 172.21.15.67:0/1980987098 >> 172.21.15.67:0/1980987098 conn(0x7f042406fe70 msgr2=0x7f04240cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 -- 172.21.15.67:0/1980987098 shutdown_connections 2024-04-25T15:54:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 -- 172.21.15.67:0/1980987098 wait complete. 2024-04-25T15:54:49.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 Processor -- start 2024-04-25T15:54:49.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.465+0000 7f042be04640 1 -- start start 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0424071b20 0x7f042414b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04240c5bb0 0x7f0424143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c7610 0x7f042413ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0424166580 con 0x7f04240c5bb0 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04240734f0 con 0x7f0424071b20 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0424166700 con 0x7f04240c7610 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04240c5bb0 0x7f0424143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04240c5bb0 0x7f0424143380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54998/0 (socket says 172.21.15.67:54998) 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 -- 172.21.15.67:0/4063985344 learned_addr learned my addr 172.21.15.67:0/4063985344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042a3a1640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c7610 0x7f042413ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f0429ba0640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0424071b20 0x7f042414b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 -- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c7610 msgr2=0x7f042413ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c7610 0x7f042413ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 -- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0424071b20 msgr2=0x7f042414b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0424071b20 0x7f042414b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 -- 172.21.15.67:0/4063985344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042415bae0 con 0x7f04240c5bb0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042939f640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04240c5bb0 0x7f0424143380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f041400e980 tx=0x7f041400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f041a7fc640 1 -- 172.21.15.67:0/4063985344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0414017070 con 0x7f04240c5bb0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f041a7fc640 1 -- 172.21.15.67:0/4063985344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f041400c9a0 con 0x7f04240c5bb0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f041a7fc640 1 -- 172.21.15.67:0/4063985344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041400cd90 con 0x7f04240c5bb0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04241560f0 con 0x7f04240c5bb0 2024-04-25T15:54:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04241496f0 con 0x7f04240c5bb0 2024-04-25T15:54:49.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.469+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f042406a480 con 0x7f04240c5bb0 2024-04-25T15:54:49.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.473+0000 7f041a7fc640 1 -- 172.21.15.67:0/4063985344 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f041401a070 con 0x7f04240c5bb0 2024-04-25T15:54:49.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.473+0000 7f041a7fc640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f040806f400 0x7f0408071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:49.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.473+0000 7f041a7fc640 1 -- 172.21.15.67:0/4063985344 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 6031+0+0 (secure 0 0 0) 0x7f041408f8a0 con 0x7f04240c5bb0 2024-04-25T15:54:49.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.473+0000 7f0429ba0640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f040806f400 0x7f0408071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:49.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.473+0000 7f0429ba0640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f040806f400 0x7f0408071910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f04200012f0 tx=0x7f0420001640 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:49.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.477+0000 7f041a7fc640 1 -- 172.21.15.67:0/4063985344 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f041405c0e0 con 0x7f04240c5bb0 2024-04-25T15:54:49.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:49.649+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7f04240c6250 con 0x7f04240c5bb0 2024-04-25T15:54:50.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.277+0000 7f041a7fc640 1 -- 172.21.15.67:0/4063985344 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v125) ==== 113+0+0 (secure 0 0 0) 0x7f04140603e0 con 0x7f04240c5bb0 2024-04-25T15:54:50.277 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-04-25T15:54:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f040806f400 msgr2=0x7f0408071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f040806f400 0x7f0408071910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f04200012f0 tx=0x7f0420001640 comp rx=0 tx=0).stop 2024-04-25T15:54:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04240c5bb0 msgr2=0x7f0424143380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04240c5bb0 0x7f0424143380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f041400e980 tx=0x7f041400ee40 comp rx=0 tx=0).stop 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 shutdown_connections 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f040806f400 0x7f0408071910 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c7610 0x7f042413ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04240c5bb0 0x7f0424143380 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 --2- 172.21.15.67:0/4063985344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0424071b20 0x7f042414b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 >> 172.21.15.67:0/4063985344 conn(0x7f042406fe70 msgr2=0x7f04240c9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 shutdown_connections 2024-04-25T15:54:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.281+0000 7f042be04640 1 -- 172.21.15.67:0/4063985344 wait complete. 2024-04-25T15:54:50.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-25T15:54:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.461+0000 7f76368a3640 1 Processor -- start 2024-04-25T15:54:50.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.461+0000 7f76368a3640 1 -- start start 2024-04-25T15:54:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f76368a3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76300723a0 0x7f76300727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:50.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f76368a3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f76300c6fb0 0x7f76300c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f76368a3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7630072dc0 con 0x7f7630074790 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f76368a3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f763015b430 con 0x7f76300723a0 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f76368a3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f763015a670 con 0x7f76300c6fb0 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f762f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76300723a0 0x7f76300727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f762f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76300723a0 0x7f76300727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55004/0 (socket says 172.21.15.67:55004) 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f762f7fe640 1 -- 172.21.15.67:0/905080685 learned_addr learned my addr 172.21.15.67:0/905080685 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f7634e40640 1 --2- 172.21.15.67:0/905080685 >> v2:172.21.15.176:3300/0 conn(0x7f76300c6fb0 0x7f76300c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f762f7fe640 1 -- 172.21.15.67:0/905080685 >> v2:172.21.15.176:3300/0 conn(0x7f76300c6fb0 msgr2=0x7f76300c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f762f7fe640 1 --2- 172.21.15.67:0/905080685 >> v2:172.21.15.176:3300/0 conn(0x7f76300c6fb0 0x7f76300c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f762f7fe640 1 -- 172.21.15.67:0/905080685 >> v1:172.21.15.176:6789/0 conn(0x7f7630074790 legacy=0x7f763015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f762f7fe640 1 -- 172.21.15.67:0/905080685 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763015a950 con 0x7f76300723a0 2024-04-25T15:54:50.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.465+0000 7f7634e40640 1 --2- 172.21.15.67:0/905080685 >> v2:172.21.15.176:3300/0 conn(0x7f76300c6fb0 0x7f76300c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:50.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f762f7fe640 1 --2- 172.21.15.67:0/905080685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76300727b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f76300c8330 tx=0x7f762401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d86ce37c4b4e0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:50.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f762effd640 1 -- 172.21.15.67:0/905080685 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76240b5070 con 0x7f76300723a0 2024-04-25T15:54:50.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f762effd640 1 -- 172.21.15.67:0/905080685 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76240a89d0 con 0x7f76300723a0 2024-04-25T15:54:50.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f762effd640 1 -- 172.21.15.67:0/905080685 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76240a8da0 con 0x7f76300723a0 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 -- 172.21.15.67:0/905080685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 msgr2=0x7f76300727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 --2- 172.21.15.67:0/905080685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76300727b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f76300c8330 tx=0x7f762401ecd0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 -- 172.21.15.67:0/905080685 shutdown_connections 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 --2- 172.21.15.67:0/905080685 >> v2:172.21.15.176:3300/0 conn(0x7f76300c6fb0 0x7f76300c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 --2- 172.21.15.67:0/905080685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76300727b0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 -- 172.21.15.67:0/905080685 >> 172.21.15.67:0/905080685 conn(0x7f763006fe70 msgr2=0x7f76300cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 -- 172.21.15.67:0/905080685 shutdown_connections 2024-04-25T15:54:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 -- 172.21.15.67:0/905080685 wait complete. 2024-04-25T15:54:50.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 Processor -- start 2024-04-25T15:54:50.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.469+0000 7f76368a3640 1 -- start start 2024-04-25T15:54:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76301520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7630074790 0x7f763014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76300c6fb0 0x7f76301e64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f763015e1f0 con 0x7f76300723a0 2024-04-25T15:54:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f763015af10 con 0x7f7630074790 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7630073080 con 0x7f76300c6fb0 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76301520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76301520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55008/0 (socket says 172.21.15.67:55008) 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 -- 172.21.15.67:0/1673012578 learned_addr learned my addr 172.21.15.67:0/1673012578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762f7fe640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7630074790 0x7f763014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f7634e40640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76300c6fb0 0x7f76301e64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 -- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76300c6fb0 msgr2=0x7f76301e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76300c6fb0 0x7f76301e64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 -- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7630074790 msgr2=0x7f763014c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7630074790 0x7f763014c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 -- 172.21.15.67:0/1673012578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763015daf0 con 0x7f76300723a0 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762f7fe640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7630074790 0x7f763014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762ffff640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76301520f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f762000b410 tx=0x7f762000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762cff9640 1 -- 172.21.15.67:0/1673012578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620015030 con 0x7f76300723a0 2024-04-25T15:54:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762cff9640 1 -- 172.21.15.67:0/1673012578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7620004020 con 0x7f76300723a0 2024-04-25T15:54:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762cff9640 1 -- 172.21.15.67:0/1673012578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620004450 con 0x7f76300723a0 2024-04-25T15:54:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76301f0ae0 con 0x7f76300723a0 2024-04-25T15:54:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76301f10a0 con 0x7f76300723a0 2024-04-25T15:54:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f763006a480 con 0x7f76300723a0 2024-04-25T15:54:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.473+0000 7f762cff9640 1 -- 172.21.15.67:0/1673012578 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7620018070 con 0x7f76300723a0 2024-04-25T15:54:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.477+0000 7f762cff9640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f761c06f400 0x7f761c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.477+0000 7f762cff9640 1 -- 172.21.15.67:0/1673012578 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 6031+0+0 (secure 0 0 0) 0x7f762008eef0 con 0x7f76300723a0 2024-04-25T15:54:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.477+0000 7f762f7fe640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f761c06f400 0x7f761c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.477+0000 7f762f7fe640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f761c06f400 0x7f761c071910 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f76240b2000 tx=0x7f7624008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.481+0000 7f762cff9640 1 -- 172.21.15.67:0/1673012578 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f762005bdc0 con 0x7f76300723a0 2024-04-25T15:54:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:50.649+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) -- 0x7f76300ca010 con 0x7f76300723a0 2024-04-25T15:54:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.277+0000 7f762cff9640 1 -- 172.21.15.67:0/1673012578 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v126) ==== 133+0+0 (secure 0 0 0) 0x7f762005b6c0 con 0x7f76300723a0 2024-04-25T15:54:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f761c06f400 msgr2=0x7f761c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f761c06f400 0x7f761c071910 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f76240b2000 tx=0x7f7624008040 comp rx=0 tx=0).stop 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 msgr2=0x7f76301520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76301520f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f762000b410 tx=0x7f762000b8d0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 shutdown_connections 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f761c06f400 0x7f761c071910 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f76300c6fb0 0x7f76301e64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7630074790 0x7f763014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 --2- 172.21.15.67:0/1673012578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76300723a0 0x7f76301520f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 >> 172.21.15.67:0/1673012578 conn(0x7f763006fe70 msgr2=0x7f763006c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 shutdown_connections 2024-04-25T15:54:51.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.281+0000 7f76368a3640 1 -- 172.21.15.67:0/1673012578 wait complete. 2024-04-25T15:54:51.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-25T15:54:51.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff120d5a640 1 Processor -- start 2024-04-25T15:54:51.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff120d5a640 1 -- start start 2024-04-25T15:54:51.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff120d5a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff11c0c6c10 0x7ff11c156c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:51.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff120d5a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c0d2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff120d5a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff11c15cc60 con 0x7ff11c0d2540 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff120d5a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff11c15cf40 con 0x7ff11c0c6c10 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff120d5a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c15d220 con 0x7ff11c0c9bc0 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff11a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff11c0c6c10 0x7ff11c156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff11a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff11c0c6c10 0x7ff11c156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55020/0 (socket says 172.21.15.67:55020) 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff11a59c640 1 -- 172.21.15.67:0/1513947726 learned_addr learned my addr 172.21.15.67:0/1513947726 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff119d9b640 1 --2- 172.21.15.67:0/1513947726 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c0d2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff11a59c640 1 -- 172.21.15.67:0/1513947726 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 msgr2=0x7ff11c0d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff11a59c640 1 --2- 172.21.15.67:0/1513947726 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c0d2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff11a59c640 1 -- 172.21.15.67:0/1513947726 >> v1:172.21.15.176:6789/0 conn(0x7ff11c0d2540 legacy=0x7ff11c0c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:51.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.469+0000 7ff11a59c640 1 -- 172.21.15.67:0/1513947726 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c15d500 con 0x7ff11c0c6c10 2024-04-25T15:54:51.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff11a59c640 1 --2- 172.21.15.67:0/1513947726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c156c60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff110009870 tx=0x7ff11001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831f93c70fc4b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff11959a640 1 -- 172.21.15.67:0/1513947726 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1100b5070 con 0x7ff11c0c6c10 2024-04-25T15:54:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff11959a640 1 -- 172.21.15.67:0/1513947726 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1100a89d0 con 0x7ff11c0c6c10 2024-04-25T15:54:51.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff11959a640 1 -- 172.21.15.67:0/1513947726 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1100a8da0 con 0x7ff11c0c6c10 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 -- 172.21.15.67:0/1513947726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 msgr2=0x7ff11c156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 --2- 172.21.15.67:0/1513947726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c156c60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff110009870 tx=0x7ff11001ed10 comp rx=0 tx=0).stop 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 -- 172.21.15.67:0/1513947726 shutdown_connections 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 --2- 172.21.15.67:0/1513947726 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c0d2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 --2- 172.21.15.67:0/1513947726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c156c60 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 -- 172.21.15.67:0/1513947726 >> 172.21.15.67:0/1513947726 conn(0x7ff11c068760 msgr2=0x7ff11c068b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 -- 172.21.15.67:0/1513947726 shutdown_connections 2024-04-25T15:54:51.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.473+0000 7ff120d5a640 1 -- 172.21.15.67:0/1513947726 wait complete. 2024-04-25T15:54:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 Processor -- start 2024-04-25T15:54:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 -- start start 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c14b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c15b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff11c0d2540 0x7ff11c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c15c8b0 con 0x7ff11c0c6c10 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c0c7ce0 con 0x7ff11c0d2540 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c1f6770 con 0x7ff11c0c9bc0 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c14b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c14b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55036/0 (socket says 172.21.15.67:55036) 2024-04-25T15:54:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 -- 172.21.15.67:0/3152073704 learned_addr learned my addr 172.21.15.67:0/3152073704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff119d9b640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c15b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 -- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 msgr2=0x7ff11c15b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c15b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 -- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff11c0d2540 msgr2=0x7ff11c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff11c0d2540 0x7ff11c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 -- 172.21.15.67:0/3152073704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c15c470 con 0x7ff11c0c6c10 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff11a59c640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c14b8b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff11001f5b0 tx=0x7ff11001bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff10affd640 1 -- 172.21.15.67:0/3152073704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1100b5070 con 0x7ff11c0c6c10 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff10affd640 1 -- 172.21.15.67:0/3152073704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1100b0a70 con 0x7ff11c0c6c10 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c0d1780 con 0x7ff11c0c6c10 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff10affd640 1 -- 172.21.15.67:0/3152073704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1100b0e60 con 0x7ff11c0c6c10 2024-04-25T15:54:51.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1f3100 con 0x7ff11c0c6c10 2024-04-25T15:54:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.477+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c06a480 con 0x7ff11c0c6c10 2024-04-25T15:54:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.481+0000 7ff10affd640 1 -- 172.21.15.67:0/3152073704 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff1100c2060 con 0x7ff11c0c6c10 2024-04-25T15:54:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.481+0000 7ff10affd640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0fc06f430 0x7ff0fc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.481+0000 7ff10affd640 1 -- 172.21.15.67:0/3152073704 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 6160+0+0 (secure 0 0 0) 0x7ff11012c640 con 0x7ff11c0c6c10 2024-04-25T15:54:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.481+0000 7ff119d9b640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0fc06f430 0x7ff0fc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:51.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.481+0000 7ff119d9b640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0fc06f430 0x7ff0fc071940 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0cb6c0 tx=0x7ff10c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.485+0000 7ff10affd640 1 -- 172.21.15.67:0/3152073704 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff110018870 con 0x7ff11c0c6c10 2024-04-25T15:54:51.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:51.649+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7ff11c0c8da0 con 0x7ff11c0c6c10 2024-04-25T15:54:52.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.281+0000 7ff10affd640 1 -- 172.21.15.67:0/3152073704 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v127) ==== 113+0+0 (secure 0 0 0) 0x7ff1100f92f0 con 0x7ff11c0c6c10 2024-04-25T15:54:52.283 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-04-25T15:54:52.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0fc06f430 msgr2=0x7ff0fc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:52.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0fc06f430 0x7ff0fc071940 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0cb6c0 tx=0x7ff10c008040 comp rx=0 tx=0).stop 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 msgr2=0x7ff11c14b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c14b8b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff11001f5b0 tx=0x7ff11001bab0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 shutdown_connections 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff0fc06f430 0x7ff0fc071940 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff11c0d2540 0x7ff11c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff11c0c9bc0 0x7ff11c15b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 --2- 172.21.15.67:0/3152073704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff11c0c6c10 0x7ff11c14b8b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 >> 172.21.15.67:0/3152073704 conn(0x7ff11c068760 msgr2=0x7ff11c153010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 shutdown_connections 2024-04-25T15:54:52.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.285+0000 7ff120d5a640 1 -- 172.21.15.67:0/3152073704 wait complete. 2024-04-25T15:54:52.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-25T15:54:52.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f442492c640 1 Processor -- start 2024-04-25T15:54:52.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f442492c640 1 -- start start 2024-04-25T15:54:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f442492c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44200b5d70 0x7f442006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:52.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f442492c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442006d780 con 0x7f442015fe00 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f442492c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4420160920 con 0x7f442006ca60 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f442492c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f442015d510 con 0x7f44200b5d70 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f4415d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44200b5d70 0x7f442006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f441ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f442015fe00 0x7f4420160210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60294/0 (socket says 172.21.15.67:60294) 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f441ed9d640 1 -- 172.21.15.67:0/1552534800 learned_addr learned my addr 172.21.15.67:0/1552534800 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f4415d9b640 1 -- 172.21.15.67:0/1552534800 >> v1:172.21.15.176:6789/0 conn(0x7f442006ca60 legacy=0x7f44200b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f4415d9b640 1 -- 172.21.15.67:0/1552534800 >> v1:172.21.15.67:6789/0 conn(0x7f442015fe00 legacy=0x7f4420160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f441dd9b640 1 -- 172.21.15.67:0/1552534800 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902144857 0 0) 0x7f442006d780 con 0x7f442015fe00 2024-04-25T15:54:52.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.481+0000 7f4415d9b640 1 -- 172.21.15.67:0/1552534800 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4420160be0 con 0x7f44200b5d70 2024-04-25T15:54:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f4415d9b640 1 --2- 172.21.15.67:0/1552534800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f442006ce70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f44080097f0 tx=0x7f440801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaaaa39b270e580e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f441dd9b640 1 -- 172.21.15.67:0/1552534800 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080b1070 con 0x7f44200b5d70 2024-04-25T15:54:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f441dd9b640 1 -- 172.21.15.67:0/1552534800 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44080ac5f0 con 0x7f44200b5d70 2024-04-25T15:54:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f441dd9b640 1 -- 172.21.15.67:0/1552534800 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080ac9c0 con 0x7f44200b5d70 2024-04-25T15:54:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f442492c640 1 -- 172.21.15.67:0/1552534800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 msgr2=0x7f442006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f442492c640 1 --2- 172.21.15.67:0/1552534800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f442006ce70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f44080097f0 tx=0x7f440801c4a0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f442492c640 1 -- 172.21.15.67:0/1552534800 shutdown_connections 2024-04-25T15:54:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f442492c640 1 --2- 172.21.15.67:0/1552534800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f442006ce70 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f442492c640 1 -- 172.21.15.67:0/1552534800 >> 172.21.15.67:0/1552534800 conn(0x7f442006e420 msgr2=0x7f442006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f442492c640 1 -- 172.21.15.67:0/1552534800 shutdown_connections 2024-04-25T15:54:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.485+0000 7f442492c640 1 -- 172.21.15.67:0/1552534800 wait complete. 2024-04-25T15:54:52.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 Processor -- start 2024-04-25T15:54:52.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 -- start start 2024-04-25T15:54:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442006ca60 0x7f4420158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f4420157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442015fe00 0x7f4420152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f44200b6e90 con 0x7f44200b5d70 2024-04-25T15:54:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f442015c890 con 0x7f442006ca60 2024-04-25T15:54:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44200b7010 con 0x7f442015fe00 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f4420157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f4420157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55052/0 (socket says 172.21.15.67:55052) 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 -- 172.21.15.67:0/3186359398 learned_addr learned my addr 172.21.15.67:0/3186359398 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f441ed9d640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442015fe00 0x7f4420152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f441e59c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442006ca60 0x7f4420158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 -- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442015fe00 msgr2=0x7f4420152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442015fe00 0x7f4420152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 -- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442006ca60 msgr2=0x7f4420158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442006ca60 0x7f4420158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 -- 172.21.15.67:0/3186359398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44201581f0 con 0x7f44200b5d70 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f441ed9d640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442015fe00 0x7f4420152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f4415d9b640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f4420157cb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f44080af040 tx=0x7f440801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f44177fe640 1 -- 172.21.15.67:0/3186359398 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44080b1050 con 0x7f44200b5d70 2024-04-25T15:54:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f44177fe640 1 -- 172.21.15.67:0/3186359398 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44080b2040 con 0x7f44200b5d70 2024-04-25T15:54:52.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f44177fe640 1 -- 172.21.15.67:0/3186359398 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408004260 con 0x7f44200b5d70 2024-04-25T15:54:52.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4420159fd0 con 0x7f44200b5d70 2024-04-25T15:54:52.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4420159cf0 con 0x7f44200b5d70 2024-04-25T15:54:52.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.489+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442006a480 con 0x7f44200b5d70 2024-04-25T15:54:52.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.493+0000 7f44177fe640 1 -- 172.21.15.67:0/3186359398 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f44080bf060 con 0x7f44200b5d70 2024-04-25T15:54:52.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.493+0000 7f44177fe640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f440006f400 0x7f4400071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:52.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.493+0000 7f44177fe640 1 -- 172.21.15.67:0/3186359398 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 6160+0+0 (secure 0 0 0) 0x7f4408128bb0 con 0x7f44200b5d70 2024-04-25T15:54:52.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.493+0000 7f441e59c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f440006f400 0x7f4400071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:52.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.493+0000 7f441e59c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f440006f400 0x7f4400071910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4410000f70 tx=0x7f4410009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:52.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.497+0000 7f44177fe640 1 -- 172.21.15.67:0/3186359398 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f44080a9070 con 0x7f44200b5d70 2024-04-25T15:54:52.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:52.689+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) -- 0x7f44200cd380 con 0x7f44200b5d70 2024-04-25T15:54:53.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.317+0000 7f44177fe640 1 -- 172.21.15.67:0/3186359398 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v128) ==== 133+0+0 (secure 0 0 0) 0x7f4408016090 con 0x7f44200b5d70 2024-04-25T15:54:53.319 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f440006f400 msgr2=0x7f4400071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f440006f400 0x7f4400071910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4410000f70 tx=0x7f4410009040 comp rx=0 tx=0).stop 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 msgr2=0x7f4420157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f4420157cb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f44080af040 tx=0x7f440801ace0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 shutdown_connections 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f440006f400 0x7f4400071910 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442015fe00 0x7f4420152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44200b5d70 0x7f4420157cb0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 --2- 172.21.15.67:0/3186359398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442006ca60 0x7f4420158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 >> 172.21.15.67:0/3186359398 conn(0x7f442006e420 msgr2=0x7f44200d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:53.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 shutdown_connections 2024-04-25T15:54:53.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.321+0000 7f442492c640 1 -- 172.21.15.67:0/3186359398 wait complete. 2024-04-25T15:54:53.359 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-25T15:54:53.359 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-25T15:54:53.360 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-25T15:54:53.360 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-25T15:54:53.360 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T15:54:53.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:54:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f16788640 1 Processor -- start 2024-04-25T15:54:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f16788640 1 -- start start 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f16788640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f100c7610 0x7f0f100c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f16788640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f100c5bb0 0x7f0f100c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f16788640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f100c6890 con 0x7f0f10071b20 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f16788640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0f1015cd40 con 0x7f0f100c7610 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f16788640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0f100c6b70 con 0x7f0f100c5bb0 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f100c7610 0x7f0f100c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0f10071b20 0x7f0f1015d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43214/0 (socket says 172.21.15.67:43214) 2024-04-25T15:54:53.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0ffff640 1 -- 172.21.15.67:0/238075218 learned_addr learned my addr 172.21.15.67:0/238075218 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0f7fe640 1 -- 172.21.15.67:0/238075218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f100c5bb0 msgr2=0x7f0f100c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f14d25640 1 --2- 172.21.15.67:0/238075218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f100c5bb0 0x7f0f100c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0effd640 1 -- 172.21.15.67:0/238075218 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3074586603 0 0) 0x7f0f100c6890 con 0x7f0f10071b20 2024-04-25T15:54:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0f7fe640 1 --2- 172.21.15.67:0/238075218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f100c5bb0 0x7f0f100c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0f7fe640 1 -- 172.21.15.67:0/238075218 >> v1:172.21.15.67:6789/0 conn(0x7f0f10071b20 legacy=0x7f0f1015d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f0f7fe640 1 -- 172.21.15.67:0/238075218 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f100c6e50 con 0x7f0f100c7610 2024-04-25T15:54:53.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.513+0000 7f0f14d25640 1 --2- 172.21.15.67:0/238075218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f100c5bb0 0x7f0f100c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:53.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f0f7fe640 1 --2- 172.21.15.67:0/238075218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c7610 0x7f0f100c7a20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400ba60 tx=0x7f0f0401eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedc05f8a2aaa3e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:53.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f0effd640 1 -- 172.21.15.67:0/238075218 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f040b4070 con 0x7f0f100c7610 2024-04-25T15:54:53.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f0effd640 1 -- 172.21.15.67:0/238075218 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f040a7a50 con 0x7f0f100c7610 2024-04-25T15:54:53.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f0effd640 1 -- 172.21.15.67:0/238075218 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f040a7e20 con 0x7f0f100c7610 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 -- 172.21.15.67:0/238075218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c7610 msgr2=0x7f0f100c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 --2- 172.21.15.67:0/238075218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c7610 0x7f0f100c7a20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400ba60 tx=0x7f0f0401eb70 comp rx=0 tx=0).stop 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 -- 172.21.15.67:0/238075218 shutdown_connections 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 --2- 172.21.15.67:0/238075218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f100c5bb0 0x7f0f100c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 --2- 172.21.15.67:0/238075218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c7610 0x7f0f100c7a20 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 -- 172.21.15.67:0/238075218 >> 172.21.15.67:0/238075218 conn(0x7f0f1006fe70 msgr2=0x7f0f100cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 -- 172.21.15.67:0/238075218 shutdown_connections 2024-04-25T15:54:53.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.517+0000 7f0f16788640 1 -- 172.21.15.67:0/238075218 wait complete. 2024-04-25T15:54:53.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 Processor -- start 2024-04-25T15:54:53.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 -- start start 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f10071b20 0x7f0f10142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c5bb0 0x7f0f1013d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f100c7610 0x7f0f10141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1006ce50 con 0x7f0f100c5bb0 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0f100734f0 con 0x7f0f100c7610 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1006cfd0 con 0x7f0f10071b20 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c5bb0 0x7f0f1013d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c5bb0 0x7f0f1013d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49370/0 (socket says 172.21.15.67:49370) 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 -- 172.21.15.67:0/2630814068 learned_addr learned my addr 172.21.15.67:0/2630814068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0ffff640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f10071b20 0x7f0f10142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 -- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f10071b20 msgr2=0x7f0f10142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f10071b20 0x7f0f10142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f14d25640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f100c7610 0x7f0f10141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 -- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f100c7610 msgr2=0x7f0f10141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f100c7610 0x7f0f10141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 -- 172.21.15.67:0/2630814068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f101490e0 con 0x7f0f100c5bb0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0ffff640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f10071b20 0x7f0f10142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0f7fe640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c5bb0 0x7f0f1013d440 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0f0401c7a0 tx=0x7f0f0401b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0cff9640 1 -- 172.21.15.67:0/2630814068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f040b4070 con 0x7f0f100c5bb0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0cff9640 1 -- 172.21.15.67:0/2630814068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f040bf070 con 0x7f0f100c5bb0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f0cff9640 1 -- 172.21.15.67:0/2630814068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f040b9850 con 0x7f0f100c5bb0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f10155850 con 0x7f0f100c5bb0 2024-04-25T15:54:53.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.521+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f10155e10 con 0x7f0f100c5bb0 2024-04-25T15:54:53.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.525+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f1006a480 con 0x7f0f100c5bb0 2024-04-25T15:54:53.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.525+0000 7f0f0cff9640 1 -- 172.21.15.67:0/2630814068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0f040b99f0 con 0x7f0f100c5bb0 2024-04-25T15:54:53.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.525+0000 7f0f0cff9640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0efc06f400 0x7f0efc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.525+0000 7f0f0cff9640 1 -- 172.21.15.67:0/2630814068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6031+0+0 (secure 0 0 0) 0x7f0f0412c650 con 0x7f0f100c5bb0 2024-04-25T15:54:53.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.525+0000 7f0f0ffff640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0efc06f400 0x7f0efc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.525+0000 7f0f0ffff640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0efc06f400 0x7f0efc071910 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007fd0 tx=0x7f0f00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:53.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.529+0000 7f0f0cff9640 1 -- 172.21.15.67:0/2630814068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0f04018760 con 0x7f0f100c5bb0 2024-04-25T15:54:53.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.693+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f100c4700 con 0x7f0f100c5bb0 2024-04-25T15:54:53.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.697+0000 7f0f0cff9640 1 -- 172.21.15.67:0/2630814068 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+17051 (secure 0 0 0) 0x7f0f040f9490 con 0x7f0f100c5bb0 2024-04-25T15:54:53.699 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:53.699 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":128,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:54:53.312253+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:53:14.990625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.a","osds":[2,6]},{"pgid":"3.18","osds":[1,4]},{"pgid":"3.27","osds":[7,6]}],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":5}]},{"pgid":"3.22","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3e","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0efc06f400 msgr2=0x7f0efc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0efc06f400 0x7f0efc071910 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007fd0 tx=0x7f0f00009040 comp rx=0 tx=0).stop 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c5bb0 msgr2=0x7f0f1013d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c5bb0 0x7f0f1013d440 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0f0401c7a0 tx=0x7f0f0401b910 comp rx=0 tx=0).stop 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 shutdown_connections 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0efc06f400 0x7f0efc071910 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f100c7610 0x7f0f10141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f100c5bb0 0x7f0f1013d440 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 --2- 172.21.15.67:0/2630814068 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f10071b20 0x7f0f10142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 >> 172.21.15.67:0/2630814068 conn(0x7f0f1006fe70 msgr2=0x7f0f100c8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:53.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 shutdown_connections 2024-04-25T15:54:53.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.701+0000 7f0f16788640 1 -- 172.21.15.67:0/2630814068 wait complete. 2024-04-25T15:54:53.724 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T15:54:53.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:54:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd65b44d640 1 Processor -- start 2024-04-25T15:54:53.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd65b44d640 1 -- start start 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd65b44d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 0x7fd6540c9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd65b44d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd654074740 0x7fd654160670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd65b44d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65406cca0 con 0x7fd65406cf70 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd65b44d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd654073520 con 0x7fd654074740 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd65b44d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd654074b50 con 0x7fd6540c8b80 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd6599ea640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd65406cf70 0x7fd65415c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43220/0 (socket says 172.21.15.67:43220) 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd6599ea640 1 -- 172.21.15.67:0/1752865431 learned_addr learned my addr 172.21.15.67:0/1752865431 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd6591e9640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 0x7fd6540c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd6589e8640 1 --2- 172.21.15.67:0/1752865431 >> v2:172.21.15.176:3300/0 conn(0x7fd654074740 0x7fd654160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110491949 0 0) 0x7fd65406cca0 con 0x7fd65406cf70 2024-04-25T15:54:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd63c003530 con 0x7fd65406cf70 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 734958506 0 0) 0x7fd63c003530 con 0x7fd65406cf70 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 msgr2=0x7fd6540c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 0x7fd6540c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 >> v2:172.21.15.176:3300/0 conn(0x7fd654074740 msgr2=0x7fd654160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 --2- 172.21.15.67:0/1752865431 >> v2:172.21.15.176:3300/0 conn(0x7fd654074740 0x7fd654160670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65406dc90 con 0x7fd65406cf70 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fd64c001680 con 0x7fd65406cf70 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 >> v1:172.21.15.67:6789/0 conn(0x7fd65406cf70 legacy=0x7fd65415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd63c0039c0 0x7fd63c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c004810 0x7fd63c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 0x7fd63c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd654073520 con 0x7fd63c0039c0 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd654074b50 con 0x7fd63c004810 2024-04-25T15:54:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.873+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd63c00b9d0 con 0x7fd6540c8b80 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6591e9640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd63c0039c0 0x7fd63c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6589e8640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c004810 0x7fd63c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6599ea640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 0x7fd63c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6591e9640 1 -- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 msgr2=0x7fd63c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6591e9640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6540c8b80 0x7fd63c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6591e9640 1 -- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c004810 msgr2=0x7fd63c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6591e9640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c004810 0x7fd63c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6591e9640 1 -- 172.21.15.67:0/1752865431 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c003530 con 0x7fd63c0039c0 2024-04-25T15:54:53.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6591e9640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd63c0039c0 0x7fd63c003e30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd644009fc0 tx=0x7fd64401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1588d3614025f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:53.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6440b2070 con 0x7fd63c0039c0 2024-04-25T15:54:53.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6440a5ba0 con 0x7fd63c0039c0 2024-04-25T15:54:53.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd653fff640 1 -- 172.21.15.67:0/1752865431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644009c10 con 0x7fd63c0039c0 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd65b44d640 1 -- 172.21.15.67:0/1752865431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd63c0039c0 msgr2=0x7fd63c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd65b44d640 1 --2- 172.21.15.67:0/1752865431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd63c0039c0 0x7fd63c003e30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd644009fc0 tx=0x7fd64401cca0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd6599ea640 1 -- 172.21.15.67:0/1752865431 reap_dead start 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd65b44d640 1 -- 172.21.15.67:0/1752865431 shutdown_connections 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd65b44d640 1 -- 172.21.15.67:0/1752865431 >> 172.21.15.67:0/1752865431 conn(0x7fd654070e70 msgr2=0x7fd6540ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd65b44d640 1 -- 172.21.15.67:0/1752865431 shutdown_connections 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.877+0000 7fd65b44d640 1 -- 172.21.15.67:0/1752865431 wait complete. 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 Processor -- start 2024-04-25T15:54:53.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 -- start start 2024-04-25T15:54:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c0039c0 0x7fd6541f9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6541ea8f0 0x7fd654154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd654153450 0x7fd654153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd65406d790 con 0x7fd654153450 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd65406c9c0 con 0x7fd63c0039c0 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd654072c30 con 0x7fd6541ea8f0 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd654153450 0x7fd654153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd654153450 0x7fd654153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49388/0 (socket says 172.21.15.67:49388) 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 -- 172.21.15.67:0/2498869648 learned_addr learned my addr 172.21.15.67:0/2498869648 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6591e9640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c0039c0 0x7fd6541f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6589e8640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6541ea8f0 0x7fd654154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 -- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6541ea8f0 msgr2=0x7fd654154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6541ea8f0 0x7fd654154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 -- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c0039c0 msgr2=0x7fd6541f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c0039c0 0x7fd6541f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 -- 172.21.15.67:0/2498869648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6541f9770 con 0x7fd654153450 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6591e9640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c0039c0 0x7fd6541f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd6599ea640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd654153450 0x7fd654153860 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd6540cb200 tx=0x7fd64c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd651ffb640 1 -- 172.21.15.67:0/2498869648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c019070 con 0x7fd654153450 2024-04-25T15:54:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd651ffb640 1 -- 172.21.15.67:0/2498869648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64c00c690 con 0x7fd654153450 2024-04-25T15:54:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd654153da0 con 0x7fd654153450 2024-04-25T15:54:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.881+0000 7fd651ffb640 1 -- 172.21.15.67:0/2498869648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c00ca80 con 0x7fd654153450 2024-04-25T15:54:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.885+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6541f62d0 con 0x7fd654153450 2024-04-25T15:54:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.885+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd65406a480 con 0x7fd654153450 2024-04-25T15:54:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.885+0000 7fd651ffb640 1 -- 172.21.15.67:0/2498869648 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd64c01c070 con 0x7fd654153450 2024-04-25T15:54:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.885+0000 7fd651ffb640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd64006f400 0x7fd640071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.885+0000 7fd651ffb640 1 -- 172.21.15.67:0/2498869648 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6031+0+0 (secure 0 0 0) 0x7fd64c08ea90 con 0x7fd654153450 2024-04-25T15:54:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.885+0000 7fd6591e9640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd64006f400 0x7fd640071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.885+0000 7fd6591e9640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd64006f400 0x7fd640071910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd6440af000 tx=0x7fd6440a7040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:53.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:53.889+0000 7fd651ffb640 1 -- 172.21.15.67:0/2498869648 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd64c05b8d0 con 0x7fd654153450 2024-04-25T15:54:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.057+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd654073300 con 0x7fd654153450 2024-04-25T15:54:54.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd651ffb640 1 -- 172.21.15.67:0/2498869648 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+17051 (secure 0 0 0) 0x7fd64c05b1d0 con 0x7fd654153450 2024-04-25T15:54:54.061 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:54.061 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":128,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:54:53.312253+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:53:14.990625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.a","osds":[2,6]},{"pgid":"3.18","osds":[1,4]},{"pgid":"3.27","osds":[7,6]}],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":5}]},{"pgid":"3.22","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3e","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd64006f400 msgr2=0x7fd640071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd64006f400 0x7fd640071910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd6440af000 tx=0x7fd6440a7040 comp rx=0 tx=0).stop 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd654153450 msgr2=0x7fd654153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd654153450 0x7fd654153860 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd6540cb200 tx=0x7fd64c00eb90 comp rx=0 tx=0).stop 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 shutdown_connections 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd64006f400 0x7fd640071910 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd654153450 0x7fd654153860 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6541ea8f0 0x7fd654154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 --2- 172.21.15.67:0/2498869648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd63c0039c0 0x7fd6541f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 >> 172.21.15.67:0/2498869648 conn(0x7fd654070e70 msgr2=0x7fd654160350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 shutdown_connections 2024-04-25T15:54:54.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.061+0000 7fd65b44d640 1 -- 172.21.15.67:0/2498869648 wait complete. 2024-04-25T15:54:54.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T15:54:54.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T15:54:54.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T15:54:54.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T15:54:54.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T15:54:54.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T15:54:54.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T15:54:54.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T15:54:54.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.305+0000 7f4439e29640 1 Processor -- start 2024-04-25T15:54:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.305+0000 7f4439e29640 1 -- start start 2024-04-25T15:54:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4439e29640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c046ad0 0x7f442c047a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4439e29640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f442c0e4070 0x7f442c0e6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4439e29640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442c0d5df0 con 0x7f442c0dc110 2024-04-25T15:54:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4439e29640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f442c046070 con 0x7f442c0e4070 2024-04-25T15:54:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4439e29640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f442c04d200 con 0x7f442c046ad0 2024-04-25T15:54:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 Processor -- start 2024-04-25T15:54:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 -- start start 2024-04-25T15:54:54.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f44337fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c046ad0 0x7f442c047a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4433fff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f442c0e4070 0x7f442c0e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f44337fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c046ad0 0x7f442c047a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50374/0 (socket says 172.21.15.67:50374) 2024-04-25T15:54:54.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f44337fe640 1 -- 172.21.15.67:0/456277327 learned_addr learned my addr 172.21.15.67:0/456277327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf1c153a00 0x7faf1c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c0725c0 0x7faf1c15a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7faf1c073f90 0x7faf1c0743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7faf1c153e10 con 0x7faf1c153a00 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7faf1c15a5f0 con 0x7faf1c073f90 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf217c1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faf1c068c40 con 0x7faf1c0725c0 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf1affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf1c153a00 0x7faf1c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf1affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faf1c153a00 0x7faf1c0c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49400/0 (socket says 172.21.15.67:49400) 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf1affd640 1 -- 172.21.15.67:0/4026715857 learned_addr learned my addr 172.21.15.67:0/4026715857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf1b7fe640 1 --2- 172.21.15.67:0/4026715857 >> v2:172.21.15.176:3300/0 conn(0x7faf1c073f90 0x7faf1c0743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7faf1a7fc640 1 --2- 172.21.15.67:0/4026715857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c0725c0 0x7faf1c15a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4433fff640 1 -- 172.21.15.67:0/456277327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c046ad0 msgr2=0x7f442c047a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4433fff640 1 --2- 172.21.15.67:0/456277327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c046ad0 0x7f442c047a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4433fff640 1 -- 172.21.15.67:0/456277327 >> v1:172.21.15.176:6789/0 conn(0x7f442c0dc110 legacy=0x7f442c04c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f4433fff640 1 -- 172.21.15.67:0/456277327 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c046690 con 0x7f442c0e4070 2024-04-25T15:54:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.309+0000 7f8abd9cd640 1 Processor -- start 2024-04-25T15:54:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8abd9cd640 1 -- start start 2024-04-25T15:54:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f4433fff640 1 --2- 172.21.15.67:0/456277327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 0x7f442c0e6490 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f442c0d5fc0 tx=0x7f443409c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c21436d171e822fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f44327fc640 1 -- 172.21.15.67:0/456277327 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4434131070 con 0x7f442c0e4070 2024-04-25T15:54:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f44327fc640 1 -- 172.21.15.67:0/456277327 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f443412c600 con 0x7f442c0e4070 2024-04-25T15:54:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f44327fc640 1 -- 172.21.15.67:0/456277327 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443412ca30 con 0x7f442c0e4070 2024-04-25T15:54:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf1affd640 1 -- 172.21.15.67:0/4026715857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c0725c0 msgr2=0x7faf1c15a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf1affd640 1 --2- 172.21.15.67:0/4026715857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c0725c0 0x7faf1c15a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf1affd640 1 -- 172.21.15.67:0/4026715857 >> v2:172.21.15.176:3300/0 conn(0x7faf1c073f90 msgr2=0x7faf1c0743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf1affd640 1 --2- 172.21.15.67:0/4026715857 >> v2:172.21.15.176:3300/0 conn(0x7faf1c073f90 0x7faf1c0743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf1affd640 1 -- 172.21.15.67:0/4026715857 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c0c3bc0 con 0x7faf1c153a00 2024-04-25T15:54:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8abd9cd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ab8160c20 0x7f8ab8161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8abd9cd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8ab806c640 0x7f8ab806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8abd9cd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab8164ff0 con 0x7f8ab815e180 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8abd9cd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8ab8164b60 con 0x7f8ab806c640 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8abd9cd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ab80c5d90 con 0x7f8ab8160c20 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf1affd640 1 --2- 172.21.15.67:0/4026715857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c153a00 0x7faf1c0c4010 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7faf1c073360 tx=0x7faf1001eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e9434bb4df6826 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf19ffb640 1 -- 172.21.15.67:0/4026715857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf100b4070 con 0x7faf1c153a00 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf19ffb640 1 -- 172.21.15.67:0/4026715857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf100a7a10 con 0x7faf1c153a00 2024-04-25T15:54:54.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7faf19ffb640 1 -- 172.21.15.67:0/4026715857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf100a7e20 con 0x7faf1c153a00 2024-04-25T15:54:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8ab6ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8ab815e180 0x7f8ab80c4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43230/0 (socket says 172.21.15.67:43230) 2024-04-25T15:54:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8ab6ffd640 1 -- 172.21.15.67:0/500742986 learned_addr learned my addr 172.21.15.67:0/500742986 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f4439e29640 1 -- 172.21.15.67:0/456277327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 msgr2=0x7f442c0e6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f4439e29640 1 --2- 172.21.15.67:0/456277327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 0x7f442c0e6490 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f442c0d5fc0 tx=0x7f443409c650 comp rx=0 tx=0).stop 2024-04-25T15:54:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3078831458 0 0) 0x7f8ab8164ff0 con 0x7f8ab815e180 2024-04-25T15:54:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.313+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8aa4003530 con 0x7f8ab815e180 2024-04-25T15:54:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 538243535 0 0) 0x7f8aa4003530 con 0x7f8ab815e180 2024-04-25T15:54:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ab8160c20 msgr2=0x7f8ab8161030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:54:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ab8160c20 0x7f8ab8161030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 -- 172.21.15.67:0/4026715857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c153a00 msgr2=0x7faf1c0c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 --2- 172.21.15.67:0/4026715857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c153a00 0x7faf1c0c4010 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7faf1c073360 tx=0x7faf1001eb80 comp rx=0 tx=0).stop 2024-04-25T15:54:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 >> v2:172.21.15.176:3300/0 conn(0x7f8ab806c640 msgr2=0x7f8ab806ca50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:54:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 --2- 172.21.15.67:0/500742986 >> v2:172.21.15.176:3300/0 conn(0x7f8ab806c640 0x7f8ab806ca50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 -- 172.21.15.67:0/4026715857 shutdown_connections 2024-04-25T15:54:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 --2- 172.21.15.67:0/4026715857 >> v2:172.21.15.176:3300/0 conn(0x7faf1c073f90 0x7faf1c0743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 --2- 172.21.15.67:0/4026715857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c0725c0 0x7faf1c15a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 --2- 172.21.15.67:0/4026715857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c153a00 0x7faf1c0c4010 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 -- 172.21.15.67:0/4026715857 >> 172.21.15.67:0/4026715857 conn(0x7faf1c0ce1c0 msgr2=0x7faf1c0ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f742d790640 1 Processor -- start 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f742d790640 1 -- start start 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 -- 172.21.15.67:0/456277327 shutdown_connections 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 --2- 172.21.15.67:0/456277327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 0x7f442c0e6490 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 --2- 172.21.15.67:0/456277327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c046ad0 0x7f442c047a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 -- 172.21.15.67:0/456277327 >> 172.21.15.67:0/456277327 conn(0x7f442c050c50 msgr2=0x7f442c051060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 -- 172.21.15.67:0/456277327 shutdown_connections 2024-04-25T15:54:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab8164230 con 0x7f8ab815e180 2024-04-25T15:54:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f8aa0001680 con 0x7f8ab815e180 2024-04-25T15:54:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 >> v1:172.21.15.67:6789/0 conn(0x7f8ab815e180 legacy=0x7f8ab80c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aa40039c0 0x7f8aa4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 -- 172.21.15.67:0/4026715857 shutdown_connections 2024-04-25T15:54:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa4004810 0x7f8aa4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ab8160c20 0x7f8aa400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8aaffff640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa4004810 0x7f8aa4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab6ffd640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aa40039c0 0x7f8aa4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab8164b60 con 0x7f8aa4004810 2024-04-25T15:54:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab80c5d90 con 0x7f8aa40039c0 2024-04-25T15:54:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8718a66640 1 Processor -- start 2024-04-25T15:54:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa400bf10 con 0x7f8ab8160c20 2024-04-25T15:54:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8ab77fe640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ab8160c20 0x7f8aa400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8aaffff640 1 -- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ab8160c20 msgr2=0x7f8aa400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8aaffff640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ab8160c20 0x7f8aa400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8aaffff640 1 -- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aa40039c0 msgr2=0x7f8aa4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8aaffff640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8aa40039c0 0x7f8aa4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8aaffff640 1 -- 172.21.15.67:0/500742986 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa4003530 con 0x7f8aa4004810 2024-04-25T15:54:54.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7faf217c1640 1 -- 172.21.15.67:0/4026715857 wait complete. 2024-04-25T15:54:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 -- 172.21.15.67:0/456277327 wait complete. 2024-04-25T15:54:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 Processor -- start 2024-04-25T15:54:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f742d790640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f74200d72a0 0x7f7420034ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f742d790640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742004dd50 con 0x7f7420035410 2024-04-25T15:54:54.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f742d790640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74200dd7c0 con 0x7f7420008a70 2024-04-25T15:54:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f742d790640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f74200dd5a0 con 0x7f74200d72a0 2024-04-25T15:54:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8718a66640 1 -- start start 2024-04-25T15:54:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8718a66640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f87140c6330 0x7f87141588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f8718a66640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8714158e20 con 0x7f8714152df0 2024-04-25T15:54:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f8718a66640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87140c9ce0 con 0x7f87141529e0 2024-04-25T15:54:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f8718a66640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f87140c5240 con 0x7f87140c6330 2024-04-25T15:54:54.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f871259c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f87140c6330 0x7f87141588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.317+0000 7f4439e29640 1 -- start start 2024-04-25T15:54:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f8711d9b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f87141529e0 0x7f871406c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56844/0 (socket says 172.21.15.67:56844) 2024-04-25T15:54:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f8711d9b640 1 -- 172.21.15.67:0/1402633078 learned_addr learned my addr 172.21.15.67:0/1402633078 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f742cf8f640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7420008a70 0x7f7420008e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56832/0 (socket says 172.21.15.67:56832) 2024-04-25T15:54:54.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f742cf8f640 1 -- 172.21.15.67:0/1519356250 learned_addr learned my addr 172.21.15.67:0/1519356250 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7faf217c1640 1 Processor -- start 2024-04-25T15:54:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7faf217c1640 1 -- start start 2024-04-25T15:54:54.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f8aaffff640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa4004810 0x7f8aa4004c20 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80097e0 tx=0x7f8aa801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a981c2c8665547b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4439e29640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442c046ad0 0x7f442c0c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4439e29640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c0dc110 0x7f442c050230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4439e29640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 0x7f442c183260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442c046ad0 0x7f442c0c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442c046ad0 0x7f442c0c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49426/0 (socket says 172.21.15.67:49426) 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 -- 172.21.15.67:0/2505926361 learned_addr learned my addr 172.21.15.67:0/2505926361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f442c0dbbf0 con 0x7f442c046ad0 2024-04-25T15:54:54.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f442c0db910 con 0x7f442c0e4070 2024-04-25T15:54:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f442c033800 con 0x7f442c0dc110 2024-04-25T15:54:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f871259c640 1 -- 172.21.15.67:0/1402633078 >> v1:172.21.15.176:6789/0 conn(0x7f87141529e0 legacy=0x7f871406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f871259c640 1 -- 172.21.15.67:0/1402633078 >> v1:172.21.15.67:6789/0 conn(0x7f8714152df0 legacy=0x7f8714163180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f871259c640 1 -- 172.21.15.67:0/1402633078 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87140c9fc0 con 0x7f87140c6330 2024-04-25T15:54:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4432ffd640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c0dc110 0x7f442c050230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4433fff640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 0x7f442c183260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 -- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c0dc110 msgr2=0x7f442c050230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f442c0dc110 0x7f442c050230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 -- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 msgr2=0x7f442c183260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 0x7f442c183260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c0c88e0 con 0x7f442c046ad0 2024-04-25T15:54:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f4433fff640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f442c0e4070 0x7f442c183260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f44337fe640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442c046ad0 0x7f442c0c83a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f442000e8a0 tx=0x7f442000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7faf217c1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c0725c0 0x7faf1c1f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7faf1affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c0725c0 0x7faf1c1f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7faf1affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c0725c0 0x7faf1c1f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49438/0 (socket says 172.21.15.67:49438) 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7faf1affd640 1 -- 172.21.15.67:0/3160004509 learned_addr learned my addr 172.21.15.67:0/3160004509 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7faf217c1640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c073f90 0x7faf1c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf217c1640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf1c153a00 0x7faf1c14c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.321+0000 7f7427fff640 1 --2- 172.21.15.67:0/1519356250 >> v2:172.21.15.176:3300/0 conn(0x7f74200d72a0 0x7f7420034ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf217c1640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c0686c0 con 0x7faf1c0725c0 2024-04-25T15:54:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf217c1640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c0c53a0 con 0x7faf1c153a00 2024-04-25T15:54:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf217c1640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faf1c0729d0 con 0x7faf1c073f90 2024-04-25T15:54:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f1e6f640 1 Processor -- start 2024-04-25T15:54:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f1e6f640 1 -- start start 2024-04-25T15:54:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1a7fc640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c073f90 0x7faf1c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1b7fe640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf1c153a00 0x7faf1c14c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420017070 con 0x7f442c046ad0 2024-04-25T15:54:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1affd640 1 -- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c073f90 msgr2=0x7faf1c159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1affd640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faf1c073f90 0x7faf1c159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1affd640 1 -- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf1c153a00 msgr2=0x7faf1c14c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1affd640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faf1c153a00 0x7faf1c14c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1affd640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c0b2900 con 0x7faf1c0725c0 2024-04-25T15:54:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7ff042874640 1 Processor -- start 2024-04-25T15:54:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7ff042874640 1 -- start start 2024-04-25T15:54:54.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f871259c640 1 --2- 172.21.15.67:0/1402633078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87140c6330 0x7f87141588e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f870c009d90 tx=0x7f870c01ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d064529713468ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf1affd640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c0725c0 0x7faf1c1f6d90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7faf1000b2c0 tx=0x7faf1001b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f7427fff640 1 -- 172.21.15.67:0/1519356250 >> v1:172.21.15.176:6789/0 conn(0x7f7420008a70 legacy=0x7f7420008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f7427fff640 1 -- 172.21.15.67:0/1519356250 >> v1:172.21.15.67:6789/0 conn(0x7f7420035410 legacy=0x7f7420007450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f7427fff640 1 -- 172.21.15.67:0/1519356250 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74200ddaa0 con 0x7f74200d72a0 2024-04-25T15:54:54.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442c0c50f0 con 0x7f442c046ad0 2024-04-25T15:54:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442c0db0d0 con 0x7f442c046ad0 2024-04-25T15:54:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9b08a4640 1 Processor -- start 2024-04-25T15:54:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f442c0c53d0 con 0x7f442c046ad0 2024-04-25T15:54:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9b08a4640 1 -- start start 2024-04-25T15:54:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f442c0c53d0 con 0x7f442c046ad0 2024-04-25T15:54:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420021060 con 0x7f442c046ad0 2024-04-25T15:54:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f1e6f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f33e4034ff0 0x7f33e40dd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf100bf070 con 0x7faf1c0725c0 2024-04-25T15:54:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf100b6050 con 0x7faf1c0725c0 2024-04-25T15:54:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf100b4030 con 0x7faf1c0725c0 2024-04-25T15:54:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f1e6f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e40dda90 con 0x7f33e40093f0 2024-04-25T15:54:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f1e6f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e40ddd70 con 0x7f33e4036c50 2024-04-25T15:54:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f1e6f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f33e4008850 con 0x7f33e4034ff0 2024-04-25T15:54:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f166e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f33e40093f0 0x7f33e40e7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43274/0 (socket says 172.21.15.67:43274) 2024-04-25T15:54:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f166e640 1 -- 172.21.15.67:0/1189771054 learned_addr learned my addr 172.21.15.67:0/1189771054 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f0e6d640 1 --2- 172.21.15.67:0/1189771054 >> v2:172.21.15.176:3300/0 conn(0x7f33e4034ff0 0x7f33e40dd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa80b0070 con 0x7f8aa4004810 2024-04-25T15:54:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa80ab620 con 0x7f8aa4004810 2024-04-25T15:54:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f8ab67fc640 1 -- 172.21.15.67:0/500742986 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa80093b0 con 0x7f8aa4004810 2024-04-25T15:54:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f0e6d640 1 -- 172.21.15.67:0/1189771054 >> v1:172.21.15.176:6789/0 conn(0x7f33e4036c50 legacy=0x7f33e4008ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:54:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f0e6d640 1 -- 172.21.15.67:0/1189771054 >> v1:172.21.15.67:6789/0 conn(0x7f33e40093f0 legacy=0x7f33e40e7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7f33f0e6d640 1 -- 172.21.15.67:0/1189771054 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e40de050 con 0x7f33e4034ff0 2024-04-25T15:54:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9b08a4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9ac0cd940 0x7fc9ac0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf217c1640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf1c1f72d0 con 0x7faf1c0725c0 2024-04-25T15:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7faf217c1640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf1c158090 con 0x7faf1c0725c0 2024-04-25T15:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff042874640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff0340374b0 0x7ff0340d94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff042874640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff034050ac0 0x7ff03404ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff042874640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff034049690 con 0x7ff03404b260 2024-04-25T15:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff042874640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff0340dff90 con 0x7ff034050ac0 2024-04-25T15:54:54.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff042874640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff0340e0270 con 0x7ff0340374b0 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f7427fff640 1 --2- 172.21.15.67:0/1519356250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 0x7f7420034ed0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7428077670 tx=0x7f742809c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63b0af763b32ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9b08a4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc9ac0d1ae0 0x7fc9ac0d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9b08a4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9ac0cca60 con 0x7fc9ac156910 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9b08a4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc9ac0ccd00 con 0x7fc9ac0cd940 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9b08a4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc9ac06d050 con 0x7fc9ac0d1ae0 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9aa59c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc9ac156910 0x7fc9ac160380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56850/0 (socket says 172.21.15.67:56850) 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.325+0000 7fc9aa59c640 1 -- 172.21.15.67:0/2319735952 learned_addr learned my addr 172.21.15.67:0/2319735952 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7fc9ab59e640 1 --2- 172.21.15.67:0/2319735952 >> v2:172.21.15.176:3300/0 conn(0x7fc9ac0d1ae0 0x7fc9ac0d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7fc9aad9d640 1 --2- 172.21.15.67:0/2319735952 >> v2:172.21.15.67:3300/0 conn(0x7fc9ac0cd940 0x7fc9ac0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8abd9cd640 1 -- 172.21.15.67:0/500742986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa4004810 msgr2=0x7f8aa4004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8abd9cd640 1 --2- 172.21.15.67:0/500742986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa4004810 0x7f8aa4004c20 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80097e0 tx=0x7f8aa801c320 comp rx=0 tx=0).stop 2024-04-25T15:54:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8ab77fe640 1 -- 172.21.15.67:0/500742986 reap_dead start 2024-04-25T15:54:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041071640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff034050ac0 0x7ff03404ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041071640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff034050ac0 0x7ff03404ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49460/0 (socket says 172.21.15.67:49460) 2024-04-25T15:54:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041071640 1 -- 172.21.15.67:0/2884803090 learned_addr learned my addr 172.21.15.67:0/2884803090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041872640 1 --2- 172.21.15.67:0/2884803090 >> v2:172.21.15.176:3300/0 conn(0x7ff0340374b0 0x7ff0340d94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8abd9cd640 1 -- 172.21.15.67:0/500742986 shutdown_connections 2024-04-25T15:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8abd9cd640 1 -- 172.21.15.67:0/500742986 >> 172.21.15.67:0/500742986 conn(0x7f8ab806fe70 msgr2=0x7f8ab80cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8abd9cd640 1 -- 172.21.15.67:0/500742986 shutdown_connections 2024-04-25T15:54:54.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7fc9aad9d640 1 -- 172.21.15.67:0/2319735952 >> v2:172.21.15.176:3300/0 conn(0x7fc9ac0d1ae0 msgr2=0x7fc9ac0d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7fc9aad9d640 1 --2- 172.21.15.67:0/2319735952 >> v2:172.21.15.176:3300/0 conn(0x7fc9ac0d1ae0 0x7fc9ac0d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7fc9aad9d640 1 -- 172.21.15.67:0/2319735952 >> v1:172.21.15.176:6789/0 conn(0x7fc9ac156910 legacy=0x7fc9ac160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8abd9cd640 1 -- 172.21.15.67:0/500742986 wait complete. 2024-04-25T15:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041071640 1 -- 172.21.15.67:0/2884803090 >> v2:172.21.15.176:3300/0 conn(0x7ff0340374b0 msgr2=0x7ff0340d94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041071640 1 --2- 172.21.15.67:0/2884803090 >> v2:172.21.15.176:3300/0 conn(0x7ff0340374b0 0x7ff0340d94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041071640 1 -- 172.21.15.67:0/2884803090 >> v1:172.21.15.176:6789/0 conn(0x7ff03404b260 legacy=0x7ff0340073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7ff041071640 1 -- 172.21.15.67:0/2884803090 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03404a520 con 0x7ff034050ac0 2024-04-25T15:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f442001a070 con 0x7f442c046ad0 2024-04-25T15:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7fc9aad9d640 1 -- 172.21.15.67:0/2319735952 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ac06d330 con 0x7fc9ac0cd940 2024-04-25T15:54:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf1c06a480 con 0x7faf1c0725c0 2024-04-25T15:54:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f441ffff640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f441806f400 0x7f4418071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7faf100afa30 con 0x7faf1c0725c0 2024-04-25T15:54:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6031+0+0 (secure 0 0 0) 0x7f442008f420 con 0x7f442c046ad0 2024-04-25T15:54:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f441ffff640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4418077b30 0x7f441807a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4420016ec0 con 0x7f4418077b30 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=128) ==== 24+0+0 (secure 0 0 0) 0x7f442008f810 con 0x7f442c046ad0 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f4432ffd640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f441806f400 0x7f4418071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf037fe640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf0806f480 0x7faf08071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 6031+0+0 (secure 0 0 0) 0x7faf10128cf0 con 0x7faf1c0725c0 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf037fe640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf08077c30 0x7faf0807a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf100c5e50 con 0x7faf08077c30 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=128) ==== 24+0+0 (secure 0 0 0) 0x7faf1012b3f0 con 0x7faf1c0725c0 2024-04-25T15:54:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f4433fff640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4418077b30 0x7f441807a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f871159a640 1 -- 172.21.15.67:0/1402633078 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c0b4070 con 0x7f87140c6330 2024-04-25T15:54:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf1a7fc640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf0806f480 0x7faf08071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f7426ffd640 1 -- 172.21.15.67:0/1519356250 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428131070 con 0x7f74200d72a0 2024-04-25T15:54:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7faf1b7fe640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf08077c30 0x7faf0807a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f7426ffd640 1 -- 172.21.15.67:0/1519356250 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f742812c600 con 0x7f74200d72a0 2024-04-25T15:54:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f7426ffd640 1 -- 172.21.15.67:0/1519356250 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742812ca30 con 0x7f74200d72a0 2024-04-25T15:54:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f33f0e6d640 1 --2- 172.21.15.67:0/1189771054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 0x7f33e40dd170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f33e00097c0 tx=0x7f33e001ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=318757d9131425d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f871159a640 1 -- 172.21.15.67:0/1402633078 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870c0a7b50 con 0x7f87140c6330 2024-04-25T15:54:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f871159a640 1 -- 172.21.15.67:0/1402633078 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c0af440 con 0x7f87140c6330 2024-04-25T15:54:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f33eb7fe640 1 -- 172.21.15.67:0/1189771054 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e00b5070 con 0x7f33e4034ff0 2024-04-25T15:54:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f8abd9cd640 1 Processor -- start 2024-04-25T15:54:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f33eb7fe640 1 -- 172.21.15.67:0/1189771054 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e00a8950 con 0x7f33e4034ff0 2024-04-25T15:54:54.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f33eb7fe640 1 -- 172.21.15.67:0/1189771054 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e00a8d20 con 0x7f33e4034ff0 2024-04-25T15:54:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.329+0000 7f4433fff640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4418077b30 0x7f441807a020 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f4432ffd640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f441806f400 0x7f4418071910 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f442c0c45b0 tx=0x7f4428009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 -- 172.21.15.67:0/1402633078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87140c6330 msgr2=0x7f87141588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 --2- 172.21.15.67:0/1402633078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87140c6330 0x7f87141588e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f870c009d90 tx=0x7f870c01ebb0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 -- 172.21.15.67:0/1402633078 shutdown_connections 2024-04-25T15:54:54.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 --2- 172.21.15.67:0/1402633078 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87140c6330 0x7f87141588e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 -- 172.21.15.67:0/1402633078 >> 172.21.15.67:0/1402633078 conn(0x7f871406fe70 msgr2=0x7f87140cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 -- 172.21.15.67:0/1519356250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 msgr2=0x7f7420034ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 --2- 172.21.15.67:0/1519356250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 0x7f7420034ed0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7428077670 tx=0x7f742809c6a0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 -- 172.21.15.67:0/1402633078 shutdown_connections 2024-04-25T15:54:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 -- 172.21.15.67:0/1519356250 shutdown_connections 2024-04-25T15:54:54.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 --2- 172.21.15.67:0/1519356250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 0x7f7420034ed0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 -- 172.21.15.67:0/1519356250 >> 172.21.15.67:0/1519356250 conn(0x7f74200514d0 msgr2=0x7f74200518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4420016ec0 con 0x7f4418077b30 2024-04-25T15:54:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7ff041071640 1 --2- 172.21.15.67:0/2884803090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff034050ac0 0x7ff03404ad20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff034011840 tx=0x7ff02c01cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccf9380ec7cfbad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 -- 172.21.15.67:0/1519356250 shutdown_connections 2024-04-25T15:54:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 -- 172.21.15.67:0/1519356250 wait complete. 2024-04-25T15:54:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 -- 172.21.15.67:0/1189771054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 msgr2=0x7f33e40dd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 --2- 172.21.15.67:0/1189771054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 0x7f33e40dd170 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f33e00097c0 tx=0x7f33e001ec50 comp rx=0 tx=0).stop 2024-04-25T15:54:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7faf1b7fe640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf08077c30 0x7faf0807a120 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7faf1a7fc640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf0806f480 0x7faf08071990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faf1c157270 tx=0x7faf0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faf100c5e50 con 0x7faf08077c30 2024-04-25T15:54:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 -- 172.21.15.67:0/1189771054 shutdown_connections 2024-04-25T15:54:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 --2- 172.21.15.67:0/1189771054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 0x7f33e40dd170 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 -- 172.21.15.67:0/1189771054 >> 172.21.15.67:0/1189771054 conn(0x7f33e40514d0 msgr2=0x7f33e40518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 -- 172.21.15.67:0/1189771054 shutdown_connections 2024-04-25T15:54:54.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 -- 172.21.15.67:0/1189771054 wait complete. 2024-04-25T15:54:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 -- 172.21.15.67:0/1402633078 wait complete. 2024-04-25T15:54:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 Processor -- start 2024-04-25T15:54:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8abd9cd640 1 -- start start 2024-04-25T15:54:54.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8718a66640 1 -- start start 2024-04-25T15:54:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8abd9cd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aa4004810 0x7f8ab81552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8abd9cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8161d80 0x7f8ab8155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8abd9cd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ab80cb6a0 0x7f8ab81538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8abd9cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab80c5a40 con 0x7f8ab8161d80 2024-04-25T15:54:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8abd9cd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab81647f0 con 0x7f8ab80cb6a0 2024-04-25T15:54:54.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f8abd9cd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ab80c6070 con 0x7f8aa4004810 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 Processor -- start 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f33f1e6f640 1 -- start start 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8718a66640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87140c6330 0x7f8714154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8718a66640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87141529e0 0x7f8714153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8718a66640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714152df0 0x7f8714157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8718a66640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f871406ced0 con 0x7f8714152df0 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8718a66640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f871406cd50 con 0x7f87141529e0 2024-04-25T15:54:54.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8718a66640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f87140c4da0 con 0x7f87140c6330 2024-04-25T15:54:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8161d80 0x7f8ab8155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8161d80 0x7f8ab8155800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49462/0 (socket says 172.21.15.67:49462) 2024-04-25T15:54:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 -- 172.21.15.67:0/3784343722 learned_addr learned my addr 172.21.15.67:0/3784343722 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f871259c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87140c6330 0x7f8714154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8ab6ffd640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aa4004810 0x7f8ab81552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f871259c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87140c6330 0x7f8714154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50430/0 (socket says 172.21.15.67:50430) 2024-04-25T15:54:54.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f871259c640 1 -- 172.21.15.67:0/1051239076 learned_addr learned my addr 172.21.15.67:0/1051239076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8ab77fe640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ab80cb6a0 0x7f8ab81538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 -- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aa4004810 msgr2=0x7f8ab81552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8aa4004810 0x7f8ab81552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 -- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ab80cb6a0 msgr2=0x7f8ab81538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ab80cb6a0 0x7f8ab81538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 -- 172.21.15.67:0/3784343722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab81fa270 con 0x7f8ab8161d80 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8ab77fe640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ab80cb6a0 0x7f8ab81538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8712d9d640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714152df0 0x7f8714157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8aaffff640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8161d80 0x7f8ab8155800 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8aa801c150 tx=0x7f8aa4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8712d9d640 1 -- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87140c6330 msgr2=0x7f8714154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8712d9d640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87140c6330 0x7f8714154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8712d9d640 1 -- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87141529e0 msgr2=0x7f8714153250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:54:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8712d9d640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87141529e0 0x7f8714153250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8712d9d640 1 -- 172.21.15.67:0/1051239076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87141550f0 con 0x7f8714152df0 2024-04-25T15:54:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f1e6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e40093f0 0x7f33e40cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.333+0000 7f742d790640 1 Processor -- start 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742d790640 1 -- start start 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742d790640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7420008a70 0x7f74200d88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742d790640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7420035410 0x7f74200d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742d790640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 0x7f74200d1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742d790640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74200085e0 con 0x7f7420035410 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742d790640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f74200de060 con 0x7f74200d72a0 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742d790640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7420008270 con 0x7f7420008a70 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f7427fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7420008a70 0x7f74200d88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f7427fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7420008a70 0x7f74200d88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50446/0 (socket says 172.21.15.67:50446) 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f74277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7420035410 0x7f74200d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f7427fff640 1 -- 172.21.15.67:0/642262564 learned_addr learned my addr 172.21.15.67:0/642262564 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742cf8f640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 0x7f74200d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f74277fe640 1 -- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7420008a70 msgr2=0x7f74200d88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f74277fe640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7420008a70 0x7f74200d88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f74277fe640 1 -- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 msgr2=0x7f74200d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f74277fe640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 0x7f74200d1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f74277fe640 1 -- 172.21.15.67:0/642262564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74200d8df0 con 0x7f7420035410 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f742cf8f640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74200d72a0 0x7f74200d1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f74277fe640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7420035410 0x7f74200d0f20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c390 tx=0x7f741c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e40093f0 0x7f33e40cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f1e6f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 0x7f33e40dae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e40093f0 0x7f33e40cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49484/0 (socket says 172.21.15.67:49484) 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 -- 172.21.15.67:0/37623424 learned_addr learned my addr 172.21.15.67:0/37623424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f1e6f640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33e4036c50 0x7f33e40cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f33e40dd6b0 con 0x7f33e40093f0 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f33e40084e0 con 0x7f33e4034ff0 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f33e40dd830 con 0x7f33e4036c50 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33ebfff640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 0x7f33e40dae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f166e640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33e4036c50 0x7f33e40cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 -- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33e4036c50 msgr2=0x7f33e40cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33e4036c50 0x7f33e40cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 -- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 msgr2=0x7f33e40dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e4034ff0 0x7f33e40dae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e40d5340 con 0x7f33e40093f0 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f166e640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33e4036c50 0x7f33e40cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f33f0e6d640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e40093f0 0x7f33e40cf400 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f33e001b860 tx=0x7f33e001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7fc9aad9d640 1 --2- 172.21.15.67:0/2319735952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0cd940 0x7fc9ac0cdfa0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0b48f0 tx=0x7fc9a001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63ef88467637286 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.337+0000 7f8712d9d640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714152df0 0x7f8714157d90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f870800c390 tx=0x7f870800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7ff040870640 1 -- 172.21.15.67:0/2884803090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0ab5f0 con 0x7ff034050ac0 2024-04-25T15:54:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e00b5040 con 0x7f33e40093f0 2024-04-25T15:54:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9a9d9b640 1 -- 172.21.15.67:0/2319735952 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00b1070 con 0x7fc9ac0cd940 2024-04-25T15:54:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e40dd4b0 con 0x7f33e40093f0 2024-04-25T15:54:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e40d4d80 con 0x7f33e40093f0 2024-04-25T15:54:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f7424ff9640 1 -- 172.21.15.67:0/642262564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c017070 con 0x7f7420035410 2024-04-25T15:54:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f7424ff9640 1 -- 172.21.15.67:0/642262564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f741c0052d0 con 0x7f7420035410 2024-04-25T15:54:54.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f7424ff9640 1 -- 172.21.15.67:0/642262564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c003860 con 0x7f7420035410 2024-04-25T15:54:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa80b0040 con 0x7f8ab8161d80 2024-04-25T15:54:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab81fa550 con 0x7f8ab8161d80 2024-04-25T15:54:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ab81fab10 con 0x7f8ab8161d80 2024-04-25T15:54:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa80be070 con 0x7f8ab8161d80 2024-04-25T15:54:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f742d790640 1 -- 172.21.15.67:0/642262564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7420174370 con 0x7f7420035410 2024-04-25T15:54:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f742d790640 1 -- 172.21.15.67:0/642262564 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7420174610 con 0x7f7420035410 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9a9d9b640 1 -- 172.21.15.67:0/2319735952 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a00ac5f0 con 0x7fc9ac0cd940 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9a9d9b640 1 -- 172.21.15.67:0/2319735952 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00ac9c0 con 0x7fc9ac0cd940 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 -- 172.21.15.67:0/2319735952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0cd940 msgr2=0x7fc9ac0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 --2- 172.21.15.67:0/2319735952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0cd940 0x7fc9ac0cdfa0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0b48f0 tx=0x7fc9a001c4a0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 -- 172.21.15.67:0/2319735952 shutdown_connections 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 --2- 172.21.15.67:0/2319735952 >> v2:172.21.15.176:3300/0 conn(0x7fc9ac0d1ae0 0x7fc9ac0d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 --2- 172.21.15.67:0/2319735952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0cd940 0x7fc9ac0cdfa0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 -- 172.21.15.67:0/2319735952 >> 172.21.15.67:0/2319735952 conn(0x7fc9ac068590 msgr2=0x7fc9ac0689a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 -- 172.21.15.67:0/2319735952 shutdown_connections 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa80b6880 con 0x7f8ab8161d80 2024-04-25T15:54:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8a9c001040 con 0x7f8ab8161d80 2024-04-25T15:54:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7ff040870640 1 -- 172.21.15.67:0/2884803090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02c0b0070 con 0x7ff034050ac0 2024-04-25T15:54:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7ff040870640 1 -- 172.21.15.67:0/2884803090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0ab9b0 con 0x7ff034050ac0 2024-04-25T15:54:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 -- 172.21.15.67:0/2884803090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff034050ac0 msgr2=0x7ff03404ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 --2- 172.21.15.67:0/2884803090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff034050ac0 0x7ff03404ad20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7ff034011840 tx=0x7ff02c01cf40 comp rx=0 tx=0).stop 2024-04-25T15:54:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e00c0070 con 0x7f33e40093f0 2024-04-25T15:54:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7f7424ff9640 1 -- 172.21.15.67:0/642262564 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f741c01a070 con 0x7f7420035410 2024-04-25T15:54:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 -- 172.21.15.67:0/2884803090 shutdown_connections 2024-04-25T15:54:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 --2- 172.21.15.67:0/2884803090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff034050ac0 0x7ff03404ad20 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 --2- 172.21.15.67:0/2884803090 >> v2:172.21.15.176:3300/0 conn(0x7ff0340374b0 0x7ff0340d94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 -- 172.21.15.67:0/2884803090 >> 172.21.15.67:0/2884803090 conn(0x7ff03400e370 msgr2=0x7ff03400e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e00b0af0 con 0x7f33e40093f0 2024-04-25T15:54:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 -- 172.21.15.67:0/2884803090 shutdown_connections 2024-04-25T15:54:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7f7424ff9640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f740c06f400 0x7f740c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7f7424ff9640 1 -- 172.21.15.67:0/642262564 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f741c08e780 con 0x7f7420035410 2024-04-25T15:54:54.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7f7427fff640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f740c06f400 0x7f740c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.341+0000 7fc9b08a4640 1 -- 172.21.15.67:0/2319735952 wait complete. 2024-04-25T15:54:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7fc9b08a4640 1 Processor -- start 2024-04-25T15:54:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 -- 172.21.15.67:0/2884803090 wait complete. 2024-04-25T15:54:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 Processor -- start 2024-04-25T15:54:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7ff042874640 1 -- start start 2024-04-25T15:54:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff042874640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0340374b0 0x7ff0340c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff042874640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03404b260 0x7ff0340e9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff042874640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff034050ac0 0x7ff034180280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff042874640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff034007c50 con 0x7ff034050ac0 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff042874640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff034007ad0 con 0x7ff03404b260 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff042874640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff0340ec510 con 0x7ff0340374b0 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0340374b0 0x7ff0340c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0340374b0 0x7ff0340c8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50472/0 (socket says 172.21.15.67:50472) 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 -- 172.21.15.67:0/2194430798 learned_addr learned my addr 172.21.15.67:0/2194430798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f7427fff640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f740c06f400 0x7f740c071910 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f742809c4d0 tx=0x7f742806f040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.345+0000 7fc9b08a4640 1 -- start start 2024-04-25T15:54:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9b08a4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac0cd940 0x7fc9ac1421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9b08a4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0d1ae0 0x7fc9ac148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9b08a4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac156910 0x7fc9ac148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9b08a4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac169a10 con 0x7fc9ac0d1ae0 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9b08a4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac169890 con 0x7fc9ac156910 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9b08a4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9ac169b90 con 0x7fc9ac0cd940 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8aa80b6c70 con 0x7f8ab8161d80 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab4ff9640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a9806f450 0x7f8a98071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f8aa812a710 con 0x7f8ab8161d80 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab4ff9640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8a98077ba0 0x7f8a9807a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8aa80ab620 con 0x7f8a98077ba0 2024-04-25T15:54:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f8aa812abf0 con 0x7f8ab8161d80 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab6ffd640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a9806f450 0x7f8a98071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab77fe640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8a98077ba0 0x7f8a9807a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080052d0 con 0x7f8714152df0 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8708005470 con 0x7f8714152df0 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87141f7fc0 con 0x7f8714152df0 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708007240 con 0x7f8714152df0 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87141f8580 con 0x7f8714152df0 2024-04-25T15:54:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8ab6ffd640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a9806f450 0x7f8a98071960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0000fc0 tx=0x7f8aa0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9aa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0d1ae0 0x7fc9ac148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9aa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0d1ae0 0x7fc9ac148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49502/0 (socket says 172.21.15.67:49502) 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9aa59c640 1 -- 172.21.15.67:0/1899943639 learned_addr learned my addr 172.21.15.67:0/1899943639 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9ab59e640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac156910 0x7fc9ac148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9ab59e640 1 -- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac0cd940 msgr2=0x7fc9ac1421e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9ab59e640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac0cd940 0x7fc9ac1421e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9ab59e640 1 -- 172.21.15.67:0/1899943639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0d1ae0 msgr2=0x7fc9ac148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9ab59e640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9ac0d1ae0 0x7fc9ac148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7fc9ab59e640 1 -- 172.21.15.67:0/1899943639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ac146300 con 0x7fc9ac156910 2024-04-25T15:54:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f871406a480 con 0x7f8714152df0 2024-04-25T15:54:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041071640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03404b260 0x7ff0340e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 -- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03404b260 msgr2=0x7ff0340e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff03404b260 0x7ff0340e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 -- 172.21.15.67:0/2194430798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff034050ac0 msgr2=0x7ff034180280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff034050ac0 0x7ff034180280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.349+0000 7ff041872640 1 -- 172.21.15.67:0/2194430798 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0340dc1c0 con 0x7ff0340374b0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7fc9ab59e640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac156910 0x7fc9ac148ba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400eba0 tx=0x7fc9a400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7ff041872640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0340374b0 0x7ff0340c8df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff03800e8a0 tx=0x7ff03800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff038017070 con 0x7ff0340374b0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0340cd8e0 con 0x7ff0340374b0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0340c94a0 con 0x7ff0340374b0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03800c940 con 0x7ff0340374b0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03800ccf0 con 0x7ff0340374b0 2024-04-25T15:54:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff024001040 con 0x7ff0340374b0 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4018020 con 0x7fc9ac156910 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a400cd20 con 0x7fc9ac156910 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7fc9b08a4640 1 -- 172.21.15.67:0/1899943639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ac1465e0 con 0x7fc9ac156910 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40146c0 con 0x7fc9ac156910 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7fc9b08a4640 1 -- 172.21.15.67:0/1899943639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ac146ba0 con 0x7fc9ac156910 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc98c001040 con 0x7fc9ac156910 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8ab77fe640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8a98077ba0 0x7f8a9807a090 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f87080073e0 con 0x7f8714152df0 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8702ffd640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86f406f430 0x7f86f4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f870808efd0 con 0x7f8714152df0 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8702ffd640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f86f4077be0 0x7f86f407a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8708009e50 con 0x7f86f4077be0 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f870808f3c0 con 0x7f8714152df0 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f871259c640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86f406f430 0x7f86f4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8711d9b640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f86f4077be0 0x7f86f407a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.353+0000 7f8711d9b640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f86f4077be0 0x7f86f407a0d0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff03801a070 con 0x7ff0340374b0 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff0327fc640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff02006f400 0x7ff020071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7ff03808faf0 con 0x7ff0340374b0 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff041071640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff02006f400 0x7ff020071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff0327fc640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff020077b30 0x7ff02007a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff038016ec0 con 0x7ff020077b30 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7ff03808ff80 con 0x7ff0340374b0 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8aa80ab620 con 0x7f8a98077ba0 2024-04-25T15:54:54.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc9a401b020 con 0x7fc9ac156910 2024-04-25T15:54:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8708009e50 con 0x7f86f4077be0 2024-04-25T15:54:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7f871259c640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86f406f430 0x7f86f4071940 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f870c009d90 tx=0x7f870c0a9070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7fc9937fe640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc99806f400 0x7fc998071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fc9a4090a00 con 0x7fc9ac156910 2024-04-25T15:54:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7ff041071640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff02006f400 0x7ff020071910 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0af000 tx=0x7ff02c004740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7fc9937fe640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc998077b30 0x7fc99807a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9a40020c0 con 0x7fc998077b30 2024-04-25T15:54:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.357+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7fc9a4090db0 con 0x7fc9ac156910 2024-04-25T15:54:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7fc9aad9d640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc99806f400 0x7fc998071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7f742d790640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 0x7f74200d9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7ff042073640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff020077b30 0x7ff02007a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7f742d790640 1 -- 172.21.15.67:0/642262564 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f74201741a0 con 0x7f742004b370 2024-04-25T15:54:54.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7ff042073640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff020077b30 0x7ff02007a020 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7f742cf8f640 1 -- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 msgr2=0x7f74200d9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-25T15:54:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7f742cf8f640 1 -- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 msgr2=0x7f74200d9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:54:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7f742cf8f640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 0x7f74200d9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-04-25T15:54:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7f742cf8f640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 0x7f74200d9b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:54:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.361+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff038016ec0 con 0x7ff020077b30 2024-04-25T15:54:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7fc9aad9d640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc99806f400 0x7fc998071910 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00ae000 tx=0x7fc9a00a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f442c0c4ea0 con 0x7f4418077b30 2024-04-25T15:54:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f441ffff640 1 -- 172.21.15.67:0/2505926361 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f442c0c4ea0 con 0x7f4418077b30 2024-04-25T15:54:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf1c072ed0 con 0x7faf08077c30 2024-04-25T15:54:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4418077b30 msgr2=0x7f441807a020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4439e29640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f4418077b30 0x7f441807a020 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f441806f400 msgr2=0x7f4418071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4439e29640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f441806f400 0x7f4418071910 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f442c0c45b0 tx=0x7f4428009040 comp rx=0 tx=0).stop 2024-04-25T15:54:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7fc9aa59c640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc998077b30 0x7fc99807a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442c046ad0 msgr2=0x7f442c0c83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4439e29640 1 --2- 172.21.15.67:0/2505926361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f442c046ad0 0x7f442c0c83a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f442000e8a0 tx=0x7f442000ed60 comp rx=0 tx=0).stop 2024-04-25T15:54:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7f4433fff640 1 -- 172.21.15.67:0/2505926361 reap_dead start 2024-04-25T15:54:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.365+0000 7faf037fe640 1 -- 172.21.15.67:0/3160004509 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faf1c072ed0 con 0x7faf08077c30 2024-04-25T15:54:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 shutdown_connections 2024-04-25T15:54:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 >> 172.21.15.67:0/2505926361 conn(0x7f442c050c50 msgr2=0x7f442c04dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 shutdown_connections 2024-04-25T15:54:54.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7f4439e29640 1 -- 172.21.15.67:0/2505926361 wait complete. 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf08077c30 msgr2=0x7faf0807a120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7faf08077c30 0x7faf0807a120 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf0806f480 msgr2=0x7faf08071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7faf0806f480 0x7faf08071990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faf1c157270 tx=0x7faf0c008040 comp rx=0 tx=0).stop 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c0725c0 msgr2=0x7faf1c1f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 --2- 172.21.15.67:0/3160004509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf1c0725c0 0x7faf1c1f6d90 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7faf1000b2c0 tx=0x7faf1001b920 comp rx=0 tx=0).stop 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf1b7fe640 1 -- 172.21.15.67:0/3160004509 reap_dead start 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 shutdown_connections 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 >> 172.21.15.67:0/3160004509 conn(0x7faf1c0ce1c0 msgr2=0x7faf1c15c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7fc9aa59c640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc998077b30 0x7fc99807a020 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 shutdown_connections 2024-04-25T15:54:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7faf017fa640 1 -- 172.21.15.67:0/3160004509 wait complete. 2024-04-25T15:54:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.369+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9a40020c0 con 0x7fc998077b30 2024-04-25T15:54:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f33cc001040 con 0x7f33e40093f0 2024-04-25T15:54:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f33e00b8070 con 0x7f33e40093f0 2024-04-25T15:54:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33e97fa640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f33d006f630 0x7f33d0071b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f33e00b08f0 con 0x7f33e40093f0 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33e97fa640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f33d0077d60 0x7f33d007a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33e00b7e50 con 0x7f33d0077d60 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f33e0133050 con 0x7f33e40093f0 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33ebfff640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f33d006f630 0x7f33d0071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.373+0000 7f33f166e640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f33d0077d60 0x7f33d007a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.377+0000 7f33f166e640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f33d0077d60 0x7f33d007a250 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.377+0000 7f33ebfff640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f33d006f630 0x7f33d0071b40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f33e4178990 tx=0x7f33d8009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.377+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f33e00b7e50 con 0x7f33d0077d60 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.377+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f871406a480 con 0x7f86f4077be0 2024-04-25T15:54:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.381+0000 7f8702ffd640 1 -- 172.21.15.67:0/1051239076 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f871406a480 con 0x7f86f4077be0 2024-04-25T15:54:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.381+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f86f4077be0 msgr2=0x7f86f407a0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.381+0000 7f8718a66640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f86f4077be0 0x7f86f407a0d0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.381+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8a9c002de0 con 0x7f8a98077ba0 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86f406f430 msgr2=0x7f86f4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f86f406f430 0x7f86f4071940 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f870c009d90 tx=0x7f870c0a9070 comp rx=0 tx=0).stop 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714152df0 msgr2=0x7f8714157d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 --2- 172.21.15.67:0/1051239076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714152df0 0x7f8714157d90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f870800c390 tx=0x7f870800c850 comp rx=0 tx=0).stop 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8712d9d640 1 -- 172.21.15.67:0/1051239076 reap_dead start 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 shutdown_connections 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 >> 172.21.15.67:0/1051239076 conn(0x7f871406fe70 msgr2=0x7f8714151840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 shutdown_connections 2024-04-25T15:54:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8718a66640 1 -- 172.21.15.67:0/1051239076 wait complete. 2024-04-25T15:54:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8ab4ff9640 1 -- 172.21.15.67:0/3784343722 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8a9c002de0 con 0x7f8a98077ba0 2024-04-25T15:54:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8a98077ba0 msgr2=0x7f8a9807a090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7f8a98077ba0 0x7f8a9807a090 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a9806f450 msgr2=0x7f8a98071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f8a9806f450 0x7f8a98071960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0000fc0 tx=0x7f8aa0009040 comp rx=0 tx=0).stop 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8161d80 msgr2=0x7f8ab8155800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 --2- 172.21.15.67:0/3784343722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8161d80 0x7f8ab8155800 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8aa801c150 tx=0x7f8aa4005840 comp rx=0 tx=0).stop 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8ab77fe640 1 -- 172.21.15.67:0/3784343722 reap_dead start 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 shutdown_connections 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 >> 172.21.15.67:0/3784343722 conn(0x7f8ab806fe70 msgr2=0x7f8ab80cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 shutdown_connections 2024-04-25T15:54:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.385+0000 7f8abd9cd640 1 -- 172.21.15.67:0/3784343722 wait complete. 2024-04-25T15:54:54.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff024002de0 con 0x7ff020077b30 2024-04-25T15:54:54.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff0327fc640 1 -- 172.21.15.67:0/2194430798 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff024002de0 con 0x7ff020077b30 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff020077b30 msgr2=0x7ff02007a020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff042874640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7ff020077b30 0x7ff02007a020 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff02006f400 msgr2=0x7ff020071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff042874640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff02006f400 0x7ff020071910 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0af000 tx=0x7ff02c004740 comp rx=0 tx=0).stop 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0340374b0 msgr2=0x7ff0340c8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.389+0000 7ff042874640 1 --2- 172.21.15.67:0/2194430798 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0340374b0 0x7ff0340c8df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff03800e8a0 tx=0x7ff03800eca0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.393+0000 7ff042073640 1 -- 172.21.15.67:0/2194430798 reap_dead start 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.393+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 shutdown_connections 2024-04-25T15:54:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.393+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 >> 172.21.15.67:0/2194430798 conn(0x7ff03400e370 msgr2=0x7ff0340d7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.393+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 shutdown_connections 2024-04-25T15:54:54.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.393+0000 7ff042874640 1 -- 172.21.15.67:0/2194430798 wait complete. 2024-04-25T15:54:54.403 INFO:teuthology.orchestra.run.smithi067.stdout:55834574907 2024-04-25T15:54:54.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:54:54.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.393+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc98c002de0 con 0x7fc998077b30 2024-04-25T15:54:54.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.393+0000 7fc9937fe640 1 -- 172.21.15.67:0/1899943639 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc98c002de0 con 0x7fc998077b30 2024-04-25T15:54:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc998077b30 msgr2=0x7fc99807a020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fc998077b30 0x7fc99807a020 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc99806f400 msgr2=0x7fc998071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc99806f400 0x7fc998071910 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00ae000 tx=0x7fc9a00a7000 comp rx=0 tx=0).stop 2024-04-25T15:54:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac156910 msgr2=0x7fc9ac148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 --2- 172.21.15.67:0/1899943639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac156910 0x7fc9ac148ba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400eba0 tx=0x7fc9a400c490 comp rx=0 tx=0).stop 2024-04-25T15:54:54.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9ab59e640 1 -- 172.21.15.67:0/1899943639 reap_dead start 2024-04-25T15:54:54.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 shutdown_connections 2024-04-25T15:54:54.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 >> 172.21.15.67:0/1899943639 conn(0x7fc9ac068590 msgr2=0x7fc9ac0cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 shutdown_connections 2024-04-25T15:54:54.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.397+0000 7fc9917fa640 1 -- 172.21.15.67:0/1899943639 wait complete. 2024-04-25T15:54:54.407 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:54.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:54:54.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f33cc002de0 con 0x7f33d0077d60 2024-04-25T15:54:54.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33e97fa640 1 -- 172.21.15.67:0/37623424 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f33cc002de0 con 0x7f33d0077d60 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f33d0077d60 msgr2=0x7f33d007a250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7f33d0077d60 0x7f33d007a250 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f33d006f630 msgr2=0x7f33d0071b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f33d006f630 0x7f33d0071b40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f33e4178990 tx=0x7f33d8009060 comp rx=0 tx=0).stop 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e40093f0 msgr2=0x7f33e40cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 --2- 172.21.15.67:0/37623424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33e40093f0 0x7f33e40cf400 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f33e001b860 tx=0x7f33e001b9f0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f166e640 1 -- 172.21.15.67:0/37623424 reap_dead start 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 shutdown_connections 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 >> 172.21.15.67:0/37623424 conn(0x7f33e40514d0 msgr2=0x7f33e40d1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 shutdown_connections 2024-04-25T15:54:54.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.405+0000 7f33f1e6f640 1 -- 172.21.15.67:0/37623424 wait complete. 2024-04-25T15:54:54.418 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:54.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:54:54.424 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:54.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:54:54.427 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:54.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:54:54.429 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:54.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:54:54.437 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:54.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:54:54.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.565+0000 7f742cf8f640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 0x7f74200d9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.565+0000 7f742cf8f640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 0x7f74200d9b30 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.565+0000 7f7424ff9640 1 -- 172.21.15.67:0/642262564 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f74201741a0 con 0x7f742004b370 2024-04-25T15:54:54.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7ff48946f640 1 Processor -- start 2024-04-25T15:54:54.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7f742d790640 1 -- 172.21.15.67:0/642262564 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7420048480 con 0x7f742004b370 2024-04-25T15:54:54.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7f7424ff9640 1 -- 172.21.15.67:0/642262564 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7420048480 con 0x7f742004b370 2024-04-25T15:54:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7fe19c673640 1 Processor -- start 2024-04-25T15:54:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7fe19c673640 1 -- start start 2024-04-25T15:54:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7fe19c673640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1940c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7fe19c673640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1940c52e0 0x7fe194164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7fe19c673640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1940c58a0 con 0x7fe1941646d0 2024-04-25T15:54:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7fe19c673640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe1940c73a0 con 0x7fe1940c52e0 2024-04-25T15:54:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7fe19c673640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe1940c6f10 con 0x7fe1940c5e70 2024-04-25T15:54:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7ff48946f640 1 -- start start 2024-04-25T15:54:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7ff48946f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff4840c6330 0x7ff4841588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7ff48946f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff484152df0 0x7ff484163180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7ff48946f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff484158e20 con 0x7ff4841529e0 2024-04-25T15:54:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7ff48946f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff4840c5240 con 0x7ff484152df0 2024-04-25T15:54:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.593+0000 7ff48946f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff4840c43e0 con 0x7ff4840c6330 2024-04-25T15:54:54.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe199c0e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1940c52e0 0x7fe194164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe199c0e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe1940c52e0 0x7fe194164190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49534/0 (socket says 172.21.15.67:49534) 2024-04-25T15:54:54.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe199c0e640 1 -- 172.21.15.67:0/731442333 learned_addr learned my addr 172.21.15.67:0/731442333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff4837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff484152df0 0x7ff484163180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff4837fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff484152df0 0x7ff484163180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49536/0 (socket says 172.21.15.67:49536) 2024-04-25T15:54:54.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff4837fe640 1 -- 172.21.15.67:0/1959155273 learned_addr learned my addr 172.21.15.67:0/1959155273 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7f74127fc640 1 -- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 msgr2=0x7f74200d9b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff482ffd640 1 --2- 172.21.15.67:0/1959155273 >> v2:172.21.15.176:3300/0 conn(0x7ff4840c6330 0x7ff4841588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe19a40f640 1 --2- 172.21.15.67:0/731442333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1940c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe199c0e640 1 -- 172.21.15.67:0/731442333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 msgr2=0x7fe1940c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe199c0e640 1 --2- 172.21.15.67:0/731442333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1940c4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe199c0e640 1 -- 172.21.15.67:0/731442333 >> v1:172.21.15.176:6789/0 conn(0x7fe1941646d0 legacy=0x7fe1940c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7fe199c0e640 1 -- 172.21.15.67:0/731442333 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1940c65e0 con 0x7fe1940c52e0 2024-04-25T15:54:54.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f742004b370 0x7f74200d9b30 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 -- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f740c06f400 msgr2=0x7f740c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f740c06f400 0x7f740c071910 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f742809c4d0 tx=0x7f742806f040 comp rx=0 tx=0).stop 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff4837fe640 1 -- 172.21.15.67:0/1959155273 >> v2:172.21.15.176:3300/0 conn(0x7ff4840c6330 msgr2=0x7ff4841588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff4837fe640 1 --2- 172.21.15.67:0/1959155273 >> v2:172.21.15.176:3300/0 conn(0x7ff4840c6330 0x7ff4841588e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff4837fe640 1 -- 172.21.15.67:0/1959155273 >> v1:172.21.15.176:6789/0 conn(0x7ff4841529e0 legacy=0x7ff48406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.597+0000 7ff4837fe640 1 -- 172.21.15.67:0/1959155273 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4840c9ce0 con 0x7ff484152df0 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe199c0e640 1 --2- 172.21.15.67:0/731442333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe194164190 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe1940c7090 tx=0x7fe18c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f6a29f0e82b724 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 -- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7420035410 msgr2=0x7f74200d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 --2- 172.21.15.67:0/642262564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7420035410 0x7f74200d0f20 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c390 tx=0x7f741c00c850 comp rx=0 tx=0).stop 2024-04-25T15:54:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f742cf8f640 1 -- 172.21.15.67:0/642262564 reap_dead start 2024-04-25T15:54:54.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19940d640 1 -- 172.21.15.67:0/731442333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c0b4070 con 0x7fe1940c52e0 2024-04-25T15:54:54.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19940d640 1 -- 172.21.15.67:0/731442333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18c0af400 con 0x7fe1940c52e0 2024-04-25T15:54:54.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19940d640 1 -- 172.21.15.67:0/731442333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c0af7d0 con 0x7fe1940c52e0 2024-04-25T15:54:54.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff4837fe640 1 --2- 172.21.15.67:0/1959155273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484163180 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff484070d30 tx=0x7ff47c01ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83bee444c436c01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff481ffb640 1 -- 172.21.15.67:0/1959155273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0ad070 con 0x7ff484152df0 2024-04-25T15:54:54.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff481ffb640 1 -- 172.21.15.67:0/1959155273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff47c0039a0 con 0x7ff484152df0 2024-04-25T15:54:54.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff481ffb640 1 -- 172.21.15.67:0/1959155273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00be60 con 0x7ff484152df0 2024-04-25T15:54:54.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 -- 172.21.15.67:0/731442333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 msgr2=0x7fe194164190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 --2- 172.21.15.67:0/731442333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe194164190 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe1940c7090 tx=0x7fe18c01f1c0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 -- 172.21.15.67:0/642262564 shutdown_connections 2024-04-25T15:54:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 -- 172.21.15.67:0/642262564 >> 172.21.15.67:0/642262564 conn(0x7f74200514d0 msgr2=0x7f74200518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 -- 172.21.15.67:0/642262564 shutdown_connections 2024-04-25T15:54:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7f74127fc640 1 -- 172.21.15.67:0/642262564 wait complete. 2024-04-25T15:54:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 -- 172.21.15.67:0/1959155273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 msgr2=0x7ff484163180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 --2- 172.21.15.67:0/1959155273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484163180 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff484070d30 tx=0x7ff47c01ed80 comp rx=0 tx=0).stop 2024-04-25T15:54:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 -- 172.21.15.67:0/731442333 shutdown_connections 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 --2- 172.21.15.67:0/731442333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe194164190 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 --2- 172.21.15.67:0/731442333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1940c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 -- 172.21.15.67:0/731442333 >> 172.21.15.67:0/731442333 conn(0x7fe19406fe70 msgr2=0x7fe1940cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 -- 172.21.15.67:0/1959155273 shutdown_connections 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 --2- 172.21.15.67:0/1959155273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484163180 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 --2- 172.21.15.67:0/1959155273 >> v2:172.21.15.176:3300/0 conn(0x7ff4840c6330 0x7ff4841588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 -- 172.21.15.67:0/1959155273 >> 172.21.15.67:0/1959155273 conn(0x7ff48406fe70 msgr2=0x7ff4840cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 -- 172.21.15.67:0/1959155273 shutdown_connections 2024-04-25T15:54:54.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 -- 172.21.15.67:0/731442333 shutdown_connections 2024-04-25T15:54:54.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7fe19c673640 1 -- 172.21.15.67:0/731442333 wait complete. 2024-04-25T15:54:54.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.601+0000 7ff48946f640 1 -- 172.21.15.67:0/1959155273 wait complete. 2024-04-25T15:54:54.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7ff48946f640 1 Processor -- start 2024-04-25T15:54:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7fe19c673640 1 Processor -- start 2024-04-25T15:54:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7fe19c673640 1 -- start start 2024-04-25T15:54:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7ff48946f640 1 -- start start 2024-04-25T15:54:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7fe19c673640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe1941fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7fe19c673640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1941eabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7fe19c673640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1941646d0 0x7fe194152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7ff48946f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4840c6330 0x7ff4840ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.605+0000 7ff48946f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4841529e0 0x7ff484143f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff48946f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19c673640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1940c5b80 con 0x7fe1940c52e0 2024-04-25T15:54:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19c673640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe1940c6ba0 con 0x7fe1941646d0 2024-04-25T15:54:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19c673640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe1940c9a00 con 0x7fe1940c5e70 2024-04-25T15:54:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe1941fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe199c0e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1941eabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe1941fb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49552/0 (socket says 172.21.15.67:49552) 2024-04-25T15:54:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 -- 172.21.15.67:0/1966075341 learned_addr learned my addr 172.21.15.67:0/1966075341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4827fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4841529e0 0x7ff484143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19ac10640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1941646d0 0x7fe194152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff48946f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff48406cd50 con 0x7ff484152df0 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4827fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4841529e0 0x7ff484143f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:32908/0 (socket says 172.21.15.67:32908) 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4827fc640 1 -- 172.21.15.67:0/3665300985 learned_addr learned my addr 172.21.15.67:0/3665300985 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff4840c6040 con 0x7ff4841529e0 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff482ffd640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4840c6330 0x7ff4840ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff48406ced0 con 0x7ff4840c6330 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 -- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 msgr2=0x7fe1941eabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1941eabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 -- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1941646d0 msgr2=0x7fe194152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1941646d0 0x7fe194152e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 -- 172.21.15.67:0/1966075341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1941fb600 con 0x7fe1940c52e0 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7fe19a40f640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe1941fb0c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe19000d5f0 tx=0x7fe19000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4837fe640 1 -- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4840c6330 msgr2=0x7ff4840ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4837fe640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4840c6330 0x7ff4840ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4837fe640 1 -- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4841529e0 msgr2=0x7ff484143f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4837fe640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4841529e0 0x7ff484143f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4837fe640 1 -- 172.21.15.67:0/3665300985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4840cc6f0 con 0x7ff484152df0 2024-04-25T15:54:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.609+0000 7ff4837fe640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484146530 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff4840ce940 tx=0x7ff47c01bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe18affd640 1 -- 172.21.15.67:0/1966075341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1900125f0 con 0x7fe1940c52e0 2024-04-25T15:54:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe194073700 con 0x7fe1940c52e0 2024-04-25T15:54:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe194163e20 con 0x7fe1940c52e0 2024-04-25T15:54:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe18affd640 1 -- 172.21.15.67:0/1966075341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe190019070 con 0x7fe1940c52e0 2024-04-25T15:54:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe18affd640 1 -- 172.21.15.67:0/1966075341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1900129b0 con 0x7fe1940c52e0 2024-04-25T15:54:54.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7ff46b7fe640 1 -- 172.21.15.67:0/3665300985 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0ad050 con 0x7ff484152df0 2024-04-25T15:54:54.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff484148fc0 con 0x7ff484152df0 2024-04-25T15:54:54.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4840ccad0 con 0x7ff484152df0 2024-04-25T15:54:54.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe18affd640 1 -- 172.21.15.67:0/1966075341 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe190012b50 con 0x7fe1940c52e0 2024-04-25T15:54:54.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48406a480 con 0x7ff484152df0 2024-04-25T15:54:54.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe18affd640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe17806f500 0x7fe178071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe18affd640 1 -- 172.21.15.67:0/1966075341 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fe19008f4f0 con 0x7fe1940c52e0 2024-04-25T15:54:54.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7fe199c0e640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe17806f500 0x7fe178071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.617+0000 7fe199c0e640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe17806f500 0x7fe178071a10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0b1000 tx=0x7fe18c0aa000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.613+0000 7ff46b7fe640 1 -- 172.21.15.67:0/3665300985 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff47c004260 con 0x7ff484152df0 2024-04-25T15:54:54.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.617+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe17c0052a0 con 0x7fe1940c52e0 2024-04-25T15:54:54.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.617+0000 7ff46b7fe640 1 -- 172.21.15.67:0/3665300985 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00be60 con 0x7ff484152df0 2024-04-25T15:54:54.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.617+0000 7ff46b7fe640 1 -- 172.21.15.67:0/3665300985 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff47c004690 con 0x7ff484152df0 2024-04-25T15:54:54.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.617+0000 7ff46b7fe640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff47006f530 0x7ff470071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.617+0000 7ff46b7fe640 1 -- 172.21.15.67:0/3665300985 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7ff47c12c020 con 0x7ff484152df0 2024-04-25T15:54:54.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.617+0000 7ff482ffd640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff47006f530 0x7ff470071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.621+0000 7ff482ffd640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff47006f530 0x7ff470071a40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff474009ac0 tx=0x7ff474009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.621+0000 7f3e76dcd640 1 Processor -- start 2024-04-25T15:54:54.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e76dcd640 1 -- start start 2024-04-25T15:54:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e76dcd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e70068590 0x7f3e70153440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e76dcd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e700c4020 con 0x7f3e700689a0 2024-04-25T15:54:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e76dcd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e700c4300 con 0x7f3e7015cce0 2024-04-25T15:54:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e76dcd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e700c45e0 con 0x7f3e70068590 2024-04-25T15:54:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e755ca640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3e700689a0 0x7f3e70071f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43284/0 (socket says 172.21.15.67:43284) 2024-04-25T15:54:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e755ca640 1 -- 172.21.15.67:0/1963465138 learned_addr learned my addr 172.21.15.67:0/1963465138 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e75dcb640 1 --2- 172.21.15.67:0/1963465138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e70068590 0x7f3e70153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7fee04d1b640 1 Processor -- start 2024-04-25T15:54:54.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7ff46b7fe640 1 -- 172.21.15.67:0/3665300985 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff47c0189a0 con 0x7ff484152df0 2024-04-25T15:54:54.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7fee04d1b640 1 -- start start 2024-04-25T15:54:54.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e75dcb640 1 -- 172.21.15.67:0/1963465138 >> v1:172.21.15.176:6789/0 conn(0x7f3e7015cce0 legacy=0x7f3e7015d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7fcf87ef5640 1 Processor -- start 2024-04-25T15:54:54.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e75dcb640 1 -- 172.21.15.67:0/1963465138 >> v1:172.21.15.67:6789/0 conn(0x7f3e700689a0 legacy=0x7f3e70071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.625+0000 7f3e75dcb640 1 -- 172.21.15.67:0/1963465138 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e700c48c0 con 0x7f3e70068590 2024-04-25T15:54:54.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf87ef5640 1 -- start start 2024-04-25T15:54:54.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fee04d1b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee0015d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fee04d1b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee00073bf0 0x7fee00163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfdd9b640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee0015d790 0x7fee0006d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43298/0 (socket says 172.21.15.67:43298) 2024-04-25T15:54:54.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfe59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee0015d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfe59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee0015d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50500/0 (socket says 172.21.15.67:50500) 2024-04-25T15:54:54.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee00073bf0 0x7fee00163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee00073bf0 0x7fee00163ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49570/0 (socket says 172.21.15.67:49570) 2024-04-25T15:54:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf87ef5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf8015d110 0x7fcf800cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf87ef5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcf800cd820 0x7fcf800ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfdd9b640 1 -- 172.21.15.67:0/4254865532 learned_addr learned my addr 172.21.15.67:0/4254865532 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee000c77a0 con 0x7fee0015d790 2024-04-25T15:54:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fee000cab60 con 0x7fee00073bf0 2024-04-25T15:54:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee00159620 con 0x7fee000caf60 2024-04-25T15:54:54.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf87ef5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf800cf000 0x7fcf8015fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf87ef5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcf800d20c0 con 0x7fcf800cf000 2024-04-25T15:54:54.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf87ef5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcf800cf410 con 0x7fcf800cd820 2024-04-25T15:54:54.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf800cf000 0x7fcf8015fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf85c91640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf8015d110 0x7fcf800cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf800cf000 0x7fcf8015fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49578/0 (socket says 172.21.15.67:49578) 2024-04-25T15:54:54.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 -- 172.21.15.67:0/2904879106 learned_addr learned my addr 172.21.15.67:0/2904879106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf87ef5640 1 -- 172.21.15.67:0/2904879106 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf800d22f0 con 0x7fcf8015d110 2024-04-25T15:54:54.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7f3e75dcb640 1 --2- 172.21.15.67:0/1963465138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e70068590 0x7f3e70153440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e600097f0 tx=0x7f3e6001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a0a18a9718a9c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 -- 172.21.15.67:0/2904879106 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf8015d110 msgr2=0x7fcf800cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 --2- 172.21.15.67:0/2904879106 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf8015d110 0x7fcf800cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 -- 172.21.15.67:0/2904879106 >> v2:172.21.15.176:3300/0 conn(0x7fcf800cd820 msgr2=0x7fcf800ceac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T15:54:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 --2- 172.21.15.67:0/2904879106 >> v2:172.21.15.176:3300/0 conn(0x7fcf800cd820 0x7fcf800ceac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fcf86492640 1 -- 172.21.15.67:0/2904879106 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf800d25d0 con 0x7fcf800cf000 2024-04-25T15:54:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e74dc9640 1 -- 172.21.15.67:0/1963465138 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e600b1070 con 0x7f3e70068590 2024-04-25T15:54:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e74dc9640 1 -- 172.21.15.67:0/1963465138 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e600a8050 con 0x7f3e70068590 2024-04-25T15:54:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e74dc9640 1 -- 172.21.15.67:0/1963465138 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e600ac860 con 0x7f3e70068590 2024-04-25T15:54:54.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfed9d640 1 -- 172.21.15.67:0/4254865532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 msgr2=0x7fee0015d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfed9d640 1 --2- 172.21.15.67:0/4254865532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee0015d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfed9d640 1 -- 172.21.15.67:0/4254865532 >> v1:172.21.15.67:6789/0 conn(0x7fee0015d790 legacy=0x7fee0006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.629+0000 7fedfed9d640 1 -- 172.21.15.67:0/4254865532 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee000c7a60 con 0x7fee00073bf0 2024-04-25T15:54:54.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e76dcd640 1 -- 172.21.15.67:0/1963465138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e70068590 msgr2=0x7f3e70153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/1963465138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e70068590 0x7f3e70153440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e600097f0 tx=0x7f3e6001c4d0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7fcf86492640 1 --2- 172.21.15.67:0/2904879106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf800cf000 0x7fcf8015fc80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcf8006d1a0 tx=0x7fcf7801f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14dd161d2a71383 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7fedfed9d640 1 --2- 172.21.15.67:0/4254865532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee00073bf0 0x7fee00163ec0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009d90 tx=0x7fedf801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9c40a34442300a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.636 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:54.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:54:54.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e76dcd640 1 -- 172.21.15.67:0/1963465138 shutdown_connections 2024-04-25T15:54:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/1963465138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e70068590 0x7f3e70153440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e76dcd640 1 -- 172.21.15.67:0/1963465138 >> 172.21.15.67:0/1963465138 conn(0x7f3e700ca960 msgr2=0x7f3e700cad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e76dcd640 1 -- 172.21.15.67:0/1963465138 shutdown_connections 2024-04-25T15:54:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.633+0000 7f3e76dcd640 1 -- 172.21.15.67:0/1963465138 wait complete. 2024-04-25T15:54:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.637+0000 7f3e76dcd640 1 Processor -- start 2024-04-25T15:54:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.637+0000 7f3e76dcd640 1 -- start start 2024-04-25T15:54:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.637+0000 7fe18affd640 1 -- 172.21.15.67:0/1966075341 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe19005bd90 con 0x7fe1940c52e0 2024-04-25T15:54:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fedfd59a640 1 -- 172.21.15.67:0/4254865532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf80b2070 con 0x7fee00073bf0 2024-04-25T15:54:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf84c8f640 1 -- 172.21.15.67:0/2904879106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf780b4070 con 0x7fcf800cf000 2024-04-25T15:54:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf84c8f640 1 -- 172.21.15.67:0/2904879106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf780af400 con 0x7fcf800cf000 2024-04-25T15:54:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf84c8f640 1 -- 172.21.15.67:0/2904879106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf780af7d0 con 0x7fcf800cf000 2024-04-25T15:54:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e70068590 0x7f3e701636b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fedfd59a640 1 -- 172.21.15.67:0/4254865532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedf80a5830 con 0x7fee00073bf0 2024-04-25T15:54:54.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e700689a0 0x7f3e701f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e700689a0 0x7f3e701f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e700689a0 0x7f3e701f43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49580/0 (socket says 172.21.15.67:49580) 2024-04-25T15:54:54.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 -- 172.21.15.67:0/3663595803 learned_addr learned my addr 172.21.15.67:0/3663595803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e75dcb640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e70068590 0x7f3e701636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e7015cce0 0x7f3e7015b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 -- 172.21.15.67:0/2904879106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf800cf000 msgr2=0x7fcf8015fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/2904879106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf800cf000 0x7fcf8015fc80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fcf8006d1a0 tx=0x7fcf7801f200 comp rx=0 tx=0).stop 2024-04-25T15:54:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e70072820 con 0x7f3e700689a0 2024-04-25T15:54:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700726a0 con 0x7f3e70068590 2024-04-25T15:54:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e70168770 con 0x7f3e7015cce0 2024-04-25T15:54:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e765cc640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e7015cce0 0x7f3e7015b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee00073bf0 msgr2=0x7fee00163ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 --2- 172.21.15.67:0/4254865532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee00073bf0 0x7fee00163ec0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009d90 tx=0x7fedf801c9b0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 shutdown_connections 2024-04-25T15:54:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 --2- 172.21.15.67:0/4254865532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee00073bf0 0x7fee00163ec0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 --2- 172.21.15.67:0/4254865532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee0015d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 -- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e7015cce0 msgr2=0x7f3e7015b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e7015cce0 0x7f3e7015b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 -- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e70068590 msgr2=0x7f3e701636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 >> 172.21.15.67:0/4254865532 conn(0x7fee00070db0 msgr2=0x7fee000ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e70068590 0x7f3e701636b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e70161fd0 con 0x7f3e700689a0 2024-04-25T15:54:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 -- 172.21.15.67:0/2904879106 shutdown_connections 2024-04-25T15:54:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/2904879106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf800cf000 0x7fcf8015fc80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/2904879106 >> v2:172.21.15.176:3300/0 conn(0x7fcf800cd820 0x7fcf800ceac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/2904879106 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf8015d110 0x7fcf800cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 -- 172.21.15.67:0/2904879106 >> 172.21.15.67:0/2904879106 conn(0x7fcf80068990 msgr2=0x7fcf80068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 shutdown_connections 2024-04-25T15:54:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fcf87ef5640 1 -- 172.21.15.67:0/2904879106 shutdown_connections 2024-04-25T15:54:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e755ca640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e700689a0 0x7f3e701f43d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c430 tx=0x7f3e6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e667fc640 1 -- 172.21.15.67:0/3663595803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c017070 con 0x7f3e700689a0 2024-04-25T15:54:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e70150c70 con 0x7f3e700689a0 2024-04-25T15:54:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e701622b0 con 0x7f3e700689a0 2024-04-25T15:54:54.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.641+0000 7fee04d1b640 1 -- 172.21.15.67:0/4254865532 wait complete. 2024-04-25T15:54:54.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.645+0000 7f3e667fc640 1 -- 172.21.15.67:0/3663595803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6c005cf0 con 0x7f3e700689a0 2024-04-25T15:54:54.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.645+0000 7fee04d1b640 1 Processor -- start 2024-04-25T15:54:54.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.645+0000 7fee04d1b640 1 -- start start 2024-04-25T15:54:54.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.645+0000 7f3e667fc640 1 -- 172.21.15.67:0/3663595803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c021060 con 0x7f3e700689a0 2024-04-25T15:54:54.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.645+0000 7fcf87ef5640 1 -- 172.21.15.67:0/2904879106 wait complete. 2024-04-25T15:54:54.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7f3e667fc640 1 -- 172.21.15.67:0/3663595803 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3e6c01a070 con 0x7f3e700689a0 2024-04-25T15:54:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7f3e667fc640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3e5406f400 0x7f3e54071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7f3e75dcb640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3e5406f400 0x7f3e54071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7f3e667fc640 1 -- 172.21.15.67:0/3663595803 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f3e6c08caa0 con 0x7f3e700689a0 2024-04-25T15:54:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fcf87ef5640 1 Processor -- start 2024-04-25T15:54:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fcf87ef5640 1 -- start start 2024-04-25T15:54:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7f3e75dcb640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3e5406f400 0x7f3e54071910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3e600ae000 tx=0x7f3e600a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fee04d1b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee001563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fee04d1b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee0015d790 0x7fee00155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fee04d1b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee000ce700 0x7fee001533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fee04d1b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fee0015a570 con 0x7fee000ce700 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fee04d1b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fee0015a3f0 con 0x7fee0015d790 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fee04d1b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fee00164c10 con 0x7fee000caf60 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fedfed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee000ce700 0x7fee001533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fedfed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee000ce700 0x7fee001533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49588/0 (socket says 172.21.15.67:49588) 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fedfed9d640 1 -- 172.21.15.67:0/388651960 learned_addr learned my addr 172.21.15.67:0/388651960 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fedfe59c640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee001563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7fedfdd9b640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee0015d790 0x7fee00155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.649+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e580052a0 con 0x7f3e700689a0 2024-04-25T15:54:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fedfed9d640 1 -- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 msgr2=0x7fee001563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fedfed9d640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee001563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fedfed9d640 1 -- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee0015d790 msgr2=0x7fee00155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fedfed9d640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee0015d790 0x7fee00155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fedfed9d640 1 -- 172.21.15.67:0/388651960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee001fd490 con 0x7fee000ce700 2024-04-25T15:54:54.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fedfed9d640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee000ce700 0x7fee001533d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fedf80b0040 tx=0x7fedf8019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf87ef5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf800cd820 0x7fcf801faa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf87ef5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf800cf000 0x7fcf80073150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf87ef5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8015d110 0x7fcf80073690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf85c91640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf800cd820 0x7fcf801faa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf85c91640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf800cd820 0x7fcf801faa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50538/0 (socket says 172.21.15.67:50538) 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf85c91640 1 -- 172.21.15.67:0/1154300000 learned_addr learned my addr 172.21.15.67:0/1154300000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf85490640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf800cf000 0x7fcf80073150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcf800ce5a0 con 0x7fcf8015d110 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcf800ce2c0 con 0x7fcf800cf000 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf86492640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8015d110 0x7fcf80073690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf800b4730 con 0x7fcf800cd820 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf86492640 1 -- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf800cd820 msgr2=0x7fcf801faa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf86492640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf800cd820 0x7fcf801faa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf86492640 1 -- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf800cf000 msgr2=0x7fcf80073150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf86492640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf800cf000 0x7fcf80073150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf86492640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf801fafc0 con 0x7fcf8015d110 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf85c91640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf800cd820 0x7fcf801faa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf85490640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf800cf000 0x7fcf80073150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.653+0000 7fcf86492640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8015d110 0x7fcf80073690 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fcf780ae000 tx=0x7fcf7801bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7f3e667fc640 1 -- 172.21.15.67:0/3663595803 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3e6c05ae20 con 0x7f3e700689a0 2024-04-25T15:54:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fedeeffd640 1 -- 172.21.15.67:0/388651960 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf80b2070 con 0x7fee000ce700 2024-04-25T15:54:54.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee00156da0 con 0x7fee000ce700 2024-04-25T15:54:54.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fedeeffd640 1 -- 172.21.15.67:0/388651960 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedf80bc070 con 0x7fee000ce700 2024-04-25T15:54:54.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fedeeffd640 1 -- 172.21.15.67:0/388651960 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf80a5c80 con 0x7fee000ce700 2024-04-25T15:54:54.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee001fda10 con 0x7fee000ce700 2024-04-25T15:54:54.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fcf767fc640 1 -- 172.21.15.67:0/1154300000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf780b4040 con 0x7fcf8015d110 2024-04-25T15:54:54.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fcf767fc640 1 -- 172.21.15.67:0/1154300000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf780c0070 con 0x7fcf8015d110 2024-04-25T15:54:54.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fcf767fc640 1 -- 172.21.15.67:0/1154300000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78009510 con 0x7fcf8015d110 2024-04-25T15:54:54.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf80160010 con 0x7fcf8015d110 2024-04-25T15:54:54.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.661+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf800cb740 con 0x7fcf8015d110 2024-04-25T15:54:54.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.665+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf800cde90 con 0x7fcf8015d110 2024-04-25T15:54:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.665+0000 7fedeeffd640 1 -- 172.21.15.67:0/388651960 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fedf8004040 con 0x7fee000ce700 2024-04-25T15:54:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.665+0000 7fedeeffd640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fede006f430 0x7fede0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.665+0000 7fedeeffd640 1 -- 172.21.15.67:0/388651960 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fedf8129a70 con 0x7fee000ce700 2024-04-25T15:54:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.665+0000 7fcf767fc640 1 -- 172.21.15.67:0/1154300000 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcf780c5460 con 0x7fcf8015d110 2024-04-25T15:54:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.665+0000 7fcf767fc640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcf6406f4d0 0x7fcf640719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.669+0000 7fcf767fc640 1 -- 172.21.15.67:0/1154300000 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fcf7812e010 con 0x7fcf8015d110 2024-04-25T15:54:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.669+0000 7fedfe59c640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fede006f430 0x7fede0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.669+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede40052a0 con 0x7fee000ce700 2024-04-25T15:54:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.669+0000 7fcf85c91640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcf6406f4d0 0x7fcf640719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.669+0000 7fcf85c91640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcf6406f4d0 0x7fcf640719e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009f70 tx=0x7fcf70009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.669+0000 7fedfe59c640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fede006f430 0x7fede0071940 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009ac0 tx=0x7fedf0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.677+0000 7fcf767fc640 1 -- 172.21.15.67:0/1154300000 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcf78018d60 con 0x7fcf8015d110 2024-04-25T15:54:54.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.685+0000 7fedeeffd640 1 -- 172.21.15.67:0/388651960 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fedf80165a0 con 0x7fee000ce700 2024-04-25T15:54:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.721+0000 7fb23b1f9640 1 Processor -- start 2024-04-25T15:54:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.721+0000 7fb23b1f9640 1 -- start start 2024-04-25T15:54:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.713+0000 7fb8b148b640 1 Processor -- start 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.713+0000 7fb8b148b640 1 -- start start 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.713+0000 7fb8b148b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8ac0c99c0 0x7fb8ac169ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.713+0000 7fb8b148b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb8ac0c9240 0x7fb8ac0c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.713+0000 7fb8b148b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ac0c7ec0 con 0x7fb8ac0c4940 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.713+0000 7fb8b148b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb8ac0c81a0 con 0x7fb8ac0c99c0 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.713+0000 7fb8b148b640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb8ac0c8480 con 0x7fb8ac0c9240 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.717+0000 7fb8abfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8ac0c99c0 0x7fb8ac169ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.717+0000 7fb8b0c8a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb8ac0c9240 0x7fb8ac0c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.717+0000 7fb8abfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8ac0c99c0 0x7fb8ac169ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49596/0 (socket says 172.21.15.67:49596) 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.717+0000 7fb8abfff640 1 -- 172.21.15.67:0/799681282 learned_addr learned my addr 172.21.15.67:0/799681282 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.717+0000 7fb8aaffd640 1 -- 172.21.15.67:0/799681282 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 137105135 0 0) 0x7fb8ac0c7ec0 con 0x7fb8ac0c4940 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.721+0000 7fb8aaffd640 1 -- 172.21.15.67:0/799681282 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb894007940 con 0x7fb8ac0c4940 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.721+0000 7fb8abfff640 1 -- 172.21.15.67:0/799681282 >> v2:172.21.15.176:3300/0 conn(0x7fb8ac0c9240 msgr2=0x7fb8ac0c51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.721+0000 7fb8abfff640 1 --2- 172.21.15.67:0/799681282 >> v2:172.21.15.176:3300/0 conn(0x7fb8ac0c9240 0x7fb8ac0c51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.721+0000 7fb8abfff640 1 -- 172.21.15.67:0/799681282 >> v1:172.21.15.176:6789/0 conn(0x7fb8ac0c4940 legacy=0x7fb8ac06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.721+0000 7fb8abfff640 1 -- 172.21.15.67:0/799681282 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ac0c8760 con 0x7fb8ac0c99c0 2024-04-25T15:54:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb23b1f9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2340caf60 0x7fb23415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb23b1f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb23415d790 0x7fb23406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb23b1f9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2340c77a0 con 0x7fb234073bf0 2024-04-25T15:54:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb23b1f9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb2340cab60 con 0x7fb2340caf60 2024-04-25T15:54:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb23b1f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb234159620 con 0x7fb23415d790 2024-04-25T15:54:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb238f95640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2340caf60 0x7fb23415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb238f95640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2340caf60 0x7fb23415d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49600/0 (socket says 172.21.15.67:49600) 2024-04-25T15:54:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb238f95640 1 -- 172.21.15.67:0/543695899 learned_addr learned my addr 172.21.15.67:0/543695899 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb233fff640 1 --2- 172.21.15.67:0/543695899 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb23415d790 0x7fb23406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb8abfff640 1 --2- 172.21.15.67:0/799681282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac0c99c0 0x7fb8ac169ce0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac16d870 tx=0x7fb8a0024d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9abf81ecbad9fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb8aaffd640 1 -- 172.21.15.67:0/799681282 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a00add60 con 0x7fb8ac0c99c0 2024-04-25T15:54:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb8aaffd640 1 -- 172.21.15.67:0/799681282 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a000ae50 con 0x7fb8ac0c99c0 2024-04-25T15:54:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb8aaffd640 1 -- 172.21.15.67:0/799681282 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a00b5610 con 0x7fb8ac0c99c0 2024-04-25T15:54:54.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb8b148b640 1 -- 172.21.15.67:0/799681282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac0c99c0 msgr2=0x7fb8ac169ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb8b148b640 1 --2- 172.21.15.67:0/799681282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac0c99c0 0x7fb8ac169ce0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac16d870 tx=0x7fb8a0024d80 comp rx=0 tx=0).stop 2024-04-25T15:54:54.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 -- 172.21.15.67:0/799681282 shutdown_connections 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 --2- 172.21.15.67:0/799681282 >> v2:172.21.15.176:3300/0 conn(0x7fb8ac0c9240 0x7fb8ac0c51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 --2- 172.21.15.67:0/799681282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac0c99c0 0x7fb8ac169ce0 secure :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac16d870 tx=0x7fb8a0024d80 comp rx=0 tx=0).stop 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 -- 172.21.15.67:0/799681282 >> 172.21.15.67:0/799681282 conn(0x7fb8ac0c9fc0 msgr2=0x7fb8ac0ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 -- 172.21.15.67:0/799681282 shutdown_connections 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 -- 172.21.15.67:0/799681282 wait complete. 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb238f95640 1 -- 172.21.15.67:0/543695899 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb23415d790 msgr2=0x7fb23406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb238f95640 1 --2- 172.21.15.67:0/543695899 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb23415d790 0x7fb23406d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb238f95640 1 -- 172.21.15.67:0/543695899 >> v1:172.21.15.176:6789/0 conn(0x7fb234073bf0 legacy=0x7fb234072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.725+0000 7fb238f95640 1 -- 172.21.15.67:0/543695899 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2340c7a60 con 0x7fb2340caf60 2024-04-25T15:54:54.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb238f95640 1 --2- 172.21.15.67:0/543695899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340caf60 0x7fb23415d250 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb22000bb10 tx=0x7fb22001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f54db341ae1525 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 Processor -- start 2024-04-25T15:54:54.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.729+0000 7fb8b148b640 1 -- start start 2024-04-25T15:54:54.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b148b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8ac0c4940 0x7fb8ac167af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b148b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8ac0c9240 0x7fb8ac159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b148b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac203220 0x7fb8ac15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8abfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8ac0c4940 0x7fb8ac167af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8abfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8ac0c4940 0x7fb8ac167af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50566/0 (socket says 172.21.15.67:50566) 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8ab7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8ac0c9240 0x7fb8ac159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b0c8a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac203220 0x7fb8ac15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8ab7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8ac0c9240 0x7fb8ac159e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:32960/0 (socket says 172.21.15.67:32960) 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b0c8a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac203220 0x7fb8ac15a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49608/0 (socket says 172.21.15.67:49608) 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8abfff640 1 -- 172.21.15.67:0/2588394894 learned_addr learned my addr 172.21.15.67:0/2588394894 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac1764a0 con 0x7fb8ac203220 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac176320 con 0x7fb8ac0c9240 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb8ac176620 con 0x7fb8ac0c4940 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb2337fe640 1 -- 172.21.15.67:0/543695899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2200b4070 con 0x7fb2340caf60 2024-04-25T15:54:54.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb2337fe640 1 -- 172.21.15.67:0/543695899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2200a7b00 con 0x7fb2340caf60 2024-04-25T15:54:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b0c8a640 1 -- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8ac0c4940 msgr2=0x7fb8ac167af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b0c8a640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8ac0c4940 0x7fb8ac167af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b0c8a640 1 -- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8ac0c9240 msgr2=0x7fb8ac159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b0c8a640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8ac0c9240 0x7fb8ac159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.733+0000 7fb8b0c8a640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ac168030 con 0x7fb8ac203220 2024-04-25T15:54:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb8b0c8a640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac203220 0x7fb8ac15a340 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0c6fc0 tx=0x7fb8a400f500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb8a8ff9640 1 -- 172.21.15.67:0/2588394894 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4007e20 con 0x7fb8ac203220 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb8a8ff9640 1 -- 172.21.15.67:0/2588394894 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a401a070 con 0x7fb8ac203220 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb8a8ff9640 1 -- 172.21.15.67:0/2588394894 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400b2f0 con 0x7fb8ac203220 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 -- 172.21.15.67:0/543695899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340caf60 msgr2=0x7fb23415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/543695899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340caf60 0x7fb23415d250 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb22000bb10 tx=0x7fb22001ec30 comp rx=0 tx=0).stop 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 -- 172.21.15.67:0/543695899 shutdown_connections 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/543695899 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb23415d790 0x7fb23406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/543695899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2340caf60 0x7fb23415d250 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 -- 172.21.15.67:0/543695899 >> 172.21.15.67:0/543695899 conn(0x7fb234070db0 msgr2=0x7fb2340ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ac15c8c0 con 0x7fb8ac203220 2024-04-25T15:54:54.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ac1672f0 con 0x7fb8ac203220 2024-04-25T15:54:54.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8ac203710 con 0x7fb8ac203220 2024-04-25T15:54:54.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb8a8ff9640 1 -- 172.21.15.67:0/2588394894 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb8a4009540 con 0x7fb8ac203220 2024-04-25T15:54:54.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb8a8ff9640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb89806f430 0x7fb898071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 -- 172.21.15.67:0/543695899 shutdown_connections 2024-04-25T15:54:54.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.737+0000 7fb23b1f9640 1 -- 172.21.15.67:0/543695899 wait complete. 2024-04-25T15:54:54.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb23b1f9640 1 Processor -- start 2024-04-25T15:54:54.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb8abfff640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb89806f430 0x7fb898071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb8abfff640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb89806f430 0x7fb898071940 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0024dc0 tx=0x7fb8a00af070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb23b1f9640 1 -- start start 2024-04-25T15:54:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb23b1f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb234073bf0 0x7fb2340c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb23b1f9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb23415d790 0x7fb2340c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.741+0000 7fb23b1f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23415bbb0 0x7fb234154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb238f95640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb234073bf0 0x7fb2340c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23415bbb0 0x7fb234154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23415bbb0 0x7fb234154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49624/0 (socket says 172.21.15.67:49624) 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb238f95640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb234073bf0 0x7fb2340c4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50574/0 (socket says 172.21.15.67:50574) 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb233fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb23415d790 0x7fb2340c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb233fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb23415d790 0x7fb2340c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:32970/0 (socket says 172.21.15.67:32970) 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 -- 172.21.15.67:0/1445680154 learned_addr learned my addr 172.21.15.67:0/1445680154 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb23406c640 con 0x7fb23415bbb0 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb23415a600 con 0x7fb23415d790 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb23406c7c0 con 0x7fb234073bf0 2024-04-25T15:54:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb8a8ff9640 1 -- 172.21.15.67:0/2588394894 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fb8a408e0d0 con 0x7fb8ac203220 2024-04-25T15:54:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 -- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb234073bf0 msgr2=0x7fb2340c4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb234073bf0 0x7fb2340c4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 -- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb23415d790 msgr2=0x7fb2340c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb23415d790 0x7fb2340c44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2340c5180 con 0x7fb23415bbb0 2024-04-25T15:54:54.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.745+0000 7fb239796640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23415bbb0 0x7fb234154520 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00c760 tx=0x7fb22c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.749+0000 7fb8a8ff9640 1 -- 172.21.15.67:0/2588394894 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb8a4024020 con 0x7fb8ac203220 2024-04-25T15:54:54.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.749+0000 7fb2317fa640 1 -- 172.21.15.67:0/1445680154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c003ba0 con 0x7fb23415bbb0 2024-04-25T15:54:54.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.749+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2340c4a30 con 0x7fb23415bbb0 2024-04-25T15:54:54.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.749+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2341fb3c0 con 0x7fb23415bbb0 2024-04-25T15:54:54.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.749+0000 7fb2317fa640 1 -- 172.21.15.67:0/1445680154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb22c001640 con 0x7fb23415bbb0 2024-04-25T15:54:54.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.753+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2140052a0 con 0x7fb23415bbb0 2024-04-25T15:54:54.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.753+0000 7fb2317fa640 1 -- 172.21.15.67:0/1445680154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22c006b60 con 0x7fb23415bbb0 2024-04-25T15:54:54.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.753+0000 7fb2317fa640 1 -- 172.21.15.67:0/1445680154 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb22c0050d0 con 0x7fb23415bbb0 2024-04-25T15:54:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.753+0000 7fb2317fa640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb21806f400 0x7fb218071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.753+0000 7fb2317fa640 1 -- 172.21.15.67:0/1445680154 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fb22c08fb60 con 0x7fb23415bbb0 2024-04-25T15:54:54.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.753+0000 7fb238f95640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb21806f400 0x7fb218071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.757+0000 7fb238f95640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb21806f400 0x7fb218071910 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb2200b1000 tx=0x7fb2200a9070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.761+0000 7fb2317fa640 1 -- 172.21.15.67:0/1445680154 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb22c020020 con 0x7fb23415bbb0 2024-04-25T15:54:54.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.817+0000 7f523c32a640 1 Processor -- start 2024-04-25T15:54:54.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.817+0000 7f523c32a640 1 -- start start 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523c32a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f52340c9230 0x7f5234174b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523c32a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52340c4f90 con 0x7f5234169c30 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523c32a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52340c8080 con 0x7f523408e940 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523c32a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f52340c8690 con 0x7f52340c9230 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f52398c5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5234169c30 0x7f52340c8b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43314/0 (socket says 172.21.15.67:43314) 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f52398c5640 1 -- 172.21.15.67:0/3671593439 learned_addr learned my addr 172.21.15.67:0/3671593439 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523a8c7640 1 --2- 172.21.15.67:0/3671593439 >> v2:172.21.15.176:3300/0 conn(0x7f52340c9230 0x7f5234174b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f52390c4640 1 -- 172.21.15.67:0/3671593439 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171684513 0 0) 0x7f52340c4f90 con 0x7f5234169c30 2024-04-25T15:54:54.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f52390c4640 1 -- 172.21.15.67:0/3671593439 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5220003530 con 0x7f5234169c30 2024-04-25T15:54:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523a8c7640 1 -- 172.21.15.67:0/3671593439 >> v1:172.21.15.176:6789/0 conn(0x7f523408e940 legacy=0x7f523406ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523a8c7640 1 -- 172.21.15.67:0/3671593439 >> v1:172.21.15.67:6789/0 conn(0x7f5234169c30 legacy=0x7f52340c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.821+0000 7f523a8c7640 1 -- 172.21.15.67:0/3671593439 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523416a040 con 0x7f52340c9230 2024-04-25T15:54:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.825+0000 7f523a8c7640 1 --2- 172.21.15.67:0/3671593439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52340c9230 0x7f5234174b90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52340c8810 tx=0x7f522401e920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25e9e4db25b8e27e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.825+0000 7f52390c4640 1 -- 172.21.15.67:0/3671593439 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52240ad070 con 0x7f52340c9230 2024-04-25T15:54:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.825+0000 7f52390c4640 1 -- 172.21.15.67:0/3671593439 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52240a79f0 con 0x7f52340c9230 2024-04-25T15:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.829+0000 7f523c32a640 1 -- 172.21.15.67:0/3671593439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52340c9230 msgr2=0x7f5234174b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.829+0000 7f523c32a640 1 --2- 172.21.15.67:0/3671593439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52340c9230 0x7f5234174b90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52340c8810 tx=0x7f522401e920 comp rx=0 tx=0).stop 2024-04-25T15:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.829+0000 7f523c32a640 1 -- 172.21.15.67:0/3671593439 shutdown_connections 2024-04-25T15:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.829+0000 7f523c32a640 1 --2- 172.21.15.67:0/3671593439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52340c9230 0x7f5234174b90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.829+0000 7f523c32a640 1 -- 172.21.15.67:0/3671593439 >> 172.21.15.67:0/3671593439 conn(0x7f52340ca050 msgr2=0x7f52340ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.829+0000 7f523c32a640 1 -- 172.21.15.67:0/3671593439 shutdown_connections 2024-04-25T15:54:54.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.829+0000 7f523c32a640 1 -- 172.21.15.67:0/3671593439 wait complete. 2024-04-25T15:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 Processor -- start 2024-04-25T15:54:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 -- start start 2024-04-25T15:54:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f523408e940 0x7f5234163410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52340c9230 0x7f5234206af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5234169c30 0x7f5234160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52340c4590 con 0x7f5234169c30 2024-04-25T15:54:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f52340c9920 con 0x7f523408e940 2024-04-25T15:54:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523c32a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f523406f520 con 0x7f52340c9230 2024-04-25T15:54:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5234169c30 0x7f5234160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5234169c30 0x7f5234160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49638/0 (socket says 172.21.15.67:49638) 2024-04-25T15:54:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 -- 172.21.15.67:0/333993451 learned_addr learned my addr 172.21.15.67:0/333993451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a0c6640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f523408e940 0x7f5234163410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f52398c5640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52340c9230 0x7f5234206af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 -- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52340c9230 msgr2=0x7f5234206af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52340c9230 0x7f5234206af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 -- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f523408e940 msgr2=0x7f5234163410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f523408e940 0x7f5234163410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 -- 172.21.15.67:0/333993451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5234207030 con 0x7f5234169c30 2024-04-25T15:54:54.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f523a8c7640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5234169c30 0x7f5234160210 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f522401b530 tx=0x7f522401b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.833+0000 7f522affd640 1 -- 172.21.15.67:0/333993451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52240ad050 con 0x7f5234169c30 2024-04-25T15:54:54.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523416b930 con 0x7f5234169c30 2024-04-25T15:54:54.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52342062f0 con 0x7f5234169c30 2024-04-25T15:54:54.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f522affd640 1 -- 172.21.15.67:0/333993451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5224004350 con 0x7f5234169c30 2024-04-25T15:54:54.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f522affd640 1 -- 172.21.15.67:0/333993451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5224003860 con 0x7f5234169c30 2024-04-25T15:54:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f522affd640 1 -- 172.21.15.67:0/333993451 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f52240a79f0 con 0x7f5234169c30 2024-04-25T15:54:54.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f522affd640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f522006f8a0 0x7f5220071db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:54.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f522affd640 1 -- 172.21.15.67:0/333993451 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f5224129b40 con 0x7f5234169c30 2024-04-25T15:54:54.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.837+0000 7f523a0c6640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f522006f8a0 0x7f5220071db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.841+0000 7f523a0c6640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f522006f8a0 0x7f5220071db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5230007fd0 tx=0x7f5230009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.841+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5234050750 con 0x7f5234169c30 2024-04-25T15:54:54.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.849+0000 7f522affd640 1 -- 172.21.15.67:0/333993451 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5224018540 con 0x7f5234169c30 2024-04-25T15:54:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.953+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff4840c7700 con 0x7ff484152df0 2024-04-25T15:54:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff46b7fe640 1 -- 172.21.15.67:0/3665300985 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff47c0f8880 con 0x7ff484152df0 2024-04-25T15:54:54.958 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe17c005590 con 0x7fe1940c52e0 2024-04-25T15:54:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7fe18affd640 1 -- 172.21.15.67:0/1966075341 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe190060090 con 0x7fe1940c52e0 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff47006f530 msgr2=0x7ff470071a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff47006f530 0x7ff470071a40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff474009ac0 tx=0x7ff474009190 comp rx=0 tx=0).stop 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 msgr2=0x7ff484146530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484146530 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff4840ce940 tx=0x7ff47c01bb20 comp rx=0 tx=0).stop 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 shutdown_connections 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff47006f530 0x7ff470071a40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff484152df0 0x7ff484146530 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4841529e0 0x7ff484143f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 --2- 172.21.15.67:0/3665300985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4840c6330 0x7ff4840ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.957+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 >> 172.21.15.67:0/3665300985 conn(0x7ff48406fe70 msgr2=0x7ff484151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.961 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.961+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 shutdown_connections 2024-04-25T15:54:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.961+0000 7ff48946f640 1 -- 172.21.15.67:0/3665300985 wait complete. 2024-04-25T15:54:54.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe17806f500 msgr2=0x7fe178071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe17806f500 0x7fe178071a10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0b1000 tx=0x7fe18c0aa000 comp rx=0 tx=0).stop 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 msgr2=0x7fe1941fb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe1941fb0c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe19000d5f0 tx=0x7fe19000dab0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 shutdown_connections 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe17806f500 0x7fe178071a10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1941646d0 0x7fe194152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1940c5e70 0x7fe1941eabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 --2- 172.21.15.67:0/1966075341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1940c52e0 0x7fe1941fb0c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 >> 172.21.15.67:0/1966075341 conn(0x7fe19406fe70 msgr2=0x7fe194150c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 shutdown_connections 2024-04-25T15:54:54.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:54.965+0000 7fe19c673640 1 -- 172.21.15.67:0/1966075341 wait complete. 2024-04-25T15:54:54.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574908 for osd.6 2024-04-25T15:54:54.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574908 for osd.7 2024-04-25T15:54:55.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.017+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3e58005860 con 0x7f3e700689a0 2024-04-25T15:54:55.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.017+0000 7f3e667fc640 1 -- 172.21.15.67:0/3663595803 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3e6c05f120 con 0x7f3e700689a0 2024-04-25T15:54:55.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.017+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fcf800c6f50 con 0x7fcf8015d110 2024-04-25T15:54:55.020 INFO:teuthology.orchestra.run.smithi067.stdout:55834574907 2024-04-25T15:54:55.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.017+0000 7fcf767fc640 1 -- 172.21.15.67:0/1154300000 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcf780ffe20 con 0x7fcf8015d110 2024-04-25T15:54:55.020 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:55.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3e5406f400 msgr2=0x7f3e54071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3e5406f400 0x7f3e54071910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3e600ae000 tx=0x7f3e600a7000 comp rx=0 tx=0).stop 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e700689a0 msgr2=0x7f3e701f43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e700689a0 0x7f3e701f43d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c430 tx=0x7f3e6c00c8f0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 shutdown_connections 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3e5406f400 0x7f3e54071910 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e7015cce0 0x7f3e7015b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e700689a0 0x7f3e701f43d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 --2- 172.21.15.67:0/3663595803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e70068590 0x7f3e701636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 >> 172.21.15.67:0/3663595803 conn(0x7f3e700ca960 msgr2=0x7f3e70165670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 shutdown_connections 2024-04-25T15:54:55.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7f3e76dcd640 1 -- 172.21.15.67:0/3663595803 wait complete. 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcf6406f4d0 msgr2=0x7fcf640719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcf6406f4d0 0x7fcf640719e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009f70 tx=0x7fcf70009190 comp rx=0 tx=0).stop 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8015d110 msgr2=0x7fcf80073690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8015d110 0x7fcf80073690 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fcf780ae000 tx=0x7fcf7801bfa0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 shutdown_connections 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcf6406f4d0 0x7fcf640719e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8015d110 0x7fcf80073690 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf800cf000 0x7fcf80073150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 --2- 172.21.15.67:0/1154300000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf800cd820 0x7fcf801faa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 >> 172.21.15.67:0/1154300000 conn(0x7fcf80068990 msgr2=0x7fcf800d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 shutdown_connections 2024-04-25T15:54:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.021+0000 7fcf87ef5640 1 -- 172.21.15.67:0/1154300000 wait complete. 2024-04-25T15:54:55.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574908 got 55834574907 for osd.5 2024-04-25T15:54:55.054 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574908 for osd.3 2024-04-25T15:54:55.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.053+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb8ac0c97a0 con 0x7fb8ac203220 2024-04-25T15:54:55.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.053+0000 7fb8a8ff9640 1 -- 172.21.15.67:0/2588394894 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb8a405c3c0 con 0x7fb8ac203220 2024-04-25T15:54:55.056 INFO:teuthology.orchestra.run.smithi067.stdout:55834574907 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb89806f430 msgr2=0x7fb898071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb89806f430 0x7fb898071940 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0024dc0 tx=0x7fb8a00af070 comp rx=0 tx=0).stop 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac203220 msgr2=0x7fb8ac15a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac203220 0x7fb8ac15a340 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0c6fc0 tx=0x7fb8a400f500 comp rx=0 tx=0).stop 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 shutdown_connections 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb89806f430 0x7fb898071940 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8ac203220 0x7fb8ac15a340 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8ac0c9240 0x7fb8ac159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 --2- 172.21.15.67:0/2588394894 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8ac0c4940 0x7fb8ac167af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 >> 172.21.15.67:0/2588394894 conn(0x7fb8ac0c9fc0 msgr2=0x7fb8ac075e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 shutdown_connections 2024-04-25T15:54:55.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.057+0000 7fb8b148b640 1 -- 172.21.15.67:0/2588394894 wait complete. 2024-04-25T15:54:55.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.061+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fede4005590 con 0x7fee000ce700 2024-04-25T15:54:55.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.061+0000 7fedeeffd640 1 -- 172.21.15.67:0/388651960 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fedf80a5e20 con 0x7fee000ce700 2024-04-25T15:54:55.063 INFO:teuthology.orchestra.run.smithi067.stdout:55834574906 2024-04-25T15:54:55.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fede006f430 msgr2=0x7fede0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fede006f430 0x7fede0071940 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009ac0 tx=0x7fedf0009190 comp rx=0 tx=0).stop 2024-04-25T15:54:55.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee000ce700 msgr2=0x7fee001533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee000ce700 0x7fee001533d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fedf80b0040 tx=0x7fedf8019750 comp rx=0 tx=0).stop 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 shutdown_connections 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fede006f430 0x7fede0071940 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee000ce700 0x7fee001533d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee0015d790 0x7fee00155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 --2- 172.21.15.67:0/388651960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fee000caf60 0x7fee001563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 >> 172.21.15.67:0/388651960 conn(0x7fee00070db0 msgr2=0x7fee0006c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 shutdown_connections 2024-04-25T15:54:55.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.065+0000 7fee04d1b640 1 -- 172.21.15.67:0/388651960 wait complete. 2024-04-25T15:54:55.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb214005860 con 0x7fb23415bbb0 2024-04-25T15:54:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb2317fa640 1 -- 172.21.15.67:0/1445680154 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb22c05c340 con 0x7fb23415bbb0 2024-04-25T15:54:55.069 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb21806f400 msgr2=0x7fb218071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb21806f400 0x7fb218071910 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb2200b1000 tx=0x7fb2200a9070 comp rx=0 tx=0).stop 2024-04-25T15:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23415bbb0 msgr2=0x7fb234154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23415bbb0 0x7fb234154520 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00c760 tx=0x7fb22c00cc20 comp rx=0 tx=0).stop 2024-04-25T15:54:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 shutdown_connections 2024-04-25T15:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb21806f400 0x7fb218071910 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb23415bbb0 0x7fb234154520 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb23415d790 0x7fb2340c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 --2- 172.21.15.67:0/1445680154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb234073bf0 0x7fb2340c4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 >> 172.21.15.67:0/1445680154 conn(0x7fb234070db0 msgr2=0x7fb234158ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 shutdown_connections 2024-04-25T15:54:55.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.069+0000 7fb23b1f9640 1 -- 172.21.15.67:0/1445680154 wait complete. 2024-04-25T15:54:55.079 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574908 got 55834574907 for osd.4 2024-04-25T15:54:55.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574908 for osd.1 2024-04-25T15:54:55.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574907 got 55834574906 for osd.0 2024-04-25T15:54:55.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f52340c9ba0 con 0x7f5234169c30 2024-04-25T15:54:55.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f522affd640 1 -- 172.21.15.67:0/333993451 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f52240f6900 con 0x7f5234169c30 2024-04-25T15:54:55.101 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:55.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f522006f8a0 msgr2=0x7f5220071db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f522006f8a0 0x7f5220071db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5230007fd0 tx=0x7f5230009040 comp rx=0 tx=0).stop 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5234169c30 msgr2=0x7f5234160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5234169c30 0x7f5234160210 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f522401b530 tx=0x7f522401b6c0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 shutdown_connections 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f522006f8a0 0x7f5220071db0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5234169c30 0x7f5234160210 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52340c9230 0x7f5234206af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 --2- 172.21.15.67:0/333993451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f523408e940 0x7f5234163410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 >> 172.21.15.67:0/333993451 conn(0x7f52340ca050 msgr2=0x7f5234073b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 shutdown_connections 2024-04-25T15:54:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:55.101+0000 7f523c32a640 1 -- 172.21.15.67:0/333993451 wait complete. 2024-04-25T15:54:55.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574908 for osd.2 2024-04-25T15:54:55.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T15:54:55.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T15:54:56.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T15:54:56.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T15:54:56.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T15:54:56.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T15:54:56.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T15:54:56.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T15:54:56.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.197+0000 7f8580da7640 1 Processor -- start 2024-04-25T15:54:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.197+0000 7f8580da7640 1 -- start start 2024-04-25T15:54:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8580da7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c152c40 0x7f857c0c5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8580da7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f857c15bc20 0x7f857c15c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8580da7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f857c06ee10 0x7f857c06f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8580da7640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f857c1587f0 con 0x7f857c06ee10 2024-04-25T15:54:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8580da7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f857c0748f0 con 0x7f857c15bc20 2024-04-25T15:54:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8580da7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f857c158a60 con 0x7f857c152c40 2024-04-25T15:54:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f857c06ee10 0x7f857c06f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8579d9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f857c15bc20 0x7f857c15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c152c40 0x7f857c0c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f8579d9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f857c15bc20 0x7f857c15c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:32994/0 (socket says 172.21.15.67:32994) 2024-04-25T15:54:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c152c40 0x7f857c0c5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50596/0 (socket says 172.21.15.67:50596) 2024-04-25T15:54:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 -- 172.21.15.67:0/3701317732 learned_addr learned my addr 172.21.15.67:0/3701317732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857ad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f857c06ee10 0x7f857c06f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49652/0 (socket says 172.21.15.67:49652) 2024-04-25T15:54:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 -- 172.21.15.67:0/3701317732 >> v2:172.21.15.176:3300/0 conn(0x7f857c15bc20 msgr2=0x7f857c15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 --2- 172.21.15.67:0/3701317732 >> v2:172.21.15.176:3300/0 conn(0x7f857c15bc20 0x7f857c15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 -- 172.21.15.67:0/3701317732 >> v2:172.21.15.67:3300/0 conn(0x7f857c06ee10 msgr2=0x7f857c06f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 --2- 172.21.15.67:0/3701317732 >> v2:172.21.15.67:3300/0 conn(0x7f857c06ee10 0x7f857c06f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.201+0000 7f857a59c640 1 -- 172.21.15.67:0/3701317732 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c158d40 con 0x7f857c152c40 2024-04-25T15:54:56.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.205+0000 7f857a59c640 1 --2- 172.21.15.67:0/3701317732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c152c40 0x7f857c0c5800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f857c06f8f0 tx=0x7f856c01f4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c7b8d02d4e0a0eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f857959a640 1 -- 172.21.15.67:0/3701317732 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c0b4070 con 0x7f857c152c40 2024-04-25T15:54:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f857959a640 1 -- 172.21.15.67:0/3701317732 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f856c007ad0 con 0x7f857c152c40 2024-04-25T15:54:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 -- 172.21.15.67:0/3701317732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c152c40 msgr2=0x7f857c0c5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 --2- 172.21.15.67:0/3701317732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c152c40 0x7f857c0c5800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f857c06f8f0 tx=0x7f856c01f4b0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 -- 172.21.15.67:0/3701317732 shutdown_connections 2024-04-25T15:54:56.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 --2- 172.21.15.67:0/3701317732 >> v2:172.21.15.67:3300/0 conn(0x7f857c06ee10 0x7f857c06f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 --2- 172.21.15.67:0/3701317732 >> v2:172.21.15.176:3300/0 conn(0x7f857c15bc20 0x7f857c15c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 --2- 172.21.15.67:0/3701317732 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c152c40 0x7f857c0c5800 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 -- 172.21.15.67:0/3701317732 >> 172.21.15.67:0/3701317732 conn(0x7f857c0ca050 msgr2=0x7f857c0ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 -- 172.21.15.67:0/3701317732 shutdown_connections 2024-04-25T15:54:56.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.209+0000 7f8580da7640 1 -- 172.21.15.67:0/3701317732 wait complete. 2024-04-25T15:54:56.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 Processor -- start 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 -- start start 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f857c06ee10 0x7f857c15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f857c152c40 0x7f857c0cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c15bc20 0x7f857c1e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f857a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f857c06ee10 0x7f857c15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f857a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f857c06ee10 0x7f857c15b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49658/0 (socket says 172.21.15.67:49658) 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f857a59c640 1 -- 172.21.15.67:0/4017527802 learned_addr learned my addr 172.21.15.67:0/4017527802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8579d9b640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f857c152c40 0x7f857c0cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f857c159540 con 0x7f857c06ee10 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f857ad9d640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c15bc20 0x7f857c1e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f857c074bd0 con 0x7f857c152c40 2024-04-25T15:54:56.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.213+0000 7f8580da7640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f857c1596c0 con 0x7f857c15bc20 2024-04-25T15:54:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.217+0000 7f857a59c640 1 -- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c15bc20 msgr2=0x7f857c1e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.217+0000 7f857a59c640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c15bc20 0x7f857c1e8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.217+0000 7f857a59c640 1 -- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f857c152c40 msgr2=0x7f857c0cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.217+0000 7f857a59c640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f857c152c40 0x7f857c0cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.217+0000 7f857a59c640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c1e9250 con 0x7f857c06ee10 2024-04-25T15:54:56.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.217+0000 7f857a59c640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f857c06ee10 0x7f857c15b600 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f856c01fd50 tx=0x7f856c01c250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.221+0000 7f856affd640 1 -- 172.21.15.67:0/4017527802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c0b4070 con 0x7f857c06ee10 2024-04-25T15:54:56.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.221+0000 7f856affd640 1 -- 172.21.15.67:0/4017527802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f856c0b5040 con 0x7f857c06ee10 2024-04-25T15:54:56.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.221+0000 7f856affd640 1 -- 172.21.15.67:0/4017527802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c007450 con 0x7f857c06ee10 2024-04-25T15:54:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.221+0000 7f8580da7640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857c1f77b0 con 0x7f857c06ee10 2024-04-25T15:54:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.221+0000 7f8580da7640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857c1e87f0 con 0x7f857c06ee10 2024-04-25T15:54:56.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.225+0000 7f856affd640 1 -- 172.21.15.67:0/4017527802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f856c0c0030 con 0x7f857c06ee10 2024-04-25T15:54:56.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.225+0000 7f856affd640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f855c06f4d0 0x7f855c0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.225+0000 7f856affd640 1 -- 172.21.15.67:0/4017527802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f856c12c200 con 0x7f857c06ee10 2024-04-25T15:54:56.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.225+0000 7f8579d9b640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f855c06f4d0 0x7f855c0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.229+0000 7f8579d9b640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f855c06f4d0 0x7f855c0719e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f857c1f48c0 tx=0x7f857400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.233+0000 7f8580da7640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f857c050750 con 0x7f857c06ee10 2024-04-25T15:54:56.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.241+0000 7f856affd640 1 -- 172.21.15.67:0/4017527802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f856c0c7020 con 0x7f857c06ee10 2024-04-25T15:54:56.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6058917640 1 Processor -- start 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6058917640 1 -- start start 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6058917640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60540c7130 0x7f605415df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6058917640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f605415e450 0x7f6054164ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6058917640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6054154a20 con 0x7f60540c6d20 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6058917640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f60540c9990 con 0x7f60540c7130 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6058917640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f60540c9c70 con 0x7f605415e450 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f605259c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f60540c6d20 0x7f60540c7650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43322/0 (socket says 172.21.15.67:43322) 2024-04-25T15:54:56.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f605259c640 1 -- 172.21.15.67:0/3337015857 learned_addr learned my addr 172.21.15.67:0/3337015857 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6052d9d640 1 --2- 172.21.15.67:0/3337015857 >> v2:172.21.15.176:3300/0 conn(0x7f605415e450 0x7f6054164ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.265+0000 7f6051d9b640 1 --2- 172.21.15.67:0/3337015857 >> v2:172.21.15.67:3300/0 conn(0x7f60540c7130 0x7f605415df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f605159a640 1 -- 172.21.15.67:0/3337015857 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2325874095 0 0) 0x7f6044001120 con 0x7f60540c6d20 2024-04-25T15:54:56.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f6052d9d640 1 -- 172.21.15.67:0/3337015857 >> v2:172.21.15.67:3300/0 conn(0x7f60540c7130 msgr2=0x7f605415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f6052d9d640 1 --2- 172.21.15.67:0/3337015857 >> v2:172.21.15.67:3300/0 conn(0x7f60540c7130 0x7f605415df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f6052d9d640 1 -- 172.21.15.67:0/3337015857 >> v1:172.21.15.67:6789/0 conn(0x7f60540c6d20 legacy=0x7f60540c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f6052d9d640 1 -- 172.21.15.67:0/3337015857 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60540ca2a0 con 0x7f605415e450 2024-04-25T15:54:56.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f6051d9b640 1 --2- 172.21.15.67:0/3337015857 >> v2:172.21.15.67:3300/0 conn(0x7f60540c7130 0x7f605415df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:56.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f6052d9d640 1 --2- 172.21.15.67:0/3337015857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f605415e450 0x7f6054164ec0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f605406d650 tx=0x7f604c01f110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b9ea13d888fa50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.269+0000 7f605159a640 1 -- 172.21.15.67:0/3337015857 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c0b4070 con 0x7f605415e450 2024-04-25T15:54:56.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f605159a640 1 -- 172.21.15.67:0/3337015857 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604c0af600 con 0x7f605415e450 2024-04-25T15:54:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 -- 172.21.15.67:0/3337015857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f605415e450 msgr2=0x7f6054164ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 --2- 172.21.15.67:0/3337015857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f605415e450 0x7f6054164ec0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f605406d650 tx=0x7f604c01f110 comp rx=0 tx=0).stop 2024-04-25T15:54:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 -- 172.21.15.67:0/3337015857 shutdown_connections 2024-04-25T15:54:56.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 --2- 172.21.15.67:0/3337015857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f605415e450 0x7f6054164ec0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 --2- 172.21.15.67:0/3337015857 >> v2:172.21.15.67:3300/0 conn(0x7f60540c7130 0x7f605415df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 -- 172.21.15.67:0/3337015857 >> 172.21.15.67:0/3337015857 conn(0x7f6054071ba0 msgr2=0x7f6054068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 -- 172.21.15.67:0/3337015857 shutdown_connections 2024-04-25T15:54:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.273+0000 7f6058917640 1 -- 172.21.15.67:0/3337015857 wait complete. 2024-04-25T15:54:56.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 Processor -- start 2024-04-25T15:54:56.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 -- start start 2024-04-25T15:54:56.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7fdc9bc89640 1 Processor -- start 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60540c6d20 0x7f60541fab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f605415e450 0x7f605406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60541492d0 0x7f6054158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f605415e860 con 0x7f60540c6d20 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f60540ca860 con 0x7f60541492d0 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6058917640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f60540c9650 con 0x7f605415e450 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6051d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f605415e450 0x7f605406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f605259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60540c6d20 0x7f60541fab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f605259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60540c6d20 0x7f60541fab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49672/0 (socket says 172.21.15.67:49672) 2024-04-25T15:54:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f605259c640 1 -- 172.21.15.67:0/155379975 learned_addr learned my addr 172.21.15.67:0/155379975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7f6052d9d640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60541492d0 0x7f6054158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7fdc9bc89640 1 -- start start 2024-04-25T15:54:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7fdc9bc89640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc940c6330 0x7fdc941588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7fdc9bc89640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc941529e0 0x7fdc9406c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7fdc9bc89640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc94158e20 con 0x7fdc94152df0 2024-04-25T15:54:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7fdc9bc89640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdc940c5240 con 0x7fdc940c6330 2024-04-25T15:54:56.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.277+0000 7fdc9bc89640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdc940c43e0 con 0x7fdc941529e0 2024-04-25T15:54:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6051d9b640 1 -- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60541492d0 msgr2=0x7f6054158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6051d9b640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60541492d0 0x7f6054158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6051d9b640 1 -- 172.21.15.67:0/155379975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60540c6d20 msgr2=0x7f60541fab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6051d9b640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60540c6d20 0x7f60541fab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6051d9b640 1 -- 172.21.15.67:0/155379975 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60541fb0a0 con 0x7f605415e450 2024-04-25T15:54:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6052d9d640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60541492d0 0x7f6054158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6051d9b640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f605415e450 0x7f605406eb60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f604800d550 tx=0x7f604800da10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99a25640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc940c6330 0x7fdc941588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99a25640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc940c6330 0x7fdc941588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49684/0 (socket says 172.21.15.67:49684) 2024-04-25T15:54:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99a25640 1 -- 172.21.15.67:0/2378693319 learned_addr learned my addr 172.21.15.67:0/2378693319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6042ffd640 1 -- 172.21.15.67:0/155379975 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048004030 con 0x7f605415e450 2024-04-25T15:54:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6042ffd640 1 -- 172.21.15.67:0/155379975 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60480041d0 con 0x7f605415e450 2024-04-25T15:54:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6042ffd640 1 -- 172.21.15.67:0/155379975 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048003620 con 0x7f605415e450 2024-04-25T15:54:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6054159c00 con 0x7f605415e450 2024-04-25T15:54:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f605415a1c0 con 0x7f605415e450 2024-04-25T15:54:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99224640 1 --2- 172.21.15.67:0/2378693319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc941529e0 0x7fdc9406c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99a25640 1 -- 172.21.15.67:0/2378693319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc941529e0 msgr2=0x7fdc9406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99a25640 1 --2- 172.21.15.67:0/2378693319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc941529e0 0x7fdc9406c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99a25640 1 -- 172.21.15.67:0/2378693319 >> v1:172.21.15.176:6789/0 conn(0x7fdc94152df0 legacy=0x7fdc9415be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7fdc99a25640 1 -- 172.21.15.67:0/2378693319 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc940c9ce0 con 0x7fdc940c6330 2024-04-25T15:54:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.281+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60540cd8d0 con 0x7f605415e450 2024-04-25T15:54:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc99a25640 1 --2- 172.21.15.67:0/2378693319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc940c6330 0x7fdc941588e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c007980 tx=0x7fdc8c020fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7252d80970b4568 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc98a23640 1 -- 172.21.15.67:0/2378693319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c0b7070 con 0x7fdc940c6330 2024-04-25T15:54:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc98a23640 1 -- 172.21.15.67:0/2378693319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8c0b2410 con 0x7fdc940c6330 2024-04-25T15:54:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 -- 172.21.15.67:0/2378693319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc940c6330 msgr2=0x7fdc941588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/2378693319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc940c6330 0x7fdc941588e0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c007980 tx=0x7fdc8c020fb0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 -- 172.21.15.67:0/2378693319 shutdown_connections 2024-04-25T15:54:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/2378693319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc941529e0 0x7fdc9406c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/2378693319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc940c6330 0x7fdc941588e0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 -- 172.21.15.67:0/2378693319 >> 172.21.15.67:0/2378693319 conn(0x7fdc9406fe70 msgr2=0x7fdc940cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7f6042ffd640 1 -- 172.21.15.67:0/155379975 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f60480037c0 con 0x7f605415e450 2024-04-25T15:54:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7f6042ffd640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f603406f400 0x7f6034071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7f6042ffd640 1 -- 172.21.15.67:0/155379975 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f6048090090 con 0x7f605415e450 2024-04-25T15:54:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 -- 172.21.15.67:0/2378693319 shutdown_connections 2024-04-25T15:54:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7fdc9bc89640 1 -- 172.21.15.67:0/2378693319 wait complete. 2024-04-25T15:54:56.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.285+0000 7f605259c640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f603406f400 0x7f6034071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7f605259c640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f603406f400 0x7f6034071910 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6044000f70 tx=0x7f6044007240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 Processor -- start 2024-04-25T15:54:56.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 -- start start 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc941529e0 0x7fdc940cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc94152df0 0x7fdc9414f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc9415ad40 0x7fdc94147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc940c4da0 con 0x7fdc9415ad40 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdc940c6040 con 0x7fdc941529e0 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.289+0000 7fdc9bc89640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdc940c4f20 con 0x7fdc94152df0 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc9415ad40 0x7fdc94147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc99a25640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc941529e0 0x7fdc940cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc9415ad40 0x7fdc94147d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49700/0 (socket says 172.21.15.67:49700) 2024-04-25T15:54:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 -- 172.21.15.67:0/274219579 learned_addr learned my addr 172.21.15.67:0/274219579 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc99224640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc94152df0 0x7fdc9414f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 -- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc94152df0 msgr2=0x7fdc9414f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc94152df0 0x7fdc9414f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 -- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc941529e0 msgr2=0x7fdc940cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc941529e0 0x7fdc940cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 -- 172.21.15.67:0/274219579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc94074a10 con 0x7fdc9415ad40 2024-04-25T15:54:56.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9a226640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc9415ad40 0x7fdc94147d30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000c390 tx=0x7fdc9000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27f59e640 1 Processor -- start 2024-04-25T15:54:56.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27f59e640 1 -- start start 2024-04-25T15:54:56.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc8a7fc640 1 -- 172.21.15.67:0/274219579 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc900052d0 con 0x7fdc9415ad40 2024-04-25T15:54:56.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc8a7fc640 1 -- 172.21.15.67:0/274219579 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc90005470 con 0x7fdc9415ad40 2024-04-25T15:54:56.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc8a7fc640 1 -- 172.21.15.67:0/274219579 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc90007240 con 0x7fdc9415ad40 2024-04-25T15:54:56.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc940cb840 con 0x7fdc9415ad40 2024-04-25T15:54:56.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc9415b260 con 0x7fdc9415ad40 2024-04-25T15:54:56.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27f59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278037410 0x7fe2780d95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27f59e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe278050060 0x7fe27804a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27f59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe278050c40 con 0x7fe278050470 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27f59e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe278037820 con 0x7fe278050060 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27f59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe2780499c0 con 0x7fe278037410 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27dd9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe278050060 0x7fe27804a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27dd9b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe278050060 0x7fe27804a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:33030/0 (socket says 172.21.15.67:33030) 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27dd9b640 1 -- 172.21.15.67:0/1488945262 learned_addr learned my addr 172.21.15.67:0/1488945262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.293+0000 7fe27ed9d640 1 --1- 172.21.15.67:0/1488945262 >> v1:172.21.15.176:6789/0 conn(0x7fe278050470 0x7fe2780e2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56930/0 (socket says 172.21.15.67:56930) 2024-04-25T15:54:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27e59c640 1 --2- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278037410 0x7fe2780d95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27dd9b640 1 -- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278037410 msgr2=0x7fe2780d95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27dd9b640 1 --2- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278037410 0x7fe2780d95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27dd9b640 1 -- 172.21.15.67:0/1488945262 >> v1:172.21.15.176:6789/0 conn(0x7fe278050470 legacy=0x7fe2780e2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27dd9b640 1 -- 172.21.15.67:0/1488945262 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe278049e90 con 0x7fe278050060 2024-04-25T15:54:56.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7c0052a0 con 0x7fdc9415ad40 2024-04-25T15:54:56.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fdc8a7fc640 1 -- 172.21.15.67:0/274219579 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdc900073e0 con 0x7fdc9415ad40 2024-04-25T15:54:56.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27dd9b640 1 --2- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050060 0x7fe27804a990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe270009870 tx=0x7fe27001ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b821b5e079e67b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fdc8a7fc640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc7806f450 0x7fdc78071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27d59a640 1 -- 172.21.15.67:0/1488945262 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700b5070 con 0x7fe278050060 2024-04-25T15:54:56.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fdc8a7fc640 1 -- 172.21.15.67:0/274219579 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fdc9008ee40 con 0x7fdc9415ad40 2024-04-25T15:54:56.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27d59a640 1 -- 172.21.15.67:0/1488945262 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2700a8a50 con 0x7fe278050060 2024-04-25T15:54:56.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fe27d59a640 1 -- 172.21.15.67:0/1488945262 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2700a8e20 con 0x7fe278050060 2024-04-25T15:54:56.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.297+0000 7fdc99a25640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc7806f450 0x7fdc78071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050060 msgr2=0x7fe27804a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 --2- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050060 0x7fe27804a990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe270009870 tx=0x7fe27001ed50 comp rx=0 tx=0).stop 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- 172.21.15.67:0/1488945262 shutdown_connections 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 --2- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050060 0x7fe27804a990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 --2- 172.21.15.67:0/1488945262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278037410 0x7fe2780d95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- 172.21.15.67:0/1488945262 >> 172.21.15.67:0/1488945262 conn(0x7fe2780547c0 msgr2=0x7fe278054bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- 172.21.15.67:0/1488945262 shutdown_connections 2024-04-25T15:54:56.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fdc99a25640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc7806f450 0x7fdc78071960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0b4000 tx=0x7fdc8c0ab000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- 172.21.15.67:0/1488945262 wait complete. 2024-04-25T15:54:56.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7f6042ffd640 1 -- 172.21.15.67:0/155379975 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f604805c860 con 0x7f605415e450 2024-04-25T15:54:56.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 Processor -- start 2024-04-25T15:54:56.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- start start 2024-04-25T15:54:56.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe278037410 0x7fe2780d8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278050060 0x7fe2781797b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050470 0x7fe2780cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe27804a170 con 0x7fe278037410 2024-04-25T15:54:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe2780e0170 con 0x7fe278050470 2024-04-25T15:54:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27f59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe278049660 con 0x7fe278050060 2024-04-25T15:54:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe278037410 0x7fe2780d8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe278037410 0x7fe2780d8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49710/0 (socket says 172.21.15.67:49710) 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27e59c640 1 -- 172.21.15.67:0/1864498878 learned_addr learned my addr 172.21.15.67:0/1864498878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.301+0000 7fe27ed9d640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050470 0x7fe2780cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.305+0000 7fe27dd9b640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278050060 0x7fe2781797b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.305+0000 7fe27e59c640 1 -- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278050060 msgr2=0x7fe2781797b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.305+0000 7fe27e59c640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278050060 0x7fe2781797b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.305+0000 7fe27e59c640 1 -- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050470 msgr2=0x7fe2780cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.305+0000 7fe27e59c640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050470 0x7fe2780cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.305+0000 7fe27e59c640 1 -- 172.21.15.67:0/1864498878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2780dd110 con 0x7fe278037410 2024-04-25T15:54:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.305+0000 7fe27e59c640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe278037410 0x7fe2780d8b20 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe27400ae10 tx=0x7fe274007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.309+0000 7fdc8a7fc640 1 -- 172.21.15.67:0/274219579 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdc9005b680 con 0x7fdc9415ad40 2024-04-25T15:54:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.309+0000 7fe26effd640 1 -- 172.21.15.67:0/1864498878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274014020 con 0x7fe278037410 2024-04-25T15:54:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.309+0000 7fe27f59e640 1 -- 172.21.15.67:0/1864498878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2781765a0 con 0x7fe278037410 2024-04-25T15:54:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.309+0000 7fe27f59e640 1 -- 172.21.15.67:0/1864498878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2781762c0 con 0x7fe278037410 2024-04-25T15:54:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.309+0000 7fe26effd640 1 -- 172.21.15.67:0/1864498878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2740141c0 con 0x7fe278037410 2024-04-25T15:54:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.309+0000 7fe26effd640 1 -- 172.21.15.67:0/1864498878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274034060 con 0x7fe278037410 2024-04-25T15:54:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.313+0000 7f043d0d3640 1 Processor -- start 2024-04-25T15:54:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.313+0000 7fe26effd640 1 -- 172.21.15.67:0/1864498878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe2740145b0 con 0x7fe278037410 2024-04-25T15:54:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.313+0000 7fe26effd640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe26006f400 0x7fe260071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.313+0000 7fe26effd640 1 -- 172.21.15.67:0/1864498878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fe27409fe60 con 0x7fe278037410 2024-04-25T15:54:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.313+0000 7fe27dd9b640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe26006f400 0x7fe260071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7fe27dd9b640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe26006f400 0x7fe260071910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe2700b2000 tx=0x7fe2700aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.313+0000 7f043d0d3640 1 -- start start 2024-04-25T15:54:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043d0d3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f043815ae00 0x7f0438072930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043d0d3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f04380732d0 0x7f04380736e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043d0d3640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0438164670 0x7f0438164a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043d0d3640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f04380c99a0 con 0x7f0438164670 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043d0d3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f04381641e0 con 0x7f04380732d0 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043d0d3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0438073c20 con 0x7f043815ae00 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0438164670 0x7f0438164a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0438164670 0x7f0438164a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49716/0 (socket says 172.21.15.67:49716) 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 -- 172.21.15.67:0/1751756633 learned_addr learned my addr 172.21.15.67:0/1751756633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043659c640 1 --2- 172.21.15.67:0/1751756633 >> v2:172.21.15.176:3300/0 conn(0x7f04380732d0 0x7f04380736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 -- 172.21.15.67:0/1751756633 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f043815ae00 msgr2=0x7f0438072930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T15:54:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 --2- 172.21.15.67:0/1751756633 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f043815ae00 0x7f0438072930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 -- 172.21.15.67:0/1751756633 >> v2:172.21.15.176:3300/0 conn(0x7f04380732d0 msgr2=0x7f04380736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 --2- 172.21.15.67:0/1751756633 >> v2:172.21.15.176:3300/0 conn(0x7f04380732d0 0x7f04380736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043759e640 1 -- 172.21.15.67:0/1751756633 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04380ca710 con 0x7f0438164670 2024-04-25T15:54:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.317+0000 7f043659c640 1 --2- 172.21.15.67:0/1751756633 >> v2:172.21.15.176:3300/0 conn(0x7f04380732d0 0x7f04380736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:56.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043759e640 1 --2- 172.21.15.67:0/1751756633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0438164670 0x7f0438164a80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f043806d6b0 tx=0x7f042801f200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809c07ce851b3ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f0435d9b640 1 -- 172.21.15.67:0/1751756633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04280b4070 con 0x7f0438164670 2024-04-25T15:54:56.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f0435d9b640 1 -- 172.21.15.67:0/1751756633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04280af400 con 0x7f0438164670 2024-04-25T15:54:56.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043d0d3640 1 -- 172.21.15.67:0/1751756633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0438164670 msgr2=0x7f0438164a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1751756633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0438164670 0x7f0438164a80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f043806d6b0 tx=0x7f042801f200 comp rx=0 tx=0).stop 2024-04-25T15:54:56.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043d0d3640 1 -- 172.21.15.67:0/1751756633 shutdown_connections 2024-04-25T15:54:56.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1751756633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0438164670 0x7f0438164a80 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1751756633 >> v2:172.21.15.176:3300/0 conn(0x7f04380732d0 0x7f04380736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1751756633 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f043815ae00 0x7f0438072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.321+0000 7f043d0d3640 1 -- 172.21.15.67:0/1751756633 >> 172.21.15.67:0/1751756633 conn(0x7f0438070920 msgr2=0x7f0438070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.325+0000 7f043d0d3640 1 -- 172.21.15.67:0/1751756633 shutdown_connections 2024-04-25T15:54:56.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.325+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2780059d0 con 0x7fe278037410 2024-04-25T15:54:56.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.325+0000 7f043d0d3640 1 -- 172.21.15.67:0/1751756633 wait complete. 2024-04-25T15:54:56.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.325+0000 7f043d0d3640 1 Processor -- start 2024-04-25T15:54:56.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.325+0000 7f043d0d3640 1 -- start start 2024-04-25T15:54:56.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043d0d3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04380732d0 0x7f043815ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043d0d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f043815ae00 0x7f043815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f043815ae00 0x7f043815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f043815ae00 0x7f043815d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49728/0 (socket says 172.21.15.67:49728) 2024-04-25T15:54:56.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043d0d3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0438164670 0x7f0438163170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 -- 172.21.15.67:0/1136444815 learned_addr learned my addr 172.21.15.67:0/1136444815 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f04380cafb0 con 0x7f043815ae00 2024-04-25T15:54:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04380cacd0 con 0x7f0438164670 2024-04-25T15:54:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0438164fc0 con 0x7f04380732d0 2024-04-25T15:54:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043759e640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0438164670 0x7f0438163170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 -- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04380732d0 msgr2=0x7f043815ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:54:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04380732d0 0x7f043815ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 -- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0438164670 msgr2=0x7f0438163170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0438164670 0x7f0438163170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043815bfb0 con 0x7f043815ae00 2024-04-25T15:54:56.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043759e640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0438164670 0x7f0438163170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:56.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.329+0000 7f043659c640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f043815ae00 0x7f043815d750 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f043000e900 tx=0x7f043000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.333+0000 7fe26effd640 1 -- 172.21.15.67:0/1864498878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe27400d070 con 0x7fe278037410 2024-04-25T15:54:56.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.333+0000 7f041f7fe640 1 -- 172.21.15.67:0/1136444815 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430017070 con 0x7f043815ae00 2024-04-25T15:54:56.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.333+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04381f7920 con 0x7f043815ae00 2024-04-25T15:54:56.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.333+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043815e5b0 con 0x7f043815ae00 2024-04-25T15:54:56.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.333+0000 7f041f7fe640 1 -- 172.21.15.67:0/1136444815 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f043000ca00 con 0x7f043815ae00 2024-04-25T15:54:56.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.333+0000 7f041f7fe640 1 -- 172.21.15.67:0/1136444815 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430021060 con 0x7f043815ae00 2024-04-25T15:54:56.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.337+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0438050750 con 0x7f043815ae00 2024-04-25T15:54:56.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.337+0000 7f041f7fe640 1 -- 172.21.15.67:0/1136444815 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f043001a070 con 0x7f043815ae00 2024-04-25T15:54:56.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.337+0000 7f041f7fe640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f042406f400 0x7f0424071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.337+0000 7f041f7fe640 1 -- 172.21.15.67:0/1136444815 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f043008ee30 con 0x7f043815ae00 2024-04-25T15:54:56.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.337+0000 7f0436d9d640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f042406f400 0x7f0424071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.341+0000 7f0436d9d640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f042406f400 0x7f0424071910 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f042c0095f0 tx=0x7f042c006b50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.349+0000 7f041f7fe640 1 -- 172.21.15.67:0/1136444815 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f043005b5a0 con 0x7f043815ae00 2024-04-25T15:54:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.365+0000 7ff6edce8640 1 Processor -- start 2024-04-25T15:54:56.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.365+0000 7ff6edce8640 1 -- start start 2024-04-25T15:54:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6edce8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff6e8164b90 0x7ff6e80689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e6ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff6e8068590 0x7ff6e8164480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43330/0 (socket says 172.21.15.67:43330) 2024-04-25T15:54:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6edce8640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6e80735f0 con 0x7ff6e8068590 2024-04-25T15:54:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6edce8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6e8072bd0 con 0x7ff6e815aa90 2024-04-25T15:54:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6edce8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff6e8073e20 con 0x7ff6e8164b90 2024-04-25T15:54:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e6ffd640 1 -- 172.21.15.67:0/3403430960 learned_addr learned my addr 172.21.15.67:0/3403430960 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e7fff640 1 --2- 172.21.15.67:0/3403430960 >> v2:172.21.15.67:3300/0 conn(0x7ff6e8164b90 0x7ff6e80689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e67fc640 1 -- 172.21.15.67:0/3403430960 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698079988 0 0) 0x7ff6e80735f0 con 0x7ff6e8068590 2024-04-25T15:54:56.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e67fc640 1 -- 172.21.15.67:0/3403430960 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d0003530 con 0x7ff6e8068590 2024-04-25T15:54:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e7fff640 1 -- 172.21.15.67:0/3403430960 >> v1:172.21.15.176:6789/0 conn(0x7ff6e815aa90 legacy=0x7ff6e80c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e7fff640 1 -- 172.21.15.67:0/3403430960 >> v1:172.21.15.67:6789/0 conn(0x7ff6e8068590 legacy=0x7ff6e8164480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7ff6e7fff640 1 -- 172.21.15.67:0/3403430960 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e8072e70 con 0x7ff6e8164b90 2024-04-25T15:54:56.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7feb237fe640 1 Processor -- start 2024-04-25T15:54:56.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.369+0000 7feb237fe640 1 -- start start 2024-04-25T15:54:56.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb237fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7feb24158ca0 0x7feb2406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb237fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb241590b0 0x7feb2406f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb237fe640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb2415bee0 con 0x7feb240c5900 2024-04-25T15:54:56.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb237fe640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7feb2415c1c0 con 0x7feb24158ca0 2024-04-25T15:54:56.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb237fe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feb2415c4a0 con 0x7feb241590b0 2024-04-25T15:54:56.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7ff6e7fff640 1 --2- 172.21.15.67:0/3403430960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8164b90 0x7ff6e80689a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0009d90 tx=0x7ff6e001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302209c3d4d6d354 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb22ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7feb240c5900 0x7feb24152680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56946/0 (socket says 172.21.15.67:56946) 2024-04-25T15:54:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb22ffd640 1 -- 172.21.15.67:0/3392408550 learned_addr learned my addr 172.21.15.67:0/3392408550 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7ff6e67fc640 1 -- 172.21.15.67:0/3403430960 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00b4070 con 0x7ff6e8164b90 2024-04-25T15:54:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7ff6e67fc640 1 -- 172.21.15.67:0/3403430960 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e00af600 con 0x7ff6e8164b90 2024-04-25T15:54:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7ff6e67fc640 1 -- 172.21.15.67:0/3403430960 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e00af9d0 con 0x7ff6e8164b90 2024-04-25T15:54:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb23fff640 1 --2- 172.21.15.67:0/3392408550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb241590b0 0x7feb2406f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb227fc640 1 --2- 172.21.15.67:0/3392408550 >> v2:172.21.15.176:3300/0 conn(0x7feb24158ca0 0x7feb2406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7ff6edce8640 1 -- 172.21.15.67:0/3403430960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8164b90 msgr2=0x7ff6e80689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7ff6edce8640 1 --2- 172.21.15.67:0/3403430960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8164b90 0x7ff6e80689a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff6e0009d90 tx=0x7ff6e001f150 comp rx=0 tx=0).stop 2024-04-25T15:54:56.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7ff6edce8640 1 -- 172.21.15.67:0/3403430960 shutdown_connections 2024-04-25T15:54:56.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7ff6edce8640 1 --2- 172.21.15.67:0/3403430960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e8164b90 0x7ff6e80689a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7ff6edce8640 1 -- 172.21.15.67:0/3403430960 >> 172.21.15.67:0/3403430960 conn(0x7ff6e80cea00 msgr2=0x7ff6e80cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb23fff640 1 -- 172.21.15.67:0/3392408550 >> v2:172.21.15.176:3300/0 conn(0x7feb24158ca0 msgr2=0x7feb2406ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb23fff640 1 --2- 172.21.15.67:0/3392408550 >> v2:172.21.15.176:3300/0 conn(0x7feb24158ca0 0x7feb2406ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb23fff640 1 -- 172.21.15.67:0/3392408550 >> v1:172.21.15.176:6789/0 conn(0x7feb240c5900 legacy=0x7feb24152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.373+0000 7feb23fff640 1 -- 172.21.15.67:0/3392408550 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2415c780 con 0x7feb241590b0 2024-04-25T15:54:56.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7ff6edce8640 1 -- 172.21.15.67:0/3403430960 shutdown_connections 2024-04-25T15:54:56.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7feb23fff640 1 --2- 172.21.15.67:0/3392408550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb241590b0 0x7feb2406f450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009d90 tx=0x7feb1c01ead0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f77fe10ffb49308 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7ff6edce8640 1 -- 172.21.15.67:0/3403430960 wait complete. 2024-04-25T15:54:56.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7feb21ffb640 1 -- 172.21.15.67:0/3392408550 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0b4070 con 0x7feb241590b0 2024-04-25T15:54:56.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7feb21ffb640 1 -- 172.21.15.67:0/3392408550 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb1c0a7a80 con 0x7feb241590b0 2024-04-25T15:54:56.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.377+0000 7feb21ffb640 1 -- 172.21.15.67:0/3392408550 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0a7e30 con 0x7feb241590b0 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 -- 172.21.15.67:0/3392408550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb241590b0 msgr2=0x7feb2406f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 --2- 172.21.15.67:0/3392408550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb241590b0 0x7feb2406f450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009d90 tx=0x7feb1c01ead0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 -- 172.21.15.67:0/3392408550 shutdown_connections 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 --2- 172.21.15.67:0/3392408550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb241590b0 0x7feb2406f450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 --2- 172.21.15.67:0/3392408550 >> v2:172.21.15.176:3300/0 conn(0x7feb24158ca0 0x7feb2406ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 -- 172.21.15.67:0/3392408550 >> 172.21.15.67:0/3392408550 conn(0x7feb240cdfd0 msgr2=0x7feb240ca090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 -- 172.21.15.67:0/3392408550 shutdown_connections 2024-04-25T15:54:56.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 -- 172.21.15.67:0/3392408550 wait complete. 2024-04-25T15:54:56.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7ff6edce8640 1 Processor -- start 2024-04-25T15:54:56.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7ff6edce8640 1 -- start start 2024-04-25T15:54:56.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7feb237fe640 1 Processor -- start 2024-04-25T15:54:56.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.381+0000 7ff6edce8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8068590 0x7ff6e8155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb237fe640 1 -- start start 2024-04-25T15:54:56.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6edce8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e815aa90 0x7ff6e81ebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e815aa90 0x7ff6e81ebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e815aa90 0x7ff6e81ebcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49750/0 (socket says 172.21.15.67:49750) 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6edce8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8164b90 0x7ff6e81e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e77fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8068590 0x7ff6e8155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 -- 172.21.15.67:0/165205833 learned_addr learned my addr 172.21.15.67:0/165205833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6edce8640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e8163ff0 con 0x7ff6e815aa90 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6edce8640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e806e010 con 0x7ff6e8068590 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6edce8640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e8073860 con 0x7ff6e8164b90 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e7fff640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8164b90 0x7ff6e81e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 -- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8164b90 msgr2=0x7ff6e81e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8164b90 0x7ff6e81e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 -- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8068590 msgr2=0x7ff6e8155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8068590 0x7ff6e8155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e8155790 con 0x7ff6e815aa90 2024-04-25T15:54:56.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6e6ffd640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e815aa90 0x7ff6e81ebcc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800c430 tx=0x7ff6d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6cffff640 1 -- 172.21.15.67:0/165205833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8017070 con 0x7ff6e815aa90 2024-04-25T15:54:56.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6cffff640 1 -- 172.21.15.67:0/165205833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d8005d10 con 0x7ff6e815aa90 2024-04-25T15:54:56.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb237fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb240c5900 0x7feb240744e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb237fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb24158ca0 0x7feb240c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb237fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb241590b0 0x7feb2414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb237fe640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb240c5fe0 con 0x7feb240c5900 2024-04-25T15:54:56.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb237fe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feb24158430 con 0x7feb241590b0 2024-04-25T15:54:56.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb237fe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feb240c9d80 con 0x7feb24158ca0 2024-04-25T15:54:56.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6cffff640 1 -- 172.21.15.67:0/165205833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8007310 con 0x7ff6e815aa90 2024-04-25T15:54:56.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6edce8640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e81ec6b0 con 0x7ff6e815aa90 2024-04-25T15:54:56.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7ff6edce8640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e81f7f50 con 0x7ff6e815aa90 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb241590b0 0x7feb2414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb241590b0 0x7feb2414bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:33094/0 (socket says 172.21.15.67:33094) 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 -- 172.21.15.67:0/752156875 learned_addr learned my addr 172.21.15.67:0/752156875 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb227fc640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb24158ca0 0x7feb240c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb22ffd640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb240c5900 0x7feb240744e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 -- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb24158ca0 msgr2=0x7feb240c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb24158ca0 0x7feb240c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 -- 172.21.15.67:0/752156875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb240c5900 msgr2=0x7feb240744e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb240c5900 0x7feb240744e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 -- 172.21.15.67:0/752156875 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb241fb310 con 0x7feb241590b0 2024-04-25T15:54:56.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.385+0000 7feb23fff640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb241590b0 0x7feb2414bfc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0b2040 tx=0x7feb1c01b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb91e2a8640 1 Processor -- start 2024-04-25T15:54:56.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb91e2a8640 1 -- start start 2024-04-25T15:54:56.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7feb0b7fe640 1 -- 172.21.15.67:0/752156875 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0c0070 con 0x7feb241590b0 2024-04-25T15:54:56.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7feb0b7fe640 1 -- 172.21.15.67:0/752156875 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb1c003d00 con 0x7feb241590b0 2024-04-25T15:54:56.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7feb0b7fe640 1 -- 172.21.15.67:0/752156875 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0b4040 con 0x7feb241590b0 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb24150710 con 0x7feb241590b0 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb241fb030 con 0x7feb241590b0 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6c00052a0 con 0x7ff6e815aa90 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb91e2a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9181609a0 0x7fb91806c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb91e2a8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb918160db0 0x7fb91806cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb91e2a8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9181612a0 con 0x7fb91815ad80 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb91e2a8640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb91815b190 con 0x7fb918160db0 2024-04-25T15:54:56.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb91e2a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb91806fb90 con 0x7fb9181609a0 2024-04-25T15:54:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7ff6cffff640 1 -- 172.21.15.67:0/165205833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff6d801a070 con 0x7ff6e815aa90 2024-04-25T15:54:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb917fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fb91815ad80 0x7fb9180c97f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:56954/0 (socket says 172.21.15.67:56954) 2024-04-25T15:54:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb917fff640 1 -- 172.21.15.67:0/2682689043 learned_addr learned my addr 172.21.15.67:0/2682689043 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7ff6cffff640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff6d406f500 0x7ff6d4071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7ff6cffff640 1 -- 172.21.15.67:0/165205833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7ff6d808e7f0 con 0x7ff6e815aa90 2024-04-25T15:54:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7ff6e77fe640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff6d406f500 0x7ff6d4071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.389+0000 7fb916ffd640 1 -- 172.21.15.67:0/2682689043 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 326212275 0 0) 0x7fb9181612a0 con 0x7fb91815ad80 2024-04-25T15:54:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb916ffd640 1 -- 172.21.15.67:0/2682689043 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb908003530 con 0x7fb91815ad80 2024-04-25T15:54:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb9177fe640 1 --2- 172.21.15.67:0/2682689043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9181609a0 0x7fb91806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb91c845640 1 --2- 172.21.15.67:0/2682689043 >> v2:172.21.15.67:3300/0 conn(0x7fb918160db0 0x7fb91806cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb9177fe640 1 -- 172.21.15.67:0/2682689043 >> v2:172.21.15.67:3300/0 conn(0x7fb918160db0 msgr2=0x7fb91806cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7ff6e77fe640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff6d406f500 0x7ff6d4071a10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0015b0 tx=0x7ff6dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb9177fe640 1 --2- 172.21.15.67:0/2682689043 >> v2:172.21.15.67:3300/0 conn(0x7fb918160db0 0x7fb91806cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7feb0b7fe640 1 -- 172.21.15.67:0/752156875 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7feb1c004280 con 0x7feb241590b0 2024-04-25T15:54:56.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb916ffd640 1 -- 172.21.15.67:0/2682689043 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1085879254 0 0) 0x7fb908003530 con 0x7fb91815ad80 2024-04-25T15:54:56.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb9177fe640 1 -- 172.21.15.67:0/2682689043 >> v1:172.21.15.176:6789/0 conn(0x7fb91815ad80 legacy=0x7fb9180c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb9177fe640 1 -- 172.21.15.67:0/2682689043 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb918161580 con 0x7fb9181609a0 2024-04-25T15:54:56.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7fb91c845640 1 --2- 172.21.15.67:0/2682689043 >> v2:172.21.15.67:3300/0 conn(0x7fb918160db0 0x7fb91806cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:56.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7feb0b7fe640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb1006f450 0x7feb10071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.393+0000 7feb0b7fe640 1 -- 172.21.15.67:0/752156875 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7feb1c129970 con 0x7feb241590b0 2024-04-25T15:54:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb9177fe640 1 --2- 172.21.15.67:0/2682689043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9181609a0 0x7fb91806c890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0097c0 tx=0x7fb90c01ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b2916e5a09ffd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb040052a0 con 0x7feb241590b0 2024-04-25T15:54:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb916ffd640 1 -- 172.21.15.67:0/2682689043 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0b5070 con 0x7fb9181609a0 2024-04-25T15:54:56.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb916ffd640 1 -- 172.21.15.67:0/2682689043 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb90c0a85f0 con 0x7fb9181609a0 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 -- 172.21.15.67:0/2682689043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9181609a0 msgr2=0x7fb91806c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/2682689043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9181609a0 0x7fb91806c890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0097c0 tx=0x7fb90c01ebb0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 -- 172.21.15.67:0/2682689043 shutdown_connections 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/2682689043 >> v2:172.21.15.67:3300/0 conn(0x7fb918160db0 0x7fb91806cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/2682689043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9181609a0 0x7fb91806c890 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 -- 172.21.15.67:0/2682689043 >> 172.21.15.67:0/2682689043 conn(0x7fb9180cdf20 msgr2=0x7fb9180ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 -- 172.21.15.67:0/2682689043 shutdown_connections 2024-04-25T15:54:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.397+0000 7fb91e2a8640 1 -- 172.21.15.67:0/2682689043 wait complete. 2024-04-25T15:54:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7feb22ffd640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb1006f450 0x7feb10071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7ff6cffff640 1 -- 172.21.15.67:0/165205833 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff6d805b610 con 0x7ff6e815aa90 2024-04-25T15:54:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 Processor -- start 2024-04-25T15:54:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 -- start start 2024-04-25T15:54:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7feb22ffd640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb1006f450 0x7feb10071960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7feb14000c10 tx=0x7feb1400d3e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb91815ad80 0x7fb9180c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb918160db0 0x7fb9180cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91815e0e0 0x7fb918154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb918161860 con 0x7fb91815e0e0 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb91806f820 con 0x7fb918160db0 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb91e2a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb9180c3570 con 0x7fb91815ad80 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb917fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb91815ad80 0x7fb9180c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.401+0000 7fb9177fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb918160db0 0x7fb9180cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb91815ad80 0x7fb9180c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:50698/0 (socket says 172.21.15.67:50698) 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 -- 172.21.15.67:0/3570576105 learned_addr learned my addr 172.21.15.67:0/3570576105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb91c845640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91815e0e0 0x7fb918154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 -- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb918160db0 msgr2=0x7fb9180cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb918160db0 0x7fb9180cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 -- 172.21.15.67:0/3570576105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91815e0e0 msgr2=0x7fb918154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91815e0e0 0x7fb918154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 -- 172.21.15.67:0/3570576105 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91815c5a0 con 0x7fb91815ad80 2024-04-25T15:54:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.405+0000 7fb917fff640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb91815ad80 0x7fb9180c77f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb90400c780 tx=0x7fb90400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7fb914ff9640 1 -- 172.21.15.67:0/3570576105 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904005cf0 con 0x7fb91815ad80 2024-04-25T15:54:56.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91815d000 con 0x7fb91815ad80 2024-04-25T15:54:56.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9180c7d30 con 0x7fb91815ad80 2024-04-25T15:54:56.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7fb914ff9640 1 -- 172.21.15.67:0/3570576105 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb904007070 con 0x7fb91815ad80 2024-04-25T15:54:56.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb918050750 con 0x7fb91815ad80 2024-04-25T15:54:56.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7fb914ff9640 1 -- 172.21.15.67:0/3570576105 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904007460 con 0x7fb91815ad80 2024-04-25T15:54:56.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7feb0b7fe640 1 -- 172.21.15.67:0/752156875 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feb1c0186c0 con 0x7feb241590b0 2024-04-25T15:54:56.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.409+0000 7fb914ff9640 1 -- 172.21.15.67:0/3570576105 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb904007600 con 0x7fb91815ad80 2024-04-25T15:54:56.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.413+0000 7fb914ff9640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8fc06f4d0 0x7fb8fc0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.413+0000 7fb914ff9640 1 -- 172.21.15.67:0/3570576105 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fb90408fc70 con 0x7fb91815ad80 2024-04-25T15:54:56.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.413+0000 7fb9177fe640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8fc06f4d0 0x7fb8fc0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.413+0000 7fb9177fe640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8fc06f4d0 0x7fb8fc0719e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0b2000 tx=0x7fb90c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.425+0000 7fb914ff9640 1 -- 172.21.15.67:0/3570576105 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb90405c3b0 con 0x7fb91815ad80 2024-04-25T15:54:56.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.565+0000 7f8580da7640 1 -- 172.21.15.67:0/4017527802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f857c0c4aa0 con 0x7f857c06ee10 2024-04-25T15:54:56.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.565+0000 7f856affd640 1 -- 172.21.15.67:0/4017527802 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f856c019010 con 0x7f857c06ee10 2024-04-25T15:54:56.568 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:56.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 -- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f855c06f4d0 msgr2=0x7f855c0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f855c06f4d0 0x7f855c0719e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f857c1f48c0 tx=0x7f857400b040 comp rx=0 tx=0).stop 2024-04-25T15:54:56.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 -- 172.21.15.67:0/4017527802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f857c06ee10 msgr2=0x7f857c15b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f857c06ee10 0x7f857c15b600 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f856c01fd50 tx=0x7f856c01c250 comp rx=0 tx=0).stop 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 -- 172.21.15.67:0/4017527802 shutdown_connections 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f855c06f4d0 0x7f855c0719e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f857c15bc20 0x7f857c1e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f857c152c40 0x7f857c0cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 --2- 172.21.15.67:0/4017527802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f857c06ee10 0x7f857c15b600 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 -- 172.21.15.67:0/4017527802 >> 172.21.15.67:0/4017527802 conn(0x7f857c0ca050 msgr2=0x7f857c0c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 -- 172.21.15.67:0/4017527802 shutdown_connections 2024-04-25T15:54:56.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.569+0000 7f8568ff9640 1 -- 172.21.15.67:0/4017527802 wait complete. 2024-04-25T15:54:56.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574909 for osd.6 2024-04-25T15:54:56.598 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.601+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdc7c005860 con 0x7fdc9415ad40 2024-04-25T15:54:56.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.605+0000 7fdc8a7fc640 1 -- 172.21.15.67:0/274219579 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdc9005f980 con 0x7fdc9415ad40 2024-04-25T15:54:56.604 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:56.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.605+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc7806f450 msgr2=0x7fdc78071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.605+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc7806f450 0x7fdc78071960 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0b4000 tx=0x7fdc8c0ab000 comp rx=0 tx=0).stop 2024-04-25T15:54:56.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.605+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc9415ad40 msgr2=0x7fdc94147d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.605+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc9415ad40 0x7fdc94147d30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000c390 tx=0x7fdc9000c850 comp rx=0 tx=0).stop 2024-04-25T15:54:56.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 shutdown_connections 2024-04-25T15:54:56.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdc7806f450 0x7fdc78071960 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc9415ad40 0x7fdc94147d30 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdc94152df0 0x7fdc9414f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 --2- 172.21.15.67:0/274219579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdc941529e0 0x7fdc940cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 >> 172.21.15.67:0/274219579 conn(0x7fdc9406fe70 msgr2=0x7fdc940c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 shutdown_connections 2024-04-25T15:54:56.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.609+0000 7fdc9bc89640 1 -- 172.21.15.67:0/274219579 wait complete. 2024-04-25T15:54:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.629+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6054050750 con 0x7f605415e450 2024-04-25T15:54:56.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.629+0000 7f6042ffd640 1 -- 172.21.15.67:0/155379975 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6048060b60 con 0x7f605415e450 2024-04-25T15:54:56.631 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:56.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f603406f400 msgr2=0x7f6034071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f603406f400 0x7f6034071910 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6044000f70 tx=0x7f6044007240 comp rx=0 tx=0).stop 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f605415e450 msgr2=0x7f605406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f605415e450 0x7f605406eb60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f604800d550 tx=0x7f604800da10 comp rx=0 tx=0).stop 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 shutdown_connections 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f603406f400 0x7f6034071910 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60541492d0 0x7f6054158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f605415e450 0x7f605406eb60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 --2- 172.21.15.67:0/155379975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60540c6d20 0x7f60541fab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 >> 172.21.15.67:0/155379975 conn(0x7f6054071ba0 msgr2=0x7f60540caeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 shutdown_connections 2024-04-25T15:54:56.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.633+0000 7f6058917640 1 -- 172.21.15.67:0/155379975 wait complete. 2024-04-25T15:54:56.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574908 got 55834574908 for osd.4 2024-04-25T15:54:56.637 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574909 for osd.7 2024-04-25T15:54:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe2780dcc90 con 0x7fe278037410 2024-04-25T15:54:56.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26effd640 1 -- 172.21.15.67:0/1864498878 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe2740148a0 con 0x7fe278037410 2024-04-25T15:54:56.665 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.666 INFO:teuthology.orchestra.run.smithi067.stdout:55834574908 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe26006f400 msgr2=0x7fe260071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe26006f400 0x7fe260071910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe2700b2000 tx=0x7fe2700aa070 comp rx=0 tx=0).stop 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe278037410 msgr2=0x7fe2780d8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe278037410 0x7fe2780d8b20 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe27400ae10 tx=0x7fe274007410 comp rx=0 tx=0).stop 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 shutdown_connections 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe26006f400 0x7fe260071910 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe278050470 0x7fe2780cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe278050060 0x7fe2781797b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 --2- 172.21.15.67:0/1864498878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe278037410 0x7fe2780d8b20 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 >> 172.21.15.67:0/1864498878 conn(0x7fe2780547c0 msgr2=0x7fe2780526a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 shutdown_connections 2024-04-25T15:54:56.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.665+0000 7fe26cff9640 1 -- 172.21.15.67:0/1864498878 wait complete. 2024-04-25T15:54:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.669+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7feb04005860 con 0x7feb241590b0 2024-04-25T15:54:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.669+0000 7feb0b7fe640 1 -- 172.21.15.67:0/752156875 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feb1c0f6240 con 0x7feb241590b0 2024-04-25T15:54:56.671 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:56.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb1006f450 msgr2=0x7feb10071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb1006f450 0x7feb10071960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7feb14000c10 tx=0x7feb1400d3e0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb241590b0 msgr2=0x7feb2414bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb241590b0 0x7feb2414bfc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0b2040 tx=0x7feb1c01b870 comp rx=0 tx=0).stop 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 shutdown_connections 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7feb1006f450 0x7feb10071960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb241590b0 0x7feb2414bfc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb24158ca0 0x7feb240c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 --2- 172.21.15.67:0/752156875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb240c5900 0x7feb240744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 >> 172.21.15.67:0/752156875 conn(0x7feb240cdfd0 msgr2=0x7feb2414f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 shutdown_connections 2024-04-25T15:54:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.673+0000 7feb237fe640 1 -- 172.21.15.67:0/752156875 wait complete. 2024-04-25T15:54:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.681+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f04380c9bb0 con 0x7f043815ae00 2024-04-25T15:54:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.681+0000 7f041f7fe640 1 -- 172.21.15.67:0/1136444815 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f043005f8a0 con 0x7f043815ae00 2024-04-25T15:54:56.682 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f042406f400 msgr2=0x7f0424071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f042406f400 0x7f0424071910 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f042c0095f0 tx=0x7f042c006b50 comp rx=0 tx=0).stop 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f043815ae00 msgr2=0x7f043815d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f043815ae00 0x7f043815d750 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f043000e900 tx=0x7f043000edc0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 shutdown_connections 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f042406f400 0x7f0424071910 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0438164670 0x7f0438163170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f043815ae00 0x7f043815d750 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 --2- 172.21.15.67:0/1136444815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04380732d0 0x7f043815ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 >> 172.21.15.67:0/1136444815 conn(0x7f0438070920 msgr2=0x7f0438158ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 shutdown_connections 2024-04-25T15:54:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.685+0000 7f043d0d3640 1 -- 172.21.15.67:0/1136444815 wait complete. 2024-04-25T15:54:56.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574908 got 55834574908 for osd.5 2024-04-25T15:54:56.689 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574909 for osd.2 2024-04-25T15:54:56.706 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.705+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb9180c3140 con 0x7fb91815ad80 2024-04-25T15:54:56.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.705+0000 7fb914ff9640 1 -- 172.21.15.67:0/3570576105 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb9040606b0 con 0x7fb91815ad80 2024-04-25T15:54:56.707 INFO:teuthology.orchestra.run.smithi067.stdout:55834574907 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8fc06f4d0 msgr2=0x7fb8fc0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8fc06f4d0 0x7fb8fc0719e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0b2000 tx=0x7fb90c008040 comp rx=0 tx=0).stop 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb91815ad80 msgr2=0x7fb9180c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb91815ad80 0x7fb9180c77f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb90400c780 tx=0x7fb90400cc40 comp rx=0 tx=0).stop 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 shutdown_connections 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb8fc06f4d0 0x7fb8fc0719e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91815e0e0 0x7fb918154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb918160db0 0x7fb9180cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 --2- 172.21.15.67:0/3570576105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb91815ad80 0x7fb9180c77f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 >> 172.21.15.67:0/3570576105 conn(0x7fb9180cdf20 msgr2=0x7fb91815f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 shutdown_connections 2024-04-25T15:54:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.709+0000 7fb91e2a8640 1 -- 172.21.15.67:0/3570576105 wait complete. 2024-04-25T15:54:56.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574909 for osd.3 2024-04-25T15:54:56.715 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.721+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff6c0005590 con 0x7ff6e815aa90 2024-04-25T15:54:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.721+0000 7ff6cffff640 1 -- 172.21.15.67:0/165205833 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff6d805af10 con 0x7ff6e815aa90 2024-04-25T15:54:56.722 INFO:teuthology.orchestra.run.smithi067.stdout:55834574909 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff6d406f500 msgr2=0x7ff6d4071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff6d406f500 0x7ff6d4071a10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0015b0 tx=0x7ff6dc009040 comp rx=0 tx=0).stop 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e815aa90 msgr2=0x7ff6e81ebcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e815aa90 0x7ff6e81ebcc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff6d800c430 tx=0x7ff6d800c8f0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 shutdown_connections 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff6d406f500 0x7ff6d4071a10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6e8164b90 0x7ff6e81e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6e815aa90 0x7ff6e81ebcc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 --2- 172.21.15.67:0/165205833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6e8068590 0x7ff6e8155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 >> 172.21.15.67:0/165205833 conn(0x7ff6e80cea00 msgr2=0x7ff6e80ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 shutdown_connections 2024-04-25T15:54:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.725+0000 7ff6cdffb640 1 -- 172.21.15.67:0/165205833 wait complete. 2024-04-25T15:54:56.727 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574907 got 55834574907 for osd.0 2024-04-25T15:54:56.727 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574909 got 55834574909 for osd.1 2024-04-25T15:54:56.743 DEBUG:teuthology.parallel:result is None 2024-04-25T15:54:56.743 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-25T15:54:56.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.917+0000 7fc477d33640 1 Processor -- start 2024-04-25T15:54:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.917+0000 7fc477d33640 1 -- start start 2024-04-25T15:54:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc477d33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc470160800 0x7fc47006d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc477d33640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc47006d8e0 con 0x7fc4700c96e0 2024-04-25T15:54:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc477d33640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc470073c70 con 0x7fc4700c92d0 2024-04-25T15:54:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc477d33640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc47006dbc0 con 0x7fc470160800 2024-04-25T15:54:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc475acf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc470160800 0x7fc47006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc475acf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc470160800 0x7fc47006d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49788/0 (socket says 172.21.15.67:49788) 2024-04-25T15:54:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc475acf640 1 -- 172.21.15.67:0/335263437 learned_addr learned my addr 172.21.15.67:0/335263437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc475acf640 1 -- 172.21.15.67:0/335263437 >> v1:172.21.15.176:6789/0 conn(0x7fc4700c92d0 legacy=0x7fc470072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc475acf640 1 -- 172.21.15.67:0/335263437 >> v1:172.21.15.67:6789/0 conn(0x7fc4700c96e0 legacy=0x7fc470167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc475acf640 1 -- 172.21.15.67:0/335263437 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc470073f50 con 0x7fc470160800 2024-04-25T15:54:56.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc475acf640 1 --2- 172.21.15.67:0/335263437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc470160800 0x7fc47006d030 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc468009870 tx=0x7fc46801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f8645a2128cf37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc474acd640 1 -- 172.21.15.67:0/335263437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680b2070 con 0x7fc470160800 2024-04-25T15:54:56.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc474acd640 1 -- 172.21.15.67:0/335263437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4680ad680 con 0x7fc470160800 2024-04-25T15:54:56.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.921+0000 7fc474acd640 1 -- 172.21.15.67:0/335263437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680ada50 con 0x7fc470160800 2024-04-25T15:54:56.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- 172.21.15.67:0/335263437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc470160800 msgr2=0x7fc47006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 --2- 172.21.15.67:0/335263437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc470160800 0x7fc47006d030 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc468009870 tx=0x7fc46801c490 comp rx=0 tx=0).stop 2024-04-25T15:54:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- 172.21.15.67:0/335263437 shutdown_connections 2024-04-25T15:54:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 --2- 172.21.15.67:0/335263437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc470160800 0x7fc47006d030 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- 172.21.15.67:0/335263437 >> 172.21.15.67:0/335263437 conn(0x7fc470070db0 msgr2=0x7fc4700ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- 172.21.15.67:0/335263437 shutdown_connections 2024-04-25T15:54:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- 172.21.15.67:0/335263437 wait complete. 2024-04-25T15:54:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 Processor -- start 2024-04-25T15:54:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- start start 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4700c92d0 0x7fc470153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4700c96e0 0x7fc47014c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc470160800 0x7fc47014a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc470160c10 con 0x7fc4700c92d0 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc47006d570 con 0x7fc470160800 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc477d33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc470160d90 con 0x7fc4700c96e0 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc475acf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4700c92d0 0x7fc470153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc475acf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4700c92d0 0x7fc470153360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49790/0 (socket says 172.21.15.67:49790) 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.925+0000 7fc475acf640 1 -- 172.21.15.67:0/2016906804 learned_addr learned my addr 172.21.15.67:0/2016906804 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4762d0640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc470160800 0x7fc47014a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4752ce640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4700c96e0 0x7fc47014c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc475acf640 1 -- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4700c96e0 msgr2=0x7fc47014c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc475acf640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4700c96e0 0x7fc47014c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc475acf640 1 -- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc470160800 msgr2=0x7fc47014a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc475acf640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc470160800 0x7fc47014a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc475acf640 1 -- 172.21.15.67:0/2016906804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47014e5f0 con 0x7fc4700c92d0 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc475acf640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4700c92d0 0x7fc470153360 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc46801a0c0 tx=0x7fc46801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4667fc640 1 -- 172.21.15.67:0/2016906804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680b2040 con 0x7fc4700c92d0 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4667fc640 1 -- 172.21.15.67:0/2016906804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4680bd070 con 0x7fc4700c92d0 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc470152d70 con 0x7fc4700c92d0 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4667fc640 1 -- 172.21.15.67:0/2016906804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680b8450 con 0x7fc4700c92d0 2024-04-25T15:54:56.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc47014eb50 con 0x7fc4700c92d0 2024-04-25T15:54:56.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc47014f480 con 0x7fc4700c92d0 2024-04-25T15:54:56.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4667fc640 1 -- 172.21.15.67:0/2016906804 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc4680b85f0 con 0x7fc4700c92d0 2024-04-25T15:54:56.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4667fc640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc45406f400 0x7fc454071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:56.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4667fc640 1 -- 172.21.15.67:0/2016906804 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fc4681295d0 con 0x7fc4700c92d0 2024-04-25T15:54:56.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4752ce640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc45406f400 0x7fc454071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:56.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.929+0000 7fc4752ce640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc45406f400 0x7fc454071910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc47015d5f0 tx=0x7fc460008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:56.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:56.933+0000 7fc4667fc640 1 -- 172.21.15.67:0/2016906804 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc47014e5f0 con 0x7fc4700c92d0 2024-04-25T15:54:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.085+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc47006a480 con 0x7fc45406f400 2024-04-25T15:54:57.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.105+0000 7fc4667fc640 1 -- 172.21.15.67:0/2016906804 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253608 (secure 0 0 0) 0x7fc47006a480 con 0x7fc45406f400 2024-04-25T15:54:57.108 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:57.110 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:54:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc45406f400 msgr2=0x7fc454071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc45406f400 0x7fc454071910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc47015d5f0 tx=0x7fc460008040 comp rx=0 tx=0).stop 2024-04-25T15:54:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4700c92d0 msgr2=0x7fc470153360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4700c92d0 0x7fc470153360 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc46801a0c0 tx=0x7fc46801acd0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 shutdown_connections 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc45406f400 0x7fc454071910 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc470160800 0x7fc47014a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4700c96e0 0x7fc47014c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 --2- 172.21.15.67:0/2016906804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4700c92d0 0x7fc470153360 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 >> 172.21.15.67:0/2016906804 conn(0x7fc470070db0 msgr2=0x7fc470151080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 shutdown_connections 2024-04-25T15:54:57.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.109+0000 7fc477d33640 1 -- 172.21.15.67:0/2016906804 wait complete. 2024-04-25T15:54:57.131 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-04-25T15:54:55.889485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":797648,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":9655,"num_write_kb":25167765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":1627779911,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9589,"ondisk_log_size":9589,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":199,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383964,"kb_used_data":13776,"kb_used_omap":12,"kb_used_meta":370035,"kb_avail":754590756,"statfs":{"total":773094113280,"available":772700934144,"internally_reserved":0,"allocated":14106624,"data_stored":7506496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":378916402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":199030,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.607980"},"pg_stats":[{"pgid":"3.3f","version":"113'138","reported_seq":366,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.339269+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:54:53.339269+0000","last_peered":"2024-04-25T15:54:53.339269+0000","last_clean":"2024-04-25T15:54:53.339269+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:54:53.339269+0000","last_undegraded":"2024-04-25T15:54:53.339269+0000","last_fullsized":"2024-04-25T15:54:53.339269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"113'126","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341661+0000","last_change":"2024-04-25T15:54:44.217073+0000","last_active":"2024-04-25T15:54:54.341661+0000","last_peered":"2024-04-25T15:54:54.341661+0000","last_clean":"2024-04-25T15:54:54.341661+0000","last_became_active":"2024-04-25T15:54:44.215261+0000","last_became_peered":"2024-04-25T15:54:44.215261+0000","last_unstale":"2024-04-25T15:54:54.341661+0000","last_undegraded":"2024-04-25T15:54:54.341661+0000","last_fullsized":"2024-04-25T15:54:54.341661+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:01.929448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"113'144","reported_seq":336,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352388+0000","last_change":"2024-04-25T15:54:40.166712+0000","last_active":"2024-04-25T15:54:54.352388+0000","last_peered":"2024-04-25T15:54:54.352388+0000","last_clean":"2024-04-25T15:54:54.352388+0000","last_became_active":"2024-04-25T15:54:40.164866+0000","last_became_peered":"2024-04-25T15:54:40.164866+0000","last_unstale":"2024-04-25T15:54:54.352388+0000","last_undegraded":"2024-04-25T15:54:54.352388+0000","last_fullsized":"2024-04-25T15:54:54.352388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:58.780901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"113'165","reported_seq":397,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344931+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:54:54.344931+0000","last_peered":"2024-04-25T15:54:54.344931+0000","last_clean":"2024-04-25T15:54:54.344931+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:54:54.344931+0000","last_undegraded":"2024-04-25T15:54:54.344931+0000","last_fullsized":"2024-04-25T15:54:54.344931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"113'144","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336952+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:54:53.336952+0000","last_peered":"2024-04-25T15:54:53.336952+0000","last_clean":"2024-04-25T15:54:53.336952+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:54:53.336952+0000","last_undegraded":"2024-04-25T15:54:53.336952+0000","last_fullsized":"2024-04-25T15:54:53.336952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"113'129","reported_seq":360,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344524+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:54:54.344524+0000","last_peered":"2024-04-25T15:54:54.344524+0000","last_clean":"2024-04-25T15:54:54.344524+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:54:54.344524+0000","last_undegraded":"2024-04-25T15:54:54.344524+0000","last_fullsized":"2024-04-25T15:54:54.344524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"113'159","reported_seq":386,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336477+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:54:53.336477+0000","last_peered":"2024-04-25T15:54:53.336477+0000","last_clean":"2024-04-25T15:54:53.336477+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:54:53.336477+0000","last_undegraded":"2024-04-25T15:54:53.336477+0000","last_fullsized":"2024-04-25T15:54:53.336477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"113'177","reported_seq":408,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349516+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:54:54.349516+0000","last_peered":"2024-04-25T15:54:54.349516+0000","last_clean":"2024-04-25T15:54:54.349516+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:54:54.349516+0000","last_undegraded":"2024-04-25T15:54:54.349516+0000","last_fullsized":"2024-04-25T15:54:54.349516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"113'141","reported_seq":365,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347174+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:54:54.347174+0000","last_peered":"2024-04-25T15:54:54.347174+0000","last_clean":"2024-04-25T15:54:54.347174+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:54:54.347174+0000","last_undegraded":"2024-04-25T15:54:54.347174+0000","last_fullsized":"2024-04-25T15:54:54.347174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"113'141","reported_seq":443,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.359559+0000","last_change":"2024-04-25T15:54:54.355672+0000","last_active":"2024-04-25T15:54:54.359559+0000","last_peered":"2024-04-25T15:54:54.359559+0000","last_clean":"2024-04-25T15:54:54.359559+0000","last_became_active":"2024-04-25T15:54:54.352223+0000","last_became_peered":"2024-04-25T15:54:54.352223+0000","last_unstale":"2024-04-25T15:54:54.359559+0000","last_undegraded":"2024-04-25T15:54:54.359559+0000","last_fullsized":"2024-04-25T15:54:54.359559+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:39.782280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"113'183","reported_seq":414,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349394+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:54:54.349394+0000","last_peered":"2024-04-25T15:54:54.349394+0000","last_clean":"2024-04-25T15:54:54.349394+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:54:54.349394+0000","last_undegraded":"2024-04-25T15:54:54.349394+0000","last_fullsized":"2024-04-25T15:54:54.349394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"113'135","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341244+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:54:54.341244+0000","last_peered":"2024-04-25T15:54:54.341244+0000","last_clean":"2024-04-25T15:54:54.341244+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:54:54.341244+0000","last_undegraded":"2024-04-25T15:54:54.341244+0000","last_fullsized":"2024-04-25T15:54:54.341244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"113'141","reported_seq":681,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.357489+0000","last_change":"2024-04-25T15:54:54.357489+0000","last_active":"2024-04-25T15:54:54.357489+0000","last_peered":"2024-04-25T15:54:54.357489+0000","last_clean":"2024-04-25T15:54:54.357489+0000","last_became_active":"2024-04-25T15:54:54.356694+0000","last_became_peered":"2024-04-25T15:54:54.356694+0000","last_unstale":"2024-04-25T15:54:54.357489+0000","last_undegraded":"2024-04-25T15:54:54.357489+0000","last_fullsized":"2024-04-25T15:54:54.357489+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.653628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"113'141","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352252+0000","last_change":"2024-04-25T15:54:40.160979+0000","last_active":"2024-04-25T15:54:54.352252+0000","last_peered":"2024-04-25T15:54:54.352252+0000","last_clean":"2024-04-25T15:54:54.352252+0000","last_became_active":"2024-04-25T15:54:40.159879+0000","last_became_peered":"2024-04-25T15:54:40.159879+0000","last_unstale":"2024-04-25T15:54:54.352252+0000","last_undegraded":"2024-04-25T15:54:54.352252+0000","last_fullsized":"2024-04-25T15:54:54.352252+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:25.499728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"113'144","reported_seq":263,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341194+0000","last_change":"2024-04-25T15:54:44.208852+0000","last_active":"2024-04-25T15:54:54.341194+0000","last_peered":"2024-04-25T15:54:54.341194+0000","last_clean":"2024-04-25T15:54:54.341194+0000","last_became_active":"2024-04-25T15:54:44.207144+0000","last_became_peered":"2024-04-25T15:54:44.207144+0000","last_unstale":"2024-04-25T15:54:54.341194+0000","last_undegraded":"2024-04-25T15:54:54.341194+0000","last_fullsized":"2024-04-25T15:54:54.341194+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:44:02.181746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"113'138","reported_seq":369,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344487+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:54:54.344487+0000","last_peered":"2024-04-25T15:54:54.344487+0000","last_clean":"2024-04-25T15:54:54.344487+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:54:54.344487+0000","last_undegraded":"2024-04-25T15:54:54.344487+0000","last_fullsized":"2024-04-25T15:54:54.344487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"113'138","reported_seq":303,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346019+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:54:54.346019+0000","last_peered":"2024-04-25T15:54:54.346019+0000","last_clean":"2024-04-25T15:54:54.346019+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:54:54.346019+0000","last_undegraded":"2024-04-25T15:54:54.346019+0000","last_fullsized":"2024-04-25T15:54:54.346019+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"113'147","reported_seq":591,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.355989+0000","last_change":"2024-04-25T15:54:54.355989+0000","last_active":"2024-04-25T15:54:54.355989+0000","last_peered":"2024-04-25T15:54:54.355989+0000","last_clean":"2024-04-25T15:54:54.355989+0000","last_became_active":"2024-04-25T15:54:54.354849+0000","last_became_peered":"2024-04-25T15:54:54.354849+0000","last_unstale":"2024-04-25T15:54:54.355989+0000","last_undegraded":"2024-04-25T15:54:54.355989+0000","last_fullsized":"2024-04-25T15:54:54.355989+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:05.626413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"113'165","reported_seq":392,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338717+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:54:53.338717+0000","last_peered":"2024-04-25T15:54:53.338717+0000","last_clean":"2024-04-25T15:54:53.338717+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:54:53.338717+0000","last_undegraded":"2024-04-25T15:54:53.338717+0000","last_fullsized":"2024-04-25T15:54:53.338717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"113'153","reported_seq":400,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348880+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:54:54.348880+0000","last_peered":"2024-04-25T15:54:54.348880+0000","last_clean":"2024-04-25T15:54:54.348880+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:54:54.348880+0000","last_undegraded":"2024-04-25T15:54:54.348880+0000","last_fullsized":"2024-04-25T15:54:54.348880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"113'126","reported_seq":316,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352187+0000","last_change":"2024-04-25T15:54:40.169787+0000","last_active":"2024-04-25T15:54:54.352187+0000","last_peered":"2024-04-25T15:54:54.352187+0000","last_clean":"2024-04-25T15:54:54.352187+0000","last_became_active":"2024-04-25T15:54:40.167469+0000","last_became_peered":"2024-04-25T15:54:40.167469+0000","last_unstale":"2024-04-25T15:54:54.352187+0000","last_undegraded":"2024-04-25T15:54:54.352187+0000","last_fullsized":"2024-04-25T15:54:54.352187+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:26.375093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"113'156","reported_seq":380,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346230+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:54:54.346230+0000","last_peered":"2024-04-25T15:54:54.346230+0000","last_clean":"2024-04-25T15:54:54.346230+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:54:54.346230+0000","last_undegraded":"2024-04-25T15:54:54.346230+0000","last_fullsized":"2024-04-25T15:54:54.346230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"113'126","reported_seq":373,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349915+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:54:54.349915+0000","last_peered":"2024-04-25T15:54:54.349915+0000","last_clean":"2024-04-25T15:54:54.349915+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:54:54.349915+0000","last_undegraded":"2024-04-25T15:54:54.349915+0000","last_fullsized":"2024-04-25T15:54:54.349915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"113'174","reported_seq":406,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345202+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:54:54.345202+0000","last_peered":"2024-04-25T15:54:54.345202+0000","last_clean":"2024-04-25T15:54:54.345202+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:54:54.345202+0000","last_undegraded":"2024-04-25T15:54:54.345202+0000","last_fullsized":"2024-04-25T15:54:54.345202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"113'132","reported_seq":233,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338631+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:54:53.338631+0000","last_peered":"2024-04-25T15:54:53.338631+0000","last_clean":"2024-04-25T15:54:53.338631+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:54:53.338631+0000","last_undegraded":"2024-04-25T15:54:53.338631+0000","last_fullsized":"2024-04-25T15:54:53.338631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"113'183","reported_seq":415,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349568+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:54:54.349568+0000","last_peered":"2024-04-25T15:54:54.349568+0000","last_clean":"2024-04-25T15:54:54.349568+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:54:54.349568+0000","last_undegraded":"2024-04-25T15:54:54.349568+0000","last_fullsized":"2024-04-25T15:54:54.349568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"113'138","reported_seq":350,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342384+0000","last_change":"2024-04-25T15:54:44.212386+0000","last_active":"2024-04-25T15:54:54.342384+0000","last_peered":"2024-04-25T15:54:54.342384+0000","last_clean":"2024-04-25T15:54:54.342384+0000","last_became_active":"2024-04-25T15:54:44.210520+0000","last_became_peered":"2024-04-25T15:54:44.210520+0000","last_unstale":"2024-04-25T15:54:54.342384+0000","last_undegraded":"2024-04-25T15:54:54.342384+0000","last_fullsized":"2024-04-25T15:54:54.342384+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:01.061276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"113'129","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350925+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:54:54.350925+0000","last_peered":"2024-04-25T15:54:54.350925+0000","last_clean":"2024-04-25T15:54:54.350925+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:54:54.350925+0000","last_undegraded":"2024-04-25T15:54:54.350925+0000","last_fullsized":"2024-04-25T15:54:54.350925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"113'111","reported_seq":304,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352326+0000","last_change":"2024-04-25T15:54:40.174462+0000","last_active":"2024-04-25T15:54:54.352326+0000","last_peered":"2024-04-25T15:54:54.352326+0000","last_clean":"2024-04-25T15:54:54.352326+0000","last_became_active":"2024-04-25T15:54:40.173209+0000","last_became_peered":"2024-04-25T15:54:40.173209+0000","last_unstale":"2024-04-25T15:54:54.352326+0000","last_undegraded":"2024-04-25T15:54:54.352326+0000","last_fullsized":"2024-04-25T15:54:54.352326+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:08.410467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"113'123","reported_seq":299,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341825+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:54:54.341825+0000","last_peered":"2024-04-25T15:54:54.341825+0000","last_clean":"2024-04-25T15:54:54.341825+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:54:54.341825+0000","last_undegraded":"2024-04-25T15:54:54.341825+0000","last_fullsized":"2024-04-25T15:54:54.341825+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"113'123","reported_seq":364,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345864+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:54:54.345864+0000","last_peered":"2024-04-25T15:54:54.345864+0000","last_clean":"2024-04-25T15:54:54.345864+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:54:54.345864+0000","last_undegraded":"2024-04-25T15:54:54.345864+0000","last_fullsized":"2024-04-25T15:54:54.345864+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"113'141","reported_seq":470,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.361042+0000","last_change":"2024-04-25T15:54:54.359681+0000","last_active":"2024-04-25T15:54:54.361042+0000","last_peered":"2024-04-25T15:54:54.361042+0000","last_clean":"2024-04-25T15:54:54.361042+0000","last_became_active":"2024-04-25T15:54:54.353824+0000","last_became_peered":"2024-04-25T15:54:54.353824+0000","last_unstale":"2024-04-25T15:54:54.361042+0000","last_undegraded":"2024-04-25T15:54:54.361042+0000","last_fullsized":"2024-04-25T15:54:54.361042+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:52.297640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"113'123","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341022+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:54:54.341022+0000","last_peered":"2024-04-25T15:54:54.341022+0000","last_clean":"2024-04-25T15:54:54.341022+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:54:54.341022+0000","last_undegraded":"2024-04-25T15:54:54.341022+0000","last_fullsized":"2024-04-25T15:54:54.341022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351432+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:54.351432+0000","last_peered":"2024-04-25T15:54:54.351432+0000","last_clean":"2024-04-25T15:54:54.351432+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:54.351432+0000","last_undegraded":"2024-04-25T15:54:54.351432+0000","last_fullsized":"2024-04-25T15:54:54.351432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"113'129","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352444+0000","last_change":"2024-04-25T15:54:40.163745+0000","last_active":"2024-04-25T15:54:54.352444+0000","last_peered":"2024-04-25T15:54:54.352444+0000","last_clean":"2024-04-25T15:54:54.352444+0000","last_became_active":"2024-04-25T15:54:40.162667+0000","last_became_peered":"2024-04-25T15:54:40.162667+0000","last_unstale":"2024-04-25T15:54:54.352444+0000","last_undegraded":"2024-04-25T15:54:54.352444+0000","last_fullsized":"2024-04-25T15:54:54.352444+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:17.528981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347310+0000","last_change":"2024-04-25T15:54:51.317865+0000","last_active":"2024-04-25T15:54:54.347310+0000","last_peered":"2024-04-25T15:54:54.347310+0000","last_clean":"2024-04-25T15:54:54.347310+0000","last_became_active":"2024-04-25T15:54:51.302422+0000","last_became_peered":"2024-04-25T15:54:51.302422+0000","last_unstale":"2024-04-25T15:54:54.347310+0000","last_undegraded":"2024-04-25T15:54:54.347310+0000","last_fullsized":"2024-04-25T15:54:54.347310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:36.956598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350037+0000","last_change":"2024-04-25T15:54:51.329012+0000","last_active":"2024-04-25T15:54:54.350037+0000","last_peered":"2024-04-25T15:54:54.350037+0000","last_clean":"2024-04-25T15:54:54.350037+0000","last_became_active":"2024-04-25T15:54:51.307662+0000","last_became_peered":"2024-04-25T15:54:51.307662+0000","last_unstale":"2024-04-25T15:54:54.350037+0000","last_undegraded":"2024-04-25T15:54:54.350037+0000","last_fullsized":"2024-04-25T15:54:54.350037+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:41.566474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"113'162","reported_seq":288,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341299+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:54:54.341299+0000","last_peered":"2024-04-25T15:54:54.341299+0000","last_clean":"2024-04-25T15:54:54.341299+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:54:54.341299+0000","last_undegraded":"2024-04-25T15:54:54.341299+0000","last_fullsized":"2024-04-25T15:54:54.341299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345518+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:54.345518+0000","last_peered":"2024-04-25T15:54:54.345518+0000","last_clean":"2024-04-25T15:54:54.345518+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:54.345518+0000","last_undegraded":"2024-04-25T15:54:54.345518+0000","last_fullsized":"2024-04-25T15:54:54.345518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"113'147","reported_seq":372,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341194+0000","last_change":"2024-04-25T15:54:44.212095+0000","last_active":"2024-04-25T15:54:54.341194+0000","last_peered":"2024-04-25T15:54:54.341194+0000","last_clean":"2024-04-25T15:54:54.341194+0000","last_became_active":"2024-04-25T15:54:44.209985+0000","last_became_peered":"2024-04-25T15:54:44.209985+0000","last_unstale":"2024-04-25T15:54:54.341194+0000","last_undegraded":"2024-04-25T15:54:54.341194+0000","last_fullsized":"2024-04-25T15:54:54.341194+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:35.906277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348019+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:54:54.348019+0000","last_peered":"2024-04-25T15:54:54.348019+0000","last_clean":"2024-04-25T15:54:54.348019+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:54:54.348019+0000","last_undegraded":"2024-04-25T15:54:54.348019+0000","last_fullsized":"2024-04-25T15:54:54.348019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"113'219","reported_seq":450,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346009+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:54:54.346009+0000","last_peered":"2024-04-25T15:54:54.346009+0000","last_clean":"2024-04-25T15:54:54.346009+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:54:54.346009+0000","last_undegraded":"2024-04-25T15:54:54.346009+0000","last_fullsized":"2024-04-25T15:54:54.346009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346366+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:54.346366+0000","last_peered":"2024-04-25T15:54:54.346366+0000","last_clean":"2024-04-25T15:54:54.346366+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:54.346366+0000","last_undegraded":"2024-04-25T15:54:54.346366+0000","last_fullsized":"2024-04-25T15:54:54.346366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"113'132","reported_seq":325,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352966+0000","last_change":"2024-04-25T15:54:40.169299+0000","last_active":"2024-04-25T15:54:54.352966+0000","last_peered":"2024-04-25T15:54:54.352966+0000","last_clean":"2024-04-25T15:54:54.352966+0000","last_became_active":"2024-04-25T15:54:40.168203+0000","last_became_peered":"2024-04-25T15:54:40.168203+0000","last_unstale":"2024-04-25T15:54:54.352966+0000","last_undegraded":"2024-04-25T15:54:54.352966+0000","last_fullsized":"2024-04-25T15:54:54.352966+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.710379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"114'11","reported_seq":46,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342159+0000","last_change":"2024-04-25T15:54:41.197909+0000","last_active":"2024-04-25T15:54:54.342159+0000","last_peered":"2024-04-25T15:54:54.342159+0000","last_clean":"2024-04-25T15:54:54.342159+0000","last_became_active":"2024-04-25T15:54:41.171190+0000","last_became_peered":"2024-04-25T15:54:41.171190+0000","last_unstale":"2024-04-25T15:54:54.342159+0000","last_undegraded":"2024-04-25T15:54:54.342159+0000","last_fullsized":"2024-04-25T15:54:54.342159+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:40.822327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2016,"num_keys_recovered":30,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"113'138","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341605+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:54:54.341605+0000","last_peered":"2024-04-25T15:54:54.341605+0000","last_clean":"2024-04-25T15:54:54.341605+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:54:54.341605+0000","last_undegraded":"2024-04-25T15:54:54.341605+0000","last_fullsized":"2024-04-25T15:54:54.341605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"117'90","reported_seq":30,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348326+0000","last_change":"2024-04-25T15:54:51.358160+0000","last_active":"2024-04-25T15:54:54.348326+0000","last_peered":"2024-04-25T15:54:54.348326+0000","last_clean":"2024-04-25T15:54:54.348326+0000","last_became_active":"2024-04-25T15:54:51.303172+0000","last_became_peered":"2024-04-25T15:54:51.303172+0000","last_unstale":"2024-04-25T15:54:54.348326+0000","last_undegraded":"2024-04-25T15:54:54.348326+0000","last_fullsized":"2024-04-25T15:54:54.348326+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:39.823475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199030,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":380717,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"113'144","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338292+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:54:53.338292+0000","last_peered":"2024-04-25T15:54:53.338292+0000","last_clean":"2024-04-25T15:54:53.338292+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:54:53.338292+0000","last_undegraded":"2024-04-25T15:54:53.338292+0000","last_fullsized":"2024-04-25T15:54:53.338292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352562+0000","last_change":"2024-04-25T15:54:40.166458+0000","last_active":"2024-04-25T15:54:54.352562+0000","last_peered":"2024-04-25T15:54:54.352562+0000","last_clean":"2024-04-25T15:54:54.352562+0000","last_became_active":"2024-04-25T15:54:40.165701+0000","last_became_peered":"2024-04-25T15:54:40.165701+0000","last_unstale":"2024-04-25T15:54:54.352562+0000","last_undegraded":"2024-04-25T15:54:54.352562+0000","last_fullsized":"2024-04-25T15:54:54.352562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:43:18.998536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"113'148","reported_seq":339,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352611+0000","last_change":"2024-04-25T15:54:40.168672+0000","last_active":"2024-04-25T15:54:54.352611+0000","last_peered":"2024-04-25T15:54:54.352611+0000","last_clean":"2024-04-25T15:54:54.352611+0000","last_became_active":"2024-04-25T15:54:40.167338+0000","last_became_peered":"2024-04-25T15:54:40.167338+0000","last_unstale":"2024-04-25T15:54:54.352611+0000","last_undegraded":"2024-04-25T15:54:54.352611+0000","last_fullsized":"2024-04-25T15:54:54.352611+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:04.063276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344550+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:54.344550+0000","last_peered":"2024-04-25T15:54:54.344550+0000","last_clean":"2024-04-25T15:54:54.344550+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:54.344550+0000","last_undegraded":"2024-04-25T15:54:54.344550+0000","last_fullsized":"2024-04-25T15:54:54.344550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"113'156","reported_seq":403,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.337345+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:54:53.337345+0000","last_peered":"2024-04-25T15:54:53.337345+0000","last_clean":"2024-04-25T15:54:53.337345+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:54:53.337345+0000","last_undegraded":"2024-04-25T15:54:53.337345+0000","last_fullsized":"2024-04-25T15:54:53.337345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346948+0000","last_change":"2024-04-25T15:54:51.333252+0000","last_active":"2024-04-25T15:54:54.346948+0000","last_peered":"2024-04-25T15:54:54.346948+0000","last_clean":"2024-04-25T15:54:54.346948+0000","last_became_active":"2024-04-25T15:54:51.302160+0000","last_became_peered":"2024-04-25T15:54:51.302160+0000","last_unstale":"2024-04-25T15:54:54.346948+0000","last_undegraded":"2024-04-25T15:54:54.346948+0000","last_fullsized":"2024-04-25T15:54:54.346948+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:25.724351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4116,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"113'159","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347624+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:54:54.347624+0000","last_peered":"2024-04-25T15:54:54.347624+0000","last_clean":"2024-04-25T15:54:54.347624+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:54:54.347624+0000","last_undegraded":"2024-04-25T15:54:54.347624+0000","last_fullsized":"2024-04-25T15:54:54.347624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"114'18","reported_seq":342,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345636+0000","last_change":"2024-04-25T15:54:41.193381+0000","last_active":"2024-04-25T15:54:54.345636+0000","last_peered":"2024-04-25T15:54:54.345636+0000","last_clean":"2024-04-25T15:54:54.345636+0000","last_became_active":"2024-04-25T15:54:41.166653+0000","last_became_peered":"2024-04-25T15:54:41.166653+0000","last_unstale":"2024-04-25T15:54:54.345636+0000","last_undegraded":"2024-04-25T15:54:54.345636+0000","last_fullsized":"2024-04-25T15:54:54.345636+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:10.229064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":448,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"113'183","reported_seq":438,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336812+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:54:53.336812+0000","last_peered":"2024-04-25T15:54:53.336812+0000","last_clean":"2024-04-25T15:54:53.336812+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:54:53.336812+0000","last_undegraded":"2024-04-25T15:54:53.336812+0000","last_fullsized":"2024-04-25T15:54:53.336812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":120,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341794+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:54.341794+0000","last_peered":"2024-04-25T15:54:54.341794+0000","last_clean":"2024-04-25T15:54:54.341794+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:54.341794+0000","last_undegraded":"2024-04-25T15:54:54.341794+0000","last_fullsized":"2024-04-25T15:54:54.341794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"113'141","reported_seq":384,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338909+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:54:53.338909+0000","last_peered":"2024-04-25T15:54:53.338909+0000","last_clean":"2024-04-25T15:54:53.338909+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:54:53.338909+0000","last_undegraded":"2024-04-25T15:54:53.338909+0000","last_fullsized":"2024-04-25T15:54:53.338909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346320+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:54:54.346320+0000","last_peered":"2024-04-25T15:54:54.346320+0000","last_clean":"2024-04-25T15:54:54.346320+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:54:54.346320+0000","last_undegraded":"2024-04-25T15:54:54.346320+0000","last_fullsized":"2024-04-25T15:54:54.346320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"113'129","reported_seq":376,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.339429+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:54:53.339429+0000","last_peered":"2024-04-25T15:54:53.339429+0000","last_clean":"2024-04-25T15:54:53.339429+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:54:53.339429+0000","last_undegraded":"2024-04-25T15:54:53.339429+0000","last_fullsized":"2024-04-25T15:54:53.339429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":40,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342273+0000","last_change":"2024-04-25T15:54:41.186336+0000","last_active":"2024-04-25T15:54:54.342273+0000","last_peered":"2024-04-25T15:54:54.342273+0000","last_clean":"2024-04-25T15:54:54.342273+0000","last_became_active":"2024-04-25T15:54:41.165683+0000","last_became_peered":"2024-04-25T15:54:41.165683+0000","last_unstale":"2024-04-25T15:54:54.342273+0000","last_undegraded":"2024-04-25T15:54:54.342273+0000","last_fullsized":"2024-04-25T15:54:54.342273+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:54.115834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"113'138","reported_seq":369,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345263+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:54:54.345263+0000","last_peered":"2024-04-25T15:54:54.345263+0000","last_clean":"2024-04-25T15:54:54.345263+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:54:54.345263+0000","last_undegraded":"2024-04-25T15:54:54.345263+0000","last_fullsized":"2024-04-25T15:54:54.345263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346836+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:54:54.346836+0000","last_peered":"2024-04-25T15:54:54.346836+0000","last_clean":"2024-04-25T15:54:54.346836+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:54:54.346836+0000","last_undegraded":"2024-04-25T15:54:54.346836+0000","last_fullsized":"2024-04-25T15:54:54.346836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"113'144","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338305+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:54:53.338305+0000","last_peered":"2024-04-25T15:54:53.338305+0000","last_clean":"2024-04-25T15:54:53.338305+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:54:53.338305+0000","last_undegraded":"2024-04-25T15:54:53.338305+0000","last_fullsized":"2024-04-25T15:54:53.338305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":326,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336462+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:53.336462+0000","last_peered":"2024-04-25T15:54:53.336462+0000","last_clean":"2024-04-25T15:54:53.336462+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:53.336462+0000","last_undegraded":"2024-04-25T15:54:53.336462+0000","last_fullsized":"2024-04-25T15:54:53.336462+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344831+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:54.344831+0000","last_peered":"2024-04-25T15:54:54.344831+0000","last_clean":"2024-04-25T15:54:54.344831+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:54.344831+0000","last_undegraded":"2024-04-25T15:54:54.344831+0000","last_fullsized":"2024-04-25T15:54:54.344831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"113'150","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350321+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:54:54.350321+0000","last_peered":"2024-04-25T15:54:54.350321+0000","last_clean":"2024-04-25T15:54:54.350321+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:54:54.350321+0000","last_undegraded":"2024-04-25T15:54:54.350321+0000","last_fullsized":"2024-04-25T15:54:54.350321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":270,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336240+0000","last_change":"2024-04-25T15:54:41.173406+0000","last_active":"2024-04-25T15:54:53.336240+0000","last_peered":"2024-04-25T15:54:53.336240+0000","last_clean":"2024-04-25T15:54:53.336240+0000","last_became_active":"2024-04-25T15:54:41.166720+0000","last_became_peered":"2024-04-25T15:54:41.166720+0000","last_unstale":"2024-04-25T15:54:53.336240+0000","last_undegraded":"2024-04-25T15:54:53.336240+0000","last_fullsized":"2024-04-25T15:54:53.336240+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:41.861800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"113'126","reported_seq":247,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352215+0000","last_change":"2024-04-25T15:54:40.164755+0000","last_active":"2024-04-25T15:54:54.352215+0000","last_peered":"2024-04-25T15:54:54.352215+0000","last_clean":"2024-04-25T15:54:54.352215+0000","last_became_active":"2024-04-25T15:54:40.163587+0000","last_became_peered":"2024-04-25T15:54:40.163587+0000","last_unstale":"2024-04-25T15:54:54.352215+0000","last_undegraded":"2024-04-25T15:54:54.352215+0000","last_fullsized":"2024-04-25T15:54:54.352215+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:58.420382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"114'12","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345989+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:54:54.345989+0000","last_peered":"2024-04-25T15:54:54.345989+0000","last_clean":"2024-04-25T15:54:54.345989+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:54:54.345989+0000","last_undegraded":"2024-04-25T15:54:54.345989+0000","last_fullsized":"2024-04-25T15:54:54.345989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"113'123","reported_seq":423,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341157+0000","last_change":"2024-04-25T15:54:44.216921+0000","last_active":"2024-04-25T15:54:54.341157+0000","last_peered":"2024-04-25T15:54:54.341157+0000","last_clean":"2024-04-25T15:54:54.341157+0000","last_became_active":"2024-04-25T15:54:44.215186+0000","last_became_peered":"2024-04-25T15:54:44.215186+0000","last_unstale":"2024-04-25T15:54:54.341157+0000","last_undegraded":"2024-04-25T15:54:54.341157+0000","last_fullsized":"2024-04-25T15:54:54.341157+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:49.565745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344849+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:54.344849+0000","last_peered":"2024-04-25T15:54:54.344849+0000","last_clean":"2024-04-25T15:54:54.344849+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:54.344849+0000","last_undegraded":"2024-04-25T15:54:54.344849+0000","last_fullsized":"2024-04-25T15:54:54.344849+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'138","reported_seq":321,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342709+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:54:54.342709+0000","last_peered":"2024-04-25T15:54:54.342709+0000","last_clean":"2024-04-25T15:54:54.342709+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:54:54.342709+0000","last_undegraded":"2024-04-25T15:54:54.342709+0000","last_fullsized":"2024-04-25T15:54:54.342709+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":36,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341899+0000","last_change":"2024-04-25T15:54:41.168286+0000","last_active":"2024-04-25T15:54:54.341899+0000","last_peered":"2024-04-25T15:54:54.341899+0000","last_clean":"2024-04-25T15:54:54.341899+0000","last_became_active":"2024-04-25T15:54:41.166643+0000","last_became_peered":"2024-04-25T15:54:41.166643+0000","last_unstale":"2024-04-25T15:54:54.341899+0000","last_undegraded":"2024-04-25T15:54:54.341899+0000","last_fullsized":"2024-04-25T15:54:54.341899+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:08.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"113'138","reported_seq":370,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346116+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:54:54.346116+0000","last_peered":"2024-04-25T15:54:54.346116+0000","last_clean":"2024-04-25T15:54:54.346116+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:54:54.346116+0000","last_undegraded":"2024-04-25T15:54:54.346116+0000","last_fullsized":"2024-04-25T15:54:54.346116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":234,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338944+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:53.338944+0000","last_peered":"2024-04-25T15:54:53.338944+0000","last_clean":"2024-04-25T15:54:53.338944+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:53.338944+0000","last_undegraded":"2024-04-25T15:54:53.338944+0000","last_fullsized":"2024-04-25T15:54:53.338944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"113'135","reported_seq":335,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342026+0000","last_change":"2024-04-25T15:54:44.217114+0000","last_active":"2024-04-25T15:54:54.342026+0000","last_peered":"2024-04-25T15:54:54.342026+0000","last_clean":"2024-04-25T15:54:54.342026+0000","last_became_active":"2024-04-25T15:54:44.215121+0000","last_became_peered":"2024-04-25T15:54:44.215121+0000","last_unstale":"2024-04-25T15:54:54.342026+0000","last_undegraded":"2024-04-25T15:54:54.342026+0000","last_fullsized":"2024-04-25T15:54:54.342026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:37.441111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'26","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341499+0000","last_change":"2024-04-25T15:54:51.320611+0000","last_active":"2024-04-25T15:54:54.341499+0000","last_peered":"2024-04-25T15:54:54.341499+0000","last_clean":"2024-04-25T15:54:54.341499+0000","last_became_active":"2024-04-25T15:54:51.307056+0000","last_became_peered":"2024-04-25T15:54:51.307056+0000","last_unstale":"2024-04-25T15:54:54.341499+0000","last_undegraded":"2024-04-25T15:54:54.341499+0000","last_fullsized":"2024-04-25T15:54:54.341499+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:15.769447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":13,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"113'138","reported_seq":370,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351602+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:54:54.351602+0000","last_peered":"2024-04-25T15:54:54.351602+0000","last_clean":"2024-04-25T15:54:54.351602+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:54:54.351602+0000","last_undegraded":"2024-04-25T15:54:54.351602+0000","last_fullsized":"2024-04-25T15:54:54.351602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"110'8","reported_seq":242,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348406+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:54.348406+0000","last_peered":"2024-04-25T15:54:54.348406+0000","last_clean":"2024-04-25T15:54:54.348406+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:54.348406+0000","last_undegraded":"2024-04-25T15:54:54.348406+0000","last_fullsized":"2024-04-25T15:54:54.348406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"113'156","reported_seq":384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352531+0000","last_change":"2024-04-25T15:54:44.208533+0000","last_active":"2024-04-25T15:54:54.352531+0000","last_peered":"2024-04-25T15:54:54.352531+0000","last_clean":"2024-04-25T15:54:54.352531+0000","last_became_active":"2024-04-25T15:54:44.207033+0000","last_became_peered":"2024-04-25T15:54:44.207033+0000","last_unstale":"2024-04-25T15:54:54.352531+0000","last_undegraded":"2024-04-25T15:54:54.352531+0000","last_fullsized":"2024-04-25T15:54:54.352531+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:44.559711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":232,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336093+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:53.336093+0000","last_peered":"2024-04-25T15:54:53.336093+0000","last_clean":"2024-04-25T15:54:53.336093+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:53.336093+0000","last_undegraded":"2024-04-25T15:54:53.336093+0000","last_fullsized":"2024-04-25T15:54:53.336093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"113'132","reported_seq":248,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341783+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:54:54.341783+0000","last_peered":"2024-04-25T15:54:54.341783+0000","last_clean":"2024-04-25T15:54:54.341783+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:54:54.341783+0000","last_undegraded":"2024-04-25T15:54:54.341783+0000","last_fullsized":"2024-04-25T15:54:54.341783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":206,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341314+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:54:54.341314+0000","last_peered":"2024-04-25T15:54:54.341314+0000","last_clean":"2024-04-25T15:54:54.341314+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:54:54.341314+0000","last_undegraded":"2024-04-25T15:54:54.341314+0000","last_fullsized":"2024-04-25T15:54:54.341314+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"113'144","reported_seq":512,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349677+0000","last_change":"2024-04-25T15:54:44.207954+0000","last_active":"2024-04-25T15:54:54.349677+0000","last_peered":"2024-04-25T15:54:54.349677+0000","last_clean":"2024-04-25T15:54:54.349677+0000","last_became_active":"2024-04-25T15:54:44.206595+0000","last_became_peered":"2024-04-25T15:54:44.206595+0000","last_unstale":"2024-04-25T15:54:54.349677+0000","last_undegraded":"2024-04-25T15:54:54.349677+0000","last_fullsized":"2024-04-25T15:54:54.349677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.107940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":69,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346110+0000","last_change":"2024-04-25T15:54:22.056875+0000","last_active":"2024-04-25T15:54:54.346110+0000","last_peered":"2024-04-25T15:54:54.346110+0000","last_clean":"2024-04-25T15:54:54.346110+0000","last_became_active":"2024-04-25T15:54:22.037756+0000","last_became_peered":"2024-04-25T15:54:22.037756+0000","last_unstale":"2024-04-25T15:54:54.346110+0000","last_undegraded":"2024-04-25T15:54:54.346110+0000","last_fullsized":"2024-04-25T15:54:54.346110+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:57.998015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"113'135","reported_seq":359,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346451+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:54:54.346451+0000","last_peered":"2024-04-25T15:54:54.346451+0000","last_clean":"2024-04-25T15:54:54.346451+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:54:54.346451+0000","last_undegraded":"2024-04-25T15:54:54.346451+0000","last_fullsized":"2024-04-25T15:54:54.346451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":256,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.337826+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:53.337826+0000","last_peered":"2024-04-25T15:54:53.337826+0000","last_clean":"2024-04-25T15:54:53.337826+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:53.337826+0000","last_undegraded":"2024-04-25T15:54:53.337826+0000","last_fullsized":"2024-04-25T15:54:53.337826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"113'123","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347926+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:54:54.347926+0000","last_peered":"2024-04-25T15:54:54.347926+0000","last_clean":"2024-04-25T15:54:54.347926+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:54:54.347926+0000","last_undegraded":"2024-04-25T15:54:54.347926+0000","last_fullsized":"2024-04-25T15:54:54.347926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"113'168","reported_seq":425,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345784+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:54:54.345784+0000","last_peered":"2024-04-25T15:54:54.345784+0000","last_clean":"2024-04-25T15:54:54.345784+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:54:54.345784+0000","last_undegraded":"2024-04-25T15:54:54.345784+0000","last_fullsized":"2024-04-25T15:54:54.345784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":234,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338200+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:53.338200+0000","last_peered":"2024-04-25T15:54:53.338200+0000","last_clean":"2024-04-25T15:54:53.338200+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:53.338200+0000","last_undegraded":"2024-04-25T15:54:53.338200+0000","last_fullsized":"2024-04-25T15:54:53.338200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"113'132","reported_seq":406,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344993+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:54:54.344993+0000","last_peered":"2024-04-25T15:54:54.344993+0000","last_clean":"2024-04-25T15:54:54.344993+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:54:54.344993+0000","last_undegraded":"2024-04-25T15:54:54.344993+0000","last_fullsized":"2024-04-25T15:54:54.344993+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":242,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.339121+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:53.339121+0000","last_peered":"2024-04-25T15:54:53.339121+0000","last_clean":"2024-04-25T15:54:53.339121+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:53.339121+0000","last_undegraded":"2024-04-25T15:54:53.339121+0000","last_fullsized":"2024-04-25T15:54:53.339121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"113'129","reported_seq":322,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352816+0000","last_change":"2024-04-25T15:54:40.169300+0000","last_active":"2024-04-25T15:54:54.352816+0000","last_peered":"2024-04-25T15:54:54.352816+0000","last_clean":"2024-04-25T15:54:54.352816+0000","last_became_active":"2024-04-25T15:54:40.168164+0000","last_became_peered":"2024-04-25T15:54:40.168164+0000","last_unstale":"2024-04-25T15:54:54.352816+0000","last_undegraded":"2024-04-25T15:54:54.352816+0000","last_fullsized":"2024-04-25T15:54:54.352816+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:50.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351250+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:54.351250+0000","last_peered":"2024-04-25T15:54:54.351250+0000","last_clean":"2024-04-25T15:54:54.351250+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:54.351250+0000","last_undegraded":"2024-04-25T15:54:54.351250+0000","last_fullsized":"2024-04-25T15:54:54.351250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"113'147","reported_seq":593,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.360954+0000","last_change":"2024-04-25T15:54:54.360954+0000","last_active":"2024-04-25T15:54:54.360954+0000","last_peered":"2024-04-25T15:54:54.360954+0000","last_clean":"2024-04-25T15:54:54.360954+0000","last_became_active":"2024-04-25T15:54:54.360118+0000","last_became_peered":"2024-04-25T15:54:54.360118+0000","last_unstale":"2024-04-25T15:54:54.360954+0000","last_undegraded":"2024-04-25T15:54:54.360954+0000","last_fullsized":"2024-04-25T15:54:54.360954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:37.822818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":103,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.353079+0000","last_change":"2024-04-25T15:54:40.170535+0000","last_active":"2024-04-25T15:54:54.353079+0000","last_peered":"2024-04-25T15:54:54.353079+0000","last_clean":"2024-04-25T15:54:54.353079+0000","last_became_active":"2024-04-25T15:54:40.169964+0000","last_became_peered":"2024-04-25T15:54:40.169964+0000","last_unstale":"2024-04-25T15:54:54.353079+0000","last_undegraded":"2024-04-25T15:54:54.353079+0000","last_fullsized":"2024-04-25T15:54:54.353079+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:24.688851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":1627389952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9217,"ondisk_log_size":9217,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":207280,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":338,"num_write_kb":313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":389959,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":414560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574909,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59664,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312176,"statfs":{"total":96636764160,"available":96575668224,"internally_reserved":0,"allocated":2162688,"data_stored":1331809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41220,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330620,"statfs":{"total":96636764160,"available":96594554880,"internally_reserved":0,"allocated":1757184,"data_stored":939161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63256,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308584,"statfs":{"total":96636764160,"available":96571990016,"internally_reserved":0,"allocated":1581056,"data_stored":743273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59072,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312768,"statfs":{"total":96636764160,"available":96576274432,"internally_reserved":0,"allocated":1556480,"data_stored":740671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":1765376,"data_stored":939700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574909,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1568768,"data_stored":741308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":740432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2154496,"data_stored":1330142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":200800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:54:57.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:54:57.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d85df640 1 Processor -- start 2024-04-25T15:54:57.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d85df640 1 -- start start 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d85df640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f92d0074790 0x7f92d015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d85df640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 0x7f92d00c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d85df640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92d0072dc0 con 0x7f92d015e020 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d85df640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f92d015b430 con 0x7f92d0074790 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d85df640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f92d015a670 con 0x7f92d00c6fb0 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5b7a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f92d015e020 0x7f92d00723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43350/0 (socket says 172.21.15.67:43350) 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5b7a640 1 -- 172.21.15.67:0/246008707 learned_addr learned my addr 172.21.15.67:0/246008707 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d6b7c640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 0x7f92d00c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d637b640 1 --2- 172.21.15.67:0/246008707 >> v2:172.21.15.176:3300/0 conn(0x7f92d0074790 0x7f92d015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021068157 0 0) 0x7f92d0072dc0 con 0x7f92d015e020 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92bc003530 con 0x7f92d015e020 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3217691930 0 0) 0x7f92bc003530 con 0x7f92d015e020 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 msgr2=0x7f92d00c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 0x7f92d00c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 >> v2:172.21.15.176:3300/0 conn(0x7f92d0074790 msgr2=0x7f92d015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 --2- 172.21.15.67:0/246008707 >> v2:172.21.15.176:3300/0 conn(0x7f92d0074790 0x7f92d015dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d015a950 con 0x7f92d015e020 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f92c0001680 con 0x7f92d015e020 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 >> v1:172.21.15.67:6789/0 conn(0x7f92d015e020 legacy=0x7f92d00723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92bc0039c0 0x7f92bc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.321+0000 7f92d5379640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 0x7f92bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5379640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92bc004d50 0x7f92bc005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f92d015b430 con 0x7f92bc0039c0 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f92d015a670 con 0x7f92bc004d50 2024-04-25T15:54:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f92bc00ba50 con 0x7f92d00c6fb0 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d637b640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92bc0039c0 0x7f92bc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5b7a640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 0x7f92bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d6b7c640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92bc004d50 0x7f92bc005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d637b640 1 -- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 msgr2=0x7f92bc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d637b640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 0x7f92bc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d637b640 1 -- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92bc004d50 msgr2=0x7f92bc005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d637b640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92bc004d50 0x7f92bc005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d637b640 1 -- 172.21.15.67:0/246008707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92bc003530 con 0x7f92bc0039c0 2024-04-25T15:54:57.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5b7a640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d00c6fb0 0x7f92bc004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d637b640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92bc0039c0 0x7f92bc003e30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009ab0 tx=0x7f92c801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e67aa22f620a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c80b2070 con 0x7f92bc0039c0 2024-04-25T15:54:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92c80a5680 con 0x7f92bc0039c0 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.325+0000 7f92d5379640 1 -- 172.21.15.67:0/246008707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c80096b0 con 0x7f92bc0039c0 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 -- 172.21.15.67:0/246008707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92bc0039c0 msgr2=0x7f92bc003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 --2- 172.21.15.67:0/246008707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92bc0039c0 0x7f92bc003e30 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009ab0 tx=0x7f92c801ca90 comp rx=0 tx=0).stop 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d6b7c640 1 -- 172.21.15.67:0/246008707 reap_dead start 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 -- 172.21.15.67:0/246008707 shutdown_connections 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 -- 172.21.15.67:0/246008707 >> 172.21.15.67:0/246008707 conn(0x7f92d006fe70 msgr2=0x7f92d00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 -- 172.21.15.67:0/246008707 shutdown_connections 2024-04-25T15:54:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 -- 172.21.15.67:0/246008707 wait complete. 2024-04-25T15:54:57.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 Processor -- start 2024-04-25T15:54:57.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.329+0000 7f92d85df640 1 -- start start 2024-04-25T15:54:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92d015a180 0x7f92d0145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d0152190 0x7f92d01525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d0153270 0x7f92d0153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f92d0073080 con 0x7f92d0153270 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f92d015af10 con 0x7f92d015a180 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f92d0072ab0 con 0x7f92d0152190 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d0153270 0x7f92d0153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d0153270 0x7f92d0153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49808/0 (socket says 172.21.15.67:49808) 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 -- 172.21.15.67:0/3301738756 learned_addr learned my addr 172.21.15.67:0/3301738756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d637b640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92d015a180 0x7f92d0145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d5b7a640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d0152190 0x7f92d01525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 -- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d0152190 msgr2=0x7f92d01525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d0152190 0x7f92d01525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 -- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92d015a180 msgr2=0x7f92d0145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92d015a180 0x7f92d0145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 -- 172.21.15.67:0/3301738756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d0152ae0 con 0x7f92d0153270 2024-04-25T15:54:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d5b7a640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d0152190 0x7f92d01525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d637b640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92d015a180 0x7f92d0145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:54:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d6b7c640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d0153270 0x7f92d0153680 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f92d00cb750 tx=0x7f92cc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92c6ffd640 1 -- 172.21.15.67:0/3301738756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc011e40 con 0x7f92d0153270 2024-04-25T15:54:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92c6ffd640 1 -- 172.21.15.67:0/3301738756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92cc01b070 con 0x7f92d0153270 2024-04-25T15:54:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d0159840 con 0x7f92d0153270 2024-04-25T15:54:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92c6ffd640 1 -- 172.21.15.67:0/3301738756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc0039a0 con 0x7f92d0153270 2024-04-25T15:54:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d015d0b0 con 0x7f92d0153270 2024-04-25T15:54:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d006a480 con 0x7f92d0153270 2024-04-25T15:54:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92c6ffd640 1 -- 172.21.15.67:0/3301738756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f92cc003b40 con 0x7f92d0153270 2024-04-25T15:54:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92c6ffd640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f92b406f400 0x7f92b4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92c6ffd640 1 -- 172.21.15.67:0/3301738756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f92cc0908f0 con 0x7f92d0153270 2024-04-25T15:54:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.333+0000 7f92d637b640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f92b406f400 0x7f92b4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.337+0000 7f92d637b640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f92b406f400 0x7f92b4071910 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f92c80af000 tx=0x7f92c8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:57.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.337+0000 7f92c6ffd640 1 -- 172.21.15.67:0/3301738756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f92cc025020 con 0x7f92d0153270 2024-04-25T15:54:57.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.489+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92d015de00 con 0x7f92b406f400 2024-04-25T15:54:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.509+0000 7f92c6ffd640 1 -- 172.21.15.67:0/3301738756 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253608 (secure 0 0 0) 0x7f92d015de00 con 0x7f92b406f400 2024-04-25T15:54:57.514 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:57.515 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f92b406f400 msgr2=0x7f92b4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f92b406f400 0x7f92b4071910 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f92c80af000 tx=0x7f92c8008040 comp rx=0 tx=0).stop 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d0153270 msgr2=0x7f92d0153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d0153270 0x7f92d0153680 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f92d00cb750 tx=0x7f92cc0115a0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 shutdown_connections 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f92b406f400 0x7f92b4071910 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92d0153270 0x7f92d0153680 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92d0152190 0x7f92d01525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 --2- 172.21.15.67:0/3301738756 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92d015a180 0x7f92d0145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 >> 172.21.15.67:0/3301738756 conn(0x7f92d006fe70 msgr2=0x7f92d0150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 shutdown_connections 2024-04-25T15:54:57.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.513+0000 7f92d85df640 1 -- 172.21.15.67:0/3301738756 wait complete. 2024-04-25T15:54:57.535 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-04-25T15:54:55.889485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":797648,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":9655,"num_write_kb":25167765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":1627779911,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9589,"ondisk_log_size":9589,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":199,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383964,"kb_used_data":13776,"kb_used_omap":12,"kb_used_meta":370035,"kb_avail":754590756,"statfs":{"total":773094113280,"available":772700934144,"internally_reserved":0,"allocated":14106624,"data_stored":7506496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":378916402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":254,"num_bytes_recovered":199030,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.607980"},"pg_stats":[{"pgid":"3.3f","version":"113'138","reported_seq":366,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.339269+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:54:53.339269+0000","last_peered":"2024-04-25T15:54:53.339269+0000","last_clean":"2024-04-25T15:54:53.339269+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:54:53.339269+0000","last_undegraded":"2024-04-25T15:54:53.339269+0000","last_fullsized":"2024-04-25T15:54:53.339269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"113'126","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341661+0000","last_change":"2024-04-25T15:54:44.217073+0000","last_active":"2024-04-25T15:54:54.341661+0000","last_peered":"2024-04-25T15:54:54.341661+0000","last_clean":"2024-04-25T15:54:54.341661+0000","last_became_active":"2024-04-25T15:54:44.215261+0000","last_became_peered":"2024-04-25T15:54:44.215261+0000","last_unstale":"2024-04-25T15:54:54.341661+0000","last_undegraded":"2024-04-25T15:54:54.341661+0000","last_fullsized":"2024-04-25T15:54:54.341661+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:01.929448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"113'144","reported_seq":336,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352388+0000","last_change":"2024-04-25T15:54:40.166712+0000","last_active":"2024-04-25T15:54:54.352388+0000","last_peered":"2024-04-25T15:54:54.352388+0000","last_clean":"2024-04-25T15:54:54.352388+0000","last_became_active":"2024-04-25T15:54:40.164866+0000","last_became_peered":"2024-04-25T15:54:40.164866+0000","last_unstale":"2024-04-25T15:54:54.352388+0000","last_undegraded":"2024-04-25T15:54:54.352388+0000","last_fullsized":"2024-04-25T15:54:54.352388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:58.780901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"113'165","reported_seq":397,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344931+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:54:54.344931+0000","last_peered":"2024-04-25T15:54:54.344931+0000","last_clean":"2024-04-25T15:54:54.344931+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:54:54.344931+0000","last_undegraded":"2024-04-25T15:54:54.344931+0000","last_fullsized":"2024-04-25T15:54:54.344931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"113'144","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336952+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:54:53.336952+0000","last_peered":"2024-04-25T15:54:53.336952+0000","last_clean":"2024-04-25T15:54:53.336952+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:54:53.336952+0000","last_undegraded":"2024-04-25T15:54:53.336952+0000","last_fullsized":"2024-04-25T15:54:53.336952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"113'129","reported_seq":360,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344524+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:54:54.344524+0000","last_peered":"2024-04-25T15:54:54.344524+0000","last_clean":"2024-04-25T15:54:54.344524+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:54:54.344524+0000","last_undegraded":"2024-04-25T15:54:54.344524+0000","last_fullsized":"2024-04-25T15:54:54.344524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"113'159","reported_seq":386,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336477+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:54:53.336477+0000","last_peered":"2024-04-25T15:54:53.336477+0000","last_clean":"2024-04-25T15:54:53.336477+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:54:53.336477+0000","last_undegraded":"2024-04-25T15:54:53.336477+0000","last_fullsized":"2024-04-25T15:54:53.336477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"113'177","reported_seq":408,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349516+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:54:54.349516+0000","last_peered":"2024-04-25T15:54:54.349516+0000","last_clean":"2024-04-25T15:54:54.349516+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:54:54.349516+0000","last_undegraded":"2024-04-25T15:54:54.349516+0000","last_fullsized":"2024-04-25T15:54:54.349516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"113'141","reported_seq":365,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347174+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:54:54.347174+0000","last_peered":"2024-04-25T15:54:54.347174+0000","last_clean":"2024-04-25T15:54:54.347174+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:54:54.347174+0000","last_undegraded":"2024-04-25T15:54:54.347174+0000","last_fullsized":"2024-04-25T15:54:54.347174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"113'141","reported_seq":443,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.359559+0000","last_change":"2024-04-25T15:54:54.355672+0000","last_active":"2024-04-25T15:54:54.359559+0000","last_peered":"2024-04-25T15:54:54.359559+0000","last_clean":"2024-04-25T15:54:54.359559+0000","last_became_active":"2024-04-25T15:54:54.352223+0000","last_became_peered":"2024-04-25T15:54:54.352223+0000","last_unstale":"2024-04-25T15:54:54.359559+0000","last_undegraded":"2024-04-25T15:54:54.359559+0000","last_fullsized":"2024-04-25T15:54:54.359559+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:39.782280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"113'183","reported_seq":414,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349394+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:54:54.349394+0000","last_peered":"2024-04-25T15:54:54.349394+0000","last_clean":"2024-04-25T15:54:54.349394+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:54:54.349394+0000","last_undegraded":"2024-04-25T15:54:54.349394+0000","last_fullsized":"2024-04-25T15:54:54.349394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"113'135","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341244+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:54:54.341244+0000","last_peered":"2024-04-25T15:54:54.341244+0000","last_clean":"2024-04-25T15:54:54.341244+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:54:54.341244+0000","last_undegraded":"2024-04-25T15:54:54.341244+0000","last_fullsized":"2024-04-25T15:54:54.341244+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"113'141","reported_seq":681,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.357489+0000","last_change":"2024-04-25T15:54:54.357489+0000","last_active":"2024-04-25T15:54:54.357489+0000","last_peered":"2024-04-25T15:54:54.357489+0000","last_clean":"2024-04-25T15:54:54.357489+0000","last_became_active":"2024-04-25T15:54:54.356694+0000","last_became_peered":"2024-04-25T15:54:54.356694+0000","last_unstale":"2024-04-25T15:54:54.357489+0000","last_undegraded":"2024-04-25T15:54:54.357489+0000","last_fullsized":"2024-04-25T15:54:54.357489+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.653628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"113'141","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352252+0000","last_change":"2024-04-25T15:54:40.160979+0000","last_active":"2024-04-25T15:54:54.352252+0000","last_peered":"2024-04-25T15:54:54.352252+0000","last_clean":"2024-04-25T15:54:54.352252+0000","last_became_active":"2024-04-25T15:54:40.159879+0000","last_became_peered":"2024-04-25T15:54:40.159879+0000","last_unstale":"2024-04-25T15:54:54.352252+0000","last_undegraded":"2024-04-25T15:54:54.352252+0000","last_fullsized":"2024-04-25T15:54:54.352252+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:25.499728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"113'144","reported_seq":263,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341194+0000","last_change":"2024-04-25T15:54:44.208852+0000","last_active":"2024-04-25T15:54:54.341194+0000","last_peered":"2024-04-25T15:54:54.341194+0000","last_clean":"2024-04-25T15:54:54.341194+0000","last_became_active":"2024-04-25T15:54:44.207144+0000","last_became_peered":"2024-04-25T15:54:44.207144+0000","last_unstale":"2024-04-25T15:54:54.341194+0000","last_undegraded":"2024-04-25T15:54:54.341194+0000","last_fullsized":"2024-04-25T15:54:54.341194+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:44:02.181746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"113'138","reported_seq":369,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344487+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:54:54.344487+0000","last_peered":"2024-04-25T15:54:54.344487+0000","last_clean":"2024-04-25T15:54:54.344487+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:54:54.344487+0000","last_undegraded":"2024-04-25T15:54:54.344487+0000","last_fullsized":"2024-04-25T15:54:54.344487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"113'138","reported_seq":303,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346019+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:54:54.346019+0000","last_peered":"2024-04-25T15:54:54.346019+0000","last_clean":"2024-04-25T15:54:54.346019+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:54:54.346019+0000","last_undegraded":"2024-04-25T15:54:54.346019+0000","last_fullsized":"2024-04-25T15:54:54.346019+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"113'147","reported_seq":591,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.355989+0000","last_change":"2024-04-25T15:54:54.355989+0000","last_active":"2024-04-25T15:54:54.355989+0000","last_peered":"2024-04-25T15:54:54.355989+0000","last_clean":"2024-04-25T15:54:54.355989+0000","last_became_active":"2024-04-25T15:54:54.354849+0000","last_became_peered":"2024-04-25T15:54:54.354849+0000","last_unstale":"2024-04-25T15:54:54.355989+0000","last_undegraded":"2024-04-25T15:54:54.355989+0000","last_fullsized":"2024-04-25T15:54:54.355989+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:05.626413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"113'165","reported_seq":392,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338717+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:54:53.338717+0000","last_peered":"2024-04-25T15:54:53.338717+0000","last_clean":"2024-04-25T15:54:53.338717+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:54:53.338717+0000","last_undegraded":"2024-04-25T15:54:53.338717+0000","last_fullsized":"2024-04-25T15:54:53.338717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"113'153","reported_seq":400,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348880+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:54:54.348880+0000","last_peered":"2024-04-25T15:54:54.348880+0000","last_clean":"2024-04-25T15:54:54.348880+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:54:54.348880+0000","last_undegraded":"2024-04-25T15:54:54.348880+0000","last_fullsized":"2024-04-25T15:54:54.348880+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"113'126","reported_seq":316,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352187+0000","last_change":"2024-04-25T15:54:40.169787+0000","last_active":"2024-04-25T15:54:54.352187+0000","last_peered":"2024-04-25T15:54:54.352187+0000","last_clean":"2024-04-25T15:54:54.352187+0000","last_became_active":"2024-04-25T15:54:40.167469+0000","last_became_peered":"2024-04-25T15:54:40.167469+0000","last_unstale":"2024-04-25T15:54:54.352187+0000","last_undegraded":"2024-04-25T15:54:54.352187+0000","last_fullsized":"2024-04-25T15:54:54.352187+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:26.375093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"113'156","reported_seq":380,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346230+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:54:54.346230+0000","last_peered":"2024-04-25T15:54:54.346230+0000","last_clean":"2024-04-25T15:54:54.346230+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:54:54.346230+0000","last_undegraded":"2024-04-25T15:54:54.346230+0000","last_fullsized":"2024-04-25T15:54:54.346230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"113'126","reported_seq":373,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349915+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:54:54.349915+0000","last_peered":"2024-04-25T15:54:54.349915+0000","last_clean":"2024-04-25T15:54:54.349915+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:54:54.349915+0000","last_undegraded":"2024-04-25T15:54:54.349915+0000","last_fullsized":"2024-04-25T15:54:54.349915+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"113'174","reported_seq":406,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345202+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:54:54.345202+0000","last_peered":"2024-04-25T15:54:54.345202+0000","last_clean":"2024-04-25T15:54:54.345202+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:54:54.345202+0000","last_undegraded":"2024-04-25T15:54:54.345202+0000","last_fullsized":"2024-04-25T15:54:54.345202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"113'132","reported_seq":233,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338631+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:54:53.338631+0000","last_peered":"2024-04-25T15:54:53.338631+0000","last_clean":"2024-04-25T15:54:53.338631+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:54:53.338631+0000","last_undegraded":"2024-04-25T15:54:53.338631+0000","last_fullsized":"2024-04-25T15:54:53.338631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"113'183","reported_seq":415,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349568+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:54:54.349568+0000","last_peered":"2024-04-25T15:54:54.349568+0000","last_clean":"2024-04-25T15:54:54.349568+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:54:54.349568+0000","last_undegraded":"2024-04-25T15:54:54.349568+0000","last_fullsized":"2024-04-25T15:54:54.349568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"113'138","reported_seq":350,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342384+0000","last_change":"2024-04-25T15:54:44.212386+0000","last_active":"2024-04-25T15:54:54.342384+0000","last_peered":"2024-04-25T15:54:54.342384+0000","last_clean":"2024-04-25T15:54:54.342384+0000","last_became_active":"2024-04-25T15:54:44.210520+0000","last_became_peered":"2024-04-25T15:54:44.210520+0000","last_unstale":"2024-04-25T15:54:54.342384+0000","last_undegraded":"2024-04-25T15:54:54.342384+0000","last_fullsized":"2024-04-25T15:54:54.342384+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:01.061276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"113'129","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350925+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:54:54.350925+0000","last_peered":"2024-04-25T15:54:54.350925+0000","last_clean":"2024-04-25T15:54:54.350925+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:54:54.350925+0000","last_undegraded":"2024-04-25T15:54:54.350925+0000","last_fullsized":"2024-04-25T15:54:54.350925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"113'111","reported_seq":304,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352326+0000","last_change":"2024-04-25T15:54:40.174462+0000","last_active":"2024-04-25T15:54:54.352326+0000","last_peered":"2024-04-25T15:54:54.352326+0000","last_clean":"2024-04-25T15:54:54.352326+0000","last_became_active":"2024-04-25T15:54:40.173209+0000","last_became_peered":"2024-04-25T15:54:40.173209+0000","last_unstale":"2024-04-25T15:54:54.352326+0000","last_undegraded":"2024-04-25T15:54:54.352326+0000","last_fullsized":"2024-04-25T15:54:54.352326+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:08.410467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"113'123","reported_seq":299,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341825+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:54:54.341825+0000","last_peered":"2024-04-25T15:54:54.341825+0000","last_clean":"2024-04-25T15:54:54.341825+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:54:54.341825+0000","last_undegraded":"2024-04-25T15:54:54.341825+0000","last_fullsized":"2024-04-25T15:54:54.341825+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"113'123","reported_seq":364,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345864+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:54:54.345864+0000","last_peered":"2024-04-25T15:54:54.345864+0000","last_clean":"2024-04-25T15:54:54.345864+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:54:54.345864+0000","last_undegraded":"2024-04-25T15:54:54.345864+0000","last_fullsized":"2024-04-25T15:54:54.345864+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"113'141","reported_seq":470,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.361042+0000","last_change":"2024-04-25T15:54:54.359681+0000","last_active":"2024-04-25T15:54:54.361042+0000","last_peered":"2024-04-25T15:54:54.361042+0000","last_clean":"2024-04-25T15:54:54.361042+0000","last_became_active":"2024-04-25T15:54:54.353824+0000","last_became_peered":"2024-04-25T15:54:54.353824+0000","last_unstale":"2024-04-25T15:54:54.361042+0000","last_undegraded":"2024-04-25T15:54:54.361042+0000","last_fullsized":"2024-04-25T15:54:54.361042+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:52.297640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"113'123","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341022+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:54:54.341022+0000","last_peered":"2024-04-25T15:54:54.341022+0000","last_clean":"2024-04-25T15:54:54.341022+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:54:54.341022+0000","last_undegraded":"2024-04-25T15:54:54.341022+0000","last_fullsized":"2024-04-25T15:54:54.341022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351432+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:54.351432+0000","last_peered":"2024-04-25T15:54:54.351432+0000","last_clean":"2024-04-25T15:54:54.351432+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:54.351432+0000","last_undegraded":"2024-04-25T15:54:54.351432+0000","last_fullsized":"2024-04-25T15:54:54.351432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"113'129","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352444+0000","last_change":"2024-04-25T15:54:40.163745+0000","last_active":"2024-04-25T15:54:54.352444+0000","last_peered":"2024-04-25T15:54:54.352444+0000","last_clean":"2024-04-25T15:54:54.352444+0000","last_became_active":"2024-04-25T15:54:40.162667+0000","last_became_peered":"2024-04-25T15:54:40.162667+0000","last_unstale":"2024-04-25T15:54:54.352444+0000","last_undegraded":"2024-04-25T15:54:54.352444+0000","last_fullsized":"2024-04-25T15:54:54.352444+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:17.528981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347310+0000","last_change":"2024-04-25T15:54:51.317865+0000","last_active":"2024-04-25T15:54:54.347310+0000","last_peered":"2024-04-25T15:54:54.347310+0000","last_clean":"2024-04-25T15:54:54.347310+0000","last_became_active":"2024-04-25T15:54:51.302422+0000","last_became_peered":"2024-04-25T15:54:51.302422+0000","last_unstale":"2024-04-25T15:54:54.347310+0000","last_undegraded":"2024-04-25T15:54:54.347310+0000","last_fullsized":"2024-04-25T15:54:54.347310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:36.956598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350037+0000","last_change":"2024-04-25T15:54:51.329012+0000","last_active":"2024-04-25T15:54:54.350037+0000","last_peered":"2024-04-25T15:54:54.350037+0000","last_clean":"2024-04-25T15:54:54.350037+0000","last_became_active":"2024-04-25T15:54:51.307662+0000","last_became_peered":"2024-04-25T15:54:51.307662+0000","last_unstale":"2024-04-25T15:54:54.350037+0000","last_undegraded":"2024-04-25T15:54:54.350037+0000","last_fullsized":"2024-04-25T15:54:54.350037+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:41.566474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"113'162","reported_seq":288,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341299+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:54:54.341299+0000","last_peered":"2024-04-25T15:54:54.341299+0000","last_clean":"2024-04-25T15:54:54.341299+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:54:54.341299+0000","last_undegraded":"2024-04-25T15:54:54.341299+0000","last_fullsized":"2024-04-25T15:54:54.341299+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345518+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:54.345518+0000","last_peered":"2024-04-25T15:54:54.345518+0000","last_clean":"2024-04-25T15:54:54.345518+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:54.345518+0000","last_undegraded":"2024-04-25T15:54:54.345518+0000","last_fullsized":"2024-04-25T15:54:54.345518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"113'147","reported_seq":372,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341194+0000","last_change":"2024-04-25T15:54:44.212095+0000","last_active":"2024-04-25T15:54:54.341194+0000","last_peered":"2024-04-25T15:54:54.341194+0000","last_clean":"2024-04-25T15:54:54.341194+0000","last_became_active":"2024-04-25T15:54:44.209985+0000","last_became_peered":"2024-04-25T15:54:44.209985+0000","last_unstale":"2024-04-25T15:54:54.341194+0000","last_undegraded":"2024-04-25T15:54:54.341194+0000","last_fullsized":"2024-04-25T15:54:54.341194+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:35.906277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"81'6","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348019+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:54:54.348019+0000","last_peered":"2024-04-25T15:54:54.348019+0000","last_clean":"2024-04-25T15:54:54.348019+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:54:54.348019+0000","last_undegraded":"2024-04-25T15:54:54.348019+0000","last_fullsized":"2024-04-25T15:54:54.348019+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"113'219","reported_seq":450,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346009+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:54:54.346009+0000","last_peered":"2024-04-25T15:54:54.346009+0000","last_clean":"2024-04-25T15:54:54.346009+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:54:54.346009+0000","last_undegraded":"2024-04-25T15:54:54.346009+0000","last_fullsized":"2024-04-25T15:54:54.346009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346366+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:54.346366+0000","last_peered":"2024-04-25T15:54:54.346366+0000","last_clean":"2024-04-25T15:54:54.346366+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:54.346366+0000","last_undegraded":"2024-04-25T15:54:54.346366+0000","last_fullsized":"2024-04-25T15:54:54.346366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"113'132","reported_seq":325,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352966+0000","last_change":"2024-04-25T15:54:40.169299+0000","last_active":"2024-04-25T15:54:54.352966+0000","last_peered":"2024-04-25T15:54:54.352966+0000","last_clean":"2024-04-25T15:54:54.352966+0000","last_became_active":"2024-04-25T15:54:40.168203+0000","last_became_peered":"2024-04-25T15:54:40.168203+0000","last_unstale":"2024-04-25T15:54:54.352966+0000","last_undegraded":"2024-04-25T15:54:54.352966+0000","last_fullsized":"2024-04-25T15:54:54.352966+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.710379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"114'11","reported_seq":46,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342159+0000","last_change":"2024-04-25T15:54:41.197909+0000","last_active":"2024-04-25T15:54:54.342159+0000","last_peered":"2024-04-25T15:54:54.342159+0000","last_clean":"2024-04-25T15:54:54.342159+0000","last_became_active":"2024-04-25T15:54:41.171190+0000","last_became_peered":"2024-04-25T15:54:41.171190+0000","last_unstale":"2024-04-25T15:54:54.342159+0000","last_undegraded":"2024-04-25T15:54:54.342159+0000","last_fullsized":"2024-04-25T15:54:54.342159+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:40.822327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2016,"num_keys_recovered":30,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"113'138","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341605+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:54:54.341605+0000","last_peered":"2024-04-25T15:54:54.341605+0000","last_clean":"2024-04-25T15:54:54.341605+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:54:54.341605+0000","last_undegraded":"2024-04-25T15:54:54.341605+0000","last_fullsized":"2024-04-25T15:54:54.341605+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"117'90","reported_seq":30,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348326+0000","last_change":"2024-04-25T15:54:51.358160+0000","last_active":"2024-04-25T15:54:54.348326+0000","last_peered":"2024-04-25T15:54:54.348326+0000","last_clean":"2024-04-25T15:54:54.348326+0000","last_became_active":"2024-04-25T15:54:51.303172+0000","last_became_peered":"2024-04-25T15:54:51.303172+0000","last_unstale":"2024-04-25T15:54:54.348326+0000","last_undegraded":"2024-04-25T15:54:54.348326+0000","last_fullsized":"2024-04-25T15:54:54.348326+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:39.823475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199030,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91,"num_write_kb":228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":380717,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"113'144","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338292+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:54:53.338292+0000","last_peered":"2024-04-25T15:54:53.338292+0000","last_clean":"2024-04-25T15:54:53.338292+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:54:53.338292+0000","last_undegraded":"2024-04-25T15:54:53.338292+0000","last_fullsized":"2024-04-25T15:54:53.338292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352562+0000","last_change":"2024-04-25T15:54:40.166458+0000","last_active":"2024-04-25T15:54:54.352562+0000","last_peered":"2024-04-25T15:54:54.352562+0000","last_clean":"2024-04-25T15:54:54.352562+0000","last_became_active":"2024-04-25T15:54:40.165701+0000","last_became_peered":"2024-04-25T15:54:40.165701+0000","last_unstale":"2024-04-25T15:54:54.352562+0000","last_undegraded":"2024-04-25T15:54:54.352562+0000","last_fullsized":"2024-04-25T15:54:54.352562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:43:18.998536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"113'148","reported_seq":339,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352611+0000","last_change":"2024-04-25T15:54:40.168672+0000","last_active":"2024-04-25T15:54:54.352611+0000","last_peered":"2024-04-25T15:54:54.352611+0000","last_clean":"2024-04-25T15:54:54.352611+0000","last_became_active":"2024-04-25T15:54:40.167338+0000","last_became_peered":"2024-04-25T15:54:40.167338+0000","last_unstale":"2024-04-25T15:54:54.352611+0000","last_undegraded":"2024-04-25T15:54:54.352611+0000","last_fullsized":"2024-04-25T15:54:54.352611+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:04.063276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344550+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:54.344550+0000","last_peered":"2024-04-25T15:54:54.344550+0000","last_clean":"2024-04-25T15:54:54.344550+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:54.344550+0000","last_undegraded":"2024-04-25T15:54:54.344550+0000","last_fullsized":"2024-04-25T15:54:54.344550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"113'156","reported_seq":403,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.337345+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:54:53.337345+0000","last_peered":"2024-04-25T15:54:53.337345+0000","last_clean":"2024-04-25T15:54:53.337345+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:54:53.337345+0000","last_undegraded":"2024-04-25T15:54:53.337345+0000","last_fullsized":"2024-04-25T15:54:53.337345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346948+0000","last_change":"2024-04-25T15:54:51.333252+0000","last_active":"2024-04-25T15:54:54.346948+0000","last_peered":"2024-04-25T15:54:54.346948+0000","last_clean":"2024-04-25T15:54:54.346948+0000","last_became_active":"2024-04-25T15:54:51.302160+0000","last_became_peered":"2024-04-25T15:54:51.302160+0000","last_unstale":"2024-04-25T15:54:54.346948+0000","last_undegraded":"2024-04-25T15:54:54.346948+0000","last_fullsized":"2024-04-25T15:54:54.346948+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:25.724351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4116,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"113'159","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347624+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:54:54.347624+0000","last_peered":"2024-04-25T15:54:54.347624+0000","last_clean":"2024-04-25T15:54:54.347624+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:54:54.347624+0000","last_undegraded":"2024-04-25T15:54:54.347624+0000","last_fullsized":"2024-04-25T15:54:54.347624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"114'18","reported_seq":342,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345636+0000","last_change":"2024-04-25T15:54:41.193381+0000","last_active":"2024-04-25T15:54:54.345636+0000","last_peered":"2024-04-25T15:54:54.345636+0000","last_clean":"2024-04-25T15:54:54.345636+0000","last_became_active":"2024-04-25T15:54:41.166653+0000","last_became_peered":"2024-04-25T15:54:41.166653+0000","last_unstale":"2024-04-25T15:54:54.345636+0000","last_undegraded":"2024-04-25T15:54:54.345636+0000","last_fullsized":"2024-04-25T15:54:54.345636+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:10.229064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":448,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"113'183","reported_seq":438,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336812+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:54:53.336812+0000","last_peered":"2024-04-25T15:54:53.336812+0000","last_clean":"2024-04-25T15:54:53.336812+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:54:53.336812+0000","last_undegraded":"2024-04-25T15:54:53.336812+0000","last_fullsized":"2024-04-25T15:54:53.336812+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":120,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341794+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:54.341794+0000","last_peered":"2024-04-25T15:54:54.341794+0000","last_clean":"2024-04-25T15:54:54.341794+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:54.341794+0000","last_undegraded":"2024-04-25T15:54:54.341794+0000","last_fullsized":"2024-04-25T15:54:54.341794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"113'141","reported_seq":384,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338909+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:54:53.338909+0000","last_peered":"2024-04-25T15:54:53.338909+0000","last_clean":"2024-04-25T15:54:53.338909+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:54:53.338909+0000","last_undegraded":"2024-04-25T15:54:53.338909+0000","last_fullsized":"2024-04-25T15:54:53.338909+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346320+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:54:54.346320+0000","last_peered":"2024-04-25T15:54:54.346320+0000","last_clean":"2024-04-25T15:54:54.346320+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:54:54.346320+0000","last_undegraded":"2024-04-25T15:54:54.346320+0000","last_fullsized":"2024-04-25T15:54:54.346320+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"113'129","reported_seq":376,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.339429+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:54:53.339429+0000","last_peered":"2024-04-25T15:54:53.339429+0000","last_clean":"2024-04-25T15:54:53.339429+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:54:53.339429+0000","last_undegraded":"2024-04-25T15:54:53.339429+0000","last_fullsized":"2024-04-25T15:54:53.339429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":40,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342273+0000","last_change":"2024-04-25T15:54:41.186336+0000","last_active":"2024-04-25T15:54:54.342273+0000","last_peered":"2024-04-25T15:54:54.342273+0000","last_clean":"2024-04-25T15:54:54.342273+0000","last_became_active":"2024-04-25T15:54:41.165683+0000","last_became_peered":"2024-04-25T15:54:41.165683+0000","last_unstale":"2024-04-25T15:54:54.342273+0000","last_undegraded":"2024-04-25T15:54:54.342273+0000","last_fullsized":"2024-04-25T15:54:54.342273+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:54.115834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"113'138","reported_seq":369,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345263+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:54:54.345263+0000","last_peered":"2024-04-25T15:54:54.345263+0000","last_clean":"2024-04-25T15:54:54.345263+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:54:54.345263+0000","last_undegraded":"2024-04-25T15:54:54.345263+0000","last_fullsized":"2024-04-25T15:54:54.345263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346836+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:54:54.346836+0000","last_peered":"2024-04-25T15:54:54.346836+0000","last_clean":"2024-04-25T15:54:54.346836+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:54:54.346836+0000","last_undegraded":"2024-04-25T15:54:54.346836+0000","last_fullsized":"2024-04-25T15:54:54.346836+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"113'144","reported_seq":371,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338305+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:54:53.338305+0000","last_peered":"2024-04-25T15:54:53.338305+0000","last_clean":"2024-04-25T15:54:53.338305+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:54:53.338305+0000","last_undegraded":"2024-04-25T15:54:53.338305+0000","last_fullsized":"2024-04-25T15:54:53.338305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":326,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336462+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:53.336462+0000","last_peered":"2024-04-25T15:54:53.336462+0000","last_clean":"2024-04-25T15:54:53.336462+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:53.336462+0000","last_undegraded":"2024-04-25T15:54:53.336462+0000","last_fullsized":"2024-04-25T15:54:53.336462+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344831+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:54.344831+0000","last_peered":"2024-04-25T15:54:54.344831+0000","last_clean":"2024-04-25T15:54:54.344831+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:54.344831+0000","last_undegraded":"2024-04-25T15:54:54.344831+0000","last_fullsized":"2024-04-25T15:54:54.344831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"113'150","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350321+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:54:54.350321+0000","last_peered":"2024-04-25T15:54:54.350321+0000","last_clean":"2024-04-25T15:54:54.350321+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:54:54.350321+0000","last_undegraded":"2024-04-25T15:54:54.350321+0000","last_fullsized":"2024-04-25T15:54:54.350321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":270,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336240+0000","last_change":"2024-04-25T15:54:41.173406+0000","last_active":"2024-04-25T15:54:53.336240+0000","last_peered":"2024-04-25T15:54:53.336240+0000","last_clean":"2024-04-25T15:54:53.336240+0000","last_became_active":"2024-04-25T15:54:41.166720+0000","last_became_peered":"2024-04-25T15:54:41.166720+0000","last_unstale":"2024-04-25T15:54:53.336240+0000","last_undegraded":"2024-04-25T15:54:53.336240+0000","last_fullsized":"2024-04-25T15:54:53.336240+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:41.861800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"113'126","reported_seq":247,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352215+0000","last_change":"2024-04-25T15:54:40.164755+0000","last_active":"2024-04-25T15:54:54.352215+0000","last_peered":"2024-04-25T15:54:54.352215+0000","last_clean":"2024-04-25T15:54:54.352215+0000","last_became_active":"2024-04-25T15:54:40.163587+0000","last_became_peered":"2024-04-25T15:54:40.163587+0000","last_unstale":"2024-04-25T15:54:54.352215+0000","last_undegraded":"2024-04-25T15:54:54.352215+0000","last_fullsized":"2024-04-25T15:54:54.352215+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:58.420382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"114'12","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345989+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:54:54.345989+0000","last_peered":"2024-04-25T15:54:54.345989+0000","last_clean":"2024-04-25T15:54:54.345989+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:54:54.345989+0000","last_undegraded":"2024-04-25T15:54:54.345989+0000","last_fullsized":"2024-04-25T15:54:54.345989+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"113'123","reported_seq":423,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341157+0000","last_change":"2024-04-25T15:54:44.216921+0000","last_active":"2024-04-25T15:54:54.341157+0000","last_peered":"2024-04-25T15:54:54.341157+0000","last_clean":"2024-04-25T15:54:54.341157+0000","last_became_active":"2024-04-25T15:54:44.215186+0000","last_became_peered":"2024-04-25T15:54:44.215186+0000","last_unstale":"2024-04-25T15:54:54.341157+0000","last_undegraded":"2024-04-25T15:54:54.341157+0000","last_fullsized":"2024-04-25T15:54:54.341157+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:49.565745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344849+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:54.344849+0000","last_peered":"2024-04-25T15:54:54.344849+0000","last_clean":"2024-04-25T15:54:54.344849+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:54.344849+0000","last_undegraded":"2024-04-25T15:54:54.344849+0000","last_fullsized":"2024-04-25T15:54:54.344849+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"113'138","reported_seq":321,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342709+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:54:54.342709+0000","last_peered":"2024-04-25T15:54:54.342709+0000","last_clean":"2024-04-25T15:54:54.342709+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:54:54.342709+0000","last_undegraded":"2024-04-25T15:54:54.342709+0000","last_fullsized":"2024-04-25T15:54:54.342709+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":36,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341899+0000","last_change":"2024-04-25T15:54:41.168286+0000","last_active":"2024-04-25T15:54:54.341899+0000","last_peered":"2024-04-25T15:54:54.341899+0000","last_clean":"2024-04-25T15:54:54.341899+0000","last_became_active":"2024-04-25T15:54:41.166643+0000","last_became_peered":"2024-04-25T15:54:41.166643+0000","last_unstale":"2024-04-25T15:54:54.341899+0000","last_undegraded":"2024-04-25T15:54:54.341899+0000","last_fullsized":"2024-04-25T15:54:54.341899+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:08.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"113'138","reported_seq":370,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346116+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:54:54.346116+0000","last_peered":"2024-04-25T15:54:54.346116+0000","last_clean":"2024-04-25T15:54:54.346116+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:54:54.346116+0000","last_undegraded":"2024-04-25T15:54:54.346116+0000","last_fullsized":"2024-04-25T15:54:54.346116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":234,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338944+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:53.338944+0000","last_peered":"2024-04-25T15:54:53.338944+0000","last_clean":"2024-04-25T15:54:53.338944+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:53.338944+0000","last_undegraded":"2024-04-25T15:54:53.338944+0000","last_fullsized":"2024-04-25T15:54:53.338944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"113'135","reported_seq":335,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342026+0000","last_change":"2024-04-25T15:54:44.217114+0000","last_active":"2024-04-25T15:54:54.342026+0000","last_peered":"2024-04-25T15:54:54.342026+0000","last_clean":"2024-04-25T15:54:54.342026+0000","last_became_active":"2024-04-25T15:54:44.215121+0000","last_became_peered":"2024-04-25T15:54:44.215121+0000","last_unstale":"2024-04-25T15:54:54.342026+0000","last_undegraded":"2024-04-25T15:54:54.342026+0000","last_fullsized":"2024-04-25T15:54:54.342026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:37.441111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'26","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341499+0000","last_change":"2024-04-25T15:54:51.320611+0000","last_active":"2024-04-25T15:54:54.341499+0000","last_peered":"2024-04-25T15:54:54.341499+0000","last_clean":"2024-04-25T15:54:54.341499+0000","last_became_active":"2024-04-25T15:54:51.307056+0000","last_became_peered":"2024-04-25T15:54:51.307056+0000","last_unstale":"2024-04-25T15:54:54.341499+0000","last_undegraded":"2024-04-25T15:54:54.341499+0000","last_fullsized":"2024-04-25T15:54:54.341499+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:15.769447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":13,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"113'138","reported_seq":370,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351602+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:54:54.351602+0000","last_peered":"2024-04-25T15:54:54.351602+0000","last_clean":"2024-04-25T15:54:54.351602+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:54:54.351602+0000","last_undegraded":"2024-04-25T15:54:54.351602+0000","last_fullsized":"2024-04-25T15:54:54.351602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"110'8","reported_seq":242,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348406+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:54.348406+0000","last_peered":"2024-04-25T15:54:54.348406+0000","last_clean":"2024-04-25T15:54:54.348406+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:54.348406+0000","last_undegraded":"2024-04-25T15:54:54.348406+0000","last_fullsized":"2024-04-25T15:54:54.348406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"113'156","reported_seq":384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352531+0000","last_change":"2024-04-25T15:54:44.208533+0000","last_active":"2024-04-25T15:54:54.352531+0000","last_peered":"2024-04-25T15:54:54.352531+0000","last_clean":"2024-04-25T15:54:54.352531+0000","last_became_active":"2024-04-25T15:54:44.207033+0000","last_became_peered":"2024-04-25T15:54:44.207033+0000","last_unstale":"2024-04-25T15:54:54.352531+0000","last_undegraded":"2024-04-25T15:54:54.352531+0000","last_fullsized":"2024-04-25T15:54:54.352531+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:44.559711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":232,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.336093+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:53.336093+0000","last_peered":"2024-04-25T15:54:53.336093+0000","last_clean":"2024-04-25T15:54:53.336093+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:53.336093+0000","last_undegraded":"2024-04-25T15:54:53.336093+0000","last_fullsized":"2024-04-25T15:54:53.336093+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"113'132","reported_seq":248,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341783+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:54:54.341783+0000","last_peered":"2024-04-25T15:54:54.341783+0000","last_clean":"2024-04-25T15:54:54.341783+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:54:54.341783+0000","last_undegraded":"2024-04-25T15:54:54.341783+0000","last_fullsized":"2024-04-25T15:54:54.341783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":206,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341314+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:54:54.341314+0000","last_peered":"2024-04-25T15:54:54.341314+0000","last_clean":"2024-04-25T15:54:54.341314+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:54:54.341314+0000","last_undegraded":"2024-04-25T15:54:54.341314+0000","last_fullsized":"2024-04-25T15:54:54.341314+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"113'144","reported_seq":512,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.349677+0000","last_change":"2024-04-25T15:54:44.207954+0000","last_active":"2024-04-25T15:54:54.349677+0000","last_peered":"2024-04-25T15:54:54.349677+0000","last_clean":"2024-04-25T15:54:54.349677+0000","last_became_active":"2024-04-25T15:54:44.206595+0000","last_became_peered":"2024-04-25T15:54:44.206595+0000","last_unstale":"2024-04-25T15:54:54.349677+0000","last_undegraded":"2024-04-25T15:54:54.349677+0000","last_fullsized":"2024-04-25T15:54:54.349677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.107940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":69,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346110+0000","last_change":"2024-04-25T15:54:22.056875+0000","last_active":"2024-04-25T15:54:54.346110+0000","last_peered":"2024-04-25T15:54:54.346110+0000","last_clean":"2024-04-25T15:54:54.346110+0000","last_became_active":"2024-04-25T15:54:22.037756+0000","last_became_peered":"2024-04-25T15:54:22.037756+0000","last_unstale":"2024-04-25T15:54:54.346110+0000","last_undegraded":"2024-04-25T15:54:54.346110+0000","last_fullsized":"2024-04-25T15:54:54.346110+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:57.998015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"113'135","reported_seq":359,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346451+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:54:54.346451+0000","last_peered":"2024-04-25T15:54:54.346451+0000","last_clean":"2024-04-25T15:54:54.346451+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:54:54.346451+0000","last_undegraded":"2024-04-25T15:54:54.346451+0000","last_fullsized":"2024-04-25T15:54:54.346451+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":256,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.337826+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:53.337826+0000","last_peered":"2024-04-25T15:54:53.337826+0000","last_clean":"2024-04-25T15:54:53.337826+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:53.337826+0000","last_undegraded":"2024-04-25T15:54:53.337826+0000","last_fullsized":"2024-04-25T15:54:53.337826+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"113'123","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347926+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:54:54.347926+0000","last_peered":"2024-04-25T15:54:54.347926+0000","last_clean":"2024-04-25T15:54:54.347926+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:54:54.347926+0000","last_undegraded":"2024-04-25T15:54:54.347926+0000","last_fullsized":"2024-04-25T15:54:54.347926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"113'168","reported_seq":425,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345784+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:54:54.345784+0000","last_peered":"2024-04-25T15:54:54.345784+0000","last_clean":"2024-04-25T15:54:54.345784+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:54:54.345784+0000","last_undegraded":"2024-04-25T15:54:54.345784+0000","last_fullsized":"2024-04-25T15:54:54.345784+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":234,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.338200+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:53.338200+0000","last_peered":"2024-04-25T15:54:53.338200+0000","last_clean":"2024-04-25T15:54:53.338200+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:53.338200+0000","last_undegraded":"2024-04-25T15:54:53.338200+0000","last_fullsized":"2024-04-25T15:54:53.338200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"113'132","reported_seq":406,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344993+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:54:54.344993+0000","last_peered":"2024-04-25T15:54:54.344993+0000","last_clean":"2024-04-25T15:54:54.344993+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:54:54.344993+0000","last_undegraded":"2024-04-25T15:54:54.344993+0000","last_fullsized":"2024-04-25T15:54:54.344993+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":242,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-04-25T15:54:53.339121+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:53.339121+0000","last_peered":"2024-04-25T15:54:53.339121+0000","last_clean":"2024-04-25T15:54:53.339121+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:53.339121+0000","last_undegraded":"2024-04-25T15:54:53.339121+0000","last_fullsized":"2024-04-25T15:54:53.339121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"113'129","reported_seq":322,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352816+0000","last_change":"2024-04-25T15:54:40.169300+0000","last_active":"2024-04-25T15:54:54.352816+0000","last_peered":"2024-04-25T15:54:54.352816+0000","last_clean":"2024-04-25T15:54:54.352816+0000","last_became_active":"2024-04-25T15:54:40.168164+0000","last_became_peered":"2024-04-25T15:54:40.168164+0000","last_unstale":"2024-04-25T15:54:54.352816+0000","last_undegraded":"2024-04-25T15:54:54.352816+0000","last_fullsized":"2024-04-25T15:54:54.352816+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:50.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351250+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:54.351250+0000","last_peered":"2024-04-25T15:54:54.351250+0000","last_clean":"2024-04-25T15:54:54.351250+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:54.351250+0000","last_undegraded":"2024-04-25T15:54:54.351250+0000","last_fullsized":"2024-04-25T15:54:54.351250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"113'147","reported_seq":593,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.360954+0000","last_change":"2024-04-25T15:54:54.360954+0000","last_active":"2024-04-25T15:54:54.360954+0000","last_peered":"2024-04-25T15:54:54.360954+0000","last_clean":"2024-04-25T15:54:54.360954+0000","last_became_active":"2024-04-25T15:54:54.360118+0000","last_became_peered":"2024-04-25T15:54:54.360118+0000","last_unstale":"2024-04-25T15:54:54.360954+0000","last_undegraded":"2024-04-25T15:54:54.360954+0000","last_fullsized":"2024-04-25T15:54:54.360954+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:37.822818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":103,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.353079+0000","last_change":"2024-04-25T15:54:40.170535+0000","last_active":"2024-04-25T15:54:54.353079+0000","last_peered":"2024-04-25T15:54:54.353079+0000","last_clean":"2024-04-25T15:54:54.353079+0000","last_became_active":"2024-04-25T15:54:40.169964+0000","last_became_peered":"2024-04-25T15:54:40.169964+0000","last_unstale":"2024-04-25T15:54:54.353079+0000","last_undegraded":"2024-04-25T15:54:54.353079+0000","last_fullsized":"2024-04-25T15:54:54.353079+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:24.688851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":1627389952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9217,"ondisk_log_size":9217,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":207280,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":338,"num_write_kb":313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":389959,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":414560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574909,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59664,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312176,"statfs":{"total":96636764160,"available":96575668224,"internally_reserved":0,"allocated":2162688,"data_stored":1331809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574909,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41220,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330620,"statfs":{"total":96636764160,"available":96594554880,"internally_reserved":0,"allocated":1757184,"data_stored":939161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63256,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308584,"statfs":{"total":96636764160,"available":96571990016,"internally_reserved":0,"allocated":1581056,"data_stored":743273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574908,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59072,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312768,"statfs":{"total":96636764160,"available":96576274432,"internally_reserved":0,"allocated":1556480,"data_stored":740671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574909,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":1765376,"data_stored":939700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574909,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1568768,"data_stored":741308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":740432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574907,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2154496,"data_stored":1330142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":200800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:54:57.538 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-25T15:54:57.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-25T15:54:57.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.729+0000 7f8430096640 1 Processor -- start 2024-04-25T15:54:57.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.729+0000 7f8430096640 1 -- start start 2024-04-25T15:54:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f8430096640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8428071b20 0x7f842815d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f8430096640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f842815d710 0x7f84280c7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f8430096640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f84280c7b50 0x7f84280c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f8430096640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f842815cd40 con 0x7f842815d710 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f8430096640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f84280c6890 con 0x7f84280c7b50 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f8430096640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f84280c6b70 con 0x7f8428071b20 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f842815d710 0x7f84280c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f842815d710 0x7f84280c7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49810/0 (socket says 172.21.15.67:49810) 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 -- 172.21.15.67:0/3043935574 learned_addr learned my addr 172.21.15.67:0/3043935574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842de32640 1 --2- 172.21.15.67:0/3043935574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8428071b20 0x7f842815d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842e633640 1 --2- 172.21.15.67:0/3043935574 >> v2:172.21.15.176:3300/0 conn(0x7f84280c7b50 0x7f84280c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 -- 172.21.15.67:0/3043935574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8428071b20 msgr2=0x7f842815d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 --2- 172.21.15.67:0/3043935574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8428071b20 0x7f842815d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 -- 172.21.15.67:0/3043935574 >> v2:172.21.15.176:3300/0 conn(0x7f84280c7b50 msgr2=0x7f84280c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:57.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 --2- 172.21.15.67:0/3043935574 >> v2:172.21.15.176:3300/0 conn(0x7f84280c7b50 0x7f84280c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 -- 172.21.15.67:0/3043935574 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84280c6e50 con 0x7f842815d710 2024-04-25T15:54:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842e633640 1 --2- 172.21.15.67:0/3043935574 >> v2:172.21.15.176:3300/0 conn(0x7f84280c7b50 0x7f84280c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842d631640 1 --2- 172.21.15.67:0/3043935574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f842815d710 0x7f84280c7610 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f84280ce940 tx=0x7f841801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a98b83d369a28f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842ce30640 1 -- 172.21.15.67:0/3043935574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84180ab5f0 con 0x7f842815d710 2024-04-25T15:54:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842ce30640 1 -- 172.21.15.67:0/3043935574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84180b0070 con 0x7f842815d710 2024-04-25T15:54:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.733+0000 7f842ce30640 1 -- 172.21.15.67:0/3043935574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84180ab9b0 con 0x7f842815d710 2024-04-25T15:54:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- 172.21.15.67:0/3043935574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f842815d710 msgr2=0x7f84280c7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 --2- 172.21.15.67:0/3043935574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f842815d710 0x7f84280c7610 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f84280ce940 tx=0x7f841801c520 comp rx=0 tx=0).stop 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- 172.21.15.67:0/3043935574 shutdown_connections 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 --2- 172.21.15.67:0/3043935574 >> v2:172.21.15.176:3300/0 conn(0x7f84280c7b50 0x7f84280c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 --2- 172.21.15.67:0/3043935574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f842815d710 0x7f84280c7610 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 --2- 172.21.15.67:0/3043935574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8428071b20 0x7f842815d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- 172.21.15.67:0/3043935574 >> 172.21.15.67:0/3043935574 conn(0x7f842806fe70 msgr2=0x7f84280cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- 172.21.15.67:0/3043935574 shutdown_connections 2024-04-25T15:54:57.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- 172.21.15.67:0/3043935574 wait complete. 2024-04-25T15:54:57.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 Processor -- start 2024-04-25T15:54:57.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- start start 2024-04-25T15:54:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8428071b20 0x7f84280cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84280c7b50 0x7f8428151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f842815d710 0x7f8428148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8428159db0 con 0x7f84280c7b50 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8428159c30 con 0x7f8428071b20 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f8430096640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8428159f30 con 0x7f842815d710 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f842d631640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84280c7b50 0x7f8428151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f842d631640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84280c7b50 0x7f8428151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49818/0 (socket says 172.21.15.67:49818) 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.737+0000 7f842d631640 1 -- 172.21.15.67:0/1350975587 learned_addr learned my addr 172.21.15.67:0/1350975587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842d631640 1 -- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f842815d710 msgr2=0x7f8428148320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842d631640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f842815d710 0x7f8428148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842d631640 1 -- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8428071b20 msgr2=0x7f84280cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842d631640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8428071b20 0x7f84280cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842d631640 1 -- 172.21.15.67:0/1350975587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84280cc790 con 0x7f84280c7b50 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842d631640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84280c7b50 0x7f8428151410 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f841801c350 tx=0x7f84180192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f841e7fc640 1 -- 172.21.15.67:0/1350975587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84180be5f0 con 0x7f84280c7b50 2024-04-25T15:54:57.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f841e7fc640 1 -- 172.21.15.67:0/1350975587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84180be790 con 0x7f84280c7b50 2024-04-25T15:54:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8428155000 con 0x7f84280c7b50 2024-04-25T15:54:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f841e7fc640 1 -- 172.21.15.67:0/1350975587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84180beb80 con 0x7f84280c7b50 2024-04-25T15:54:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84281560a0 con 0x7f84280c7b50 2024-04-25T15:54:57.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f842806a480 con 0x7f84280c7b50 2024-04-25T15:54:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f841e7fc640 1 -- 172.21.15.67:0/1350975587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f84180b43f0 con 0x7f84280c7b50 2024-04-25T15:54:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f841e7fc640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f840c06f430 0x7f840c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f841e7fc640 1 -- 172.21.15.67:0/1350975587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f8418129920 con 0x7f84280c7b50 2024-04-25T15:54:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842de32640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f840c06f430 0x7f840c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.741+0000 7f842de32640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f840c06f430 0x7f840c071940 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f84280c5960 tx=0x7f8424008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:57.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.745+0000 7f841e7fc640 1 -- 172.21.15.67:0/1350975587 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8418016110 con 0x7f84280c7b50 2024-04-25T15:54:57.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.901+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) -- 0x7f842806a480 con 0x7f84280c7b50 2024-04-25T15:54:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.905+0000 7f841e7fc640 1 -- 172.21.15.67:0/1350975587 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v11) ==== 118+0+0 (secure 0 0 0) 0x7f84180f6880 con 0x7f84280c7b50 2024-04-25T15:54:57.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f840c06f430 msgr2=0x7f840c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f840c06f430 0x7f840c071940 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f84280c5960 tx=0x7f8424008040 comp rx=0 tx=0).stop 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84280c7b50 msgr2=0x7f8428151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84280c7b50 0x7f8428151410 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f841801c350 tx=0x7f84180192c0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 shutdown_connections 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f840c06f430 0x7f840c071940 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f842815d710 0x7f8428148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84280c7b50 0x7f8428151410 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 --2- 172.21.15.67:0/1350975587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8428071b20 0x7f84280cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 >> 172.21.15.67:0/1350975587 conn(0x7f842806fe70 msgr2=0x7f84280c9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 shutdown_connections 2024-04-25T15:54:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:57.909+0000 7f8430096640 1 -- 172.21.15.67:0/1350975587 wait complete. 2024-04-25T15:54:57.929 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-25T15:54:57.939 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-25T15:54:57.939 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:54:57.939 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:54:57.945 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:54:57.945 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:54:57.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:54:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50e28d640 1 Processor -- start 2024-04-25T15:54:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50e28d640 1 -- start start 2024-04-25T15:54:58.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50e28d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc50806c640 0x7fc50806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50e28d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc5080c8680 0x7fc5080c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50e28d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc508158d20 con 0x7fc508152b00 2024-04-25T15:54:58.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50e28d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fc5080722c0 con 0x7fc5080c8680 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50e28d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc508158fe0 con 0x7fc50806c640 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc507fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc508152b00 0x7fc5080c4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43354/0 (socket says 172.21.15.67:43354) 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc507fff640 1 -- 172.21.15.67:0/1413982533 learned_addr learned my addr 172.21.15.67:0/1413982533 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50c82a640 1 --2- 172.21.15.67:0/1413982533 >> v2:172.21.15.67:3300/0 conn(0x7fc5080c8680 0x7fc5080c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc5077fe640 1 --2- 172.21.15.67:0/1413982533 >> v2:172.21.15.176:3300/0 conn(0x7fc50806c640 0x7fc50806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc506ffd640 1 -- 172.21.15.67:0/1413982533 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996281079 0 0) 0x7fc508158d20 con 0x7fc508152b00 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc506ffd640 1 -- 172.21.15.67:0/1413982533 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4f4003530 con 0x7fc508152b00 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50c82a640 1 -- 172.21.15.67:0/1413982533 >> v2:172.21.15.176:3300/0 conn(0x7fc50806c640 msgr2=0x7fc50806ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50c82a640 1 --2- 172.21.15.67:0/1413982533 >> v2:172.21.15.176:3300/0 conn(0x7fc50806c640 0x7fc50806ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50c82a640 1 -- 172.21.15.67:0/1413982533 >> v1:172.21.15.67:6789/0 conn(0x7fc508152b00 legacy=0x7fc5080c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc50c82a640 1 -- 172.21.15.67:0/1413982533 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5081592c0 con 0x7fc5080c8680 2024-04-25T15:54:58.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.185+0000 7fc5077fe640 1 --2- 172.21.15.67:0/1413982533 >> v2:172.21.15.176:3300/0 conn(0x7fc50806c640 0x7fc50806ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50c82a640 1 --2- 172.21.15.67:0/1413982533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5080c8680 0x7fc5080c8a90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc5080c5be0 tx=0x7fc50001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c9832e1260d166 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc506ffd640 1 -- 172.21.15.67:0/1413982533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000a6670 con 0x7fc5080c8680 2024-04-25T15:54:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc506ffd640 1 -- 172.21.15.67:0/1413982533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5000a6810 con 0x7fc5080c8680 2024-04-25T15:54:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc506ffd640 1 -- 172.21.15.67:0/1413982533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000a6be0 con 0x7fc5080c8680 2024-04-25T15:54:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 -- 172.21.15.67:0/1413982533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5080c8680 msgr2=0x7fc5080c8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 --2- 172.21.15.67:0/1413982533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5080c8680 0x7fc5080c8a90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc5080c5be0 tx=0x7fc50001c930 comp rx=0 tx=0).stop 2024-04-25T15:54:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 -- 172.21.15.67:0/1413982533 shutdown_connections 2024-04-25T15:54:58.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 --2- 172.21.15.67:0/1413982533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5080c8680 0x7fc5080c8a90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 --2- 172.21.15.67:0/1413982533 >> v2:172.21.15.176:3300/0 conn(0x7fc50806c640 0x7fc50806ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 -- 172.21.15.67:0/1413982533 >> 172.21.15.67:0/1413982533 conn(0x7fc50806fe70 msgr2=0x7fc5080cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:58.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 -- 172.21.15.67:0/1413982533 shutdown_connections 2024-04-25T15:54:58.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 -- 172.21.15.67:0/1413982533 wait complete. 2024-04-25T15:54:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 Processor -- start 2024-04-25T15:54:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.189+0000 7fc50e28d640 1 -- start start 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50806c640 0x7fc508165f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5080c8680 0x7fc508143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc508152b00 0x7fc508074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc508152f10 con 0x7fc50806c640 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc5080ca100 con 0x7fc5080c8680 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc5081677e0 con 0x7fc508152b00 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50806c640 0x7fc508165f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50806c640 0x7fc508165f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49838/0 (socket says 172.21.15.67:49838) 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 -- 172.21.15.67:0/4063000987 learned_addr learned my addr 172.21.15.67:0/4063000987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:58.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 -- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc508152b00 msgr2=0x7fc508074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50c82a640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc508152b00 0x7fc508074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc5077fe640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5080c8680 0x7fc508143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc508152b00 0x7fc508074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 -- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5080c8680 msgr2=0x7fc508143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5080c8680 0x7fc508143bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 -- 172.21.15.67:0/4063000987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc508166490 con 0x7fc50806c640 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc507fff640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50806c640 0x7fc508165f50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c3c0 tx=0x7fc4f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc504ff9640 1 -- 172.21.15.67:0/4063000987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8017070 con 0x7fc50806c640 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc504ff9640 1 -- 172.21.15.67:0/4063000987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f8005cf0 con 0x7fc50806c640 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5081f6530 con 0x7fc50806c640 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc504ff9640 1 -- 172.21.15.67:0/4063000987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f80072f0 con 0x7fc50806c640 2024-04-25T15:54:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc508149b50 con 0x7fc50806c640 2024-04-25T15:54:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc508076730 con 0x7fc50806c640 2024-04-25T15:54:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.193+0000 7fc504ff9640 1 -- 172.21.15.67:0/4063000987 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc4f801a070 con 0x7fc50806c640 2024-04-25T15:54:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.197+0000 7fc504ff9640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4e806f400 0x7fc4e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.197+0000 7fc504ff9640 1 -- 172.21.15.67:0/4063000987 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fc4f808e7f0 con 0x7fc50806c640 2024-04-25T15:54:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.197+0000 7fc5077fe640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4e806f400 0x7fc4e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.197+0000 7fc5077fe640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4e806f400 0x7fc4e8071910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0006fd0 tx=0x7fc4f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.201+0000 7fc504ff9640 1 -- 172.21.15.67:0/4063000987 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc4f805b630 con 0x7fc50806c640 2024-04-25T15:54:58.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.361+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc50814a2f0 con 0x7fc50806c640 2024-04-25T15:54:58.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc504ff9640 1 -- 172.21.15.67:0/4063000987 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+17051 (secure 0 0 0) 0x7fc4f805af30 con 0x7fc50806c640 2024-04-25T15:54:58.364 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:54:58.364 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":129,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:54:54.321839+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:53:14.990625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.a","osds":[2,6]},{"pgid":"3.18","osds":[1,4]},{"pgid":"3.27","osds":[7,6]}],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":5}]},{"pgid":"3.22","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3e","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4e806f400 msgr2=0x7fc4e8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4e806f400 0x7fc4e8071910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0006fd0 tx=0x7fc4f0008040 comp rx=0 tx=0).stop 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50806c640 msgr2=0x7fc508165f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50806c640 0x7fc508165f50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c3c0 tx=0x7fc4f800c880 comp rx=0 tx=0).stop 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 shutdown_connections 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc4e806f400 0x7fc4e8071910 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc508152b00 0x7fc508074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5080c8680 0x7fc508143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 --2- 172.21.15.67:0/4063000987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc50806c640 0x7fc508165f50 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 >> 172.21.15.67:0/4063000987 conn(0x7fc50806fe70 msgr2=0x7fc50814e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 shutdown_connections 2024-04-25T15:54:58.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.365+0000 7fc50e28d640 1 -- 172.21.15.67:0/4063000987 wait complete. 2024-04-25T15:54:58.386 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:54:58.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:54:58.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae8c3e640 1 Processor -- start 2024-04-25T15:54:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae8c3e640 1 -- start start 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae8c3e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2ae40c9440 0x7f2ae40c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae8c3e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ae40c5620 0x7f2ae40c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae8c3e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae406d260 con 0x7f2ae415ce00 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae8c3e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae406c6e0 con 0x7f2ae40c5620 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae8c3e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae406c9c0 con 0x7f2ae40c9440 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ad9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ae40c5620 0x7f2ae40c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ad9d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ae40c5620 0x7f2ae40c5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49854/0 (socket says 172.21.15.67:49854) 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ad9d9b640 1 -- 172.21.15.67:0/256982777 learned_addr learned my addr 172.21.15.67:0/256982777 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ad9d9b640 1 -- 172.21.15.67:0/256982777 >> v2:172.21.15.176:3300/0 conn(0x7f2ae40c9440 msgr2=0x7f2ae40c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae259c640 1 --2- 172.21.15.67:0/256982777 >> v2:172.21.15.176:3300/0 conn(0x7f2ae40c9440 0x7f2ae40c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ad9d9b640 1 --2- 172.21.15.67:0/256982777 >> v2:172.21.15.176:3300/0 conn(0x7f2ae40c9440 0x7f2ae40c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ad9d9b640 1 -- 172.21.15.67:0/256982777 >> v1:172.21.15.67:6789/0 conn(0x7f2ae415ce00 legacy=0x7f2ae415fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ad9d9b640 1 -- 172.21.15.67:0/256982777 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae406cca0 con 0x7f2ae40c5620 2024-04-25T15:54:58.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.537+0000 7f2ae1d9b640 1 -- 172.21.15.67:0/256982777 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677519675 0 0) 0x7f2ae406d260 con 0x7f2ae415ce00 2024-04-25T15:54:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ad9d9b640 1 --2- 172.21.15.67:0/256982777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae40c5620 0x7f2ae40c5a30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40715d0 tx=0x7f2acc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ed5cb2bb0d6c4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae1d9b640 1 -- 172.21.15.67:0/256982777 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0b0070 con 0x7f2ae40c5620 2024-04-25T15:54:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae1d9b640 1 -- 172.21.15.67:0/256982777 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc0ab600 con 0x7f2ae40c5620 2024-04-25T15:54:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae1d9b640 1 -- 172.21.15.67:0/256982777 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0ab9d0 con 0x7f2ae40c5620 2024-04-25T15:54:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/256982777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae40c5620 msgr2=0x7f2ae40c5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/256982777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae40c5620 0x7f2ae40c5a30 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40715d0 tx=0x7f2acc01c520 comp rx=0 tx=0).stop 2024-04-25T15:54:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/256982777 shutdown_connections 2024-04-25T15:54:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/256982777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae40c5620 0x7f2ae40c5a30 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/256982777 >> v2:172.21.15.176:3300/0 conn(0x7f2ae40c9440 0x7f2ae40c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/256982777 >> 172.21.15.67:0/256982777 conn(0x7f2ae406fe70 msgr2=0x7f2ae40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:58.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/256982777 shutdown_connections 2024-04-25T15:54:58.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.541+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/256982777 wait complete. 2024-04-25T15:54:58.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 Processor -- start 2024-04-25T15:54:58.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 -- start start 2024-04-25T15:54:58.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae40c5620 0x7f2ae4149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae40c9440 0x7f2ae414c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae415ce00 0x7f2ae414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae40c9be0 con 0x7f2ae415ce00 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae40c9a60 con 0x7f2ae40c5620 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae416a080 con 0x7f2ae40c9440 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae415ce00 0x7f2ae414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae415ce00 0x7f2ae414cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49858/0 (socket says 172.21.15.67:49858) 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 -- 172.21.15.67:0/457868156 learned_addr learned my addr 172.21.15.67:0/457868156 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae259c640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae40c5620 0x7f2ae4149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ad9d9b640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae40c9440 0x7f2ae414c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 -- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae40c9440 msgr2=0x7f2ae414c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae40c9440 0x7f2ae414c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 -- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae40c5620 msgr2=0x7f2ae4149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae40c5620 0x7f2ae4149d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 -- 172.21.15.67:0/457868156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae4155200 con 0x7f2ae415ce00 2024-04-25T15:54:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ad9d9b640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae40c9440 0x7f2ae414c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae2d9d640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae415ce00 0x7f2ae414cf00 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00c7b0 tx=0x7f2adc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2adb7fe640 1 -- 172.21.15.67:0/457868156 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc00ce60 con 0x7f2ae415ce00 2024-04-25T15:54:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2adb7fe640 1 -- 172.21.15.67:0/457868156 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2adc003db0 con 0x7f2ae415ce00 2024-04-25T15:54:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae4154f60 con 0x7f2ae415ce00 2024-04-25T15:54:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2adb7fe640 1 -- 172.21.15.67:0/457868156 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2adc0052b0 con 0x7f2ae415ce00 2024-04-25T15:54:58.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.545+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae41f6680 con 0x7f2ae415ce00 2024-04-25T15:54:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.549+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae406a480 con 0x7f2ae415ce00 2024-04-25T15:54:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.549+0000 7f2adb7fe640 1 -- 172.21.15.67:0/457868156 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2adc005450 con 0x7f2ae415ce00 2024-04-25T15:54:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.549+0000 7f2adb7fe640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac406f430 0x7f2ac4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.549+0000 7f2adb7fe640 1 -- 172.21.15.67:0/457868156 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f2adc08f1c0 con 0x7f2ae415ce00 2024-04-25T15:54:58.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.549+0000 7f2ae259c640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac406f430 0x7f2ac4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.549+0000 7f2ae259c640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac406f430 0x7f2ac4071940 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400a8d0 tx=0x7f2ad4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.553+0000 7f2adb7fe640 1 -- 172.21.15.67:0/457868156 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2ae4155200 con 0x7f2ae415ce00 2024-04-25T15:54:58.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.713+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2ae40c9e50 con 0x7f2ae415ce00 2024-04-25T15:54:58.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.713+0000 7f2adb7fe640 1 -- 172.21.15.67:0/457868156 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7f2adc05c000 con 0x7f2ae415ce00 2024-04-25T15:54:58.715 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac406f430 msgr2=0x7f2ac4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac406f430 0x7f2ac4071940 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400a8d0 tx=0x7f2ad4008040 comp rx=0 tx=0).stop 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae415ce00 msgr2=0x7f2ae414cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae415ce00 0x7f2ae414cf00 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00c7b0 tx=0x7f2adc00cc70 comp rx=0 tx=0).stop 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 shutdown_connections 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2ac406f430 0x7f2ac4071940 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ae415ce00 0x7f2ae414cf00 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2ae40c9440 0x7f2ae414c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 --2- 172.21.15.67:0/457868156 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ae40c5620 0x7f2ae4149d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 >> 172.21.15.67:0/457868156 conn(0x7f2ae406fe70 msgr2=0x7f2ae40c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 shutdown_connections 2024-04-25T15:54:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.717+0000 7f2ae8c3e640 1 -- 172.21.15.67:0/457868156 wait complete. 2024-04-25T15:54:58.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:54:58.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efeceaae640 1 Processor -- start 2024-04-25T15:54:58.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efeceaae640 1 -- start start 2024-04-25T15:54:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efeceaae640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7efec80c8d90 0x7efec806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efeceaae640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec8158d20 con 0x7efec806c640 2024-04-25T15:54:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efeceaae640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec8158fe0 con 0x7efec8152b00 2024-04-25T15:54:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efeceaae640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7efec80722c0 con 0x7efec80c8d90 2024-04-25T15:54:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec7fff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efec806c640 0x7efec80c8680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43384/0 (socket says 172.21.15.67:43384) 2024-04-25T15:54:58.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec7fff640 1 -- 172.21.15.67:0/1805553954 learned_addr learned my addr 172.21.15.67:0/1805553954 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efecd04b640 1 --2- 172.21.15.67:0/1805553954 >> v2:172.21.15.176:3300/0 conn(0x7efec80c8d90 0x7efec806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025861533 0 0) 0x7efec8158d20 con 0x7efec806c640 2024-04-25T15:54:58.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeb0003530 con 0x7efec806c640 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1497101333 0 0) 0x7efeb0003530 con 0x7efec806c640 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 >> v2:172.21.15.176:3300/0 conn(0x7efec80c8d90 msgr2=0x7efec806ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 --2- 172.21.15.67:0/1805553954 >> v2:172.21.15.176:3300/0 conn(0x7efec80c8d90 0x7efec806ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 >> v1:172.21.15.176:6789/0 conn(0x7efec8152b00 legacy=0x7efec80c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec81592c0 con 0x7efec806c640 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7efebc001680 con 0x7efec806c640 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 >> v1:172.21.15.67:6789/0 conn(0x7efec806c640 legacy=0x7efec80c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb00039c0 0x7efeb0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb0004810 0x7efeb0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb00051d0 0x7efeb000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efec80722c0 con 0x7efeb00039c0 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efeb000d280 con 0x7efeb0004810 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.881+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efeb000c1c0 con 0x7efeb00051d0 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecc84a640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb00039c0 0x7efeb0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecd04b640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb00051d0 0x7efeb000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efec7fff640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb0004810 0x7efeb0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecc84a640 1 -- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb00051d0 msgr2=0x7efeb000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecc84a640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efeb00051d0 0x7efeb000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecc84a640 1 -- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb0004810 msgr2=0x7efeb0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecc84a640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb0004810 0x7efeb0004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecd04b640 1 -- 172.21.15.67:0/1805553954 reap_dead start 2024-04-25T15:54:58.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.885+0000 7efecc84a640 1 -- 172.21.15.67:0/1805553954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb0003530 con 0x7efeb00039c0 2024-04-25T15:54:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efecc84a640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb00039c0 0x7efeb0003e30 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7efeb8001780 tx=0x7efeb801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607de1274e9bb57e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb80013a0 con 0x7efeb00039c0 2024-04-25T15:54:58.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb80050d0 con 0x7efeb00039c0 2024-04-25T15:54:58.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efec77fe640 1 -- 172.21.15.67:0/1805553954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb8003600 con 0x7efeb00039c0 2024-04-25T15:54:58.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 -- 172.21.15.67:0/1805553954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb00039c0 msgr2=0x7efeb0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb00039c0 0x7efeb0003e30 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7efeb8001780 tx=0x7efeb801c360 comp rx=0 tx=0).stop 2024-04-25T15:54:58.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 -- 172.21.15.67:0/1805553954 shutdown_connections 2024-04-25T15:54:58.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 --2- 172.21.15.67:0/1805553954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efeb00039c0 0x7efeb0003e30 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 -- 172.21.15.67:0/1805553954 >> 172.21.15.67:0/1805553954 conn(0x7efec806fe70 msgr2=0x7efec80cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:58.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 -- 172.21.15.67:0/1805553954 shutdown_connections 2024-04-25T15:54:58.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 -- 172.21.15.67:0/1805553954 wait complete. 2024-04-25T15:54:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 Processor -- start 2024-04-25T15:54:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.889+0000 7efeceaae640 1 -- start start 2024-04-25T15:54:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb00039c0 0x7efec81e8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec815b770 0x7efec815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec8073cb0 0x7efec80740c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efec8152f10 con 0x7efec815b770 2024-04-25T15:54:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efec80ca100 con 0x7efeb00039c0 2024-04-25T15:54:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efec80c8a90 con 0x7efec8073cb0 2024-04-25T15:54:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec815b770 0x7efec815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec815b770 0x7efec815bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49864/0 (socket says 172.21.15.67:49864) 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 -- 172.21.15.67:0/1806093150 learned_addr learned my addr 172.21.15.67:0/1806093150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 -- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec8073cb0 msgr2=0x7efec80740c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec8073cb0 0x7efec80740c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 -- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb00039c0 msgr2=0x7efec81e8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb00039c0 0x7efec81e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 -- 172.21.15.67:0/1806093150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec81494f0 con 0x7efec815b770 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec7fff640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec815b770 0x7efec815bb80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7efebc004f00 tx=0x7efebc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec57fa640 1 -- 172.21.15.67:0/1806093150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc00f6d0 con 0x7efec815b770 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec57fa640 1 -- 172.21.15.67:0/1806093150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efebc018070 con 0x7efec815b770 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec81f9230 con 0x7efec815b770 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec57fa640 1 -- 172.21.15.67:0/1806093150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc00fad0 con 0x7efec815b770 2024-04-25T15:54:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec814e6c0 con 0x7efec815b770 2024-04-25T15:54:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec8076730 con 0x7efec815b770 2024-04-25T15:54:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec57fa640 1 -- 172.21.15.67:0/1806093150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efebc00fc70 con 0x7efec815b770 2024-04-25T15:54:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec57fa640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeb406f400 0x7efeb4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efec57fa640 1 -- 172.21.15.67:0/1806093150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7efebc08f340 con 0x7efec815b770 2024-04-25T15:54:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.893+0000 7efecc84a640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeb406f400 0x7efeb4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.897+0000 7efecc84a640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeb406f400 0x7efeb4071910 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efeb8001750 tx=0x7efeb8001130 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:58.901+0000 7efec57fa640 1 -- 172.21.15.67:0/1806093150 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efebc05c0f0 con 0x7efec815b770 2024-04-25T15:54:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.053+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efec8149210 con 0x7efec815b770 2024-04-25T15:54:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efec57fa640 1 -- 172.21.15.67:0/1806093150 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) ==== 104+0+11 (secure 0 0 0) 0x7efebc064170 con 0x7efec815b770 2024-04-25T15:54:59.057 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:54:59.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeb406f400 msgr2=0x7efeb4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeb406f400 0x7efeb4071910 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7efeb8001750 tx=0x7efeb8001130 comp rx=0 tx=0).stop 2024-04-25T15:54:59.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec815b770 msgr2=0x7efec815bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec815b770 0x7efec815bb80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7efebc004f00 tx=0x7efebc007430 comp rx=0 tx=0).stop 2024-04-25T15:54:59.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 shutdown_connections 2024-04-25T15:54:59.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7efeb406f400 0x7efeb4071910 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efec8073cb0 0x7efec80740c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efec815b770 0x7efec815bb80 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 --2- 172.21.15.67:0/1806093150 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efeb00039c0 0x7efec81e8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 >> 172.21.15.67:0/1806093150 conn(0x7efec806fe70 msgr2=0x7efec80c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 shutdown_connections 2024-04-25T15:54:59.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.057+0000 7efeceaae640 1 -- 172.21.15.67:0/1806093150 wait complete. 2024-04-25T15:54:59.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:54:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f0371e71640 1 Processor -- start 2024-04-25T15:54:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f0371e71640 1 -- start start 2024-04-25T15:54:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f0371e71640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c073a90 0x7f036c068590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f0371e71640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f036c0c8fa0 con 0x7f036c15c750 2024-04-25T15:54:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f0371e71640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f036c0c9600 con 0x7f036c06c640 2024-04-25T15:54:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f0371e71640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f036c15cb60 con 0x7f036c073a90 2024-04-25T15:54:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f036c15c750 0x7f036c1568a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43394/0 (socket says 172.21.15.67:43394) 2024-04-25T15:54:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036b7fe640 1 -- 172.21.15.67:0/1651365385 learned_addr learned my addr 172.21.15.67:0/1651365385 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:54:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036affd640 1 --2- 172.21.15.67:0/1651365385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c073a90 0x7f036c068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036a7fc640 1 -- 172.21.15.67:0/1651365385 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3840102006 0 0) 0x7f036c0c8fa0 con 0x7f036c15c750 2024-04-25T15:54:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036a7fc640 1 -- 172.21.15.67:0/1651365385 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0354003530 con 0x7f036c15c750 2024-04-25T15:54:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036affd640 1 -- 172.21.15.67:0/1651365385 >> v1:172.21.15.176:6789/0 conn(0x7f036c06c640 legacy=0x7f036c1670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036affd640 1 -- 172.21.15.67:0/1651365385 >> v1:172.21.15.67:6789/0 conn(0x7f036c15c750 legacy=0x7f036c1568a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036a7fc640 1 -- 172.21.15.67:0/1651365385 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4236880278 0 0) 0x7f0354003530 con 0x7f036c15c750 2024-04-25T15:54:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.233+0000 7f036affd640 1 -- 172.21.15.67:0/1651365385 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036c0c98e0 con 0x7f036c073a90 2024-04-25T15:54:59.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f036affd640 1 --2- 172.21.15.67:0/1651365385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c073a90 0x7f036c068590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f03600097f0 tx=0x7f036001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dcd03ffc64feab1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:59.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f036a7fc640 1 -- 172.21.15.67:0/1651365385 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03600b0070 con 0x7f036c073a90 2024-04-25T15:54:59.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f036a7fc640 1 -- 172.21.15.67:0/1651365385 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03600ab400 con 0x7f036c073a90 2024-04-25T15:54:59.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f036a7fc640 1 -- 172.21.15.67:0/1651365385 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c001680 con 0x7f036c073a90 2024-04-25T15:54:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f0371e71640 1 -- 172.21.15.67:0/1651365385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c073a90 msgr2=0x7f036c068590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f0371e71640 1 --2- 172.21.15.67:0/1651365385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c073a90 0x7f036c068590 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f03600097f0 tx=0x7f036001c400 comp rx=0 tx=0).stop 2024-04-25T15:54:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f0371e71640 1 -- 172.21.15.67:0/1651365385 shutdown_connections 2024-04-25T15:54:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f0371e71640 1 --2- 172.21.15.67:0/1651365385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c073a90 0x7f036c068590 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f0371e71640 1 -- 172.21.15.67:0/1651365385 >> 172.21.15.67:0/1651365385 conn(0x7f036c069220 msgr2=0x7f036c0cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f0371e71640 1 -- 172.21.15.67:0/1651365385 shutdown_connections 2024-04-25T15:54:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.237+0000 7f0371e71640 1 -- 172.21.15.67:0/1651365385 wait complete. 2024-04-25T15:54:59.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 Processor -- start 2024-04-25T15:54:59.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 -- start start 2024-04-25T15:54:59.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c06c640 0x7f036c1f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036c073a90 0x7f036c0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f036c15c750 0x7f036c143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f036c068ad0 con 0x7f036c073a90 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f036c06ce60 con 0x7f036c15c750 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f036c0c92b0 con 0x7f036c06c640 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036c073a90 0x7f036c0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036c073a90 0x7f036c0cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49866/0 (socket says 172.21.15.67:49866) 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 -- 172.21.15.67:0/512382886 learned_addr learned my addr 172.21.15.67:0/512382886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036bfff640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f036c15c750 0x7f036c143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036b7fe640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c06c640 0x7f036c1f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 -- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c06c640 msgr2=0x7f036c1f6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c06c640 0x7f036c1f6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 -- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f036c15c750 msgr2=0x7f036c143a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f036c15c750 0x7f036c143a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 -- 172.21.15.67:0/512382886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036c150ea0 con 0x7f036c073a90 2024-04-25T15:54:59.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f036affd640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036c073a90 0x7f036c0cd190 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f036001a030 tx=0x7f03600191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:59.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0353fff640 1 -- 172.21.15.67:0/512382886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03600b0040 con 0x7f036c073a90 2024-04-25T15:54:59.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0353fff640 1 -- 172.21.15.67:0/512382886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03600036a0 con 0x7f036c073a90 2024-04-25T15:54:59.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0353fff640 1 -- 172.21.15.67:0/512382886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360003a50 con 0x7f036c073a90 2024-04-25T15:54:59.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036c151180 con 0x7f036c073a90 2024-04-25T15:54:59.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.241+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f036c1f6a00 con 0x7f036c073a90 2024-04-25T15:54:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.245+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f036c06a480 con 0x7f036c073a90 2024-04-25T15:54:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.245+0000 7f0353fff640 1 -- 172.21.15.67:0/512382886 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f03600bd020 con 0x7f036c073a90 2024-04-25T15:54:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.245+0000 7f0353fff640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f035806f400 0x7f0358071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.245+0000 7f0353fff640 1 -- 172.21.15.67:0/512382886 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f0360128e10 con 0x7f036c073a90 2024-04-25T15:54:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.245+0000 7f036b7fe640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f035806f400 0x7f0358071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.245+0000 7f036b7fe640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f035806f400 0x7f0358071910 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f035c0011f0 tx=0x7f035c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:59.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.249+0000 7f0353fff640 1 -- 172.21.15.67:0/512382886 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0360015ff0 con 0x7f036c073a90 2024-04-25T15:54:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.405+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f036c15a170 con 0x7f036c073a90 2024-04-25T15:54:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.409+0000 7f0353fff640 1 -- 172.21.15.67:0/512382886 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) ==== 100+0+11 (secure 0 0 0) 0x7f03600f5bc0 con 0x7f036c073a90 2024-04-25T15:54:59.409 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:54:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f035806f400 msgr2=0x7f0358071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f035806f400 0x7f0358071910 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f035c0011f0 tx=0x7f035c009040 comp rx=0 tx=0).stop 2024-04-25T15:54:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036c073a90 msgr2=0x7f036c0cd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036c073a90 0x7f036c0cd190 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f036001a030 tx=0x7f03600191a0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 shutdown_connections 2024-04-25T15:54:59.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f035806f400 0x7f0358071910 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f036c15c750 0x7f036c143a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f036c073a90 0x7f036c0cd190 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 --2- 172.21.15.67:0/512382886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f036c06c640 0x7f036c1f6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 >> 172.21.15.67:0/512382886 conn(0x7f036c069220 msgr2=0x7f036c154d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 shutdown_connections 2024-04-25T15:54:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.413+0000 7f0371e71640 1 -- 172.21.15.67:0/512382886 wait complete. 2024-04-25T15:54:59.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:54:59.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0ea002640 1 Processor -- start 2024-04-25T15:54:59.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0ea002640 1 -- start start 2024-04-25T15:54:59.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0ea002640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0e4074790 0x7fe0e415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0ea002640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe0e415e020 0x7fe0e40723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0ea002640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0e4072dc0 con 0x7fe0e40728e0 2024-04-25T15:54:59.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0ea002640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e415b430 con 0x7fe0e4074790 2024-04-25T15:54:59.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0ea002640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e415a670 con 0x7fe0e415e020 2024-04-25T15:54:59.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0e37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0e4074790 0x7fe0e415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0e37fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0e4074790 0x7fe0e415dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49872/0 (socket says 172.21.15.67:49872) 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.585+0000 7fe0e37fe640 1 -- 172.21.15.67:0/2277513443 learned_addr learned my addr 172.21.15.67:0/2277513443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e2ffd640 1 --2- 172.21.15.67:0/2277513443 >> v2:172.21.15.176:3300/0 conn(0x7fe0e415e020 0x7fe0e40723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e37fe640 1 -- 172.21.15.67:0/2277513443 >> v2:172.21.15.176:3300/0 conn(0x7fe0e415e020 msgr2=0x7fe0e40723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e37fe640 1 --2- 172.21.15.67:0/2277513443 >> v2:172.21.15.176:3300/0 conn(0x7fe0e415e020 0x7fe0e40723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e37fe640 1 -- 172.21.15.67:0/2277513443 >> v1:172.21.15.67:6789/0 conn(0x7fe0e40728e0 legacy=0x7fe0e40c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e37fe640 1 -- 172.21.15.67:0/2277513443 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e415a950 con 0x7fe0e4074790 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e27fc640 1 -- 172.21.15.67:0/2277513443 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252304408 0 0) 0x7fe0e4072dc0 con 0x7fe0e40728e0 2024-04-25T15:54:59.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e2ffd640 1 --2- 172.21.15.67:0/2277513443 >> v2:172.21.15.176:3300/0 conn(0x7fe0e415e020 0x7fe0e40723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:59.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e37fe640 1 --2- 172.21.15.67:0/2277513443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e4074790 0x7fe0e415dae0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4070a50 tx=0x7fe0d401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde4bf34b1fad86b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:59.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e27fc640 1 -- 172.21.15.67:0/2277513443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40b5070 con 0x7fe0e4074790 2024-04-25T15:54:59.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e27fc640 1 -- 172.21.15.67:0/2277513443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d40a8a10 con 0x7fe0e4074790 2024-04-25T15:54:59.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0e27fc640 1 -- 172.21.15.67:0/2277513443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40a8de0 con 0x7fe0e4074790 2024-04-25T15:54:59.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 -- 172.21.15.67:0/2277513443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e4074790 msgr2=0x7fe0e415dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 --2- 172.21.15.67:0/2277513443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e4074790 0x7fe0e415dae0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4070a50 tx=0x7fe0d401ed10 comp rx=0 tx=0).stop 2024-04-25T15:54:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 -- 172.21.15.67:0/2277513443 shutdown_connections 2024-04-25T15:54:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 --2- 172.21.15.67:0/2277513443 >> v2:172.21.15.176:3300/0 conn(0x7fe0e415e020 0x7fe0e40723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 --2- 172.21.15.67:0/2277513443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e4074790 0x7fe0e415dae0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 -- 172.21.15.67:0/2277513443 >> 172.21.15.67:0/2277513443 conn(0x7fe0e406fe70 msgr2=0x7fe0e40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 -- 172.21.15.67:0/2277513443 shutdown_connections 2024-04-25T15:54:59.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.589+0000 7fe0ea002640 1 -- 172.21.15.67:0/2277513443 wait complete. 2024-04-25T15:54:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 Processor -- start 2024-04-25T15:54:59.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 -- start start 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0e40728e0 0x7fe0e40c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0e4074790 0x7fe0e41531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e415e020 0x7fe0e415d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e4073080 con 0x7fe0e415e020 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e415af10 con 0x7fe0e4074790 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e4074ba0 con 0x7fe0e40728e0 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e415e020 0x7fe0e415d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e415e020 0x7fe0e415d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49886/0 (socket says 172.21.15.67:49886) 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 -- 172.21.15.67:0/3366083153 learned_addr learned my addr 172.21.15.67:0/3366083153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e2ffd640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0e4074790 0x7fe0e41531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e37fe640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0e40728e0 0x7fe0e40c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 -- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0e40728e0 msgr2=0x7fe0e40c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0e40728e0 0x7fe0e40c53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 -- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0e4074790 msgr2=0x7fe0e41531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0e4074790 0x7fe0e41531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 -- 172.21.15.67:0/3366083153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e40c5910 con 0x7fe0e415e020 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0e3fff640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e415e020 0x7fe0e415d140 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00c700 tx=0x7fe0dc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0cbfff640 1 -- 172.21.15.67:0/3366083153 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc017070 con 0x7fe0e415e020 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0cbfff640 1 -- 172.21.15.67:0/3366083153 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0dc003db0 con 0x7fe0e415e020 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e415c110 con 0x7fe0e415e020 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0cbfff640 1 -- 172.21.15.67:0/3366083153 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc005310 con 0x7fe0e415e020 2024-04-25T15:54:59.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.593+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e415c6d0 con 0x7fe0e415e020 2024-04-25T15:54:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.597+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e406a480 con 0x7fe0e415e020 2024-04-25T15:54:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.597+0000 7fe0cbfff640 1 -- 172.21.15.67:0/3366083153 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe0dc01a070 con 0x7fe0e415e020 2024-04-25T15:54:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.597+0000 7fe0cbfff640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe0d006f400 0x7fe0d0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.597+0000 7fe0cbfff640 1 -- 172.21.15.67:0/3366083153 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fe0dc08e9a0 con 0x7fe0e415e020 2024-04-25T15:54:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.597+0000 7fe0e37fe640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe0d006f400 0x7fe0d0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.597+0000 7fe0e37fe640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe0d006f400 0x7fe0d0071910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40b2000 tx=0x7fe0d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:59.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.601+0000 7fe0cbfff640 1 -- 172.21.15.67:0/3366083153 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe0dc05b870 con 0x7fe0e415e020 2024-04-25T15:54:59.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:54:59.744 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:54:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.777+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe0e415be90 con 0x7fe0e415e020 2024-04-25T15:54:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.777+0000 7fe0cbfff640 1 -- 172.21.15.67:0/3366083153 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v129) ==== 80+0+0 (secure 0 0 0) 0x7fe0e415be90 con 0x7fe0e415e020 2024-04-25T15:54:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:54:59.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe0d006f400 msgr2=0x7fe0d0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe0d006f400 0x7fe0d0071910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40b2000 tx=0x7fe0d4008040 comp rx=0 tx=0).stop 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e415e020 msgr2=0x7fe0e415d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e415e020 0x7fe0e415d140 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00c700 tx=0x7fe0dc00cbc0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 shutdown_connections 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe0d006f400 0x7fe0d0071910 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0e415e020 0x7fe0e415d140 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0e4074790 0x7fe0e41531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 --2- 172.21.15.67:0/3366083153 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0e40728e0 0x7fe0e40c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 >> 172.21.15.67:0/3366083153 conn(0x7fe0e406fe70 msgr2=0x7fe0e4071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 shutdown_connections 2024-04-25T15:54:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:54:59.781+0000 7fe0ea002640 1 -- 172.21.15.67:0/3366083153 wait complete. 2024-04-25T15:54:59.804 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-25T15:54:59.804 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T15:54:59.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:55:00.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda89439640 1 Processor -- start 2024-04-25T15:55:00.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda89439640 1 -- start start 2024-04-25T15:55:00.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda89439640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fda8415dd00 0x7fda84068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda89439640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 0x7fda84167190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda89439640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda84160990 con 0x7fda841569a0 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda89439640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fda8415e110 con 0x7fda8415dd00 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda89439640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda84160c70 con 0x7fda84156db0 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda827fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fda841569a0 0x7fda8406c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43414/0 (socket says 172.21.15.67:43414) 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda827fc640 1 -- 172.21.15.67:0/859399200 learned_addr learned my addr 172.21.15.67:0/859399200 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda82ffd640 1 --2- 172.21.15.67:0/859399200 >> v2:172.21.15.176:3300/0 conn(0x7fda8415dd00 0x7fda84068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda837fe640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 0x7fda84167190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3301456972 0 0) 0x7fda84160990 con 0x7fda841569a0 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda6c003530 con 0x7fda841569a0 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 454320751 0 0) 0x7fda6c003530 con 0x7fda841569a0 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 msgr2=0x7fda84167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 0x7fda84167190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 >> v2:172.21.15.176:3300/0 conn(0x7fda8415dd00 msgr2=0x7fda84068830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 --2- 172.21.15.67:0/859399200 >> v2:172.21.15.176:3300/0 conn(0x7fda8415dd00 0x7fda84068830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda840c3bf0 con 0x7fda841569a0 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fda78001800 con 0x7fda841569a0 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 >> v1:172.21.15.67:6789/0 conn(0x7fda841569a0 legacy=0x7fda8406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 0x7fda6c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda6c003f40 0x7fda6c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c004d50 0x7fda6c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda8415e110 con 0x7fda6c003f40 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda84160c70 con 0x7fda6c004d50 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.073+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda6c00ba20 con 0x7fda84156db0 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda827fc640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda6c003f40 0x7fda6c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda82ffd640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 0x7fda6c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda837fe640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c004d50 0x7fda6c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda827fc640 1 -- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 msgr2=0x7fda6c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda827fc640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda84156db0 0x7fda6c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda827fc640 1 -- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c004d50 msgr2=0x7fda6c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda827fc640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c004d50 0x7fda6c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda827fc640 1 -- 172.21.15.67:0/859399200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c003530 con 0x7fda6c003f40 2024-04-25T15:55:00.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda827fc640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda6c003f40 0x7fda6c004810 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fda84076bb0 tx=0x7fda7801f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae123b05a5b031d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78007d30 con 0x7fda6c003f40 2024-04-25T15:55:00.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda78007290 con 0x7fda6c003f40 2024-04-25T15:55:00.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda81ffb640 1 -- 172.21.15.67:0/859399200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780057b0 con 0x7fda6c003f40 2024-04-25T15:55:00.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda89439640 1 -- 172.21.15.67:0/859399200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda6c003f40 msgr2=0x7fda6c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda89439640 1 --2- 172.21.15.67:0/859399200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda6c003f40 0x7fda6c004810 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fda84076bb0 tx=0x7fda7801f370 comp rx=0 tx=0).stop 2024-04-25T15:55:00.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda837fe640 1 -- 172.21.15.67:0/859399200 reap_dead start 2024-04-25T15:55:00.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda89439640 1 -- 172.21.15.67:0/859399200 shutdown_connections 2024-04-25T15:55:00.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda89439640 1 -- 172.21.15.67:0/859399200 >> 172.21.15.67:0/859399200 conn(0x7fda840ce1c0 msgr2=0x7fda840ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:00.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda89439640 1 -- 172.21.15.67:0/859399200 shutdown_connections 2024-04-25T15:55:00.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.077+0000 7fda89439640 1 -- 172.21.15.67:0/859399200 wait complete. 2024-04-25T15:55:00.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 Processor -- start 2024-04-25T15:55:00.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 -- start start 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c003f40 0x7fda84072550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84074190 0x7fda840cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415b860 0x7fda840b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda84074900 con 0x7fda84074190 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda840c4240 con 0x7fda6c003f40 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda84074a80 con 0x7fda8415b860 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84074190 0x7fda840cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84074190 0x7fda840cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49900/0 (socket says 172.21.15.67:49900) 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 -- 172.21.15.67:0/4077949384 learned_addr learned my addr 172.21.15.67:0/4077949384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c003f40 0x7fda84072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda837fe640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415b860 0x7fda840b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 -- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415b860 msgr2=0x7fda840b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415b860 0x7fda840b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 -- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c003f40 msgr2=0x7fda84072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c003f40 0x7fda84072550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 -- 172.21.15.67:0/4077949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda84072a90 con 0x7fda84074190 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda827fc640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84074190 0x7fda840cd250 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fda78000f00 tx=0x7fda6c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda6b7fe640 1 -- 172.21.15.67:0/4077949384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78000f80 con 0x7fda84074190 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda6b7fe640 1 -- 172.21.15.67:0/4077949384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda78007440 con 0x7fda84074190 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda841f2dd0 con 0x7fda84074190 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda6b7fe640 1 -- 172.21.15.67:0/4077949384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780057b0 con 0x7fda84074190 2024-04-25T15:55:00.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.081+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda841f3390 con 0x7fda84074190 2024-04-25T15:55:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.085+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8406a480 con 0x7fda84074190 2024-04-25T15:55:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.085+0000 7fda6b7fe640 1 -- 172.21.15.67:0/4077949384 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fda78005950 con 0x7fda84074190 2024-04-25T15:55:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.085+0000 7fda6b7fe640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f430 0x7fda70071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.085+0000 7fda6b7fe640 1 -- 172.21.15.67:0/4077949384 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fda7812c710 con 0x7fda84074190 2024-04-25T15:55:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.085+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f430 0x7fda70071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.085+0000 7fda82ffd640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f430 0x7fda70071940 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fda74009a30 tx=0x7fda74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.093+0000 7fda6b7fe640 1 -- 172.21.15.67:0/4077949384 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fda78019080 con 0x7fda84074190 2024-04-25T15:55:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.253+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda840cdfa0 con 0x7fda84074190 2024-04-25T15:55:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.253+0000 7fda6b7fe640 1 -- 172.21.15.67:0/4077949384 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+17051 (secure 0 0 0) 0x7fda780f9630 con 0x7fda84074190 2024-04-25T15:55:00.255 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:00.255 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":129,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:54:54.321839+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:53:14.990625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.a","osds":[2,6]},{"pgid":"3.18","osds":[1,4]},{"pgid":"3.27","osds":[7,6]}],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":5}]},{"pgid":"3.22","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3e","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:55:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f430 msgr2=0x7fda70071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f430 0x7fda70071940 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fda74009a30 tx=0x7fda74008040 comp rx=0 tx=0).stop 2024-04-25T15:55:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84074190 msgr2=0x7fda840cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84074190 0x7fda840cd250 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fda78000f00 tx=0x7fda6c005840 comp rx=0 tx=0).stop 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 shutdown_connections 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fda7006f430 0x7fda70071940 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda8415b860 0x7fda840b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda84074190 0x7fda840cd250 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 --2- 172.21.15.67:0/4077949384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda6c003f40 0x7fda84072550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 >> 172.21.15.67:0/4077949384 conn(0x7fda840ce1c0 msgr2=0x7fda8406c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 shutdown_connections 2024-04-25T15:55:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.257+0000 7fda89439640 1 -- 172.21.15.67:0/4077949384 wait complete. 2024-04-25T15:55:00.278 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:50:17.846623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:50:28.408547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:50:29.569527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:55:00.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:55:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.425+0000 7effd391b640 1 Processor -- start 2024-04-25T15:55:00.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd391b640 1 -- start start 2024-04-25T15:55:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd391b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 0x7effcc0cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd391b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7effcc15e210 0x7effcc165190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd391b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effcc0ca8a0 con 0x7effcc15de00 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd391b640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7effcc0cab80 con 0x7effcc15e210 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd391b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7effcc15a920 con 0x7effcc154ac0 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7effcc15de00 0x7effcc0cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43418/0 (socket says 172.21.15.67:43418) 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 -- 172.21.15.67:0/3216454171 learned_addr learned my addr 172.21.15.67:0/3216454171 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd1eb8640 1 --2- 172.21.15.67:0/3216454171 >> v2:172.21.15.176:3300/0 conn(0x7effcc15e210 0x7effcc165190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd16b7640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 0x7effcc0cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3760303804 0 0) 0x7effcc0ca8a0 con 0x7effcc15de00 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effb8003530 con 0x7effcc15de00 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2000948873 0 0) 0x7effb8003530 con 0x7effcc15de00 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 msgr2=0x7effcc0cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 0x7effcc0cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 >> v2:172.21.15.176:3300/0 conn(0x7effcc15e210 msgr2=0x7effcc165190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 --2- 172.21.15.67:0/3216454171 >> v2:172.21.15.176:3300/0 conn(0x7effcc15e210 0x7effcc165190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effcc15ac00 con 0x7effcc15de00 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7effc80016a0 con 0x7effcc15de00 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 >> v1:172.21.15.67:6789/0 conn(0x7effcc15de00 legacy=0x7effcc0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb80039c0 0x7effb8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8004810 0x7effb8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 0x7effb8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effcc0cab80 con 0x7effb8004810 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7effcc15a920 con 0x7effb80039c0 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7effb800ba80 con 0x7effcc154ac0 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8004810 0x7effb8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd1eb8640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 0x7effb8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd16b7640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb80039c0 0x7effb8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 -- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 msgr2=0x7effb8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effcc154ac0 0x7effb8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 -- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb80039c0 msgr2=0x7effb8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb80039c0 0x7effb8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd0eb6640 1 -- 172.21.15.67:0/3216454171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb8003530 con 0x7effb8004810 2024-04-25T15:55:00.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.429+0000 7effd16b7640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effb80039c0 0x7effb8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd0eb6640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8004810 0x7effb8004c20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7effcc0cd1c0 tx=0x7effc801cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cfbcf6b1c21d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8007dc0 con 0x7effb8004810 2024-04-25T15:55:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc80a9020 con 0x7effb8004810 2024-04-25T15:55:00.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effc3fff640 1 -- 172.21.15.67:0/3216454171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8005b00 con 0x7effb8004810 2024-04-25T15:55:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd391b640 1 -- 172.21.15.67:0/3216454171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8004810 msgr2=0x7effb8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd391b640 1 --2- 172.21.15.67:0/3216454171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effb8004810 0x7effb8004c20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7effcc0cd1c0 tx=0x7effc801cbd0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd1eb8640 1 -- 172.21.15.67:0/3216454171 reap_dead start 2024-04-25T15:55:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd391b640 1 -- 172.21.15.67:0/3216454171 shutdown_connections 2024-04-25T15:55:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd391b640 1 -- 172.21.15.67:0/3216454171 >> 172.21.15.67:0/3216454171 conn(0x7effcc068c70 msgr2=0x7effcc071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd391b640 1 -- 172.21.15.67:0/3216454171 shutdown_connections 2024-04-25T15:55:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.433+0000 7effd391b640 1 -- 172.21.15.67:0/3216454171 wait complete. 2024-04-25T15:55:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 Processor -- start 2024-04-25T15:55:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 -- start start 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb8004810 0x7effcc1f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effcc15bf70 0x7effcc14dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effcc15a340 0x7effcc14a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effcc15b6e0 con 0x7effcc15bf70 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7effcc0cc2a0 con 0x7effcc15a340 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7effcc0ccd10 con 0x7effb8004810 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effcc15bf70 0x7effcc14dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effcc15bf70 0x7effcc14dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49920/0 (socket says 172.21.15.67:49920) 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 -- 172.21.15.67:0/1868168332 learned_addr learned my addr 172.21.15.67:0/1868168332 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd16b7640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb8004810 0x7effcc1f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd1eb8640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effcc15a340 0x7effcc14a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 -- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb8004810 msgr2=0x7effcc1f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb8004810 0x7effcc1f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 -- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effcc15a340 msgr2=0x7effcc14a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effcc15a340 0x7effcc14a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 -- 172.21.15.67:0/1868168332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effcc14cba0 con 0x7effcc15bf70 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd0eb6640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effcc15bf70 0x7effcc14dc40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7effc80ab040 tx=0x7effb8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effc1ffb640 1 -- 172.21.15.67:0/1868168332 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8001d00 con 0x7effcc15bf70 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effc1ffb640 1 -- 172.21.15.67:0/1868168332 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc80a5ab0 con 0x7effcc15bf70 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effcc14ce80 con 0x7effcc15bf70 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.437+0000 7effc1ffb640 1 -- 172.21.15.67:0/1868168332 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc80a5e60 con 0x7effcc15bf70 2024-04-25T15:55:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.441+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effcc14d440 con 0x7effcc15bf70 2024-04-25T15:55:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.441+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effcc06a480 con 0x7effcc15bf70 2024-04-25T15:55:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.441+0000 7effc1ffb640 1 -- 172.21.15.67:0/1868168332 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7effc80047f0 con 0x7effcc15bf70 2024-04-25T15:55:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.441+0000 7effc1ffb640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7effb006f430 0x7effb0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.441+0000 7effc1ffb640 1 -- 172.21.15.67:0/1868168332 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7effc8129420 con 0x7effcc15bf70 2024-04-25T15:55:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.441+0000 7effd16b7640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7effb006f430 0x7effb0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.445+0000 7effd16b7640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7effb006f430 0x7effb0071940 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7effbc0090e0 tx=0x7effbc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.445+0000 7effc1ffb640 1 -- 172.21.15.67:0/1868168332 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7effc80bf030 con 0x7effcc15bf70 2024-04-25T15:55:00.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.609+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7effcc0cb260 con 0x7effcc15bf70 2024-04-25T15:55:00.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.609+0000 7effc1ffb640 1 -- 172.21.15.67:0/1868168332 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7effc8016910 con 0x7effcc15bf70 2024-04-25T15:55:00.610 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-25T15:55:00.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.609+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7effb006f430 msgr2=0x7effb0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.609+0000 7effd391b640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7effb006f430 0x7effb0071940 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7effbc0090e0 tx=0x7effbc008040 comp rx=0 tx=0).stop 2024-04-25T15:55:00.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effcc15bf70 msgr2=0x7effcc14dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effcc15bf70 0x7effcc14dc40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7effc80ab040 tx=0x7effb8005840 comp rx=0 tx=0).stop 2024-04-25T15:55:00.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 shutdown_connections 2024-04-25T15:55:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7effb006f430 0x7effb0071940 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effcc15a340 0x7effcc14a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effcc15bf70 0x7effcc14dc40 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 --2- 172.21.15.67:0/1868168332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effb8004810 0x7effcc1f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 >> 172.21.15.67:0/1868168332 conn(0x7effcc068c70 msgr2=0x7effcc14f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 shutdown_connections 2024-04-25T15:55:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.613+0000 7effd391b640 1 -- 172.21.15.67:0/1868168332 wait complete. 2024-04-25T15:55:00.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T15:55:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb0213ce640 1 Processor -- start 2024-04-25T15:55:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb0213ce640 1 -- start start 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb0213ce640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb01c156f10 0x7fb01c0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb0213ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c15d620 0x7fb01c1673f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb0213ce640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb01c160fa0 con 0x7fb01c15d210 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb0213ce640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb01c157320 con 0x7fb01c156f10 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb0213ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb01c0c6d10 con 0x7fb01c15d620 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb01affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb01c156f10 0x7fb01c0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb01affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb01c156f10 0x7fb01c0cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49926/0 (socket says 172.21.15.67:49926) 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb01affd640 1 -- 172.21.15.67:0/3788320000 learned_addr learned my addr 172.21.15.67:0/3788320000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb01b7fe640 1 --2- 172.21.15.67:0/3788320000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c15d620 0x7fb01c1673f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb01affd640 1 -- 172.21.15.67:0/3788320000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c15d620 msgr2=0x7fb01c1673f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb01affd640 1 --2- 172.21.15.67:0/3788320000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c15d620 0x7fb01c1673f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb01affd640 1 -- 172.21.15.67:0/3788320000 >> v1:172.21.15.67:6789/0 conn(0x7fb01c15d210 legacy=0x7fb01c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.789+0000 7fb019ffb640 1 -- 172.21.15.67:0/3788320000 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 206949407 0 0) 0x7fb01c160fa0 con 0x7fb01c15d210 2024-04-25T15:55:00.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.793+0000 7fb01affd640 1 -- 172.21.15.67:0/3788320000 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c160420 con 0x7fb01c156f10 2024-04-25T15:55:00.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.793+0000 7fb01affd640 1 --2- 172.21.15.67:0/3788320000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c156f10 0x7fb01c0cbfc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009870 tx=0x7fb00c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fba94b2260a8e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.793+0000 7fb019ffb640 1 -- 172.21.15.67:0/3788320000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0ab650 con 0x7fb01c156f10 2024-04-25T15:55:00.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.793+0000 7fb019ffb640 1 -- 172.21.15.67:0/3788320000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c0b0070 con 0x7fb01c156f10 2024-04-25T15:55:00.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.793+0000 7fb019ffb640 1 -- 172.21.15.67:0/3788320000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0aba10 con 0x7fb01c156f10 2024-04-25T15:55:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- 172.21.15.67:0/3788320000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c156f10 msgr2=0x7fb01c0cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 --2- 172.21.15.67:0/3788320000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c156f10 0x7fb01c0cbfc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009870 tx=0x7fb00c01c560 comp rx=0 tx=0).stop 2024-04-25T15:55:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- 172.21.15.67:0/3788320000 shutdown_connections 2024-04-25T15:55:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 --2- 172.21.15.67:0/3788320000 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c15d620 0x7fb01c1673f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 --2- 172.21.15.67:0/3788320000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c156f10 0x7fb01c0cbfc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- 172.21.15.67:0/3788320000 >> 172.21.15.67:0/3788320000 conn(0x7fb01c068c70 msgr2=0x7fb01c071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- 172.21.15.67:0/3788320000 shutdown_connections 2024-04-25T15:55:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- 172.21.15.67:0/3788320000 wait complete. 2024-04-25T15:55:00.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 Processor -- start 2024-04-25T15:55:00.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- start start 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c156f10 0x7fb01c15f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c15d210 0x7fb01c15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb01c15d620 0x7fb01c14b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c15da30 con 0x7fb01c15d210 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb01c15ccf0 con 0x7fb01c15d620 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb0213ce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb01c1600c0 con 0x7fb01c156f10 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb01a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c15d210 0x7fb01c15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb01a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c15d210 0x7fb01c15f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49936/0 (socket says 172.21.15.67:49936) 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.797+0000 7fb01a7fc640 1 -- 172.21.15.67:0/1812064842 learned_addr learned my addr 172.21.15.67:0/1812064842 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01affd640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c156f10 0x7fb01c15f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01b7fe640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb01c15d620 0x7fb01c14b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01a7fc640 1 -- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c156f10 msgr2=0x7fb01c15f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01a7fc640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c156f10 0x7fb01c15f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01a7fc640 1 -- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb01c15d620 msgr2=0x7fb01c14b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01a7fc640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb01c15d620 0x7fb01c14b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01a7fc640 1 -- 172.21.15.67:0/1812064842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01c15e640 con 0x7fb01c15d210 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01affd640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c156f10 0x7fb01c15f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01b7fe640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb01c15d620 0x7fb01c14b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01a7fc640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c15d210 0x7fb01c15f9f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c450 tx=0x7fb01000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0037fe640 1 -- 172.21.15.67:0/1812064842 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010017070 con 0x7fb01c15d210 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0037fe640 1 -- 172.21.15.67:0/1812064842 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb010005480 con 0x7fb01c15d210 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0037fe640 1 -- 172.21.15.67:0/1812064842 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010003840 con 0x7fb01c15d210 2024-04-25T15:55:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01c15b620 con 0x7fb01c15d210 2024-04-25T15:55:00.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb01c1507b0 con 0x7fb01c15d210 2024-04-25T15:55:00.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb01c06a480 con 0x7fb01c15d210 2024-04-25T15:55:00.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0037fe640 1 -- 172.21.15.67:0/1812064842 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb01001a070 con 0x7fb01c15d210 2024-04-25T15:55:00.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0037fe640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb00806f400 0x7fb008071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:00.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb0037fe640 1 -- 172.21.15.67:0/1812064842 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fb01008e5e0 con 0x7fb01c15d210 2024-04-25T15:55:00.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.801+0000 7fb01affd640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb00806f400 0x7fb008071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:00.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.805+0000 7fb01affd640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb00806f400 0x7fb008071910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0af000 tx=0x7fb00c0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:00.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.809+0000 7fb0037fe640 1 -- 172.21.15.67:0/1812064842 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb01005b4b0 con 0x7fb01c15d210 2024-04-25T15:55:00.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.969+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb01c15e3d0 con 0x7fb01c15d210 2024-04-25T15:55:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0037fe640 1 -- 172.21.15.67:0/1812064842 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v129) ==== 104+0+11 (secure 0 0 0) 0x7fb01005adb0 con 0x7fb01c15d210 2024-04-25T15:55:00.973 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-25T15:55:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb00806f400 msgr2=0x7fb008071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb00806f400 0x7fb008071910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0af000 tx=0x7fb00c0a7000 comp rx=0 tx=0).stop 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c15d210 msgr2=0x7fb01c15f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c15d210 0x7fb01c15f9f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c450 tx=0x7fb01000c910 comp rx=0 tx=0).stop 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 shutdown_connections 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb00806f400 0x7fb008071910 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb01c15d620 0x7fb01c14b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb01c15d210 0x7fb01c15f9f0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 --2- 172.21.15.67:0/1812064842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb01c156f10 0x7fb01c15f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 >> 172.21.15.67:0/1812064842 conn(0x7fb01c068c70 msgr2=0x7fb01c154fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 shutdown_connections 2024-04-25T15:55:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:00.973+0000 7fb0213ce640 1 -- 172.21.15.67:0/1812064842 wait complete. 2024-04-25T15:55:00.997 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T15:55:01.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a3bae640 1 Processor -- start 2024-04-25T15:55:01.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a3bae640 1 -- start start 2024-04-25T15:55:01.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a3bae640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0727b0 0x7fc59c165e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a3bae640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59c072dc0 con 0x7fc59c074790 2024-04-25T15:55:01.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a3bae640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc59c15b430 con 0x7fc59c0723a0 2024-04-25T15:55:01.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a3bae640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc59c15a670 con 0x7fc59c0727b0 2024-04-25T15:55:01.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a194a640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc59c074790 0x7fc59c15dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43442/0 (socket says 172.21.15.67:43442) 2024-04-25T15:55:01.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a194a640 1 -- 172.21.15.67:0/387326970 learned_addr learned my addr 172.21.15.67:0/387326970 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:01.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a214b640 1 --2- 172.21.15.67:0/387326970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0727b0 0x7fc59c165e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460035082 0 0) 0x7fc59c072dc0 con 0x7fc59c074790 2024-04-25T15:55:01.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc588003530 con 0x7fc59c074790 2024-04-25T15:55:01.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795919532 0 0) 0x7fc59c15b430 con 0x7fc59c0723a0 2024-04-25T15:55:01.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc59c072dc0 con 0x7fc59c0723a0 2024-04-25T15:55:01.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3491750077 0 0) 0x7fc588003530 con 0x7fc59c074790 2024-04-25T15:55:01.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a214b640 1 -- 172.21.15.67:0/387326970 >> v1:172.21.15.176:6789/0 conn(0x7fc59c0723a0 legacy=0x7fc59c0c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a214b640 1 -- 172.21.15.67:0/387326970 >> v1:172.21.15.67:6789/0 conn(0x7fc59c074790 legacy=0x7fc59c15dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.145+0000 7fc5a214b640 1 -- 172.21.15.67:0/387326970 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c15a950 con 0x7fc59c0727b0 2024-04-25T15:55:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a214b640 1 --2- 172.21.15.67:0/387326970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0727b0 0x7fc59c165e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc594009dc0 tx=0x7fc59401f0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da67bbf2372d1dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940b5070 con 0x7fc59c0727b0 2024-04-25T15:55:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5940a8c30 con 0x7fc59c0727b0 2024-04-25T15:55:01.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a0948640 1 -- 172.21.15.67:0/387326970 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598001680 con 0x7fc59c0727b0 2024-04-25T15:55:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a3bae640 1 -- 172.21.15.67:0/387326970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0727b0 msgr2=0x7fc59c165e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/387326970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0727b0 0x7fc59c165e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc594009dc0 tx=0x7fc59401f0a0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a3bae640 1 -- 172.21.15.67:0/387326970 shutdown_connections 2024-04-25T15:55:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/387326970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0727b0 0x7fc59c165e80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a3bae640 1 -- 172.21.15.67:0/387326970 >> 172.21.15.67:0/387326970 conn(0x7fc59c06fe70 msgr2=0x7fc59c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a3bae640 1 -- 172.21.15.67:0/387326970 shutdown_connections 2024-04-25T15:55:01.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.149+0000 7fc5a3bae640 1 -- 172.21.15.67:0/387326970 wait complete. 2024-04-25T15:55:01.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 Processor -- start 2024-04-25T15:55:01.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 -- start start 2024-04-25T15:55:01.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0723a0 0x7fc59c1e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc59c0727b0 0x7fc59c14db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc59c074790 0x7fc59c14e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c15e1f0 con 0x7fc59c0727b0 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc59c15af10 con 0x7fc59c074790 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc59c073080 con 0x7fc59c0723a0 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc59c0727b0 0x7fc59c14db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc59c0727b0 0x7fc59c14db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49952/0 (socket says 172.21.15.67:49952) 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 -- 172.21.15.67:0/3066701216 learned_addr learned my addr 172.21.15.67:0/3066701216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a194a640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0723a0 0x7fc59c1e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a214b640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc59c074790 0x7fc59c14e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 -- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0723a0 msgr2=0x7fc59c1e52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0723a0 0x7fc59c1e52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 -- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc59c074790 msgr2=0x7fc59c14e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc59c074790 0x7fc59c14e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 -- 172.21.15.67:0/3066701216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc59c153fa0 con 0x7fc59c0727b0 2024-04-25T15:55:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a214b640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc59c074790 0x7fc59c14e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a194a640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0723a0 0x7fc59c1e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a1149640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc59c0727b0 0x7fc59c14db70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00c490 tx=0x7fc58c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5927fc640 1 -- 172.21.15.67:0/3066701216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0052d0 con 0x7fc59c0727b0 2024-04-25T15:55:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5927fc640 1 -- 172.21.15.67:0/3066701216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc58c005490 con 0x7fc59c0727b0 2024-04-25T15:55:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5927fc640 1 -- 172.21.15.67:0/3066701216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c012850 con 0x7fc59c0727b0 2024-04-25T15:55:01.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc59c1f08a0 con 0x7fc59c0727b0 2024-04-25T15:55:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.153+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc59c1f0e60 con 0x7fc59c0727b0 2024-04-25T15:55:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.157+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc59c1599e0 con 0x7fc59c0727b0 2024-04-25T15:55:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.157+0000 7fc5927fc640 1 -- 172.21.15.67:0/3066701216 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc58c0129f0 con 0x7fc59c0727b0 2024-04-25T15:55:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.157+0000 7fc5927fc640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc58006f430 0x7fc580071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.157+0000 7fc5927fc640 1 -- 172.21.15.67:0/3066701216 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fc58c08f0a0 con 0x7fc59c0727b0 2024-04-25T15:55:01.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.157+0000 7fc5a194a640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc58006f430 0x7fc580071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.157+0000 7fc5a194a640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc58006f430 0x7fc580071940 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc5980017f0 tx=0x7fc598009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:01.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.161+0000 7fc5927fc640 1 -- 172.21.15.67:0/3066701216 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc58c05bdb0 con 0x7fc59c0727b0 2024-04-25T15:55:01.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.325+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc59c1f0690 con 0x7fc59c0727b0 2024-04-25T15:55:01.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.325+0000 7fc5927fc640 1 -- 172.21.15.67:0/3066701216 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v129) ==== 100+0+11 (secure 0 0 0) 0x7fc58c05b6b0 con 0x7fc59c0727b0 2024-04-25T15:55:01.330 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-25T15:55:01.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc58006f430 msgr2=0x7fc580071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc58006f430 0x7fc580071940 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc5980017f0 tx=0x7fc598009040 comp rx=0 tx=0).stop 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc59c0727b0 msgr2=0x7fc59c14db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc59c0727b0 0x7fc59c14db70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fc58c00c490 tx=0x7fc58c00c950 comp rx=0 tx=0).stop 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 shutdown_connections 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fc58006f430 0x7fc580071940 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc59c074790 0x7fc59c14e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc59c0727b0 0x7fc59c14db70 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 --2- 172.21.15.67:0/3066701216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59c0723a0 0x7fc59c1e52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 >> 172.21.15.67:0/3066701216 conn(0x7fc59c06fe70 msgr2=0x7fc59c0ca990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 shutdown_connections 2024-04-25T15:55:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.329+0000 7fc5a3bae640 1 -- 172.21.15.67:0/3066701216 wait complete. 2024-04-25T15:55:01.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T15:55:01.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d43b0640 1 Processor -- start 2024-04-25T15:55:01.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d43b0640 1 -- start start 2024-04-25T15:55:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d43b0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 0x7ff5cc15dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d43b0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5cc072dc0 con 0x7ff5cc0c6fb0 2024-04-25T15:55:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d43b0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5cc15b430 con 0x7ff5cc15e020 2024-04-25T15:55:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d43b0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff5cc15a670 con 0x7ff5cc074790 2024-04-25T15:55:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d294d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff5cc0c6fb0 0x7ff5cc164e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43450/0 (socket says 172.21.15.67:43450) 2024-04-25T15:55:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d294d640 1 -- 172.21.15.67:0/627573917 learned_addr learned my addr 172.21.15.67:0/627573917 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d214c640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 0x7ff5cc15dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 902368761 0 0) 0x7ff5cc072dc0 con 0x7ff5cc0c6fb0 2024-04-25T15:55:01.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5b4003530 con 0x7ff5cc0c6fb0 2024-04-25T15:55:01.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 49829234 0 0) 0x7ff5b4003530 con 0x7ff5cc0c6fb0 2024-04-25T15:55:01.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 msgr2=0x7ff5cc15dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:01.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 0x7ff5cc15dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 >> v1:172.21.15.176:6789/0 conn(0x7ff5cc15e020 legacy=0x7ff5cc0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5cc15a950 con 0x7ff5cc0c6fb0 2024-04-25T15:55:01.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.501+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7ff5c4001680 con 0x7ff5cc0c6fb0 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 >> v1:172.21.15.67:6789/0 conn(0x7ff5cc0c6fb0 legacy=0x7ff5cc164e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b40039f0 0x7ff5b4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b4004810 0x7ff5b4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 0x7ff5b4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc15a670 con 0x7ff5b4004810 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b400bb90 con 0x7ff5b40039f0 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b400bd90 con 0x7ff5cc074790 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d194b640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b4004810 0x7ff5b4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d294d640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 0x7ff5b4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d214c640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b40039f0 0x7ff5b4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d194b640 1 -- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 msgr2=0x7ff5b4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d194b640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc074790 0x7ff5b4005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d194b640 1 -- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b40039f0 msgr2=0x7ff5b4003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d194b640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b40039f0 0x7ff5b4003e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d194b640 1 -- 172.21.15.67:0/627573917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b4003530 con 0x7ff5b4004810 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d194b640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b4004810 0x7ff5b4004c20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0001e40 tx=0x7ff5c001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be9d42cb92aba4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0001a90 con 0x7ff5b4004810 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c0003b80 con 0x7ff5b4004810 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d114a640 1 -- 172.21.15.67:0/627573917 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0005080 con 0x7ff5b4004810 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d43b0640 1 -- 172.21.15.67:0/627573917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b4004810 msgr2=0x7ff5b4004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d43b0640 1 --2- 172.21.15.67:0/627573917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5b4004810 0x7ff5b4004c20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0001e40 tx=0x7ff5c001c760 comp rx=0 tx=0).stop 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d294d640 1 -- 172.21.15.67:0/627573917 reap_dead start 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d43b0640 1 -- 172.21.15.67:0/627573917 shutdown_connections 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d43b0640 1 -- 172.21.15.67:0/627573917 >> 172.21.15.67:0/627573917 conn(0x7ff5cc06fe70 msgr2=0x7ff5cc0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d43b0640 1 -- 172.21.15.67:0/627573917 shutdown_connections 2024-04-25T15:55:01.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.505+0000 7ff5d43b0640 1 -- 172.21.15.67:0/627573917 wait complete. 2024-04-25T15:55:01.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 Processor -- start 2024-04-25T15:55:01.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 -- start start 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b4004810 0x7ff5cc1e3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc0cd480 0x7ff5cc0cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5cc14a180 0x7ff5cc14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc073080 con 0x7ff5cc14a180 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff5cc15af10 con 0x7ff5b4004810 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff5cc072ab0 con 0x7ff5cc0cd480 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5cc14a180 0x7ff5cc14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5cc14a180 0x7ff5cc14a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49972/0 (socket says 172.21.15.67:49972) 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 -- 172.21.15.67:0/2100147589 learned_addr learned my addr 172.21.15.67:0/2100147589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d194b640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc0cd480 0x7ff5cc0cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d214c640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b4004810 0x7ff5cc1e3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 -- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc0cd480 msgr2=0x7ff5cc0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc0cd480 0x7ff5cc0cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 -- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b4004810 msgr2=0x7ff5cc1e3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b4004810 0x7ff5cc1e3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 -- 172.21.15.67:0/2100147589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5cc1e3fa0 con 0x7ff5cc14a180 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d214c640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b4004810 0x7ff5cc1e3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d294d640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5cc14a180 0x7ff5cc14a590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0c8350 tx=0x7ff5c400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5caffd640 1 -- 172.21.15.67:0/2100147589 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4007230 con 0x7ff5cc14a180 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5caffd640 1 -- 172.21.15.67:0/2100147589 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c40073f0 con 0x7ff5cc14a180 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5caffd640 1 -- 172.21.15.67:0/2100147589 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c40047f0 con 0x7ff5cc14a180 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5cc1f18b0 con 0x7ff5cc14a180 2024-04-25T15:55:01.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.509+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5cc14b060 con 0x7ff5cc14a180 2024-04-25T15:55:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.513+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5cc076730 con 0x7ff5cc14a180 2024-04-25T15:55:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.513+0000 7ff5caffd640 1 -- 172.21.15.67:0/2100147589 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff5c401b020 con 0x7ff5cc14a180 2024-04-25T15:55:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.513+0000 7ff5caffd640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff5b806f400 0x7ff5b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.513+0000 7ff5caffd640 1 -- 172.21.15.67:0/2100147589 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7ff5c408eff0 con 0x7ff5cc14a180 2024-04-25T15:55:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.513+0000 7ff5d214c640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff5b806f400 0x7ff5b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:01.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.513+0000 7ff5d214c640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff5b806f400 0x7ff5b8071910 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009a00 tx=0x7ff5bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:01.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.517+0000 7ff5caffd640 1 -- 172.21.15.67:0/2100147589 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff5c405b7e0 con 0x7ff5cc14a180 2024-04-25T15:55:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.685+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff5cc1491d0 con 0x7ff5cc14a180 2024-04-25T15:55:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.685+0000 7ff5caffd640 1 -- 172.21.15.67:0/2100147589 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v129) ==== 80+0+0 (secure 0 0 0) 0x7ff5cc1491d0 con 0x7ff5cc14a180 2024-04-25T15:55:01.687 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-25T15:55:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff5b806f400 msgr2=0x7ff5b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff5b806f400 0x7ff5b8071910 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009a00 tx=0x7ff5bc008040 comp rx=0 tx=0).stop 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5cc14a180 msgr2=0x7ff5cc14a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5cc14a180 0x7ff5cc14a590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0c8350 tx=0x7ff5c400dc00 comp rx=0 tx=0).stop 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 shutdown_connections 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff5b806f400 0x7ff5b8071910 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5cc14a180 0x7ff5cc14a590 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5cc0cd480 0x7ff5cc0cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 --2- 172.21.15.67:0/2100147589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5b4004810 0x7ff5cc1e3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 >> 172.21.15.67:0/2100147589 conn(0x7ff5cc06fe70 msgr2=0x7ff5cc14e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 shutdown_connections 2024-04-25T15:55:01.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:01.689+0000 7ff5d43b0640 1 -- 172.21.15.67:0/2100147589 wait complete. 2024-04-25T15:55:01.709 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi067.front.sepia.ceph.com... 2024-04-25T15:55:01.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:55:01.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T15:55:01.732 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:01.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi067.stderr:ceph-fuse[35381]: fuse finished with error 0 and tester_r 0 2024-04-25T15:55:07.833 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:55:07.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:55:07.834 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T15:55:07.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:55:07.848 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-25T15:55:07.897 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T15:55:07.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:55:07.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T15:55:07.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T15:55:07.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-25T15:55:08.130 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T15:55:08.140 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T15:55:08.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:55:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdac557f640 1 Processor -- start 2024-04-25T15:55:08.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdac557f640 1 -- start start 2024-04-25T15:55:08.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdac557f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00723a0 0x7fdac00727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdac557f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdac00c6fb0 0x7fdac00c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdac557f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdac0072dc0 con 0x7fdac0074790 2024-04-25T15:55:08.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdac557f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdac015b430 con 0x7fdac00c6fb0 2024-04-25T15:55:08.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdac557f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdac015a670 con 0x7fdac00723a0 2024-04-25T15:55:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabf7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdac00c6fb0 0x7fdac00c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabe7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00723a0 0x7fdac00727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabf7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdac00c6fb0 0x7fdac00c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:60616/0 (socket says 172.21.15.67:60616) 2024-04-25T15:55:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabf7fe640 1 -- 172.21.15.67:0/830751298 learned_addr learned my addr 172.21.15.67:0/830751298 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabf7fe640 1 -- 172.21.15.67:0/830751298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00723a0 msgr2=0x7fdac00727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabf7fe640 1 --2- 172.21.15.67:0/830751298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00723a0 0x7fdac00727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabf7fe640 1 -- 172.21.15.67:0/830751298 >> v1:172.21.15.176:6789/0 conn(0x7fdac0074790 legacy=0x7fdac015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.301+0000 7fdabf7fe640 1 -- 172.21.15.67:0/830751298 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac015a950 con 0x7fdac00c6fb0 2024-04-25T15:55:08.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdabf7fe640 1 --2- 172.21.15.67:0/830751298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00c6fb0 0x7fdac00c73c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009e50 tx=0x7fdab801f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e913b5a075a7631c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdabdffb640 1 -- 172.21.15.67:0/830751298 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab80b4070 con 0x7fdac00c6fb0 2024-04-25T15:55:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdabdffb640 1 -- 172.21.15.67:0/830751298 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab80af400 con 0x7fdac00c6fb0 2024-04-25T15:55:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdabdffb640 1 -- 172.21.15.67:0/830751298 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab80af7d0 con 0x7fdac00c6fb0 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 -- 172.21.15.67:0/830751298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00c6fb0 msgr2=0x7fdac00c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 --2- 172.21.15.67:0/830751298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00c6fb0 0x7fdac00c73c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009e50 tx=0x7fdab801f210 comp rx=0 tx=0).stop 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 -- 172.21.15.67:0/830751298 shutdown_connections 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 --2- 172.21.15.67:0/830751298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00c6fb0 0x7fdac00c73c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 --2- 172.21.15.67:0/830751298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00723a0 0x7fdac00727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 -- 172.21.15.67:0/830751298 >> 172.21.15.67:0/830751298 conn(0x7fdac006fe70 msgr2=0x7fdac00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 -- 172.21.15.67:0/830751298 shutdown_connections 2024-04-25T15:55:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.305+0000 7fdac557f640 1 -- 172.21.15.67:0/830751298 wait complete. 2024-04-25T15:55:08.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 Processor -- start 2024-04-25T15:55:08.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 -- start start 2024-04-25T15:55:08.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00723a0 0x7fdac0157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac0074790 0x7fdac014cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00c6fb0 0x7fdac015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdac015e1f0 con 0x7fdac0074790 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdac015af10 con 0x7fdac00723a0 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdac0073080 con 0x7fdac00c6fb0 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac0074790 0x7fdac014cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac0074790 0x7fdac014cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55168/0 (socket says 172.21.15.67:55168) 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 -- 172.21.15.67:0/1359741220 learned_addr learned my addr 172.21.15.67:0/1359741220 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabeffd640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00723a0 0x7fdac0157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabf7fe640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00c6fb0 0x7fdac015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 -- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00c6fb0 msgr2=0x7fdac015c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00c6fb0 0x7fdac015c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 -- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00723a0 msgr2=0x7fdac0157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00723a0 0x7fdac0157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 -- 172.21.15.67:0/1359741220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac01534a0 con 0x7fdac0074790 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabeffd640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00723a0 0x7fdac0157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdabe7fc640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac0074790 0x7fdac014cfa0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fdab000e8a0 tx=0x7fdab000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdaa77fe640 1 -- 172.21.15.67:0/1359741220 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab0017070 con 0x7fdac0074790 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdaa77fe640 1 -- 172.21.15.67:0/1359741220 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab000c940 con 0x7fdac0074790 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac015b7e0 con 0x7fdac0074790 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdaa77fe640 1 -- 172.21.15.67:0/1359741220 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab000ccf0 con 0x7fdac0074790 2024-04-25T15:55:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.309+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac015bda0 con 0x7fdac0074790 2024-04-25T15:55:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.313+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac006a480 con 0x7fdac0074790 2024-04-25T15:55:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.313+0000 7fdaa77fe640 1 -- 172.21.15.67:0/1359741220 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdab001a070 con 0x7fdac0074790 2024-04-25T15:55:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.313+0000 7fdaa77fe640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdaac06f400 0x7fdaac071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.313+0000 7fdaa77fe640 1 -- 172.21.15.67:0/1359741220 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fdab008f560 con 0x7fdac0074790 2024-04-25T15:55:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.313+0000 7fdabeffd640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdaac06f400 0x7fdaac071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.313+0000 7fdabeffd640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdaac06f400 0x7fdaac071910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdab4001670 tx=0x7fdab4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:08.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.317+0000 7fdaa77fe640 1 -- 172.21.15.67:0/1359741220 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdac01534a0 con 0x7fdac0074790 2024-04-25T15:55:08.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.477+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdac00c8e90 con 0x7fdaac06f400 2024-04-25T15:55:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.497+0000 7fdaa77fe640 1 -- 172.21.15.67:0/1359741220 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253557 (secure 0 0 0) 0x7fdac00c8e90 con 0x7fdaac06f400 2024-04-25T15:55:08.500 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:08.502 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:55:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdaac06f400 msgr2=0x7fdaac071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdaac06f400 0x7fdaac071910 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdab4001670 tx=0x7fdab4009040 comp rx=0 tx=0).stop 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac0074790 msgr2=0x7fdac014cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac0074790 0x7fdac014cfa0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fdab000e8a0 tx=0x7fdab000eca0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 shutdown_connections 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdaac06f400 0x7fdaac071910 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdac00c6fb0 0x7fdac015c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdac0074790 0x7fdac014cfa0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 --2- 172.21.15.67:0/1359741220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdac00723a0 0x7fdac0157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 >> 172.21.15.67:0/1359741220 conn(0x7fdac006fe70 msgr2=0x7fdac014fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 shutdown_connections 2024-04-25T15:55:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.501+0000 7fdac557f640 1 -- 172.21.15.67:0/1359741220 wait complete. 2024-04-25T15:55:08.525 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-04-25T15:55:07.894613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798320,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":9664,"num_write_kb":25167774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":1627779911,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10098,"ondisk_log_size":10098,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383984,"kb_used_data":13872,"kb_used_omap":12,"kb_used_meta":370035,"kb_avail":754590736,"statfs":{"total":773094113280,"available":772700913664,"internally_reserved":0,"allocated":14204928,"data_stored":7557728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":378916402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005128"},"pg_stats":[{"pgid":"3.3f","version":"129'146","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831956+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:55:01.831956+0000","last_peered":"2024-04-25T15:55:01.831956+0000","last_clean":"2024-04-25T15:55:01.831956+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:55:01.831956+0000","last_undegraded":"2024-04-25T15:55:01.831956+0000","last_fullsized":"2024-04-25T15:55:01.831956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"129'130","reported_seq":255,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.789093+0000","last_change":"2024-04-25T15:54:44.217073+0000","last_active":"2024-04-25T15:55:01.789093+0000","last_peered":"2024-04-25T15:55:01.789093+0000","last_clean":"2024-04-25T15:55:01.789093+0000","last_became_active":"2024-04-25T15:54:44.215261+0000","last_became_peered":"2024-04-25T15:54:44.215261+0000","last_unstale":"2024-04-25T15:55:01.789093+0000","last_undegraded":"2024-04-25T15:55:01.789093+0000","last_fullsized":"2024-04-25T15:55:01.789093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:01.929448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"129'151","reported_seq":343,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.811643+0000","last_change":"2024-04-25T15:54:40.166712+0000","last_active":"2024-04-25T15:55:01.811643+0000","last_peered":"2024-04-25T15:55:01.811643+0000","last_clean":"2024-04-25T15:55:01.811643+0000","last_became_active":"2024-04-25T15:54:40.164866+0000","last_became_peered":"2024-04-25T15:54:40.164866+0000","last_unstale":"2024-04-25T15:55:01.811643+0000","last_undegraded":"2024-04-25T15:55:01.811643+0000","last_fullsized":"2024-04-25T15:55:01.811643+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:58.780901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"129'173","reported_seq":405,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831009+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:55:01.831009+0000","last_peered":"2024-04-25T15:55:01.831009+0000","last_clean":"2024-04-25T15:55:01.831009+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:55:01.831009+0000","last_undegraded":"2024-04-25T15:55:01.831009+0000","last_fullsized":"2024-04-25T15:55:01.831009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"129'148","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"129'135","reported_seq":366,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830601+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:55:01.830601+0000","last_peered":"2024-04-25T15:55:01.830601+0000","last_clean":"2024-04-25T15:55:01.830601+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:55:01.830601+0000","last_undegraded":"2024-04-25T15:55:01.830601+0000","last_fullsized":"2024-04-25T15:55:01.830601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"129'164","reported_seq":393,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.804532+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:55:01.804532+0000","last_peered":"2024-04-25T15:55:01.804532+0000","last_clean":"2024-04-25T15:55:01.804532+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:55:01.804532+0000","last_undegraded":"2024-04-25T15:55:01.804532+0000","last_fullsized":"2024-04-25T15:55:01.804532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"129'186","reported_seq":417,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833169+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:55:01.833169+0000","last_peered":"2024-04-25T15:55:01.833169+0000","last_clean":"2024-04-25T15:55:01.833169+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:55:01.833169+0000","last_undegraded":"2024-04-25T15:55:01.833169+0000","last_fullsized":"2024-04-25T15:55:01.833169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"129'152","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836442+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:55:01.836442+0000","last_peered":"2024-04-25T15:55:01.836442+0000","last_clean":"2024-04-25T15:55:01.836442+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:55:01.836442+0000","last_undegraded":"2024-04-25T15:55:01.836442+0000","last_fullsized":"2024-04-25T15:55:01.836442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"129'156","reported_seq":458,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831117+0000","last_change":"2024-04-25T15:54:54.355672+0000","last_active":"2024-04-25T15:55:01.831117+0000","last_peered":"2024-04-25T15:55:01.831117+0000","last_clean":"2024-04-25T15:55:01.831117+0000","last_became_active":"2024-04-25T15:54:54.352223+0000","last_became_peered":"2024-04-25T15:54:54.352223+0000","last_unstale":"2024-04-25T15:55:01.831117+0000","last_undegraded":"2024-04-25T15:55:01.831117+0000","last_fullsized":"2024-04-25T15:55:01.831117+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:39.782280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"129'189","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830161+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:55:01.830161+0000","last_peered":"2024-04-25T15:55:01.830161+0000","last_clean":"2024-04-25T15:55:01.830161+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:55:01.830161+0000","last_undegraded":"2024-04-25T15:55:01.830161+0000","last_fullsized":"2024-04-25T15:55:01.830161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"129'142","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827413+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:55:01.827413+0000","last_peered":"2024-04-25T15:55:01.827413+0000","last_clean":"2024-04-25T15:55:01.827413+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:55:01.827413+0000","last_undegraded":"2024-04-25T15:55:01.827413+0000","last_fullsized":"2024-04-25T15:55:01.827413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"129'151","reported_seq":691,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833332+0000","last_change":"2024-04-25T15:54:54.357489+0000","last_active":"2024-04-25T15:55:01.833332+0000","last_peered":"2024-04-25T15:55:01.833332+0000","last_clean":"2024-04-25T15:55:01.833332+0000","last_became_active":"2024-04-25T15:54:54.356694+0000","last_became_peered":"2024-04-25T15:54:54.356694+0000","last_unstale":"2024-04-25T15:55:01.833332+0000","last_undegraded":"2024-04-25T15:55:01.833332+0000","last_fullsized":"2024-04-25T15:55:01.833332+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.653628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"129'149","reported_seq":387,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.813061+0000","last_change":"2024-04-25T15:54:40.160979+0000","last_active":"2024-04-25T15:55:01.813061+0000","last_peered":"2024-04-25T15:55:01.813061+0000","last_clean":"2024-04-25T15:55:01.813061+0000","last_became_active":"2024-04-25T15:54:40.159879+0000","last_became_peered":"2024-04-25T15:54:40.159879+0000","last_unstale":"2024-04-25T15:55:01.813061+0000","last_undegraded":"2024-04-25T15:55:01.813061+0000","last_fullsized":"2024-04-25T15:55:01.813061+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:25.499728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"129'151","reported_seq":270,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.808743+0000","last_change":"2024-04-25T15:54:44.208852+0000","last_active":"2024-04-25T15:55:01.808743+0000","last_peered":"2024-04-25T15:55:01.808743+0000","last_clean":"2024-04-25T15:55:01.808743+0000","last_became_active":"2024-04-25T15:54:44.207144+0000","last_became_peered":"2024-04-25T15:54:44.207144+0000","last_unstale":"2024-04-25T15:55:01.808743+0000","last_undegraded":"2024-04-25T15:55:01.808743+0000","last_fullsized":"2024-04-25T15:55:01.808743+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:44:02.181746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"129'147","reported_seq":378,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830760+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:55:01.830760+0000","last_peered":"2024-04-25T15:55:01.830760+0000","last_clean":"2024-04-25T15:55:01.830760+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:55:01.830760+0000","last_undegraded":"2024-04-25T15:55:01.830760+0000","last_fullsized":"2024-04-25T15:55:01.830760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"129'146","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833273+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:55:01.833273+0000","last_peered":"2024-04-25T15:55:01.833273+0000","last_clean":"2024-04-25T15:55:01.833273+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:55:01.833273+0000","last_undegraded":"2024-04-25T15:55:01.833273+0000","last_fullsized":"2024-04-25T15:55:01.833273+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"129'157","reported_seq":601,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.837202+0000","last_change":"2024-04-25T15:54:54.355989+0000","last_active":"2024-04-25T15:55:01.837202+0000","last_peered":"2024-04-25T15:55:01.837202+0000","last_clean":"2024-04-25T15:55:01.837202+0000","last_became_active":"2024-04-25T15:54:54.354849+0000","last_became_peered":"2024-04-25T15:54:54.354849+0000","last_unstale":"2024-04-25T15:55:01.837202+0000","last_undegraded":"2024-04-25T15:55:01.837202+0000","last_fullsized":"2024-04-25T15:55:01.837202+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:05.626413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"129'172","reported_seq":402,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.818938+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:55:01.818938+0000","last_peered":"2024-04-25T15:55:01.818938+0000","last_clean":"2024-04-25T15:55:01.818938+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:55:01.818938+0000","last_undegraded":"2024-04-25T15:55:01.818938+0000","last_fullsized":"2024-04-25T15:55:01.818938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"129'161","reported_seq":409,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.829727+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:55:01.829727+0000","last_peered":"2024-04-25T15:55:01.829727+0000","last_clean":"2024-04-25T15:55:01.829727+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:55:01.829727+0000","last_undegraded":"2024-04-25T15:55:01.829727+0000","last_fullsized":"2024-04-25T15:55:01.829727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"129'137","reported_seq":327,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827376+0000","last_change":"2024-04-25T15:54:40.169787+0000","last_active":"2024-04-25T15:55:01.827376+0000","last_peered":"2024-04-25T15:55:01.827376+0000","last_clean":"2024-04-25T15:55:01.827376+0000","last_became_active":"2024-04-25T15:54:40.167469+0000","last_became_peered":"2024-04-25T15:54:40.167469+0000","last_unstale":"2024-04-25T15:55:01.827376+0000","last_undegraded":"2024-04-25T15:55:01.827376+0000","last_fullsized":"2024-04-25T15:55:01.827376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:26.375093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"129'166","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819478+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:55:01.819478+0000","last_peered":"2024-04-25T15:55:01.819478+0000","last_clean":"2024-04-25T15:55:01.819478+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:55:01.819478+0000","last_undegraded":"2024-04-25T15:55:01.819478+0000","last_fullsized":"2024-04-25T15:55:01.819478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"129'134","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.815642+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:55:01.815642+0000","last_peered":"2024-04-25T15:55:01.815642+0000","last_clean":"2024-04-25T15:55:01.815642+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:55:01.815642+0000","last_undegraded":"2024-04-25T15:55:01.815642+0000","last_fullsized":"2024-04-25T15:55:01.815642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"129'180","reported_seq":412,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827531+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:55:01.827531+0000","last_peered":"2024-04-25T15:55:01.827531+0000","last_clean":"2024-04-25T15:55:01.827531+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:55:01.827531+0000","last_undegraded":"2024-04-25T15:55:01.827531+0000","last_fullsized":"2024-04-25T15:55:01.827531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"129'134","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819773+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:55:01.819773+0000","last_peered":"2024-04-25T15:55:01.819773+0000","last_clean":"2024-04-25T15:55:01.819773+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:55:01.819773+0000","last_undegraded":"2024-04-25T15:55:01.819773+0000","last_fullsized":"2024-04-25T15:55:01.819773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"129'198","reported_seq":430,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834987+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:55:01.834987+0000","last_peered":"2024-04-25T15:55:01.834987+0000","last_clean":"2024-04-25T15:55:01.834987+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:55:01.834987+0000","last_undegraded":"2024-04-25T15:55:01.834987+0000","last_fullsized":"2024-04-25T15:55:01.834987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"129'144","reported_seq":356,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.805991+0000","last_change":"2024-04-25T15:54:44.212386+0000","last_active":"2024-04-25T15:55:01.805991+0000","last_peered":"2024-04-25T15:55:01.805991+0000","last_clean":"2024-04-25T15:55:01.805991+0000","last_became_active":"2024-04-25T15:54:44.210520+0000","last_became_peered":"2024-04-25T15:54:44.210520+0000","last_unstale":"2024-04-25T15:55:01.805991+0000","last_undegraded":"2024-04-25T15:55:01.805991+0000","last_fullsized":"2024-04-25T15:55:01.805991+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:01.061276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"129'133","reported_seq":400,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"129'118","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.813169+0000","last_change":"2024-04-25T15:54:40.174462+0000","last_active":"2024-04-25T15:55:01.813169+0000","last_peered":"2024-04-25T15:55:01.813169+0000","last_clean":"2024-04-25T15:55:01.813169+0000","last_became_active":"2024-04-25T15:54:40.173209+0000","last_became_peered":"2024-04-25T15:54:40.173209+0000","last_unstale":"2024-04-25T15:55:01.813169+0000","last_undegraded":"2024-04-25T15:55:01.813169+0000","last_fullsized":"2024-04-25T15:55:01.813169+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:08.410467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"129'131","reported_seq":307,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.789509+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:55:01.789509+0000","last_peered":"2024-04-25T15:55:01.789509+0000","last_clean":"2024-04-25T15:55:01.789509+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:55:01.789509+0000","last_undegraded":"2024-04-25T15:55:01.789509+0000","last_fullsized":"2024-04-25T15:55:01.789509+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"129'135","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836486+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:55:01.836486+0000","last_peered":"2024-04-25T15:55:01.836486+0000","last_clean":"2024-04-25T15:55:01.836486+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:55:01.836486+0000","last_undegraded":"2024-04-25T15:55:01.836486+0000","last_fullsized":"2024-04-25T15:55:01.836486+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"129'150","reported_seq":479,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826757+0000","last_change":"2024-04-25T15:54:54.359681+0000","last_active":"2024-04-25T15:55:01.826757+0000","last_peered":"2024-04-25T15:55:01.826757+0000","last_clean":"2024-04-25T15:55:01.826757+0000","last_became_active":"2024-04-25T15:54:54.353824+0000","last_became_peered":"2024-04-25T15:54:54.353824+0000","last_unstale":"2024-04-25T15:55:01.826757+0000","last_undegraded":"2024-04-25T15:55:01.826757+0000","last_fullsized":"2024-04-25T15:55:01.826757+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:52.297640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"129'130","reported_seq":246,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816679+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:55:01.816679+0000","last_peered":"2024-04-25T15:55:01.816679+0000","last_clean":"2024-04-25T15:55:01.816679+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:55:01.816679+0000","last_undegraded":"2024-04-25T15:55:01.816679+0000","last_fullsized":"2024-04-25T15:55:01.816679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"129'7","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351493+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:54.351493+0000","last_peered":"2024-04-25T15:54:54.351493+0000","last_clean":"2024-04-25T15:54:54.351493+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:54.351493+0000","last_undegraded":"2024-04-25T15:54:54.351493+0000","last_fullsized":"2024-04-25T15:54:54.351493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"129'137","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.815609+0000","last_change":"2024-04-25T15:54:40.163745+0000","last_active":"2024-04-25T15:55:01.815609+0000","last_peered":"2024-04-25T15:55:01.815609+0000","last_clean":"2024-04-25T15:55:01.815609+0000","last_became_active":"2024-04-25T15:54:40.162667+0000","last_became_peered":"2024-04-25T15:54:40.162667+0000","last_unstale":"2024-04-25T15:55:01.815609+0000","last_undegraded":"2024-04-25T15:55:01.815609+0000","last_fullsized":"2024-04-25T15:55:01.815609+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:17.528981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347310+0000","last_change":"2024-04-25T15:54:51.317865+0000","last_active":"2024-04-25T15:54:54.347310+0000","last_peered":"2024-04-25T15:54:54.347310+0000","last_clean":"2024-04-25T15:54:54.347310+0000","last_became_active":"2024-04-25T15:54:51.302422+0000","last_became_peered":"2024-04-25T15:54:51.302422+0000","last_unstale":"2024-04-25T15:54:54.347310+0000","last_undegraded":"2024-04-25T15:54:54.347310+0000","last_fullsized":"2024-04-25T15:54:54.347310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:36.956598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350037+0000","last_change":"2024-04-25T15:54:51.329012+0000","last_active":"2024-04-25T15:54:54.350037+0000","last_peered":"2024-04-25T15:54:54.350037+0000","last_clean":"2024-04-25T15:54:54.350037+0000","last_became_active":"2024-04-25T15:54:51.307662+0000","last_became_peered":"2024-04-25T15:54:51.307662+0000","last_unstale":"2024-04-25T15:54:54.350037+0000","last_undegraded":"2024-04-25T15:54:54.350037+0000","last_fullsized":"2024-04-25T15:54:54.350037+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:41.566474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"129'170","reported_seq":296,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.804884+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:55:01.804884+0000","last_peered":"2024-04-25T15:55:01.804884+0000","last_clean":"2024-04-25T15:55:01.804884+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:55:01.804884+0000","last_undegraded":"2024-04-25T15:55:01.804884+0000","last_fullsized":"2024-04-25T15:55:01.804884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345518+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:54.345518+0000","last_peered":"2024-04-25T15:54:54.345518+0000","last_clean":"2024-04-25T15:54:54.345518+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:54.345518+0000","last_undegraded":"2024-04-25T15:54:54.345518+0000","last_fullsized":"2024-04-25T15:54:54.345518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"129'157","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816623+0000","last_change":"2024-04-25T15:54:44.212095+0000","last_active":"2024-04-25T15:55:01.816623+0000","last_peered":"2024-04-25T15:55:01.816623+0000","last_clean":"2024-04-25T15:55:01.816623+0000","last_became_active":"2024-04-25T15:54:44.209985+0000","last_became_peered":"2024-04-25T15:54:44.209985+0000","last_unstale":"2024-04-25T15:55:01.816623+0000","last_undegraded":"2024-04-25T15:55:01.816623+0000","last_fullsized":"2024-04-25T15:55:01.816623+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:35.906277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"129'8","reported_seq":241,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740819+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:55:01.740819+0000","last_peered":"2024-04-25T15:55:01.740819+0000","last_clean":"2024-04-25T15:55:01.740819+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:55:01.740819+0000","last_undegraded":"2024-04-25T15:55:01.740819+0000","last_fullsized":"2024-04-25T15:55:01.740819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"129'229","reported_seq":460,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834138+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:55:01.834138+0000","last_peered":"2024-04-25T15:55:01.834138+0000","last_clean":"2024-04-25T15:55:01.834138+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:55:01.834138+0000","last_undegraded":"2024-04-25T15:55:01.834138+0000","last_fullsized":"2024-04-25T15:55:01.834138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346366+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:54.346366+0000","last_peered":"2024-04-25T15:54:54.346366+0000","last_clean":"2024-04-25T15:54:54.346366+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:54.346366+0000","last_undegraded":"2024-04-25T15:54:54.346366+0000","last_fullsized":"2024-04-25T15:54:54.346366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"129'141","reported_seq":334,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833434+0000","last_change":"2024-04-25T15:54:40.169299+0000","last_active":"2024-04-25T15:55:01.833434+0000","last_peered":"2024-04-25T15:55:01.833434+0000","last_clean":"2024-04-25T15:55:01.833434+0000","last_became_active":"2024-04-25T15:54:40.168203+0000","last_became_peered":"2024-04-25T15:54:40.168203+0000","last_unstale":"2024-04-25T15:55:01.833434+0000","last_undegraded":"2024-04-25T15:55:01.833434+0000","last_fullsized":"2024-04-25T15:55:01.833434+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.710379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"114'11","reported_seq":46,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342159+0000","last_change":"2024-04-25T15:54:41.197909+0000","last_active":"2024-04-25T15:54:54.342159+0000","last_peered":"2024-04-25T15:54:54.342159+0000","last_clean":"2024-04-25T15:54:54.342159+0000","last_became_active":"2024-04-25T15:54:41.171190+0000","last_became_peered":"2024-04-25T15:54:41.171190+0000","last_unstale":"2024-04-25T15:54:54.342159+0000","last_undegraded":"2024-04-25T15:54:54.342159+0000","last_fullsized":"2024-04-25T15:54:54.342159+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:40.822327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2016,"num_keys_recovered":30,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"129'149","reported_seq":265,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825846+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:55:01.825846+0000","last_peered":"2024-04-25T15:55:01.825846+0000","last_clean":"2024-04-25T15:55:01.825846+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:55:01.825846+0000","last_undegraded":"2024-04-25T15:55:01.825846+0000","last_fullsized":"2024-04-25T15:55:01.825846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"129'93","reported_seq":33,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740355+0000","last_change":"2024-04-25T15:54:51.358160+0000","last_active":"2024-04-25T15:55:01.740355+0000","last_peered":"2024-04-25T15:55:01.740355+0000","last_clean":"2024-04-25T15:55:01.740355+0000","last_became_active":"2024-04-25T15:54:51.303172+0000","last_became_peered":"2024-04-25T15:54:51.303172+0000","last_unstale":"2024-04-25T15:55:01.740355+0000","last_undegraded":"2024-04-25T15:55:01.740355+0000","last_fullsized":"2024-04-25T15:55:01.740355+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:39.823475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199279,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":380717,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"129'150","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819314+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:55:01.819314+0000","last_peered":"2024-04-25T15:55:01.819314+0000","last_clean":"2024-04-25T15:55:01.819314+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:55:01.819314+0000","last_undegraded":"2024-04-25T15:55:01.819314+0000","last_fullsized":"2024-04-25T15:55:01.819314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352562+0000","last_change":"2024-04-25T15:54:40.166458+0000","last_active":"2024-04-25T15:54:54.352562+0000","last_peered":"2024-04-25T15:54:54.352562+0000","last_clean":"2024-04-25T15:54:54.352562+0000","last_became_active":"2024-04-25T15:54:40.165701+0000","last_became_peered":"2024-04-25T15:54:40.165701+0000","last_unstale":"2024-04-25T15:54:54.352562+0000","last_undegraded":"2024-04-25T15:54:54.352562+0000","last_fullsized":"2024-04-25T15:54:54.352562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:43:18.998536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"129'157","reported_seq":348,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827452+0000","last_change":"2024-04-25T15:54:40.168672+0000","last_active":"2024-04-25T15:55:01.827452+0000","last_peered":"2024-04-25T15:55:01.827452+0000","last_clean":"2024-04-25T15:55:01.827452+0000","last_became_active":"2024-04-25T15:54:40.167338+0000","last_became_peered":"2024-04-25T15:54:40.167338+0000","last_unstale":"2024-04-25T15:55:01.827452+0000","last_undegraded":"2024-04-25T15:55:01.827452+0000","last_fullsized":"2024-04-25T15:55:01.827452+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:04.063276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344550+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:54.344550+0000","last_peered":"2024-04-25T15:54:54.344550+0000","last_clean":"2024-04-25T15:54:54.344550+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:54.344550+0000","last_undegraded":"2024-04-25T15:54:54.344550+0000","last_fullsized":"2024-04-25T15:54:54.344550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"129'171","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828601+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:55:01.828601+0000","last_peered":"2024-04-25T15:55:01.828601+0000","last_clean":"2024-04-25T15:55:01.828601+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:55:01.828601+0000","last_undegraded":"2024-04-25T15:55:01.828601+0000","last_fullsized":"2024-04-25T15:55:01.828601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"129'7","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347111+0000","last_change":"2024-04-25T15:54:51.333252+0000","last_active":"2024-04-25T15:54:54.347111+0000","last_peered":"2024-04-25T15:54:54.347111+0000","last_clean":"2024-04-25T15:54:54.347111+0000","last_became_active":"2024-04-25T15:54:51.302160+0000","last_became_peered":"2024-04-25T15:54:51.302160+0000","last_unstale":"2024-04-25T15:54:54.347111+0000","last_undegraded":"2024-04-25T15:54:54.347111+0000","last_fullsized":"2024-04-25T15:54:54.347111+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:25.724351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4116,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"129'165","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.824551+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:55:01.824551+0000","last_peered":"2024-04-25T15:55:01.824551+0000","last_clean":"2024-04-25T15:55:01.824551+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:55:01.824551+0000","last_undegraded":"2024-04-25T15:55:01.824551+0000","last_fullsized":"2024-04-25T15:55:01.824551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"114'18","reported_seq":342,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345636+0000","last_change":"2024-04-25T15:54:41.193381+0000","last_active":"2024-04-25T15:54:54.345636+0000","last_peered":"2024-04-25T15:54:54.345636+0000","last_clean":"2024-04-25T15:54:54.345636+0000","last_became_active":"2024-04-25T15:54:41.166653+0000","last_became_peered":"2024-04-25T15:54:41.166653+0000","last_unstale":"2024-04-25T15:54:54.345636+0000","last_undegraded":"2024-04-25T15:54:54.345636+0000","last_fullsized":"2024-04-25T15:54:54.345636+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:10.229064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":448,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"129'189","reported_seq":446,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.820824+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:55:01.820824+0000","last_peered":"2024-04-25T15:55:01.820824+0000","last_clean":"2024-04-25T15:55:01.820824+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:55:01.820824+0000","last_undegraded":"2024-04-25T15:55:01.820824+0000","last_fullsized":"2024-04-25T15:55:01.820824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":120,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341794+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:54.341794+0000","last_peered":"2024-04-25T15:54:54.341794+0000","last_clean":"2024-04-25T15:54:54.341794+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:54.341794+0000","last_undegraded":"2024-04-25T15:54:54.341794+0000","last_fullsized":"2024-04-25T15:54:54.341794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"129'146","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825150+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:55:01.825150+0000","last_peered":"2024-04-25T15:55:01.825150+0000","last_clean":"2024-04-25T15:55:01.825150+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:55:01.825150+0000","last_undegraded":"2024-04-25T15:55:01.825150+0000","last_fullsized":"2024-04-25T15:55:01.825150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346353+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:54:54.346353+0000","last_peered":"2024-04-25T15:54:54.346353+0000","last_clean":"2024-04-25T15:54:54.346353+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:54:54.346353+0000","last_undegraded":"2024-04-25T15:54:54.346353+0000","last_fullsized":"2024-04-25T15:54:54.346353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"129'138","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828396+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:55:01.828396+0000","last_peered":"2024-04-25T15:55:01.828396+0000","last_clean":"2024-04-25T15:55:01.828396+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:55:01.828396+0000","last_undegraded":"2024-04-25T15:55:01.828396+0000","last_fullsized":"2024-04-25T15:55:01.828396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":40,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342273+0000","last_change":"2024-04-25T15:54:41.186336+0000","last_active":"2024-04-25T15:54:54.342273+0000","last_peered":"2024-04-25T15:54:54.342273+0000","last_clean":"2024-04-25T15:54:54.342273+0000","last_became_active":"2024-04-25T15:54:41.165683+0000","last_became_peered":"2024-04-25T15:54:41.165683+0000","last_unstale":"2024-04-25T15:54:54.342273+0000","last_undegraded":"2024-04-25T15:54:54.342273+0000","last_fullsized":"2024-04-25T15:54:54.342273+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:54.115834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"129'144","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830924+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:55:01.830924+0000","last_peered":"2024-04-25T15:55:01.830924+0000","last_clean":"2024-04-25T15:55:01.830924+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:55:01.830924+0000","last_undegraded":"2024-04-25T15:55:01.830924+0000","last_fullsized":"2024-04-25T15:55:01.830924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"129'7","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.750762+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:55:01.750762+0000","last_peered":"2024-04-25T15:55:01.750762+0000","last_clean":"2024-04-25T15:55:01.750762+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:55:01.750762+0000","last_undegraded":"2024-04-25T15:55:01.750762+0000","last_fullsized":"2024-04-25T15:55:01.750762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"129'151","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:02.222181+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:55:02.222181+0000","last_peered":"2024-04-25T15:55:02.222181+0000","last_clean":"2024-04-25T15:55:02.222181+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:55:02.222181+0000","last_undegraded":"2024-04-25T15:55:02.222181+0000","last_fullsized":"2024-04-25T15:55:02.222181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630501+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:54.630501+0000","last_peered":"2024-04-25T15:54:54.630501+0000","last_clean":"2024-04-25T15:54:54.630501+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:54.630501+0000","last_undegraded":"2024-04-25T15:54:54.630501+0000","last_fullsized":"2024-04-25T15:54:54.630501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344831+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:54.344831+0000","last_peered":"2024-04-25T15:54:54.344831+0000","last_clean":"2024-04-25T15:54:54.344831+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:54.344831+0000","last_undegraded":"2024-04-25T15:54:54.344831+0000","last_fullsized":"2024-04-25T15:54:54.344831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"129'158","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.817195+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:55:01.817195+0000","last_peered":"2024-04-25T15:55:01.817195+0000","last_clean":"2024-04-25T15:55:01.817195+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:55:01.817195+0000","last_undegraded":"2024-04-25T15:55:01.817195+0000","last_fullsized":"2024-04-25T15:55:01.817195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":272,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630140+0000","last_change":"2024-04-25T15:54:41.173406+0000","last_active":"2024-04-25T15:54:54.630140+0000","last_peered":"2024-04-25T15:54:54.630140+0000","last_clean":"2024-04-25T15:54:54.630140+0000","last_became_active":"2024-04-25T15:54:41.166720+0000","last_became_peered":"2024-04-25T15:54:41.166720+0000","last_unstale":"2024-04-25T15:54:54.630140+0000","last_undegraded":"2024-04-25T15:54:54.630140+0000","last_fullsized":"2024-04-25T15:54:54.630140+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:41.861800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"129'131","reported_seq":252,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.820767+0000","last_change":"2024-04-25T15:54:40.164755+0000","last_active":"2024-04-25T15:55:01.820767+0000","last_peered":"2024-04-25T15:55:01.820767+0000","last_clean":"2024-04-25T15:55:01.820767+0000","last_became_active":"2024-04-25T15:54:40.163587+0000","last_became_peered":"2024-04-25T15:54:40.163587+0000","last_unstale":"2024-04-25T15:55:01.820767+0000","last_undegraded":"2024-04-25T15:55:01.820767+0000","last_fullsized":"2024-04-25T15:55:01.820767+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:58.420382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"129'13","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346047+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:54:54.346047+0000","last_peered":"2024-04-25T15:54:54.346047+0000","last_clean":"2024-04-25T15:54:54.346047+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:54:54.346047+0000","last_undegraded":"2024-04-25T15:54:54.346047+0000","last_fullsized":"2024-04-25T15:54:54.346047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"129'131","reported_seq":431,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.793091+0000","last_change":"2024-04-25T15:54:44.216921+0000","last_active":"2024-04-25T15:55:01.793091+0000","last_peered":"2024-04-25T15:55:01.793091+0000","last_clean":"2024-04-25T15:55:01.793091+0000","last_became_active":"2024-04-25T15:54:44.215186+0000","last_became_peered":"2024-04-25T15:54:44.215186+0000","last_unstale":"2024-04-25T15:55:01.793091+0000","last_undegraded":"2024-04-25T15:55:01.793091+0000","last_fullsized":"2024-04-25T15:55:01.793091+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:49.565745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344849+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:54.344849+0000","last_peered":"2024-04-25T15:54:54.344849+0000","last_clean":"2024-04-25T15:54:54.344849+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:54.344849+0000","last_undegraded":"2024-04-25T15:54:54.344849+0000","last_fullsized":"2024-04-25T15:54:54.344849+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"129'145","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.787876+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:55:01.787876+0000","last_peered":"2024-04-25T15:55:01.787876+0000","last_clean":"2024-04-25T15:55:01.787876+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:55:01.787876+0000","last_undegraded":"2024-04-25T15:55:01.787876+0000","last_fullsized":"2024-04-25T15:55:01.787876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":36,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341899+0000","last_change":"2024-04-25T15:54:41.168286+0000","last_active":"2024-04-25T15:54:54.341899+0000","last_peered":"2024-04-25T15:54:54.341899+0000","last_clean":"2024-04-25T15:54:54.341899+0000","last_became_active":"2024-04-25T15:54:41.166643+0000","last_became_peered":"2024-04-25T15:54:41.166643+0000","last_unstale":"2024-04-25T15:54:54.341899+0000","last_undegraded":"2024-04-25T15:54:54.341899+0000","last_fullsized":"2024-04-25T15:54:54.341899+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:08.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828419+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:55:01.828419+0000","last_peered":"2024-04-25T15:55:01.828419+0000","last_clean":"2024-04-25T15:55:01.828419+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:55:01.828419+0000","last_undegraded":"2024-04-25T15:55:01.828419+0000","last_fullsized":"2024-04-25T15:55:01.828419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631647+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:54.631647+0000","last_peered":"2024-04-25T15:54:54.631647+0000","last_clean":"2024-04-25T15:54:54.631647+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:54.631647+0000","last_undegraded":"2024-04-25T15:54:54.631647+0000","last_fullsized":"2024-04-25T15:54:54.631647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"129'144","reported_seq":344,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.803519+0000","last_change":"2024-04-25T15:54:44.217114+0000","last_active":"2024-04-25T15:55:01.803519+0000","last_peered":"2024-04-25T15:55:01.803519+0000","last_clean":"2024-04-25T15:55:01.803519+0000","last_became_active":"2024-04-25T15:54:44.215121+0000","last_became_peered":"2024-04-25T15:54:44.215121+0000","last_unstale":"2024-04-25T15:55:01.803519+0000","last_undegraded":"2024-04-25T15:55:01.803519+0000","last_fullsized":"2024-04-25T15:55:01.803519+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:37.441111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'26","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341499+0000","last_change":"2024-04-25T15:54:51.320611+0000","last_active":"2024-04-25T15:54:54.341499+0000","last_peered":"2024-04-25T15:54:54.341499+0000","last_clean":"2024-04-25T15:54:54.341499+0000","last_became_active":"2024-04-25T15:54:51.307056+0000","last_became_peered":"2024-04-25T15:54:51.307056+0000","last_unstale":"2024-04-25T15:54:54.341499+0000","last_undegraded":"2024-04-25T15:54:54.341499+0000","last_fullsized":"2024-04-25T15:54:54.341499+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:15.769447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":13,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.822154+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:55:01.822154+0000","last_peered":"2024-04-25T15:55:01.822154+0000","last_clean":"2024-04-25T15:55:01.822154+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:55:01.822154+0000","last_undegraded":"2024-04-25T15:55:01.822154+0000","last_fullsized":"2024-04-25T15:55:01.822154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"110'8","reported_seq":242,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348406+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:54.348406+0000","last_peered":"2024-04-25T15:54:54.348406+0000","last_clean":"2024-04-25T15:54:54.348406+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:54.348406+0000","last_undegraded":"2024-04-25T15:54:54.348406+0000","last_fullsized":"2024-04-25T15:54:54.348406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"129'163","reported_seq":391,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.812671+0000","last_change":"2024-04-25T15:54:44.208533+0000","last_active":"2024-04-25T15:55:01.812671+0000","last_peered":"2024-04-25T15:55:01.812671+0000","last_clean":"2024-04-25T15:55:01.812671+0000","last_became_active":"2024-04-25T15:54:44.207033+0000","last_became_peered":"2024-04-25T15:54:44.207033+0000","last_unstale":"2024-04-25T15:55:01.812671+0000","last_undegraded":"2024-04-25T15:55:01.812671+0000","last_fullsized":"2024-04-25T15:55:01.812671+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:44.559711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":234,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630110+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:54.630110+0000","last_peered":"2024-04-25T15:54:54.630110+0000","last_clean":"2024-04-25T15:54:54.630110+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:54.630110+0000","last_undegraded":"2024-04-25T15:54:54.630110+0000","last_fullsized":"2024-04-25T15:54:54.630110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"129'134","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.751001+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:55:01.751001+0000","last_peered":"2024-04-25T15:55:01.751001+0000","last_clean":"2024-04-25T15:55:01.751001+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:55:01.751001+0000","last_undegraded":"2024-04-25T15:55:01.751001+0000","last_fullsized":"2024-04-25T15:55:01.751001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":206,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341314+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:54:54.341314+0000","last_peered":"2024-04-25T15:54:54.341314+0000","last_clean":"2024-04-25T15:54:54.341314+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:54:54.341314+0000","last_undegraded":"2024-04-25T15:54:54.341314+0000","last_fullsized":"2024-04-25T15:54:54.341314+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"129'153","reported_seq":521,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.811691+0000","last_change":"2024-04-25T15:54:44.207954+0000","last_active":"2024-04-25T15:55:01.811691+0000","last_peered":"2024-04-25T15:55:01.811691+0000","last_clean":"2024-04-25T15:55:01.811691+0000","last_became_active":"2024-04-25T15:54:44.206595+0000","last_became_peered":"2024-04-25T15:54:44.206595+0000","last_unstale":"2024-04-25T15:55:01.811691+0000","last_undegraded":"2024-04-25T15:55:01.811691+0000","last_fullsized":"2024-04-25T15:55:01.811691+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.107940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":69,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346110+0000","last_change":"2024-04-25T15:54:22.056875+0000","last_active":"2024-04-25T15:54:54.346110+0000","last_peered":"2024-04-25T15:54:54.346110+0000","last_clean":"2024-04-25T15:54:54.346110+0000","last_became_active":"2024-04-25T15:54:22.037756+0000","last_became_peered":"2024-04-25T15:54:22.037756+0000","last_unstale":"2024-04-25T15:54:54.346110+0000","last_undegraded":"2024-04-25T15:54:54.346110+0000","last_fullsized":"2024-04-25T15:54:54.346110+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:57.998015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"129'143","reported_seq":367,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.801641+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:55:01.801641+0000","last_peered":"2024-04-25T15:55:01.801641+0000","last_clean":"2024-04-25T15:55:01.801641+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:55:01.801641+0000","last_undegraded":"2024-04-25T15:55:01.801641+0000","last_fullsized":"2024-04-25T15:55:01.801641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":259,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631676+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:54.631676+0000","last_peered":"2024-04-25T15:54:54.631676+0000","last_clean":"2024-04-25T15:54:54.631676+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:54.631676+0000","last_undegraded":"2024-04-25T15:54:54.631676+0000","last_fullsized":"2024-04-25T15:54:54.631676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"129'128","reported_seq":386,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826678+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:55:01.826678+0000","last_peered":"2024-04-25T15:55:01.826678+0000","last_clean":"2024-04-25T15:55:01.826678+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:55:01.826678+0000","last_undegraded":"2024-04-25T15:55:01.826678+0000","last_fullsized":"2024-04-25T15:55:01.826678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"129'183","reported_seq":440,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.835785+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:55:01.835785+0000","last_peered":"2024-04-25T15:55:01.835785+0000","last_clean":"2024-04-25T15:55:01.835785+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:55:01.835785+0000","last_undegraded":"2024-04-25T15:55:01.835785+0000","last_fullsized":"2024-04-25T15:55:01.835785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631338+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:54.631338+0000","last_peered":"2024-04-25T15:54:54.631338+0000","last_clean":"2024-04-25T15:54:54.631338+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:54.631338+0000","last_undegraded":"2024-04-25T15:54:54.631338+0000","last_fullsized":"2024-04-25T15:54:54.631338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"129'141","reported_seq":415,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834610+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:55:01.834610+0000","last_peered":"2024-04-25T15:55:01.834610+0000","last_clean":"2024-04-25T15:55:01.834610+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:55:01.834610+0000","last_undegraded":"2024-04-25T15:55:01.834610+0000","last_fullsized":"2024-04-25T15:55:01.834610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":244,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631249+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:54.631249+0000","last_peered":"2024-04-25T15:54:54.631249+0000","last_clean":"2024-04-25T15:54:54.631249+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:54.631249+0000","last_undegraded":"2024-04-25T15:54:54.631249+0000","last_fullsized":"2024-04-25T15:54:54.631249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"129'135","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836536+0000","last_change":"2024-04-25T15:54:40.169300+0000","last_active":"2024-04-25T15:55:01.836536+0000","last_peered":"2024-04-25T15:55:01.836536+0000","last_clean":"2024-04-25T15:55:01.836536+0000","last_became_active":"2024-04-25T15:54:40.168164+0000","last_became_peered":"2024-04-25T15:54:40.168164+0000","last_unstale":"2024-04-25T15:55:01.836536+0000","last_undegraded":"2024-04-25T15:55:01.836536+0000","last_fullsized":"2024-04-25T15:55:01.836536+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:50.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351250+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:54.351250+0000","last_peered":"2024-04-25T15:54:54.351250+0000","last_clean":"2024-04-25T15:54:54.351250+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:54.351250+0000","last_undegraded":"2024-04-25T15:54:54.351250+0000","last_fullsized":"2024-04-25T15:54:54.351250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"129'152","reported_seq":598,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.832040+0000","last_change":"2024-04-25T15:54:54.360954+0000","last_active":"2024-04-25T15:55:01.832040+0000","last_peered":"2024-04-25T15:55:01.832040+0000","last_clean":"2024-04-25T15:55:01.832040+0000","last_became_active":"2024-04-25T15:54:54.360118+0000","last_became_peered":"2024-04-25T15:54:54.360118+0000","last_unstale":"2024-04-25T15:55:01.832040+0000","last_undegraded":"2024-04-25T15:55:01.832040+0000","last_fullsized":"2024-04-25T15:55:01.832040+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:37.822818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":103,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.353079+0000","last_change":"2024-04-25T15:54:40.170535+0000","last_active":"2024-04-25T15:54:54.353079+0000","last_peered":"2024-04-25T15:54:54.353079+0000","last_clean":"2024-04-25T15:54:54.353079+0000","last_became_active":"2024-04-25T15:54:40.169964+0000","last_became_peered":"2024-04-25T15:54:40.169964+0000","last_unstale":"2024-04-25T15:54:54.353079+0000","last_undegraded":"2024-04-25T15:54:54.353079+0000","last_fullsized":"2024-04-25T15:54:54.353079+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:24.688851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":1627389952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9717,"ondisk_log_size":9717,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":207952,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":347,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":389959,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":415904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59668,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312172,"statfs":{"total":96636764160,"available":96575664128,"internally_reserved":0,"allocated":2174976,"data_stored":1338205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41224,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1769472,"data_stored":945730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63260,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308580,"statfs":{"total":96636764160,"available":96571985920,"internally_reserved":0,"allocated":1593344,"data_stored":749669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1568768,"data_stored":747067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1777664,"data_stored":946269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1581056,"data_stored":747704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1572864,"data_stored":746687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2166784,"data_stored":1336397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":201190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:55:08.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:55:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0adc1640 1 Processor -- start 2024-04-25T15:55:08.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0adc1640 1 -- start start 2024-04-25T15:55:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0adc1640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb040c7610 0x7ffb040c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0adc1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ffb040c5bb0 0x7ffb040c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0adc1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb040c6890 con 0x7ffb04071b20 2024-04-25T15:55:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0adc1640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ffb0415cd40 con 0x7ffb040c7610 2024-04-25T15:55:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0adc1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ffb040c6b70 con 0x7ffb040c5bb0 2024-04-25T15:55:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb03fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb040c7610 0x7ffb040c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb03fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb040c7610 0x7ffb040c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55184/0 (socket says 172.21.15.67:55184) 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb03fff640 1 -- 172.21.15.67:0/1014977915 learned_addr learned my addr 172.21.15.67:0/1014977915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0935e640 1 --2- 172.21.15.67:0/1014977915 >> v2:172.21.15.176:3300/0 conn(0x7ffb040c5bb0 0x7ffb040c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb03fff640 1 -- 172.21.15.67:0/1014977915 >> v2:172.21.15.176:3300/0 conn(0x7ffb040c5bb0 msgr2=0x7ffb040c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb03fff640 1 --2- 172.21.15.67:0/1014977915 >> v2:172.21.15.176:3300/0 conn(0x7ffb040c5bb0 0x7ffb040c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb03fff640 1 -- 172.21.15.67:0/1014977915 >> v1:172.21.15.176:6789/0 conn(0x7ffb04071b20 legacy=0x7ffb0415d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb03fff640 1 -- 172.21.15.67:0/1014977915 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb040c6e50 con 0x7ffb040c7610 2024-04-25T15:55:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.725+0000 7ffb0935e640 1 --2- 172.21.15.67:0/1014977915 >> v2:172.21.15.176:3300/0 conn(0x7ffb040c5bb0 0x7ffb040c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb03fff640 1 --2- 172.21.15.67:0/1014977915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb040c7a20 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffb04073410 tx=0x7ffaf801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa3ff577112ee15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb037fe640 1 -- 172.21.15.67:0/1014977915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80ab5f0 con 0x7ffb040c7610 2024-04-25T15:55:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb037fe640 1 -- 172.21.15.67:0/1014977915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf80b0070 con 0x7ffb040c7610 2024-04-25T15:55:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb037fe640 1 -- 172.21.15.67:0/1014977915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80ab9b0 con 0x7ffb040c7610 2024-04-25T15:55:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 -- 172.21.15.67:0/1014977915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 msgr2=0x7ffb040c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/1014977915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb040c7a20 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ffb04073410 tx=0x7ffaf801c4e0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 -- 172.21.15.67:0/1014977915 shutdown_connections 2024-04-25T15:55:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/1014977915 >> v2:172.21.15.176:3300/0 conn(0x7ffb040c5bb0 0x7ffb040c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/1014977915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb040c7a20 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 -- 172.21.15.67:0/1014977915 >> 172.21.15.67:0/1014977915 conn(0x7ffb0406fe70 msgr2=0x7ffb040cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 -- 172.21.15.67:0/1014977915 shutdown_connections 2024-04-25T15:55:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.729+0000 7ffb0adc1640 1 -- 172.21.15.67:0/1014977915 wait complete. 2024-04-25T15:55:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 Processor -- start 2024-04-25T15:55:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 -- start start 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb04071b20 0x7ffb0414bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb040c5bb0 0x7ffb041f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb0413f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04166580 con 0x7ffb040c7610 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffb040734f0 con 0x7ffb040c5bb0 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffb04166700 con 0x7ffb04071b20 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb0413f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb0413f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55196/0 (socket says 172.21.15.67:55196) 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 -- 172.21.15.67:0/2459034830 learned_addr learned my addr 172.21.15.67:0/2459034830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb03fff640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb040c5bb0 0x7ffb041f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb08b5d640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb04071b20 0x7ffb0414bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 -- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb04071b20 msgr2=0x7ffb0414bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb04071b20 0x7ffb0414bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 -- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb040c5bb0 msgr2=0x7ffb041f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb040c5bb0 0x7ffb041f3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 -- 172.21.15.67:0/2459034830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0415bff0 con 0x7ffb040c7610 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0935e640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb0413f360 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00cdb0 tx=0x7ffafc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb017fa640 1 -- 172.21.15.67:0/2459034830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc003ba0 con 0x7ffb040c7610 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb017fa640 1 -- 172.21.15.67:0/2459034830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffafc003d60 con 0x7ffb040c7610 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0415b860 con 0x7ffb040c7610 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb017fa640 1 -- 172.21.15.67:0/2459034830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc004260 con 0x7ffb040c7610 2024-04-25T15:55:08.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.733+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0415bb40 con 0x7ffb040c7610 2024-04-25T15:55:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.737+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb0406a480 con 0x7ffb040c7610 2024-04-25T15:55:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.737+0000 7ffb017fa640 1 -- 172.21.15.67:0/2459034830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ffafc004400 con 0x7ffb040c7610 2024-04-25T15:55:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.737+0000 7ffb017fa640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffaf006f430 0x7ffaf0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.737+0000 7ffb017fa640 1 -- 172.21.15.67:0/2459034830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7ffafc090190 con 0x7ffb040c7610 2024-04-25T15:55:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.737+0000 7ffb08b5d640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffaf006f430 0x7ffaf0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.737+0000 7ffb08b5d640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffaf006f430 0x7ffaf0071940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ffb040c5960 tx=0x7ffaf4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:08.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.741+0000 7ffb017fa640 1 -- 172.21.15.67:0/2459034830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffafc014030 con 0x7ffb040c7610 2024-04-25T15:55:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.905+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffb04147530 con 0x7ffaf006f430 2024-04-25T15:55:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.921+0000 7ffb017fa640 1 -- 172.21.15.67:0/2459034830 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253557 (secure 0 0 0) 0x7ffb04147530 con 0x7ffaf006f430 2024-04-25T15:55:08.926 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:55:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffaf006f430 msgr2=0x7ffaf0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffaf006f430 0x7ffaf0071940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ffb040c5960 tx=0x7ffaf4008040 comp rx=0 tx=0).stop 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 msgr2=0x7ffb0413f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb0413f360 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00cdb0 tx=0x7ffafc00a650 comp rx=0 tx=0).stop 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 shutdown_connections 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ffaf006f430 0x7ffaf0071940 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb040c7610 0x7ffb0413f360 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb040c5bb0 0x7ffb041f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 --2- 172.21.15.67:0/2459034830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb04071b20 0x7ffb0414bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 >> 172.21.15.67:0/2459034830 conn(0x7ffb0406fe70 msgr2=0x7ffb04073700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 shutdown_connections 2024-04-25T15:55:08.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:08.929+0000 7ffb0adc1640 1 -- 172.21.15.67:0/2459034830 wait complete. 2024-04-25T15:55:08.951 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-04-25T15:55:07.894613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798320,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":9664,"num_write_kb":25167774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":1627779911,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10098,"ondisk_log_size":10098,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383984,"kb_used_data":13872,"kb_used_omap":12,"kb_used_meta":370035,"kb_avail":754590736,"statfs":{"total":773094113280,"available":772700913664,"internally_reserved":0,"allocated":14204928,"data_stored":7557728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":378916402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005128"},"pg_stats":[{"pgid":"3.3f","version":"129'146","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831956+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:55:01.831956+0000","last_peered":"2024-04-25T15:55:01.831956+0000","last_clean":"2024-04-25T15:55:01.831956+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:55:01.831956+0000","last_undegraded":"2024-04-25T15:55:01.831956+0000","last_fullsized":"2024-04-25T15:55:01.831956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"129'130","reported_seq":255,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.789093+0000","last_change":"2024-04-25T15:54:44.217073+0000","last_active":"2024-04-25T15:55:01.789093+0000","last_peered":"2024-04-25T15:55:01.789093+0000","last_clean":"2024-04-25T15:55:01.789093+0000","last_became_active":"2024-04-25T15:54:44.215261+0000","last_became_peered":"2024-04-25T15:54:44.215261+0000","last_unstale":"2024-04-25T15:55:01.789093+0000","last_undegraded":"2024-04-25T15:55:01.789093+0000","last_fullsized":"2024-04-25T15:55:01.789093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:01.929448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"129'151","reported_seq":343,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.811643+0000","last_change":"2024-04-25T15:54:40.166712+0000","last_active":"2024-04-25T15:55:01.811643+0000","last_peered":"2024-04-25T15:55:01.811643+0000","last_clean":"2024-04-25T15:55:01.811643+0000","last_became_active":"2024-04-25T15:54:40.164866+0000","last_became_peered":"2024-04-25T15:54:40.164866+0000","last_unstale":"2024-04-25T15:55:01.811643+0000","last_undegraded":"2024-04-25T15:55:01.811643+0000","last_fullsized":"2024-04-25T15:55:01.811643+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:58.780901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"129'173","reported_seq":405,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831009+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:55:01.831009+0000","last_peered":"2024-04-25T15:55:01.831009+0000","last_clean":"2024-04-25T15:55:01.831009+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:55:01.831009+0000","last_undegraded":"2024-04-25T15:55:01.831009+0000","last_fullsized":"2024-04-25T15:55:01.831009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"129'148","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"129'135","reported_seq":366,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830601+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:55:01.830601+0000","last_peered":"2024-04-25T15:55:01.830601+0000","last_clean":"2024-04-25T15:55:01.830601+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:55:01.830601+0000","last_undegraded":"2024-04-25T15:55:01.830601+0000","last_fullsized":"2024-04-25T15:55:01.830601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"129'164","reported_seq":393,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.804532+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:55:01.804532+0000","last_peered":"2024-04-25T15:55:01.804532+0000","last_clean":"2024-04-25T15:55:01.804532+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:55:01.804532+0000","last_undegraded":"2024-04-25T15:55:01.804532+0000","last_fullsized":"2024-04-25T15:55:01.804532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"129'186","reported_seq":417,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833169+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:55:01.833169+0000","last_peered":"2024-04-25T15:55:01.833169+0000","last_clean":"2024-04-25T15:55:01.833169+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:55:01.833169+0000","last_undegraded":"2024-04-25T15:55:01.833169+0000","last_fullsized":"2024-04-25T15:55:01.833169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"129'152","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836442+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:55:01.836442+0000","last_peered":"2024-04-25T15:55:01.836442+0000","last_clean":"2024-04-25T15:55:01.836442+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:55:01.836442+0000","last_undegraded":"2024-04-25T15:55:01.836442+0000","last_fullsized":"2024-04-25T15:55:01.836442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"129'156","reported_seq":458,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831117+0000","last_change":"2024-04-25T15:54:54.355672+0000","last_active":"2024-04-25T15:55:01.831117+0000","last_peered":"2024-04-25T15:55:01.831117+0000","last_clean":"2024-04-25T15:55:01.831117+0000","last_became_active":"2024-04-25T15:54:54.352223+0000","last_became_peered":"2024-04-25T15:54:54.352223+0000","last_unstale":"2024-04-25T15:55:01.831117+0000","last_undegraded":"2024-04-25T15:55:01.831117+0000","last_fullsized":"2024-04-25T15:55:01.831117+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:39.782280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"129'189","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830161+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:55:01.830161+0000","last_peered":"2024-04-25T15:55:01.830161+0000","last_clean":"2024-04-25T15:55:01.830161+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:55:01.830161+0000","last_undegraded":"2024-04-25T15:55:01.830161+0000","last_fullsized":"2024-04-25T15:55:01.830161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"129'142","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827413+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:55:01.827413+0000","last_peered":"2024-04-25T15:55:01.827413+0000","last_clean":"2024-04-25T15:55:01.827413+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:55:01.827413+0000","last_undegraded":"2024-04-25T15:55:01.827413+0000","last_fullsized":"2024-04-25T15:55:01.827413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"129'151","reported_seq":691,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833332+0000","last_change":"2024-04-25T15:54:54.357489+0000","last_active":"2024-04-25T15:55:01.833332+0000","last_peered":"2024-04-25T15:55:01.833332+0000","last_clean":"2024-04-25T15:55:01.833332+0000","last_became_active":"2024-04-25T15:54:54.356694+0000","last_became_peered":"2024-04-25T15:54:54.356694+0000","last_unstale":"2024-04-25T15:55:01.833332+0000","last_undegraded":"2024-04-25T15:55:01.833332+0000","last_fullsized":"2024-04-25T15:55:01.833332+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.653628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"129'149","reported_seq":387,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.813061+0000","last_change":"2024-04-25T15:54:40.160979+0000","last_active":"2024-04-25T15:55:01.813061+0000","last_peered":"2024-04-25T15:55:01.813061+0000","last_clean":"2024-04-25T15:55:01.813061+0000","last_became_active":"2024-04-25T15:54:40.159879+0000","last_became_peered":"2024-04-25T15:54:40.159879+0000","last_unstale":"2024-04-25T15:55:01.813061+0000","last_undegraded":"2024-04-25T15:55:01.813061+0000","last_fullsized":"2024-04-25T15:55:01.813061+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:25.499728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"129'151","reported_seq":270,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.808743+0000","last_change":"2024-04-25T15:54:44.208852+0000","last_active":"2024-04-25T15:55:01.808743+0000","last_peered":"2024-04-25T15:55:01.808743+0000","last_clean":"2024-04-25T15:55:01.808743+0000","last_became_active":"2024-04-25T15:54:44.207144+0000","last_became_peered":"2024-04-25T15:54:44.207144+0000","last_unstale":"2024-04-25T15:55:01.808743+0000","last_undegraded":"2024-04-25T15:55:01.808743+0000","last_fullsized":"2024-04-25T15:55:01.808743+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:44:02.181746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"129'147","reported_seq":378,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830760+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:55:01.830760+0000","last_peered":"2024-04-25T15:55:01.830760+0000","last_clean":"2024-04-25T15:55:01.830760+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:55:01.830760+0000","last_undegraded":"2024-04-25T15:55:01.830760+0000","last_fullsized":"2024-04-25T15:55:01.830760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"129'146","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833273+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:55:01.833273+0000","last_peered":"2024-04-25T15:55:01.833273+0000","last_clean":"2024-04-25T15:55:01.833273+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:55:01.833273+0000","last_undegraded":"2024-04-25T15:55:01.833273+0000","last_fullsized":"2024-04-25T15:55:01.833273+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"129'157","reported_seq":601,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.837202+0000","last_change":"2024-04-25T15:54:54.355989+0000","last_active":"2024-04-25T15:55:01.837202+0000","last_peered":"2024-04-25T15:55:01.837202+0000","last_clean":"2024-04-25T15:55:01.837202+0000","last_became_active":"2024-04-25T15:54:54.354849+0000","last_became_peered":"2024-04-25T15:54:54.354849+0000","last_unstale":"2024-04-25T15:55:01.837202+0000","last_undegraded":"2024-04-25T15:55:01.837202+0000","last_fullsized":"2024-04-25T15:55:01.837202+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:05.626413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"129'172","reported_seq":402,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.818938+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:55:01.818938+0000","last_peered":"2024-04-25T15:55:01.818938+0000","last_clean":"2024-04-25T15:55:01.818938+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:55:01.818938+0000","last_undegraded":"2024-04-25T15:55:01.818938+0000","last_fullsized":"2024-04-25T15:55:01.818938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"129'161","reported_seq":409,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.829727+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:55:01.829727+0000","last_peered":"2024-04-25T15:55:01.829727+0000","last_clean":"2024-04-25T15:55:01.829727+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:55:01.829727+0000","last_undegraded":"2024-04-25T15:55:01.829727+0000","last_fullsized":"2024-04-25T15:55:01.829727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"129'137","reported_seq":327,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827376+0000","last_change":"2024-04-25T15:54:40.169787+0000","last_active":"2024-04-25T15:55:01.827376+0000","last_peered":"2024-04-25T15:55:01.827376+0000","last_clean":"2024-04-25T15:55:01.827376+0000","last_became_active":"2024-04-25T15:54:40.167469+0000","last_became_peered":"2024-04-25T15:54:40.167469+0000","last_unstale":"2024-04-25T15:55:01.827376+0000","last_undegraded":"2024-04-25T15:55:01.827376+0000","last_fullsized":"2024-04-25T15:55:01.827376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:26.375093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"129'166","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819478+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:55:01.819478+0000","last_peered":"2024-04-25T15:55:01.819478+0000","last_clean":"2024-04-25T15:55:01.819478+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:55:01.819478+0000","last_undegraded":"2024-04-25T15:55:01.819478+0000","last_fullsized":"2024-04-25T15:55:01.819478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"129'134","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.815642+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:55:01.815642+0000","last_peered":"2024-04-25T15:55:01.815642+0000","last_clean":"2024-04-25T15:55:01.815642+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:55:01.815642+0000","last_undegraded":"2024-04-25T15:55:01.815642+0000","last_fullsized":"2024-04-25T15:55:01.815642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"129'180","reported_seq":412,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827531+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:55:01.827531+0000","last_peered":"2024-04-25T15:55:01.827531+0000","last_clean":"2024-04-25T15:55:01.827531+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:55:01.827531+0000","last_undegraded":"2024-04-25T15:55:01.827531+0000","last_fullsized":"2024-04-25T15:55:01.827531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"129'134","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819773+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:55:01.819773+0000","last_peered":"2024-04-25T15:55:01.819773+0000","last_clean":"2024-04-25T15:55:01.819773+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:55:01.819773+0000","last_undegraded":"2024-04-25T15:55:01.819773+0000","last_fullsized":"2024-04-25T15:55:01.819773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"129'198","reported_seq":430,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834987+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:55:01.834987+0000","last_peered":"2024-04-25T15:55:01.834987+0000","last_clean":"2024-04-25T15:55:01.834987+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:55:01.834987+0000","last_undegraded":"2024-04-25T15:55:01.834987+0000","last_fullsized":"2024-04-25T15:55:01.834987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"129'144","reported_seq":356,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.805991+0000","last_change":"2024-04-25T15:54:44.212386+0000","last_active":"2024-04-25T15:55:01.805991+0000","last_peered":"2024-04-25T15:55:01.805991+0000","last_clean":"2024-04-25T15:55:01.805991+0000","last_became_active":"2024-04-25T15:54:44.210520+0000","last_became_peered":"2024-04-25T15:54:44.210520+0000","last_unstale":"2024-04-25T15:55:01.805991+0000","last_undegraded":"2024-04-25T15:55:01.805991+0000","last_fullsized":"2024-04-25T15:55:01.805991+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:01.061276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"129'133","reported_seq":400,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"129'118","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.813169+0000","last_change":"2024-04-25T15:54:40.174462+0000","last_active":"2024-04-25T15:55:01.813169+0000","last_peered":"2024-04-25T15:55:01.813169+0000","last_clean":"2024-04-25T15:55:01.813169+0000","last_became_active":"2024-04-25T15:54:40.173209+0000","last_became_peered":"2024-04-25T15:54:40.173209+0000","last_unstale":"2024-04-25T15:55:01.813169+0000","last_undegraded":"2024-04-25T15:55:01.813169+0000","last_fullsized":"2024-04-25T15:55:01.813169+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:08.410467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"129'131","reported_seq":307,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.789509+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:55:01.789509+0000","last_peered":"2024-04-25T15:55:01.789509+0000","last_clean":"2024-04-25T15:55:01.789509+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:55:01.789509+0000","last_undegraded":"2024-04-25T15:55:01.789509+0000","last_fullsized":"2024-04-25T15:55:01.789509+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"129'135","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836486+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:55:01.836486+0000","last_peered":"2024-04-25T15:55:01.836486+0000","last_clean":"2024-04-25T15:55:01.836486+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:55:01.836486+0000","last_undegraded":"2024-04-25T15:55:01.836486+0000","last_fullsized":"2024-04-25T15:55:01.836486+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"129'150","reported_seq":479,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826757+0000","last_change":"2024-04-25T15:54:54.359681+0000","last_active":"2024-04-25T15:55:01.826757+0000","last_peered":"2024-04-25T15:55:01.826757+0000","last_clean":"2024-04-25T15:55:01.826757+0000","last_became_active":"2024-04-25T15:54:54.353824+0000","last_became_peered":"2024-04-25T15:54:54.353824+0000","last_unstale":"2024-04-25T15:55:01.826757+0000","last_undegraded":"2024-04-25T15:55:01.826757+0000","last_fullsized":"2024-04-25T15:55:01.826757+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:52.297640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"129'130","reported_seq":246,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816679+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:55:01.816679+0000","last_peered":"2024-04-25T15:55:01.816679+0000","last_clean":"2024-04-25T15:55:01.816679+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:55:01.816679+0000","last_undegraded":"2024-04-25T15:55:01.816679+0000","last_fullsized":"2024-04-25T15:55:01.816679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"129'7","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351493+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:54.351493+0000","last_peered":"2024-04-25T15:54:54.351493+0000","last_clean":"2024-04-25T15:54:54.351493+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:54.351493+0000","last_undegraded":"2024-04-25T15:54:54.351493+0000","last_fullsized":"2024-04-25T15:54:54.351493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"129'137","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.815609+0000","last_change":"2024-04-25T15:54:40.163745+0000","last_active":"2024-04-25T15:55:01.815609+0000","last_peered":"2024-04-25T15:55:01.815609+0000","last_clean":"2024-04-25T15:55:01.815609+0000","last_became_active":"2024-04-25T15:54:40.162667+0000","last_became_peered":"2024-04-25T15:54:40.162667+0000","last_unstale":"2024-04-25T15:55:01.815609+0000","last_undegraded":"2024-04-25T15:55:01.815609+0000","last_fullsized":"2024-04-25T15:55:01.815609+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:17.528981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347310+0000","last_change":"2024-04-25T15:54:51.317865+0000","last_active":"2024-04-25T15:54:54.347310+0000","last_peered":"2024-04-25T15:54:54.347310+0000","last_clean":"2024-04-25T15:54:54.347310+0000","last_became_active":"2024-04-25T15:54:51.302422+0000","last_became_peered":"2024-04-25T15:54:51.302422+0000","last_unstale":"2024-04-25T15:54:54.347310+0000","last_undegraded":"2024-04-25T15:54:54.347310+0000","last_fullsized":"2024-04-25T15:54:54.347310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:36.956598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350037+0000","last_change":"2024-04-25T15:54:51.329012+0000","last_active":"2024-04-25T15:54:54.350037+0000","last_peered":"2024-04-25T15:54:54.350037+0000","last_clean":"2024-04-25T15:54:54.350037+0000","last_became_active":"2024-04-25T15:54:51.307662+0000","last_became_peered":"2024-04-25T15:54:51.307662+0000","last_unstale":"2024-04-25T15:54:54.350037+0000","last_undegraded":"2024-04-25T15:54:54.350037+0000","last_fullsized":"2024-04-25T15:54:54.350037+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:41.566474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"129'170","reported_seq":296,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.804884+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:55:01.804884+0000","last_peered":"2024-04-25T15:55:01.804884+0000","last_clean":"2024-04-25T15:55:01.804884+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:55:01.804884+0000","last_undegraded":"2024-04-25T15:55:01.804884+0000","last_fullsized":"2024-04-25T15:55:01.804884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345518+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:54.345518+0000","last_peered":"2024-04-25T15:54:54.345518+0000","last_clean":"2024-04-25T15:54:54.345518+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:54.345518+0000","last_undegraded":"2024-04-25T15:54:54.345518+0000","last_fullsized":"2024-04-25T15:54:54.345518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"129'157","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816623+0000","last_change":"2024-04-25T15:54:44.212095+0000","last_active":"2024-04-25T15:55:01.816623+0000","last_peered":"2024-04-25T15:55:01.816623+0000","last_clean":"2024-04-25T15:55:01.816623+0000","last_became_active":"2024-04-25T15:54:44.209985+0000","last_became_peered":"2024-04-25T15:54:44.209985+0000","last_unstale":"2024-04-25T15:55:01.816623+0000","last_undegraded":"2024-04-25T15:55:01.816623+0000","last_fullsized":"2024-04-25T15:55:01.816623+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:35.906277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"129'8","reported_seq":241,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740819+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:55:01.740819+0000","last_peered":"2024-04-25T15:55:01.740819+0000","last_clean":"2024-04-25T15:55:01.740819+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:55:01.740819+0000","last_undegraded":"2024-04-25T15:55:01.740819+0000","last_fullsized":"2024-04-25T15:55:01.740819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"129'229","reported_seq":460,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834138+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:55:01.834138+0000","last_peered":"2024-04-25T15:55:01.834138+0000","last_clean":"2024-04-25T15:55:01.834138+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:55:01.834138+0000","last_undegraded":"2024-04-25T15:55:01.834138+0000","last_fullsized":"2024-04-25T15:55:01.834138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346366+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:54.346366+0000","last_peered":"2024-04-25T15:54:54.346366+0000","last_clean":"2024-04-25T15:54:54.346366+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:54.346366+0000","last_undegraded":"2024-04-25T15:54:54.346366+0000","last_fullsized":"2024-04-25T15:54:54.346366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"129'141","reported_seq":334,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833434+0000","last_change":"2024-04-25T15:54:40.169299+0000","last_active":"2024-04-25T15:55:01.833434+0000","last_peered":"2024-04-25T15:55:01.833434+0000","last_clean":"2024-04-25T15:55:01.833434+0000","last_became_active":"2024-04-25T15:54:40.168203+0000","last_became_peered":"2024-04-25T15:54:40.168203+0000","last_unstale":"2024-04-25T15:55:01.833434+0000","last_undegraded":"2024-04-25T15:55:01.833434+0000","last_fullsized":"2024-04-25T15:55:01.833434+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.710379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"114'11","reported_seq":46,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342159+0000","last_change":"2024-04-25T15:54:41.197909+0000","last_active":"2024-04-25T15:54:54.342159+0000","last_peered":"2024-04-25T15:54:54.342159+0000","last_clean":"2024-04-25T15:54:54.342159+0000","last_became_active":"2024-04-25T15:54:41.171190+0000","last_became_peered":"2024-04-25T15:54:41.171190+0000","last_unstale":"2024-04-25T15:54:54.342159+0000","last_undegraded":"2024-04-25T15:54:54.342159+0000","last_fullsized":"2024-04-25T15:54:54.342159+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:40.822327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2016,"num_keys_recovered":30,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"129'149","reported_seq":265,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825846+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:55:01.825846+0000","last_peered":"2024-04-25T15:55:01.825846+0000","last_clean":"2024-04-25T15:55:01.825846+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:55:01.825846+0000","last_undegraded":"2024-04-25T15:55:01.825846+0000","last_fullsized":"2024-04-25T15:55:01.825846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"129'93","reported_seq":33,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740355+0000","last_change":"2024-04-25T15:54:51.358160+0000","last_active":"2024-04-25T15:55:01.740355+0000","last_peered":"2024-04-25T15:55:01.740355+0000","last_clean":"2024-04-25T15:55:01.740355+0000","last_became_active":"2024-04-25T15:54:51.303172+0000","last_became_peered":"2024-04-25T15:54:51.303172+0000","last_unstale":"2024-04-25T15:55:01.740355+0000","last_undegraded":"2024-04-25T15:55:01.740355+0000","last_fullsized":"2024-04-25T15:55:01.740355+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:39.823475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199279,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":380717,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"129'150","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819314+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:55:01.819314+0000","last_peered":"2024-04-25T15:55:01.819314+0000","last_clean":"2024-04-25T15:55:01.819314+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:55:01.819314+0000","last_undegraded":"2024-04-25T15:55:01.819314+0000","last_fullsized":"2024-04-25T15:55:01.819314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352562+0000","last_change":"2024-04-25T15:54:40.166458+0000","last_active":"2024-04-25T15:54:54.352562+0000","last_peered":"2024-04-25T15:54:54.352562+0000","last_clean":"2024-04-25T15:54:54.352562+0000","last_became_active":"2024-04-25T15:54:40.165701+0000","last_became_peered":"2024-04-25T15:54:40.165701+0000","last_unstale":"2024-04-25T15:54:54.352562+0000","last_undegraded":"2024-04-25T15:54:54.352562+0000","last_fullsized":"2024-04-25T15:54:54.352562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:43:18.998536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"129'157","reported_seq":348,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827452+0000","last_change":"2024-04-25T15:54:40.168672+0000","last_active":"2024-04-25T15:55:01.827452+0000","last_peered":"2024-04-25T15:55:01.827452+0000","last_clean":"2024-04-25T15:55:01.827452+0000","last_became_active":"2024-04-25T15:54:40.167338+0000","last_became_peered":"2024-04-25T15:54:40.167338+0000","last_unstale":"2024-04-25T15:55:01.827452+0000","last_undegraded":"2024-04-25T15:55:01.827452+0000","last_fullsized":"2024-04-25T15:55:01.827452+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:04.063276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344550+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:54.344550+0000","last_peered":"2024-04-25T15:54:54.344550+0000","last_clean":"2024-04-25T15:54:54.344550+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:54.344550+0000","last_undegraded":"2024-04-25T15:54:54.344550+0000","last_fullsized":"2024-04-25T15:54:54.344550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"129'171","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828601+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:55:01.828601+0000","last_peered":"2024-04-25T15:55:01.828601+0000","last_clean":"2024-04-25T15:55:01.828601+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:55:01.828601+0000","last_undegraded":"2024-04-25T15:55:01.828601+0000","last_fullsized":"2024-04-25T15:55:01.828601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"129'7","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347111+0000","last_change":"2024-04-25T15:54:51.333252+0000","last_active":"2024-04-25T15:54:54.347111+0000","last_peered":"2024-04-25T15:54:54.347111+0000","last_clean":"2024-04-25T15:54:54.347111+0000","last_became_active":"2024-04-25T15:54:51.302160+0000","last_became_peered":"2024-04-25T15:54:51.302160+0000","last_unstale":"2024-04-25T15:54:54.347111+0000","last_undegraded":"2024-04-25T15:54:54.347111+0000","last_fullsized":"2024-04-25T15:54:54.347111+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:25.724351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4116,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"129'165","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.824551+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:55:01.824551+0000","last_peered":"2024-04-25T15:55:01.824551+0000","last_clean":"2024-04-25T15:55:01.824551+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:55:01.824551+0000","last_undegraded":"2024-04-25T15:55:01.824551+0000","last_fullsized":"2024-04-25T15:55:01.824551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"114'18","reported_seq":342,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345636+0000","last_change":"2024-04-25T15:54:41.193381+0000","last_active":"2024-04-25T15:54:54.345636+0000","last_peered":"2024-04-25T15:54:54.345636+0000","last_clean":"2024-04-25T15:54:54.345636+0000","last_became_active":"2024-04-25T15:54:41.166653+0000","last_became_peered":"2024-04-25T15:54:41.166653+0000","last_unstale":"2024-04-25T15:54:54.345636+0000","last_undegraded":"2024-04-25T15:54:54.345636+0000","last_fullsized":"2024-04-25T15:54:54.345636+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:10.229064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":448,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"129'189","reported_seq":446,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.820824+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:55:01.820824+0000","last_peered":"2024-04-25T15:55:01.820824+0000","last_clean":"2024-04-25T15:55:01.820824+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:55:01.820824+0000","last_undegraded":"2024-04-25T15:55:01.820824+0000","last_fullsized":"2024-04-25T15:55:01.820824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":120,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341794+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:54.341794+0000","last_peered":"2024-04-25T15:54:54.341794+0000","last_clean":"2024-04-25T15:54:54.341794+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:54.341794+0000","last_undegraded":"2024-04-25T15:54:54.341794+0000","last_fullsized":"2024-04-25T15:54:54.341794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"129'146","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825150+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:55:01.825150+0000","last_peered":"2024-04-25T15:55:01.825150+0000","last_clean":"2024-04-25T15:55:01.825150+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:55:01.825150+0000","last_undegraded":"2024-04-25T15:55:01.825150+0000","last_fullsized":"2024-04-25T15:55:01.825150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346353+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:54:54.346353+0000","last_peered":"2024-04-25T15:54:54.346353+0000","last_clean":"2024-04-25T15:54:54.346353+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:54:54.346353+0000","last_undegraded":"2024-04-25T15:54:54.346353+0000","last_fullsized":"2024-04-25T15:54:54.346353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"129'138","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828396+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:55:01.828396+0000","last_peered":"2024-04-25T15:55:01.828396+0000","last_clean":"2024-04-25T15:55:01.828396+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:55:01.828396+0000","last_undegraded":"2024-04-25T15:55:01.828396+0000","last_fullsized":"2024-04-25T15:55:01.828396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":40,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342273+0000","last_change":"2024-04-25T15:54:41.186336+0000","last_active":"2024-04-25T15:54:54.342273+0000","last_peered":"2024-04-25T15:54:54.342273+0000","last_clean":"2024-04-25T15:54:54.342273+0000","last_became_active":"2024-04-25T15:54:41.165683+0000","last_became_peered":"2024-04-25T15:54:41.165683+0000","last_unstale":"2024-04-25T15:54:54.342273+0000","last_undegraded":"2024-04-25T15:54:54.342273+0000","last_fullsized":"2024-04-25T15:54:54.342273+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:54.115834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"129'144","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830924+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:55:01.830924+0000","last_peered":"2024-04-25T15:55:01.830924+0000","last_clean":"2024-04-25T15:55:01.830924+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:55:01.830924+0000","last_undegraded":"2024-04-25T15:55:01.830924+0000","last_fullsized":"2024-04-25T15:55:01.830924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"129'7","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.750762+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:55:01.750762+0000","last_peered":"2024-04-25T15:55:01.750762+0000","last_clean":"2024-04-25T15:55:01.750762+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:55:01.750762+0000","last_undegraded":"2024-04-25T15:55:01.750762+0000","last_fullsized":"2024-04-25T15:55:01.750762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"129'151","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:02.222181+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:55:02.222181+0000","last_peered":"2024-04-25T15:55:02.222181+0000","last_clean":"2024-04-25T15:55:02.222181+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:55:02.222181+0000","last_undegraded":"2024-04-25T15:55:02.222181+0000","last_fullsized":"2024-04-25T15:55:02.222181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630501+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:54.630501+0000","last_peered":"2024-04-25T15:54:54.630501+0000","last_clean":"2024-04-25T15:54:54.630501+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:54.630501+0000","last_undegraded":"2024-04-25T15:54:54.630501+0000","last_fullsized":"2024-04-25T15:54:54.630501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344831+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:54.344831+0000","last_peered":"2024-04-25T15:54:54.344831+0000","last_clean":"2024-04-25T15:54:54.344831+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:54.344831+0000","last_undegraded":"2024-04-25T15:54:54.344831+0000","last_fullsized":"2024-04-25T15:54:54.344831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"129'158","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.817195+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:55:01.817195+0000","last_peered":"2024-04-25T15:55:01.817195+0000","last_clean":"2024-04-25T15:55:01.817195+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:55:01.817195+0000","last_undegraded":"2024-04-25T15:55:01.817195+0000","last_fullsized":"2024-04-25T15:55:01.817195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":272,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630140+0000","last_change":"2024-04-25T15:54:41.173406+0000","last_active":"2024-04-25T15:54:54.630140+0000","last_peered":"2024-04-25T15:54:54.630140+0000","last_clean":"2024-04-25T15:54:54.630140+0000","last_became_active":"2024-04-25T15:54:41.166720+0000","last_became_peered":"2024-04-25T15:54:41.166720+0000","last_unstale":"2024-04-25T15:54:54.630140+0000","last_undegraded":"2024-04-25T15:54:54.630140+0000","last_fullsized":"2024-04-25T15:54:54.630140+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:41.861800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"129'131","reported_seq":252,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.820767+0000","last_change":"2024-04-25T15:54:40.164755+0000","last_active":"2024-04-25T15:55:01.820767+0000","last_peered":"2024-04-25T15:55:01.820767+0000","last_clean":"2024-04-25T15:55:01.820767+0000","last_became_active":"2024-04-25T15:54:40.163587+0000","last_became_peered":"2024-04-25T15:54:40.163587+0000","last_unstale":"2024-04-25T15:55:01.820767+0000","last_undegraded":"2024-04-25T15:55:01.820767+0000","last_fullsized":"2024-04-25T15:55:01.820767+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:58.420382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"129'13","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346047+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:54:54.346047+0000","last_peered":"2024-04-25T15:54:54.346047+0000","last_clean":"2024-04-25T15:54:54.346047+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:54:54.346047+0000","last_undegraded":"2024-04-25T15:54:54.346047+0000","last_fullsized":"2024-04-25T15:54:54.346047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"129'131","reported_seq":431,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.793091+0000","last_change":"2024-04-25T15:54:44.216921+0000","last_active":"2024-04-25T15:55:01.793091+0000","last_peered":"2024-04-25T15:55:01.793091+0000","last_clean":"2024-04-25T15:55:01.793091+0000","last_became_active":"2024-04-25T15:54:44.215186+0000","last_became_peered":"2024-04-25T15:54:44.215186+0000","last_unstale":"2024-04-25T15:55:01.793091+0000","last_undegraded":"2024-04-25T15:55:01.793091+0000","last_fullsized":"2024-04-25T15:55:01.793091+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:49.565745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344849+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:54.344849+0000","last_peered":"2024-04-25T15:54:54.344849+0000","last_clean":"2024-04-25T15:54:54.344849+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:54.344849+0000","last_undegraded":"2024-04-25T15:54:54.344849+0000","last_fullsized":"2024-04-25T15:54:54.344849+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"129'145","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.787876+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:55:01.787876+0000","last_peered":"2024-04-25T15:55:01.787876+0000","last_clean":"2024-04-25T15:55:01.787876+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:55:01.787876+0000","last_undegraded":"2024-04-25T15:55:01.787876+0000","last_fullsized":"2024-04-25T15:55:01.787876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":36,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341899+0000","last_change":"2024-04-25T15:54:41.168286+0000","last_active":"2024-04-25T15:54:54.341899+0000","last_peered":"2024-04-25T15:54:54.341899+0000","last_clean":"2024-04-25T15:54:54.341899+0000","last_became_active":"2024-04-25T15:54:41.166643+0000","last_became_peered":"2024-04-25T15:54:41.166643+0000","last_unstale":"2024-04-25T15:54:54.341899+0000","last_undegraded":"2024-04-25T15:54:54.341899+0000","last_fullsized":"2024-04-25T15:54:54.341899+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:08.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828419+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:55:01.828419+0000","last_peered":"2024-04-25T15:55:01.828419+0000","last_clean":"2024-04-25T15:55:01.828419+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:55:01.828419+0000","last_undegraded":"2024-04-25T15:55:01.828419+0000","last_fullsized":"2024-04-25T15:55:01.828419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631647+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:54.631647+0000","last_peered":"2024-04-25T15:54:54.631647+0000","last_clean":"2024-04-25T15:54:54.631647+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:54.631647+0000","last_undegraded":"2024-04-25T15:54:54.631647+0000","last_fullsized":"2024-04-25T15:54:54.631647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"129'144","reported_seq":344,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.803519+0000","last_change":"2024-04-25T15:54:44.217114+0000","last_active":"2024-04-25T15:55:01.803519+0000","last_peered":"2024-04-25T15:55:01.803519+0000","last_clean":"2024-04-25T15:55:01.803519+0000","last_became_active":"2024-04-25T15:54:44.215121+0000","last_became_peered":"2024-04-25T15:54:44.215121+0000","last_unstale":"2024-04-25T15:55:01.803519+0000","last_undegraded":"2024-04-25T15:55:01.803519+0000","last_fullsized":"2024-04-25T15:55:01.803519+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:37.441111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'26","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341499+0000","last_change":"2024-04-25T15:54:51.320611+0000","last_active":"2024-04-25T15:54:54.341499+0000","last_peered":"2024-04-25T15:54:54.341499+0000","last_clean":"2024-04-25T15:54:54.341499+0000","last_became_active":"2024-04-25T15:54:51.307056+0000","last_became_peered":"2024-04-25T15:54:51.307056+0000","last_unstale":"2024-04-25T15:54:54.341499+0000","last_undegraded":"2024-04-25T15:54:54.341499+0000","last_fullsized":"2024-04-25T15:54:54.341499+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:15.769447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":13,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.822154+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:55:01.822154+0000","last_peered":"2024-04-25T15:55:01.822154+0000","last_clean":"2024-04-25T15:55:01.822154+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:55:01.822154+0000","last_undegraded":"2024-04-25T15:55:01.822154+0000","last_fullsized":"2024-04-25T15:55:01.822154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"110'8","reported_seq":242,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348406+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:54.348406+0000","last_peered":"2024-04-25T15:54:54.348406+0000","last_clean":"2024-04-25T15:54:54.348406+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:54.348406+0000","last_undegraded":"2024-04-25T15:54:54.348406+0000","last_fullsized":"2024-04-25T15:54:54.348406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"129'163","reported_seq":391,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.812671+0000","last_change":"2024-04-25T15:54:44.208533+0000","last_active":"2024-04-25T15:55:01.812671+0000","last_peered":"2024-04-25T15:55:01.812671+0000","last_clean":"2024-04-25T15:55:01.812671+0000","last_became_active":"2024-04-25T15:54:44.207033+0000","last_became_peered":"2024-04-25T15:54:44.207033+0000","last_unstale":"2024-04-25T15:55:01.812671+0000","last_undegraded":"2024-04-25T15:55:01.812671+0000","last_fullsized":"2024-04-25T15:55:01.812671+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:44.559711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":234,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630110+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:54.630110+0000","last_peered":"2024-04-25T15:54:54.630110+0000","last_clean":"2024-04-25T15:54:54.630110+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:54.630110+0000","last_undegraded":"2024-04-25T15:54:54.630110+0000","last_fullsized":"2024-04-25T15:54:54.630110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"129'134","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.751001+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:55:01.751001+0000","last_peered":"2024-04-25T15:55:01.751001+0000","last_clean":"2024-04-25T15:55:01.751001+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:55:01.751001+0000","last_undegraded":"2024-04-25T15:55:01.751001+0000","last_fullsized":"2024-04-25T15:55:01.751001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":206,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341314+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:54:54.341314+0000","last_peered":"2024-04-25T15:54:54.341314+0000","last_clean":"2024-04-25T15:54:54.341314+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:54:54.341314+0000","last_undegraded":"2024-04-25T15:54:54.341314+0000","last_fullsized":"2024-04-25T15:54:54.341314+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"129'153","reported_seq":521,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.811691+0000","last_change":"2024-04-25T15:54:44.207954+0000","last_active":"2024-04-25T15:55:01.811691+0000","last_peered":"2024-04-25T15:55:01.811691+0000","last_clean":"2024-04-25T15:55:01.811691+0000","last_became_active":"2024-04-25T15:54:44.206595+0000","last_became_peered":"2024-04-25T15:54:44.206595+0000","last_unstale":"2024-04-25T15:55:01.811691+0000","last_undegraded":"2024-04-25T15:55:01.811691+0000","last_fullsized":"2024-04-25T15:55:01.811691+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.107940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":69,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346110+0000","last_change":"2024-04-25T15:54:22.056875+0000","last_active":"2024-04-25T15:54:54.346110+0000","last_peered":"2024-04-25T15:54:54.346110+0000","last_clean":"2024-04-25T15:54:54.346110+0000","last_became_active":"2024-04-25T15:54:22.037756+0000","last_became_peered":"2024-04-25T15:54:22.037756+0000","last_unstale":"2024-04-25T15:54:54.346110+0000","last_undegraded":"2024-04-25T15:54:54.346110+0000","last_fullsized":"2024-04-25T15:54:54.346110+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:57.998015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"129'143","reported_seq":367,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.801641+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:55:01.801641+0000","last_peered":"2024-04-25T15:55:01.801641+0000","last_clean":"2024-04-25T15:55:01.801641+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:55:01.801641+0000","last_undegraded":"2024-04-25T15:55:01.801641+0000","last_fullsized":"2024-04-25T15:55:01.801641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":259,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631676+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:54.631676+0000","last_peered":"2024-04-25T15:54:54.631676+0000","last_clean":"2024-04-25T15:54:54.631676+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:54.631676+0000","last_undegraded":"2024-04-25T15:54:54.631676+0000","last_fullsized":"2024-04-25T15:54:54.631676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"129'128","reported_seq":386,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826678+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:55:01.826678+0000","last_peered":"2024-04-25T15:55:01.826678+0000","last_clean":"2024-04-25T15:55:01.826678+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:55:01.826678+0000","last_undegraded":"2024-04-25T15:55:01.826678+0000","last_fullsized":"2024-04-25T15:55:01.826678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"129'183","reported_seq":440,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.835785+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:55:01.835785+0000","last_peered":"2024-04-25T15:55:01.835785+0000","last_clean":"2024-04-25T15:55:01.835785+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:55:01.835785+0000","last_undegraded":"2024-04-25T15:55:01.835785+0000","last_fullsized":"2024-04-25T15:55:01.835785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631338+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:54.631338+0000","last_peered":"2024-04-25T15:54:54.631338+0000","last_clean":"2024-04-25T15:54:54.631338+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:54.631338+0000","last_undegraded":"2024-04-25T15:54:54.631338+0000","last_fullsized":"2024-04-25T15:54:54.631338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"129'141","reported_seq":415,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834610+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:55:01.834610+0000","last_peered":"2024-04-25T15:55:01.834610+0000","last_clean":"2024-04-25T15:55:01.834610+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:55:01.834610+0000","last_undegraded":"2024-04-25T15:55:01.834610+0000","last_fullsized":"2024-04-25T15:55:01.834610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":244,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631249+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:54.631249+0000","last_peered":"2024-04-25T15:54:54.631249+0000","last_clean":"2024-04-25T15:54:54.631249+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:54.631249+0000","last_undegraded":"2024-04-25T15:54:54.631249+0000","last_fullsized":"2024-04-25T15:54:54.631249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"129'135","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836536+0000","last_change":"2024-04-25T15:54:40.169300+0000","last_active":"2024-04-25T15:55:01.836536+0000","last_peered":"2024-04-25T15:55:01.836536+0000","last_clean":"2024-04-25T15:55:01.836536+0000","last_became_active":"2024-04-25T15:54:40.168164+0000","last_became_peered":"2024-04-25T15:54:40.168164+0000","last_unstale":"2024-04-25T15:55:01.836536+0000","last_undegraded":"2024-04-25T15:55:01.836536+0000","last_fullsized":"2024-04-25T15:55:01.836536+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:50.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351250+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:54.351250+0000","last_peered":"2024-04-25T15:54:54.351250+0000","last_clean":"2024-04-25T15:54:54.351250+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:54.351250+0000","last_undegraded":"2024-04-25T15:54:54.351250+0000","last_fullsized":"2024-04-25T15:54:54.351250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"129'152","reported_seq":598,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.832040+0000","last_change":"2024-04-25T15:54:54.360954+0000","last_active":"2024-04-25T15:55:01.832040+0000","last_peered":"2024-04-25T15:55:01.832040+0000","last_clean":"2024-04-25T15:55:01.832040+0000","last_became_active":"2024-04-25T15:54:54.360118+0000","last_became_peered":"2024-04-25T15:54:54.360118+0000","last_unstale":"2024-04-25T15:55:01.832040+0000","last_undegraded":"2024-04-25T15:55:01.832040+0000","last_fullsized":"2024-04-25T15:55:01.832040+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:37.822818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":103,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.353079+0000","last_change":"2024-04-25T15:54:40.170535+0000","last_active":"2024-04-25T15:54:54.353079+0000","last_peered":"2024-04-25T15:54:54.353079+0000","last_clean":"2024-04-25T15:54:54.353079+0000","last_became_active":"2024-04-25T15:54:40.169964+0000","last_became_peered":"2024-04-25T15:54:40.169964+0000","last_unstale":"2024-04-25T15:54:54.353079+0000","last_undegraded":"2024-04-25T15:54:54.353079+0000","last_fullsized":"2024-04-25T15:54:54.353079+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:24.688851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":1627389952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9717,"ondisk_log_size":9717,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":207952,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":347,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":389959,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":415904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59668,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312172,"statfs":{"total":96636764160,"available":96575664128,"internally_reserved":0,"allocated":2174976,"data_stored":1338205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41224,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1769472,"data_stored":945730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63260,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308580,"statfs":{"total":96636764160,"available":96571985920,"internally_reserved":0,"allocated":1593344,"data_stored":749669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1568768,"data_stored":747067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1777664,"data_stored":946269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1581056,"data_stored":747704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1572864,"data_stored":746687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2166784,"data_stored":1336397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":201190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:55:08.954 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T15:55:08.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:55:09.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bff4b2640 1 Processor -- start 2024-04-25T15:55:09.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bff4b2640 1 -- start start 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bff4b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6bf815c750 0x7f6bf8073a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bff4b2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bf81568a0 0x7f6bf8068590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bff4b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf80c8fa0 con 0x7f6bf806c640 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bff4b2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf815cb60 con 0x7f6bf81568a0 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bff4b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf80c9600 con 0x7f6bf815c750 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfca4d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bf81568a0 0x7f6bf8068590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfca4d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bf81568a0 0x7f6bf8068590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55200/0 (socket says 172.21.15.67:55200) 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfca4d640 1 -- 172.21.15.67:0/1800287023 learned_addr learned my addr 172.21.15.67:0/1800287023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfd24e640 1 --2- 172.21.15.67:0/1800287023 >> v2:172.21.15.176:3300/0 conn(0x7f6bf815c750 0x7f6bf8073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfca4d640 1 -- 172.21.15.67:0/1800287023 >> v2:172.21.15.176:3300/0 conn(0x7f6bf815c750 msgr2=0x7f6bf8073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfca4d640 1 --2- 172.21.15.67:0/1800287023 >> v2:172.21.15.176:3300/0 conn(0x7f6bf815c750 0x7f6bf8073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfca4d640 1 -- 172.21.15.67:0/1800287023 >> v1:172.21.15.176:6789/0 conn(0x7f6bf806c640 legacy=0x7f6bf815fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.149+0000 7f6bfca4d640 1 -- 172.21.15.67:0/1800287023 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf80c98e0 con 0x7f6bf81568a0 2024-04-25T15:55:09.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bfca4d640 1 --2- 172.21.15.67:0/1800287023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf8068590 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8072730 tx=0x7f6be801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b538483c2f44840 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6beffff640 1 -- 172.21.15.67:0/1800287023 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80b5070 con 0x7f6bf81568a0 2024-04-25T15:55:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6beffff640 1 -- 172.21.15.67:0/1800287023 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be80a89d0 con 0x7f6bf81568a0 2024-04-25T15:55:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6beffff640 1 -- 172.21.15.67:0/1800287023 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80a8da0 con 0x7f6bf81568a0 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 -- 172.21.15.67:0/1800287023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 msgr2=0x7f6bf8068590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/1800287023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf8068590 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8072730 tx=0x7f6be801ecd0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 -- 172.21.15.67:0/1800287023 shutdown_connections 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/1800287023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf8068590 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/1800287023 >> v2:172.21.15.176:3300/0 conn(0x7f6bf815c750 0x7f6bf8073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 -- 172.21.15.67:0/1800287023 >> 172.21.15.67:0/1800287023 conn(0x7f6bf8069220 msgr2=0x7f6bf80cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 -- 172.21.15.67:0/1800287023 shutdown_connections 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.153+0000 7f6bff4b2640 1 -- 172.21.15.67:0/1800287023 wait complete. 2024-04-25T15:55:09.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 Processor -- start 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 -- start start 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bf806c640 0x7f6bf80cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf81e4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bf815c750 0x7f6bf81e51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf8068ad0 con 0x7f6bf81568a0 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf8072490 con 0x7f6bf806c640 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf80c92b0 con 0x7f6bf815c750 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf81e4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf81e4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55212/0 (socket says 172.21.15.67:55212) 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 -- 172.21.15.67:0/3320528458 learned_addr learned my addr 172.21.15.67:0/3320528458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfd24e640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bf806c640 0x7f6bf80cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 -- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bf815c750 msgr2=0x7f6bf81e51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfda4f640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bf815c750 0x7f6bf81e51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bf815c750 0x7f6bf81e51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 -- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bf806c640 msgr2=0x7f6bf80cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bf806c640 0x7f6bf80cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 -- 172.21.15.67:0/3320528458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf80ce050 con 0x7f6bf81568a0 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfd24e640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bf806c640 0x7f6bf80cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bfca4d640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf81e4c60 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6be80b3040 tx=0x7f6be801ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bedffb640 1 -- 172.21.15.67:0/3320528458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80c0070 con 0x7f6bf81568a0 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bedffb640 1 -- 172.21.15.67:0/3320528458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be80b0a50 con 0x7f6bf81568a0 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf80cc180 con 0x7f6bf81568a0 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bedffb640 1 -- 172.21.15.67:0/3320528458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80b5070 con 0x7f6bf81568a0 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.157+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf80cc8a0 con 0x7f6bf81568a0 2024-04-25T15:55:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.161+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf806a480 con 0x7f6bf81568a0 2024-04-25T15:55:09.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.161+0000 7f6bedffb640 1 -- 172.21.15.67:0/3320528458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6be80c3070 con 0x7f6bf81568a0 2024-04-25T15:55:09.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.161+0000 7f6bedffb640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bdc06f400 0x7f6bdc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.161+0000 7f6bedffb640 1 -- 172.21.15.67:0/3320528458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f6be812c810 con 0x7f6bf81568a0 2024-04-25T15:55:09.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.161+0000 7f6bfd24e640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bdc06f400 0x7f6bdc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.161+0000 7f6bfd24e640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bdc06f400 0x7f6bdc071910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40097a0 tx=0x7f6bf4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:09.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.165+0000 7f6bedffb640 1 -- 172.21.15.67:0/3320528458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6be8018830 con 0x7f6bf81568a0 2024-04-25T15:55:09.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.325+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6bf80cbf10 con 0x7f6bdc06f400 2024-04-25T15:55:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.345+0000 7f6bedffb640 1 -- 172.21.15.67:0/3320528458 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253557 (secure 0 0 0) 0x7f6bf80cbf10 con 0x7f6bdc06f400 2024-04-25T15:55:09.348 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:55:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bdc06f400 msgr2=0x7f6bdc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bdc06f400 0x7f6bdc071910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40097a0 tx=0x7f6bf4008040 comp rx=0 tx=0).stop 2024-04-25T15:55:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 msgr2=0x7f6bf81e4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf81e4c60 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f6be80b3040 tx=0x7f6be801ba70 comp rx=0 tx=0).stop 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 shutdown_connections 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f6bdc06f400 0x7f6bdc071910 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6bf815c750 0x7f6bf81e51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bf81568a0 0x7f6bf81e4c60 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 --2- 172.21.15.67:0/3320528458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6bf806c640 0x7f6bf80cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 >> 172.21.15.67:0/3320528458 conn(0x7f6bf8069220 msgr2=0x7f6bf814ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 shutdown_connections 2024-04-25T15:55:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.349+0000 7f6bff4b2640 1 -- 172.21.15.67:0/3320528458 wait complete. 2024-04-25T15:55:09.372 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-04-25T15:55:07.894613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798320,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":9664,"num_write_kb":25167774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":1627779911,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10098,"ondisk_log_size":10098,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383984,"kb_used_data":13872,"kb_used_omap":12,"kb_used_meta":370035,"kb_avail":754590736,"statfs":{"total":773094113280,"available":772700913664,"internally_reserved":0,"allocated":14204928,"data_stored":7557728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":378916402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005128"},"pg_stats":[{"pgid":"3.3f","version":"129'146","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831956+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:55:01.831956+0000","last_peered":"2024-04-25T15:55:01.831956+0000","last_clean":"2024-04-25T15:55:01.831956+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:55:01.831956+0000","last_undegraded":"2024-04-25T15:55:01.831956+0000","last_fullsized":"2024-04-25T15:55:01.831956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"129'130","reported_seq":255,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.789093+0000","last_change":"2024-04-25T15:54:44.217073+0000","last_active":"2024-04-25T15:55:01.789093+0000","last_peered":"2024-04-25T15:55:01.789093+0000","last_clean":"2024-04-25T15:55:01.789093+0000","last_became_active":"2024-04-25T15:54:44.215261+0000","last_became_peered":"2024-04-25T15:54:44.215261+0000","last_unstale":"2024-04-25T15:55:01.789093+0000","last_undegraded":"2024-04-25T15:55:01.789093+0000","last_fullsized":"2024-04-25T15:55:01.789093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:47:01.929448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"129'151","reported_seq":343,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.811643+0000","last_change":"2024-04-25T15:54:40.166712+0000","last_active":"2024-04-25T15:55:01.811643+0000","last_peered":"2024-04-25T15:55:01.811643+0000","last_clean":"2024-04-25T15:55:01.811643+0000","last_became_active":"2024-04-25T15:54:40.164866+0000","last_became_peered":"2024-04-25T15:54:40.164866+0000","last_unstale":"2024-04-25T15:55:01.811643+0000","last_undegraded":"2024-04-25T15:55:01.811643+0000","last_fullsized":"2024-04-25T15:55:01.811643+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:23:58.780901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"129'173","reported_seq":405,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831009+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:55:01.831009+0000","last_peered":"2024-04-25T15:55:01.831009+0000","last_clean":"2024-04-25T15:55:01.831009+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:55:01.831009+0000","last_undegraded":"2024-04-25T15:55:01.831009+0000","last_fullsized":"2024-04-25T15:55:01.831009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"129'148","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"129'135","reported_seq":366,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830601+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:55:01.830601+0000","last_peered":"2024-04-25T15:55:01.830601+0000","last_clean":"2024-04-25T15:55:01.830601+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:55:01.830601+0000","last_undegraded":"2024-04-25T15:55:01.830601+0000","last_fullsized":"2024-04-25T15:55:01.830601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"129'164","reported_seq":393,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.804532+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:55:01.804532+0000","last_peered":"2024-04-25T15:55:01.804532+0000","last_clean":"2024-04-25T15:55:01.804532+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:55:01.804532+0000","last_undegraded":"2024-04-25T15:55:01.804532+0000","last_fullsized":"2024-04-25T15:55:01.804532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"129'186","reported_seq":417,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833169+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:55:01.833169+0000","last_peered":"2024-04-25T15:55:01.833169+0000","last_clean":"2024-04-25T15:55:01.833169+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:55:01.833169+0000","last_undegraded":"2024-04-25T15:55:01.833169+0000","last_fullsized":"2024-04-25T15:55:01.833169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"129'152","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836442+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:55:01.836442+0000","last_peered":"2024-04-25T15:55:01.836442+0000","last_clean":"2024-04-25T15:55:01.836442+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:55:01.836442+0000","last_undegraded":"2024-04-25T15:55:01.836442+0000","last_fullsized":"2024-04-25T15:55:01.836442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:23.039316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"129'156","reported_seq":458,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831117+0000","last_change":"2024-04-25T15:54:54.355672+0000","last_active":"2024-04-25T15:55:01.831117+0000","last_peered":"2024-04-25T15:55:01.831117+0000","last_clean":"2024-04-25T15:55:01.831117+0000","last_became_active":"2024-04-25T15:54:54.352223+0000","last_became_peered":"2024-04-25T15:54:54.352223+0000","last_unstale":"2024-04-25T15:55:01.831117+0000","last_undegraded":"2024-04-25T15:55:01.831117+0000","last_fullsized":"2024-04-25T15:55:01.831117+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:39.782280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"129'189","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830161+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:55:01.830161+0000","last_peered":"2024-04-25T15:55:01.830161+0000","last_clean":"2024-04-25T15:55:01.830161+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:55:01.830161+0000","last_undegraded":"2024-04-25T15:55:01.830161+0000","last_fullsized":"2024-04-25T15:55:01.830161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"129'142","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827413+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:55:01.827413+0000","last_peered":"2024-04-25T15:55:01.827413+0000","last_clean":"2024-04-25T15:55:01.827413+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:55:01.827413+0000","last_undegraded":"2024-04-25T15:55:01.827413+0000","last_fullsized":"2024-04-25T15:55:01.827413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"129'151","reported_seq":691,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833332+0000","last_change":"2024-04-25T15:54:54.357489+0000","last_active":"2024-04-25T15:55:01.833332+0000","last_peered":"2024-04-25T15:55:01.833332+0000","last_clean":"2024-04-25T15:55:01.833332+0000","last_became_active":"2024-04-25T15:54:54.356694+0000","last_became_peered":"2024-04-25T15:54:54.356694+0000","last_unstale":"2024-04-25T15:55:01.833332+0000","last_undegraded":"2024-04-25T15:55:01.833332+0000","last_fullsized":"2024-04-25T15:55:01.833332+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.653628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"129'149","reported_seq":387,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.813061+0000","last_change":"2024-04-25T15:54:40.160979+0000","last_active":"2024-04-25T15:55:01.813061+0000","last_peered":"2024-04-25T15:55:01.813061+0000","last_clean":"2024-04-25T15:55:01.813061+0000","last_became_active":"2024-04-25T15:54:40.159879+0000","last_became_peered":"2024-04-25T15:54:40.159879+0000","last_unstale":"2024-04-25T15:55:01.813061+0000","last_undegraded":"2024-04-25T15:55:01.813061+0000","last_fullsized":"2024-04-25T15:55:01.813061+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:25.499728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"129'151","reported_seq":270,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.808743+0000","last_change":"2024-04-25T15:54:44.208852+0000","last_active":"2024-04-25T15:55:01.808743+0000","last_peered":"2024-04-25T15:55:01.808743+0000","last_clean":"2024-04-25T15:55:01.808743+0000","last_became_active":"2024-04-25T15:54:44.207144+0000","last_became_peered":"2024-04-25T15:54:44.207144+0000","last_unstale":"2024-04-25T15:55:01.808743+0000","last_undegraded":"2024-04-25T15:55:01.808743+0000","last_fullsized":"2024-04-25T15:55:01.808743+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:44:02.181746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"129'147","reported_seq":378,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830760+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:55:01.830760+0000","last_peered":"2024-04-25T15:55:01.830760+0000","last_clean":"2024-04-25T15:55:01.830760+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:55:01.830760+0000","last_undegraded":"2024-04-25T15:55:01.830760+0000","last_fullsized":"2024-04-25T15:55:01.830760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"129'146","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833273+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:55:01.833273+0000","last_peered":"2024-04-25T15:55:01.833273+0000","last_clean":"2024-04-25T15:55:01.833273+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:55:01.833273+0000","last_undegraded":"2024-04-25T15:55:01.833273+0000","last_fullsized":"2024-04-25T15:55:01.833273+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"129'157","reported_seq":601,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.837202+0000","last_change":"2024-04-25T15:54:54.355989+0000","last_active":"2024-04-25T15:55:01.837202+0000","last_peered":"2024-04-25T15:55:01.837202+0000","last_clean":"2024-04-25T15:55:01.837202+0000","last_became_active":"2024-04-25T15:54:54.354849+0000","last_became_peered":"2024-04-25T15:54:54.354849+0000","last_unstale":"2024-04-25T15:55:01.837202+0000","last_undegraded":"2024-04-25T15:55:01.837202+0000","last_fullsized":"2024-04-25T15:55:01.837202+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:05.626413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"129'172","reported_seq":402,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.818938+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:55:01.818938+0000","last_peered":"2024-04-25T15:55:01.818938+0000","last_clean":"2024-04-25T15:55:01.818938+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:55:01.818938+0000","last_undegraded":"2024-04-25T15:55:01.818938+0000","last_fullsized":"2024-04-25T15:55:01.818938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"129'161","reported_seq":409,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.829727+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:55:01.829727+0000","last_peered":"2024-04-25T15:55:01.829727+0000","last_clean":"2024-04-25T15:55:01.829727+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:55:01.829727+0000","last_undegraded":"2024-04-25T15:55:01.829727+0000","last_fullsized":"2024-04-25T15:55:01.829727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"129'137","reported_seq":327,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827376+0000","last_change":"2024-04-25T15:54:40.169787+0000","last_active":"2024-04-25T15:55:01.827376+0000","last_peered":"2024-04-25T15:55:01.827376+0000","last_clean":"2024-04-25T15:55:01.827376+0000","last_became_active":"2024-04-25T15:54:40.167469+0000","last_became_peered":"2024-04-25T15:54:40.167469+0000","last_unstale":"2024-04-25T15:55:01.827376+0000","last_undegraded":"2024-04-25T15:55:01.827376+0000","last_fullsized":"2024-04-25T15:55:01.827376+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:26.375093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"129'166","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819478+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:55:01.819478+0000","last_peered":"2024-04-25T15:55:01.819478+0000","last_clean":"2024-04-25T15:55:01.819478+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:55:01.819478+0000","last_undegraded":"2024-04-25T15:55:01.819478+0000","last_fullsized":"2024-04-25T15:55:01.819478+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:36.303332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"129'134","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.815642+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:55:01.815642+0000","last_peered":"2024-04-25T15:55:01.815642+0000","last_clean":"2024-04-25T15:55:01.815642+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:55:01.815642+0000","last_undegraded":"2024-04-25T15:55:01.815642+0000","last_fullsized":"2024-04-25T15:55:01.815642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"129'180","reported_seq":412,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827531+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:55:01.827531+0000","last_peered":"2024-04-25T15:55:01.827531+0000","last_clean":"2024-04-25T15:55:01.827531+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:55:01.827531+0000","last_undegraded":"2024-04-25T15:55:01.827531+0000","last_fullsized":"2024-04-25T15:55:01.827531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"129'134","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819773+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:55:01.819773+0000","last_peered":"2024-04-25T15:55:01.819773+0000","last_clean":"2024-04-25T15:55:01.819773+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:55:01.819773+0000","last_undegraded":"2024-04-25T15:55:01.819773+0000","last_fullsized":"2024-04-25T15:55:01.819773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"129'198","reported_seq":430,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834987+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:55:01.834987+0000","last_peered":"2024-04-25T15:55:01.834987+0000","last_clean":"2024-04-25T15:55:01.834987+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:55:01.834987+0000","last_undegraded":"2024-04-25T15:55:01.834987+0000","last_fullsized":"2024-04-25T15:55:01.834987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"129'144","reported_seq":356,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.805991+0000","last_change":"2024-04-25T15:54:44.212386+0000","last_active":"2024-04-25T15:55:01.805991+0000","last_peered":"2024-04-25T15:55:01.805991+0000","last_clean":"2024-04-25T15:55:01.805991+0000","last_became_active":"2024-04-25T15:54:44.210520+0000","last_became_peered":"2024-04-25T15:54:44.210520+0000","last_unstale":"2024-04-25T15:55:01.805991+0000","last_undegraded":"2024-04-25T15:55:01.805991+0000","last_fullsized":"2024-04-25T15:55:01.805991+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:01.061276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"129'133","reported_seq":400,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"129'118","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.813169+0000","last_change":"2024-04-25T15:54:40.174462+0000","last_active":"2024-04-25T15:55:01.813169+0000","last_peered":"2024-04-25T15:55:01.813169+0000","last_clean":"2024-04-25T15:55:01.813169+0000","last_became_active":"2024-04-25T15:54:40.173209+0000","last_became_peered":"2024-04-25T15:54:40.173209+0000","last_unstale":"2024-04-25T15:55:01.813169+0000","last_undegraded":"2024-04-25T15:55:01.813169+0000","last_fullsized":"2024-04-25T15:55:01.813169+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:08.410467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"129'131","reported_seq":307,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.789509+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:55:01.789509+0000","last_peered":"2024-04-25T15:55:01.789509+0000","last_clean":"2024-04-25T15:55:01.789509+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:55:01.789509+0000","last_undegraded":"2024-04-25T15:55:01.789509+0000","last_fullsized":"2024-04-25T15:55:01.789509+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:01.445034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"129'135","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836486+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:55:01.836486+0000","last_peered":"2024-04-25T15:55:01.836486+0000","last_clean":"2024-04-25T15:55:01.836486+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:55:01.836486+0000","last_undegraded":"2024-04-25T15:55:01.836486+0000","last_fullsized":"2024-04-25T15:55:01.836486+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:25.100827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"129'150","reported_seq":479,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826757+0000","last_change":"2024-04-25T15:54:54.359681+0000","last_active":"2024-04-25T15:55:01.826757+0000","last_peered":"2024-04-25T15:55:01.826757+0000","last_clean":"2024-04-25T15:55:01.826757+0000","last_became_active":"2024-04-25T15:54:54.353824+0000","last_became_peered":"2024-04-25T15:54:54.353824+0000","last_unstale":"2024-04-25T15:55:01.826757+0000","last_undegraded":"2024-04-25T15:55:01.826757+0000","last_fullsized":"2024-04-25T15:55:01.826757+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:52.297640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"129'130","reported_seq":246,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816679+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:55:01.816679+0000","last_peered":"2024-04-25T15:55:01.816679+0000","last_clean":"2024-04-25T15:55:01.816679+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:55:01.816679+0000","last_undegraded":"2024-04-25T15:55:01.816679+0000","last_fullsized":"2024-04-25T15:55:01.816679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"129'7","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351493+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:54.351493+0000","last_peered":"2024-04-25T15:54:54.351493+0000","last_clean":"2024-04-25T15:54:54.351493+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:54.351493+0000","last_undegraded":"2024-04-25T15:54:54.351493+0000","last_fullsized":"2024-04-25T15:54:54.351493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"129'137","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.815609+0000","last_change":"2024-04-25T15:54:40.163745+0000","last_active":"2024-04-25T15:55:01.815609+0000","last_peered":"2024-04-25T15:55:01.815609+0000","last_clean":"2024-04-25T15:55:01.815609+0000","last_became_active":"2024-04-25T15:54:40.162667+0000","last_became_peered":"2024-04-25T15:54:40.162667+0000","last_unstale":"2024-04-25T15:55:01.815609+0000","last_undegraded":"2024-04-25T15:55:01.815609+0000","last_fullsized":"2024-04-25T15:55:01.815609+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:22:17.528981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347310+0000","last_change":"2024-04-25T15:54:51.317865+0000","last_active":"2024-04-25T15:54:54.347310+0000","last_peered":"2024-04-25T15:54:54.347310+0000","last_clean":"2024-04-25T15:54:54.347310+0000","last_became_active":"2024-04-25T15:54:51.302422+0000","last_became_peered":"2024-04-25T15:54:51.302422+0000","last_unstale":"2024-04-25T15:54:54.347310+0000","last_undegraded":"2024-04-25T15:54:54.347310+0000","last_fullsized":"2024-04-25T15:54:54.347310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:36.956598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350037+0000","last_change":"2024-04-25T15:54:51.329012+0000","last_active":"2024-04-25T15:54:54.350037+0000","last_peered":"2024-04-25T15:54:54.350037+0000","last_clean":"2024-04-25T15:54:54.350037+0000","last_became_active":"2024-04-25T15:54:51.307662+0000","last_became_peered":"2024-04-25T15:54:51.307662+0000","last_unstale":"2024-04-25T15:54:54.350037+0000","last_undegraded":"2024-04-25T15:54:54.350037+0000","last_fullsized":"2024-04-25T15:54:54.350037+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:41.566474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"129'170","reported_seq":296,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.804884+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:55:01.804884+0000","last_peered":"2024-04-25T15:55:01.804884+0000","last_clean":"2024-04-25T15:55:01.804884+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:55:01.804884+0000","last_undegraded":"2024-04-25T15:55:01.804884+0000","last_fullsized":"2024-04-25T15:55:01.804884+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:07:19.358779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345518+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:54.345518+0000","last_peered":"2024-04-25T15:54:54.345518+0000","last_clean":"2024-04-25T15:54:54.345518+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:54.345518+0000","last_undegraded":"2024-04-25T15:54:54.345518+0000","last_fullsized":"2024-04-25T15:54:54.345518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"129'157","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816623+0000","last_change":"2024-04-25T15:54:44.212095+0000","last_active":"2024-04-25T15:55:01.816623+0000","last_peered":"2024-04-25T15:55:01.816623+0000","last_clean":"2024-04-25T15:55:01.816623+0000","last_became_active":"2024-04-25T15:54:44.209985+0000","last_became_peered":"2024-04-25T15:54:44.209985+0000","last_unstale":"2024-04-25T15:55:01.816623+0000","last_undegraded":"2024-04-25T15:55:01.816623+0000","last_fullsized":"2024-04-25T15:55:01.816623+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:35.906277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"129'8","reported_seq":241,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740819+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:55:01.740819+0000","last_peered":"2024-04-25T15:55:01.740819+0000","last_clean":"2024-04-25T15:55:01.740819+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:55:01.740819+0000","last_undegraded":"2024-04-25T15:55:01.740819+0000","last_fullsized":"2024-04-25T15:55:01.740819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"129'229","reported_seq":460,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834138+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:55:01.834138+0000","last_peered":"2024-04-25T15:55:01.834138+0000","last_clean":"2024-04-25T15:55:01.834138+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:55:01.834138+0000","last_undegraded":"2024-04-25T15:55:01.834138+0000","last_fullsized":"2024-04-25T15:55:01.834138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346366+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:54.346366+0000","last_peered":"2024-04-25T15:54:54.346366+0000","last_clean":"2024-04-25T15:54:54.346366+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:54.346366+0000","last_undegraded":"2024-04-25T15:54:54.346366+0000","last_fullsized":"2024-04-25T15:54:54.346366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"129'141","reported_seq":334,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833434+0000","last_change":"2024-04-25T15:54:40.169299+0000","last_active":"2024-04-25T15:55:01.833434+0000","last_peered":"2024-04-25T15:55:01.833434+0000","last_clean":"2024-04-25T15:55:01.833434+0000","last_became_active":"2024-04-25T15:54:40.168203+0000","last_became_peered":"2024-04-25T15:54:40.168203+0000","last_unstale":"2024-04-25T15:55:01.833434+0000","last_undegraded":"2024-04-25T15:55:01.833434+0000","last_fullsized":"2024-04-25T15:55:01.833434+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:53.710379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"114'11","reported_seq":46,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342159+0000","last_change":"2024-04-25T15:54:41.197909+0000","last_active":"2024-04-25T15:54:54.342159+0000","last_peered":"2024-04-25T15:54:54.342159+0000","last_clean":"2024-04-25T15:54:54.342159+0000","last_became_active":"2024-04-25T15:54:41.171190+0000","last_became_peered":"2024-04-25T15:54:41.171190+0000","last_unstale":"2024-04-25T15:54:54.342159+0000","last_undegraded":"2024-04-25T15:54:54.342159+0000","last_fullsized":"2024-04-25T15:54:54.342159+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:40.822327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2016,"num_keys_recovered":30,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"129'149","reported_seq":265,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825846+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:55:01.825846+0000","last_peered":"2024-04-25T15:55:01.825846+0000","last_clean":"2024-04-25T15:55:01.825846+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:55:01.825846+0000","last_undegraded":"2024-04-25T15:55:01.825846+0000","last_fullsized":"2024-04-25T15:55:01.825846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"129'93","reported_seq":33,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740355+0000","last_change":"2024-04-25T15:54:51.358160+0000","last_active":"2024-04-25T15:55:01.740355+0000","last_peered":"2024-04-25T15:55:01.740355+0000","last_clean":"2024-04-25T15:55:01.740355+0000","last_became_active":"2024-04-25T15:54:51.303172+0000","last_became_peered":"2024-04-25T15:54:51.303172+0000","last_unstale":"2024-04-25T15:55:01.740355+0000","last_undegraded":"2024-04-25T15:55:01.740355+0000","last_fullsized":"2024-04-25T15:55:01.740355+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:39.823475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199279,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":380717,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"129'150","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819314+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:55:01.819314+0000","last_peered":"2024-04-25T15:55:01.819314+0000","last_clean":"2024-04-25T15:55:01.819314+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:55:01.819314+0000","last_undegraded":"2024-04-25T15:55:01.819314+0000","last_fullsized":"2024-04-25T15:55:01.819314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352562+0000","last_change":"2024-04-25T15:54:40.166458+0000","last_active":"2024-04-25T15:54:54.352562+0000","last_peered":"2024-04-25T15:54:54.352562+0000","last_clean":"2024-04-25T15:54:54.352562+0000","last_became_active":"2024-04-25T15:54:40.165701+0000","last_became_peered":"2024-04-25T15:54:40.165701+0000","last_unstale":"2024-04-25T15:54:54.352562+0000","last_undegraded":"2024-04-25T15:54:54.352562+0000","last_fullsized":"2024-04-25T15:54:54.352562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:43:18.998536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"129'157","reported_seq":348,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827452+0000","last_change":"2024-04-25T15:54:40.168672+0000","last_active":"2024-04-25T15:55:01.827452+0000","last_peered":"2024-04-25T15:55:01.827452+0000","last_clean":"2024-04-25T15:55:01.827452+0000","last_became_active":"2024-04-25T15:54:40.167338+0000","last_became_peered":"2024-04-25T15:54:40.167338+0000","last_unstale":"2024-04-25T15:55:01.827452+0000","last_undegraded":"2024-04-25T15:55:01.827452+0000","last_fullsized":"2024-04-25T15:55:01.827452+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:13:04.063276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344550+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:54.344550+0000","last_peered":"2024-04-25T15:54:54.344550+0000","last_clean":"2024-04-25T15:54:54.344550+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:54.344550+0000","last_undegraded":"2024-04-25T15:54:54.344550+0000","last_fullsized":"2024-04-25T15:54:54.344550+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"129'171","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828601+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:55:01.828601+0000","last_peered":"2024-04-25T15:55:01.828601+0000","last_clean":"2024-04-25T15:55:01.828601+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:55:01.828601+0000","last_undegraded":"2024-04-25T15:55:01.828601+0000","last_fullsized":"2024-04-25T15:55:01.828601+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"129'7","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347111+0000","last_change":"2024-04-25T15:54:51.333252+0000","last_active":"2024-04-25T15:54:54.347111+0000","last_peered":"2024-04-25T15:54:54.347111+0000","last_clean":"2024-04-25T15:54:54.347111+0000","last_became_active":"2024-04-25T15:54:51.302160+0000","last_became_peered":"2024-04-25T15:54:51.302160+0000","last_unstale":"2024-04-25T15:54:54.347111+0000","last_undegraded":"2024-04-25T15:54:54.347111+0000","last_fullsized":"2024-04-25T15:54:54.347111+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:25.724351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4116,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"129'165","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.824551+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:55:01.824551+0000","last_peered":"2024-04-25T15:55:01.824551+0000","last_clean":"2024-04-25T15:55:01.824551+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:55:01.824551+0000","last_undegraded":"2024-04-25T15:55:01.824551+0000","last_fullsized":"2024-04-25T15:55:01.824551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"114'18","reported_seq":342,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345636+0000","last_change":"2024-04-25T15:54:41.193381+0000","last_active":"2024-04-25T15:54:54.345636+0000","last_peered":"2024-04-25T15:54:54.345636+0000","last_clean":"2024-04-25T15:54:54.345636+0000","last_became_active":"2024-04-25T15:54:41.166653+0000","last_became_peered":"2024-04-25T15:54:41.166653+0000","last_unstale":"2024-04-25T15:54:54.345636+0000","last_undegraded":"2024-04-25T15:54:54.345636+0000","last_fullsized":"2024-04-25T15:54:54.345636+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:07:10.229064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":448,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"129'189","reported_seq":446,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.820824+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:55:01.820824+0000","last_peered":"2024-04-25T15:55:01.820824+0000","last_clean":"2024-04-25T15:55:01.820824+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:55:01.820824+0000","last_undegraded":"2024-04-25T15:55:01.820824+0000","last_fullsized":"2024-04-25T15:55:01.820824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":120,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341794+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:54.341794+0000","last_peered":"2024-04-25T15:54:54.341794+0000","last_clean":"2024-04-25T15:54:54.341794+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:54.341794+0000","last_undegraded":"2024-04-25T15:54:54.341794+0000","last_fullsized":"2024-04-25T15:54:54.341794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"129'146","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825150+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:55:01.825150+0000","last_peered":"2024-04-25T15:55:01.825150+0000","last_clean":"2024-04-25T15:55:01.825150+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:55:01.825150+0000","last_undegraded":"2024-04-25T15:55:01.825150+0000","last_fullsized":"2024-04-25T15:55:01.825150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346353+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:54:54.346353+0000","last_peered":"2024-04-25T15:54:54.346353+0000","last_clean":"2024-04-25T15:54:54.346353+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:54:54.346353+0000","last_undegraded":"2024-04-25T15:54:54.346353+0000","last_fullsized":"2024-04-25T15:54:54.346353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:22.021000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"129'138","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828396+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:55:01.828396+0000","last_peered":"2024-04-25T15:55:01.828396+0000","last_clean":"2024-04-25T15:55:01.828396+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:55:01.828396+0000","last_undegraded":"2024-04-25T15:55:01.828396+0000","last_fullsized":"2024-04-25T15:55:01.828396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":40,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342273+0000","last_change":"2024-04-25T15:54:41.186336+0000","last_active":"2024-04-25T15:54:54.342273+0000","last_peered":"2024-04-25T15:54:54.342273+0000","last_clean":"2024-04-25T15:54:54.342273+0000","last_became_active":"2024-04-25T15:54:41.165683+0000","last_became_peered":"2024-04-25T15:54:41.165683+0000","last_unstale":"2024-04-25T15:54:54.342273+0000","last_undegraded":"2024-04-25T15:54:54.342273+0000","last_fullsized":"2024-04-25T15:54:54.342273+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:54.115834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"129'144","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830924+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:55:01.830924+0000","last_peered":"2024-04-25T15:55:01.830924+0000","last_clean":"2024-04-25T15:55:01.830924+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:55:01.830924+0000","last_undegraded":"2024-04-25T15:55:01.830924+0000","last_fullsized":"2024-04-25T15:55:01.830924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"129'7","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.750762+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:55:01.750762+0000","last_peered":"2024-04-25T15:55:01.750762+0000","last_clean":"2024-04-25T15:55:01.750762+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:55:01.750762+0000","last_undegraded":"2024-04-25T15:55:01.750762+0000","last_fullsized":"2024-04-25T15:55:01.750762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:44.029919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"129'151","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:02.222181+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:55:02.222181+0000","last_peered":"2024-04-25T15:55:02.222181+0000","last_clean":"2024-04-25T15:55:02.222181+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:55:02.222181+0000","last_undegraded":"2024-04-25T15:55:02.222181+0000","last_fullsized":"2024-04-25T15:55:02.222181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630501+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:54.630501+0000","last_peered":"2024-04-25T15:54:54.630501+0000","last_clean":"2024-04-25T15:54:54.630501+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:54.630501+0000","last_undegraded":"2024-04-25T15:54:54.630501+0000","last_fullsized":"2024-04-25T15:54:54.630501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344831+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:54.344831+0000","last_peered":"2024-04-25T15:54:54.344831+0000","last_clean":"2024-04-25T15:54:54.344831+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:54.344831+0000","last_undegraded":"2024-04-25T15:54:54.344831+0000","last_fullsized":"2024-04-25T15:54:54.344831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"129'158","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.817195+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:55:01.817195+0000","last_peered":"2024-04-25T15:55:01.817195+0000","last_clean":"2024-04-25T15:55:01.817195+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:55:01.817195+0000","last_undegraded":"2024-04-25T15:55:01.817195+0000","last_fullsized":"2024-04-25T15:55:01.817195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":272,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630140+0000","last_change":"2024-04-25T15:54:41.173406+0000","last_active":"2024-04-25T15:54:54.630140+0000","last_peered":"2024-04-25T15:54:54.630140+0000","last_clean":"2024-04-25T15:54:54.630140+0000","last_became_active":"2024-04-25T15:54:41.166720+0000","last_became_peered":"2024-04-25T15:54:41.166720+0000","last_unstale":"2024-04-25T15:54:54.630140+0000","last_undegraded":"2024-04-25T15:54:54.630140+0000","last_fullsized":"2024-04-25T15:54:54.630140+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:41.861800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"129'131","reported_seq":252,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.820767+0000","last_change":"2024-04-25T15:54:40.164755+0000","last_active":"2024-04-25T15:55:01.820767+0000","last_peered":"2024-04-25T15:55:01.820767+0000","last_clean":"2024-04-25T15:55:01.820767+0000","last_became_active":"2024-04-25T15:54:40.163587+0000","last_became_peered":"2024-04-25T15:54:40.163587+0000","last_unstale":"2024-04-25T15:55:01.820767+0000","last_undegraded":"2024-04-25T15:55:01.820767+0000","last_fullsized":"2024-04-25T15:55:01.820767+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:58.420382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"129'13","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346047+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:54:54.346047+0000","last_peered":"2024-04-25T15:54:54.346047+0000","last_clean":"2024-04-25T15:54:54.346047+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:54:54.346047+0000","last_undegraded":"2024-04-25T15:54:54.346047+0000","last_fullsized":"2024-04-25T15:54:54.346047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:50:12.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"129'131","reported_seq":431,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.793091+0000","last_change":"2024-04-25T15:54:44.216921+0000","last_active":"2024-04-25T15:55:01.793091+0000","last_peered":"2024-04-25T15:55:01.793091+0000","last_clean":"2024-04-25T15:55:01.793091+0000","last_became_active":"2024-04-25T15:54:44.215186+0000","last_became_peered":"2024-04-25T15:54:44.215186+0000","last_unstale":"2024-04-25T15:55:01.793091+0000","last_undegraded":"2024-04-25T15:55:01.793091+0000","last_fullsized":"2024-04-25T15:55:01.793091+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:37:49.565745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344849+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:54.344849+0000","last_peered":"2024-04-25T15:54:54.344849+0000","last_clean":"2024-04-25T15:54:54.344849+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:54.344849+0000","last_undegraded":"2024-04-25T15:54:54.344849+0000","last_fullsized":"2024-04-25T15:54:54.344849+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"129'145","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.787876+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:55:01.787876+0000","last_peered":"2024-04-25T15:55:01.787876+0000","last_clean":"2024-04-25T15:55:01.787876+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:55:01.787876+0000","last_undegraded":"2024-04-25T15:55:01.787876+0000","last_fullsized":"2024-04-25T15:55:01.787876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":36,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341899+0000","last_change":"2024-04-25T15:54:41.168286+0000","last_active":"2024-04-25T15:54:54.341899+0000","last_peered":"2024-04-25T15:54:54.341899+0000","last_clean":"2024-04-25T15:54:54.341899+0000","last_became_active":"2024-04-25T15:54:41.166643+0000","last_became_peered":"2024-04-25T15:54:41.166643+0000","last_unstale":"2024-04-25T15:54:54.341899+0000","last_undegraded":"2024-04-25T15:54:54.341899+0000","last_fullsized":"2024-04-25T15:54:54.341899+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:21:08.426377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828419+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:55:01.828419+0000","last_peered":"2024-04-25T15:55:01.828419+0000","last_clean":"2024-04-25T15:55:01.828419+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:55:01.828419+0000","last_undegraded":"2024-04-25T15:55:01.828419+0000","last_fullsized":"2024-04-25T15:55:01.828419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631647+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:54.631647+0000","last_peered":"2024-04-25T15:54:54.631647+0000","last_clean":"2024-04-25T15:54:54.631647+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:54.631647+0000","last_undegraded":"2024-04-25T15:54:54.631647+0000","last_fullsized":"2024-04-25T15:54:54.631647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"129'144","reported_seq":344,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.803519+0000","last_change":"2024-04-25T15:54:44.217114+0000","last_active":"2024-04-25T15:55:01.803519+0000","last_peered":"2024-04-25T15:55:01.803519+0000","last_clean":"2024-04-25T15:55:01.803519+0000","last_became_active":"2024-04-25T15:54:44.215121+0000","last_became_peered":"2024-04-25T15:54:44.215121+0000","last_unstale":"2024-04-25T15:55:01.803519+0000","last_undegraded":"2024-04-25T15:55:01.803519+0000","last_fullsized":"2024-04-25T15:55:01.803519+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:30:37.441111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'26","reported_seq":61,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341499+0000","last_change":"2024-04-25T15:54:51.320611+0000","last_active":"2024-04-25T15:54:54.341499+0000","last_peered":"2024-04-25T15:54:54.341499+0000","last_clean":"2024-04-25T15:54:54.341499+0000","last_became_active":"2024-04-25T15:54:51.307056+0000","last_became_peered":"2024-04-25T15:54:51.307056+0000","last_unstale":"2024-04-25T15:54:54.341499+0000","last_undegraded":"2024-04-25T15:54:54.341499+0000","last_fullsized":"2024-04-25T15:54:54.341499+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:15.769447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":13,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.822154+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:55:01.822154+0000","last_peered":"2024-04-25T15:55:01.822154+0000","last_clean":"2024-04-25T15:55:01.822154+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:55:01.822154+0000","last_undegraded":"2024-04-25T15:55:01.822154+0000","last_fullsized":"2024-04-25T15:55:01.822154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"110'8","reported_seq":242,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348406+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:54.348406+0000","last_peered":"2024-04-25T15:54:54.348406+0000","last_clean":"2024-04-25T15:54:54.348406+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:54.348406+0000","last_undegraded":"2024-04-25T15:54:54.348406+0000","last_fullsized":"2024-04-25T15:54:54.348406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"129'163","reported_seq":391,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.812671+0000","last_change":"2024-04-25T15:54:44.208533+0000","last_active":"2024-04-25T15:55:01.812671+0000","last_peered":"2024-04-25T15:55:01.812671+0000","last_clean":"2024-04-25T15:55:01.812671+0000","last_became_active":"2024-04-25T15:54:44.207033+0000","last_became_peered":"2024-04-25T15:54:44.207033+0000","last_unstale":"2024-04-25T15:55:01.812671+0000","last_undegraded":"2024-04-25T15:55:01.812671+0000","last_fullsized":"2024-04-25T15:55:01.812671+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:44.559711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":234,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630110+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:54.630110+0000","last_peered":"2024-04-25T15:54:54.630110+0000","last_clean":"2024-04-25T15:54:54.630110+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:54.630110+0000","last_undegraded":"2024-04-25T15:54:54.630110+0000","last_fullsized":"2024-04-25T15:54:54.630110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"129'134","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.751001+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:55:01.751001+0000","last_peered":"2024-04-25T15:55:01.751001+0000","last_clean":"2024-04-25T15:55:01.751001+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:55:01.751001+0000","last_undegraded":"2024-04-25T15:55:01.751001+0000","last_fullsized":"2024-04-25T15:55:01.751001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":206,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341314+0000","last_change":"2024-04-25T15:51:47.281909+0000","last_active":"2024-04-25T15:54:54.341314+0000","last_peered":"2024-04-25T15:54:54.341314+0000","last_clean":"2024-04-25T15:54:54.341314+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:54:54.341314+0000","last_undegraded":"2024-04-25T15:54:54.341314+0000","last_fullsized":"2024-04-25T15:54:54.341314+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:35:14.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"129'153","reported_seq":521,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.811691+0000","last_change":"2024-04-25T15:54:44.207954+0000","last_active":"2024-04-25T15:55:01.811691+0000","last_peered":"2024-04-25T15:55:01.811691+0000","last_clean":"2024-04-25T15:55:01.811691+0000","last_became_active":"2024-04-25T15:54:44.206595+0000","last_became_peered":"2024-04-25T15:54:44.206595+0000","last_unstale":"2024-04-25T15:55:01.811691+0000","last_undegraded":"2024-04-25T15:55:01.811691+0000","last_fullsized":"2024-04-25T15:55:01.811691+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.107940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":69,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346110+0000","last_change":"2024-04-25T15:54:22.056875+0000","last_active":"2024-04-25T15:54:54.346110+0000","last_peered":"2024-04-25T15:54:54.346110+0000","last_clean":"2024-04-25T15:54:54.346110+0000","last_became_active":"2024-04-25T15:54:22.037756+0000","last_became_peered":"2024-04-25T15:54:22.037756+0000","last_unstale":"2024-04-25T15:54:54.346110+0000","last_undegraded":"2024-04-25T15:54:54.346110+0000","last_fullsized":"2024-04-25T15:54:54.346110+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:13:57.998015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"129'143","reported_seq":367,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.801641+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:55:01.801641+0000","last_peered":"2024-04-25T15:55:01.801641+0000","last_clean":"2024-04-25T15:55:01.801641+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:55:01.801641+0000","last_undegraded":"2024-04-25T15:55:01.801641+0000","last_fullsized":"2024-04-25T15:55:01.801641+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:51.872431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":259,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631676+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:54.631676+0000","last_peered":"2024-04-25T15:54:54.631676+0000","last_clean":"2024-04-25T15:54:54.631676+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:54.631676+0000","last_undegraded":"2024-04-25T15:54:54.631676+0000","last_fullsized":"2024-04-25T15:54:54.631676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"129'128","reported_seq":386,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826678+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:55:01.826678+0000","last_peered":"2024-04-25T15:55:01.826678+0000","last_clean":"2024-04-25T15:55:01.826678+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:55:01.826678+0000","last_undegraded":"2024-04-25T15:55:01.826678+0000","last_fullsized":"2024-04-25T15:55:01.826678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"129'183","reported_seq":440,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.835785+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:55:01.835785+0000","last_peered":"2024-04-25T15:55:01.835785+0000","last_clean":"2024-04-25T15:55:01.835785+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:55:01.835785+0000","last_undegraded":"2024-04-25T15:55:01.835785+0000","last_fullsized":"2024-04-25T15:55:01.835785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631338+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:54.631338+0000","last_peered":"2024-04-25T15:54:54.631338+0000","last_clean":"2024-04-25T15:54:54.631338+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:54.631338+0000","last_undegraded":"2024-04-25T15:54:54.631338+0000","last_fullsized":"2024-04-25T15:54:54.631338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"129'141","reported_seq":415,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834610+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:55:01.834610+0000","last_peered":"2024-04-25T15:55:01.834610+0000","last_clean":"2024-04-25T15:55:01.834610+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:55:01.834610+0000","last_undegraded":"2024-04-25T15:55:01.834610+0000","last_fullsized":"2024-04-25T15:55:01.834610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":244,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631249+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:54.631249+0000","last_peered":"2024-04-25T15:54:54.631249+0000","last_clean":"2024-04-25T15:54:54.631249+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:54.631249+0000","last_undegraded":"2024-04-25T15:54:54.631249+0000","last_fullsized":"2024-04-25T15:54:54.631249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"129'135","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836536+0000","last_change":"2024-04-25T15:54:40.169300+0000","last_active":"2024-04-25T15:55:01.836536+0000","last_peered":"2024-04-25T15:55:01.836536+0000","last_clean":"2024-04-25T15:55:01.836536+0000","last_became_active":"2024-04-25T15:54:40.168164+0000","last_became_peered":"2024-04-25T15:54:40.168164+0000","last_unstale":"2024-04-25T15:55:01.836536+0000","last_undegraded":"2024-04-25T15:55:01.836536+0000","last_fullsized":"2024-04-25T15:55:01.836536+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:50.627407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351250+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:54.351250+0000","last_peered":"2024-04-25T15:54:54.351250+0000","last_clean":"2024-04-25T15:54:54.351250+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:54.351250+0000","last_undegraded":"2024-04-25T15:54:54.351250+0000","last_fullsized":"2024-04-25T15:54:54.351250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"129'152","reported_seq":598,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.832040+0000","last_change":"2024-04-25T15:54:54.360954+0000","last_active":"2024-04-25T15:55:01.832040+0000","last_peered":"2024-04-25T15:55:01.832040+0000","last_clean":"2024-04-25T15:55:01.832040+0000","last_became_active":"2024-04-25T15:54:54.360118+0000","last_became_peered":"2024-04-25T15:54:54.360118+0000","last_unstale":"2024-04-25T15:55:01.832040+0000","last_undegraded":"2024-04-25T15:55:01.832040+0000","last_fullsized":"2024-04-25T15:55:01.832040+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:37.822818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":103,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.353079+0000","last_change":"2024-04-25T15:54:40.170535+0000","last_active":"2024-04-25T15:54:54.353079+0000","last_peered":"2024-04-25T15:54:54.353079+0000","last_clean":"2024-04-25T15:54:54.353079+0000","last_became_active":"2024-04-25T15:54:40.169964+0000","last_became_peered":"2024-04-25T15:54:40.169964+0000","last_unstale":"2024-04-25T15:54:54.353079+0000","last_undegraded":"2024-04-25T15:54:54.353079+0000","last_fullsized":"2024-04-25T15:54:54.353079+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:58:24.688851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":1627389952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9717,"ondisk_log_size":9717,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":207952,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":347,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":389959,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":415904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574912,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59668,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312172,"statfs":{"total":96636764160,"available":96575664128,"internally_reserved":0,"allocated":2174976,"data_stored":1338205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41224,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1769472,"data_stored":945730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63260,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308580,"statfs":{"total":96636764160,"available":96571985920,"internally_reserved":0,"allocated":1593344,"data_stored":749669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1568768,"data_stored":747067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1777664,"data_stored":946269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1581056,"data_stored":747704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1572864,"data_stored":746687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2166784,"data_stored":1336397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":201190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:55:09.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:55:09.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.577+0000 7f9f38ac5640 1 Processor -- start 2024-04-25T15:55:09.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.577+0000 7f9f38ac5640 1 -- start start 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f38ac5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f3415a720 0x7f9f3415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f38ac5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f340ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f38ac5640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f3415de10 con 0x7f9f34071bc0 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f38ac5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f9f3415ebb0 con 0x7f9f3415a720 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f38ac5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3415e720 con 0x7f9f340ca540 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f31d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f3415a720 0x7f9f3415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f31d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f3415a720 0x7f9f3415ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55220/0 (socket says 172.21.15.67:55220) 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f31d9b640 1 -- 172.21.15.67:0/2252672168 learned_addr learned my addr 172.21.15.67:0/2252672168 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f32d9d640 1 --2- 172.21.15.67:0/2252672168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f340ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f31d9b640 1 -- 172.21.15.67:0/2252672168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 msgr2=0x7f9f340ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:09.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f31d9b640 1 --2- 172.21.15.67:0/2252672168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f340ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f31d9b640 1 -- 172.21.15.67:0/2252672168 >> v1:172.21.15.67:6789/0 conn(0x7f9f34071bc0 legacy=0x7f9f340cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.581+0000 7f9f31d9b640 1 -- 172.21.15.67:0/2252672168 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f34072390 con 0x7f9f3415a720 2024-04-25T15:55:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f31d9b640 1 --2- 172.21.15.67:0/2252672168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415ab30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009e50 tx=0x7f9f1c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35156a25cd82dabc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f3159a640 1 -- 172.21.15.67:0/2252672168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c0a5810 con 0x7f9f3415a720 2024-04-25T15:55:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f3159a640 1 -- 172.21.15.67:0/2252672168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f1c0a59b0 con 0x7f9f3415a720 2024-04-25T15:55:09.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f3159a640 1 -- 172.21.15.67:0/2252672168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c0a5d80 con 0x7f9f3415a720 2024-04-25T15:55:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 -- 172.21.15.67:0/2252672168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 msgr2=0x7f9f3415ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/2252672168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415ab30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009e50 tx=0x7f9f1c01ca30 comp rx=0 tx=0).stop 2024-04-25T15:55:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 -- 172.21.15.67:0/2252672168 shutdown_connections 2024-04-25T15:55:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/2252672168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f340ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/2252672168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415ab30 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 -- 172.21.15.67:0/2252672168 >> 172.21.15.67:0/2252672168 conn(0x7f9f340700b0 msgr2=0x7f9f340704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:09.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 -- 172.21.15.67:0/2252672168 shutdown_connections 2024-04-25T15:55:09.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 -- 172.21.15.67:0/2252672168 wait complete. 2024-04-25T15:55:09.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 Processor -- start 2024-04-25T15:55:09.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.585+0000 7f9f38ac5640 1 -- start start 2024-04-25T15:55:09.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f34071bc0 0x7f9f340cd940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f341f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f34071fd0 con 0x7f9f3415a720 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9f340cc2d0 con 0x7f9f34071bc0 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9f34072150 con 0x7f9f340ca540 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55234/0 (socket says 172.21.15.67:55234) 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 -- 172.21.15.67:0/74320227 learned_addr learned my addr 172.21.15.67:0/74320227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f31d9b640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f341f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f3259c640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f34071bc0 0x7f9f340cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 -- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 msgr2=0x7f9f341f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f341f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 -- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f34071bc0 msgr2=0x7f9f340cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f34071bc0 0x7f9f340cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 -- 172.21.15.67:0/74320227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f341f7ca0 con 0x7f9f3415a720 2024-04-25T15:55:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f3259c640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f34071bc0 0x7f9f340cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f32d9d640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415cde0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9f24006e90 tx=0x7f9f2400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f2affd640 1 -- 172.21.15.67:0/74320227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f240125f0 con 0x7f9f3415a720 2024-04-25T15:55:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f2affd640 1 -- 172.21.15.67:0/74320227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f24019070 con 0x7f9f3415a720 2024-04-25T15:55:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f2affd640 1 -- 172.21.15.67:0/74320227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f240129d0 con 0x7f9f3415a720 2024-04-25T15:55:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f34150f60 con 0x7f9f3415a720 2024-04-25T15:55:09.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f34151520 con 0x7f9f3415a720 2024-04-25T15:55:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f3406a480 con 0x7f9f3415a720 2024-04-25T15:55:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f2affd640 1 -- 172.21.15.67:0/74320227 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9f24012b70 con 0x7f9f3415a720 2024-04-25T15:55:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.589+0000 7f9f2affd640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9f1406f400 0x7f9f14071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.593+0000 7f9f2affd640 1 -- 172.21.15.67:0/74320227 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f9f2408f750 con 0x7f9f3415a720 2024-04-25T15:55:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.593+0000 7f9f3259c640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9f1406f400 0x7f9f14071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.593+0000 7f9f3259c640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9f1406f400 0x7f9f14071910 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c000f70 tx=0x7f9f2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:09.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.597+0000 7f9f2affd640 1 -- 172.21.15.67:0/74320227 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9f2405c500 con 0x7f9f3415a720 2024-04-25T15:55:09.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.761+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f34150cf0 con 0x7f9f3415a720 2024-04-25T15:55:09.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f2affd640 1 -- 172.21.15.67:0/74320227 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+17051 (secure 0 0 0) 0x7f9f2405be00 con 0x7f9f3415a720 2024-04-25T15:55:09.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:09.765 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":129,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:54:54.321839+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:53:14.990625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.a","osds":[2,6]},{"pgid":"3.18","osds":[1,4]},{"pgid":"3.27","osds":[7,6]}],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":5}]},{"pgid":"3.22","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3e","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:55:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9f1406f400 msgr2=0x7f9f14071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9f1406f400 0x7f9f14071910 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c000f70 tx=0x7f9f2c009040 comp rx=0 tx=0).stop 2024-04-25T15:55:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 msgr2=0x7f9f3415cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415cde0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9f24006e90 tx=0x7f9f2400d9e0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 shutdown_connections 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f9f1406f400 0x7f9f14071910 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f3415a720 0x7f9f3415cde0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f340ca540 0x7f9f341f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 --2- 172.21.15.67:0/74320227 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f34071bc0 0x7f9f340cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 >> 172.21.15.67:0/74320227 conn(0x7f9f340700b0 msgr2=0x7f9f340704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 shutdown_connections 2024-04-25T15:55:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:09.765+0000 7f9f38ac5640 1 -- 172.21.15.67:0/74320227 wait complete. 2024-04-25T15:55:10.791 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T15:55:10.791 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:10.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.961+0000 7ff8be61f640 1 Processor -- start 2024-04-25T15:55:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.961+0000 7ff8be61f640 1 -- start start 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8be61f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b8157940 0x7ff8b80ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8be61f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8b80caef0 0x7ff8b815a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8be61f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8b80c3980 con 0x7ff8b815aa70 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8be61f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7ff8b80c2fb0 con 0x7ff8b80caef0 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8be61f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8b80c3290 con 0x7ff8b8157940 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8b80caef0 0x7ff8b815a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8b80caef0 0x7ff8b815a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55236/0 (socket says 172.21.15.67:55236) 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8bcbbc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8b815aa70 0x7ff8b806c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57616/0 (socket says 172.21.15.67:57616) 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 -- 172.21.15.67:0/4218760138 learned_addr learned my addr 172.21.15.67:0/4218760138 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/4218760138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b8157940 0x7ff8b80ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 -- 172.21.15.67:0/4218760138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b8157940 msgr2=0x7ff8b80ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/4218760138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b8157940 0x7ff8b80ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 -- 172.21.15.67:0/4218760138 >> v1:172.21.15.67:6789/0 conn(0x7ff8b815aa70 legacy=0x7ff8b806c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 -- 172.21.15.67:0/4218760138 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b80c3c40 con 0x7ff8b80caef0 2024-04-25T15:55:10.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/4218760138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b815a530 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009870 tx=0x7ff8a001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc695e9ba21879e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:10.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b6ffd640 1 -- 172.21.15.67:0/4218760138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a00b5070 con 0x7ff8b80caef0 2024-04-25T15:55:10.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b6ffd640 1 -- 172.21.15.67:0/4218760138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8a00a8a50 con 0x7ff8b80caef0 2024-04-25T15:55:10.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.965+0000 7ff8b6ffd640 1 -- 172.21.15.67:0/4218760138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a00a8e20 con 0x7ff8b80caef0 2024-04-25T15:55:10.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 -- 172.21.15.67:0/4218760138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 msgr2=0x7ff8b815a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 --2- 172.21.15.67:0/4218760138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b815a530 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009870 tx=0x7ff8a001ed50 comp rx=0 tx=0).stop 2024-04-25T15:55:10.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 -- 172.21.15.67:0/4218760138 shutdown_connections 2024-04-25T15:55:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 --2- 172.21.15.67:0/4218760138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b815a530 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 --2- 172.21.15.67:0/4218760138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b8157940 0x7ff8b80ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 -- 172.21.15.67:0/4218760138 >> 172.21.15.67:0/4218760138 conn(0x7ff8b806fd30 msgr2=0x7ff8b8070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 -- 172.21.15.67:0/4218760138 shutdown_connections 2024-04-25T15:55:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 -- 172.21.15.67:0/4218760138 wait complete. 2024-04-25T15:55:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 Processor -- start 2024-04-25T15:55:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.969+0000 7ff8be61f640 1 -- start start 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b80734b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8b8157940 0x7ff8b81f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815aa70 0x7ff8b81f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b806d9f0 con 0x7ff8b80caef0 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b815ae80 con 0x7ff8b8157940 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8b80c84b0 con 0x7ff8b815aa70 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b80734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b80734b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55238/0 (socket says 172.21.15.67:55238) 2024-04-25T15:55:10.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 -- 172.21.15.67:0/920282805 learned_addr learned my addr 172.21.15.67:0/920282805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8b8157940 0x7ff8b81f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8bcbbc640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815aa70 0x7ff8b81f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 -- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815aa70 msgr2=0x7ff8b81f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8b815aa70 0x7ff8b81f7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 -- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8b8157940 msgr2=0x7ff8b81f6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8b8157940 0x7ff8b81f6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 -- 172.21.15.67:0/920282805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b80739f0 con 0x7ff8b80caef0 2024-04-25T15:55:10.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b7fff640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b80734b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800d650 tx=0x7ff8a800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:10.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800b6c0 con 0x7ff8b80caef0 2024-04-25T15:55:10.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8a8019070 con 0x7ff8b80caef0 2024-04-25T15:55:10.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b8159b90 con 0x7ff8b80caef0 2024-04-25T15:55:10.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a800ba80 con 0x7ff8b80caef0 2024-04-25T15:55:10.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b80c6cb0 con 0x7ff8b80caef0 2024-04-25T15:55:10.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff8b80c6770 con 0x7ff8b80caef0 2024-04-25T15:55:10.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.973+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff8a800bc20 con 0x7ff8b80caef0 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b4ff9640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89806f430 0x7ff898071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7ff8a8090270 con 0x7ff8b80caef0 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b4ff9640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff898077b60 0x7ff89807a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8a8013e50 con 0x7ff898077b60 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7ff8a8090700 con 0x7ff8b80caef0 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8bcbbc640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff898077b60 0x7ff89807a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89806f430 0x7ff898071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8bcbbc640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff898077b60 0x7ff89807a050 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:10.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b77fe640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89806f430 0x7ff898071940 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00b2000 tx=0x7ff8a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:10.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.977+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== osd.0 v2:172.21.15.67:6800/851806635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8a8013e50 con 0x7ff898077b60 2024-04-25T15:55:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.989+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 --> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff8b80c6ef0 con 0x7ff898077b60 2024-04-25T15:55:10.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.989+0000 7ff8b4ff9640 1 -- 172.21.15.67:0/920282805 <== osd.0 v2:172.21.15.67:6800/851806635 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff8b0007620 con 0x7ff898077b60 2024-04-25T15:55:10.992 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:10.992 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:10.992 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff898077b60 msgr2=0x7ff89807a050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:6800/851806635,v1:172.21.15.67:6801/851806635] conn(0x7ff898077b60 0x7ff89807a050 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89806f430 msgr2=0x7ff898071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7ff89806f430 0x7ff898071940 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00b2000 tx=0x7ff8a0008040 comp rx=0 tx=0).stop 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 msgr2=0x7ff8b80734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 --2- 172.21.15.67:0/920282805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8b80caef0 0x7ff8b80734b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800d650 tx=0x7ff8a800db10 comp rx=0 tx=0).stop 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8bcbbc640 1 -- 172.21.15.67:0/920282805 reap_dead start 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 shutdown_connections 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 >> 172.21.15.67:0/920282805 conn(0x7ff8b806fd30 msgr2=0x7ff8b8154760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 shutdown_connections 2024-04-25T15:55:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:10.993+0000 7ff8be61f640 1 -- 172.21.15.67:0/920282805 wait complete. 2024-04-25T15:55:11.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T15:55:11.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f465a5b2640 1 Processor -- start 2024-04-25T15:55:11.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f465a5b2640 1 -- start start 2024-04-25T15:55:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f465a5b2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f46540ca2c0 0x7f465415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f465a5b2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46540ccef0 con 0x7f46540ca6d0 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f465a5b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46540cafa0 con 0x7f465415d3f0 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f465a5b2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f465415ca20 con 0x7f46540ca2c0 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f4658b4f640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46540ca6d0 0x7f4654167040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57620/0 (socket says 172.21.15.67:57620) 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f4658b4f640 1 -- 172.21.15.67:0/1758630917 learned_addr learned my addr 172.21.15.67:0/1758630917 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.165+0000 7f4653fff640 1 --2- 172.21.15.67:0/1758630917 >> v2:172.21.15.176:3300/0 conn(0x7f46540ca2c0 0x7f465415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4652ffd640 1 -- 172.21.15.67:0/1758630917 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194105174 0 0) 0x7f46540ccef0 con 0x7f46540ca6d0 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4652ffd640 1 -- 172.21.15.67:0/1758630917 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4640003530 con 0x7f46540ca6d0 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4652ffd640 1 -- 172.21.15.67:0/1758630917 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1141062856 0 0) 0x7f4640003530 con 0x7f46540ca6d0 2024-04-25T15:55:11.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4653fff640 1 -- 172.21.15.67:0/1758630917 >> v1:172.21.15.176:6789/0 conn(0x7f465415d3f0 legacy=0x7f465406e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4653fff640 1 -- 172.21.15.67:0/1758630917 >> v1:172.21.15.67:6789/0 conn(0x7f46540ca6d0 legacy=0x7f4654167040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4653fff640 1 -- 172.21.15.67:0/1758630917 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46540cb280 con 0x7f46540ca2c0 2024-04-25T15:55:11.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4653fff640 1 --2- 172.21.15.67:0/1758630917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 0x7f465415ceb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f463c0097c0 tx=0x7f463c01ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b994b74abed6f41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4652ffd640 1 -- 172.21.15.67:0/1758630917 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0b5070 con 0x7f46540ca2c0 2024-04-25T15:55:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4652ffd640 1 -- 172.21.15.67:0/1758630917 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463c0a8630 con 0x7f46540ca2c0 2024-04-25T15:55:11.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f4652ffd640 1 -- 172.21.15.67:0/1758630917 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c001a00 con 0x7f46540ca2c0 2024-04-25T15:55:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f465a5b2640 1 -- 172.21.15.67:0/1758630917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 msgr2=0x7f465415ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f465a5b2640 1 --2- 172.21.15.67:0/1758630917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 0x7f465415ceb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f463c0097c0 tx=0x7f463c01ebb0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f465a5b2640 1 -- 172.21.15.67:0/1758630917 shutdown_connections 2024-04-25T15:55:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f465a5b2640 1 --2- 172.21.15.67:0/1758630917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 0x7f465415ceb0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f465a5b2640 1 -- 172.21.15.67:0/1758630917 >> 172.21.15.67:0/1758630917 conn(0x7f4654073c50 msgr2=0x7f4654074060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f465a5b2640 1 -- 172.21.15.67:0/1758630917 shutdown_connections 2024-04-25T15:55:11.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.169+0000 7f465a5b2640 1 -- 172.21.15.67:0/1758630917 wait complete. 2024-04-25T15:55:11.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 Processor -- start 2024-04-25T15:55:11.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 -- start start 2024-04-25T15:55:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 0x7f46541e8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46540ca6d0 0x7f46541f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f465415d3f0 0x7f46540c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f465406e7e0 con 0x7f46540ca6d0 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f46540cb840 con 0x7f46540ca2c0 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f465406e960 con 0x7f465415d3f0 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46540ca6d0 0x7f46541f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46540ca6d0 0x7f46541f2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55246/0 (socket says 172.21.15.67:55246) 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 -- 172.21.15.67:0/275521738 learned_addr learned my addr 172.21.15.67:0/275521738 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 -- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f465415d3f0 msgr2=0x7f46540c7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f465415d3f0 0x7f46540c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 -- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 msgr2=0x7f46541e8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 0x7f46541e8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 -- 172.21.15.67:0/275521738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46540cc760 con 0x7f46540ca6d0 2024-04-25T15:55:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f46537fe640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46540ca6d0 0x7f46541f2b10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f464400c3c0 tx=0x7f464400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f4650ff9640 1 -- 172.21.15.67:0/275521738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644017070 con 0x7f46540ca6d0 2024-04-25T15:55:11.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f4650ff9640 1 -- 172.21.15.67:0/275521738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46440052d0 con 0x7f46540ca6d0 2024-04-25T15:55:11.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46541f6020 con 0x7f46540ca6d0 2024-04-25T15:55:11.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f4650ff9640 1 -- 172.21.15.67:0/275521738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644003840 con 0x7f46540ca6d0 2024-04-25T15:55:11.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.173+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46541f65e0 con 0x7f46540ca6d0 2024-04-25T15:55:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.177+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f465406a480 con 0x7f46540ca6d0 2024-04-25T15:55:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.177+0000 7f4650ff9640 1 -- 172.21.15.67:0/275521738 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f464401a070 con 0x7f46540ca6d0 2024-04-25T15:55:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.177+0000 7f4650ff9640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f463406f430 0x7f4634071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.177+0000 7f4650ff9640 1 -- 172.21.15.67:0/275521738 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f464408e6c0 con 0x7f46540ca6d0 2024-04-25T15:55:11.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.177+0000 7f4653fff640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f463406f430 0x7f4634071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.177+0000 7f4653fff640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f463406f430 0x7f4634071940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b2000 tx=0x7f463c004740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.181+0000 7f4650ff9640 1 -- 172.21.15.67:0/275521738 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4644005480 con 0x7f46540ca6d0 2024-04-25T15:55:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.345+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f465406dca0 con 0x7f463406f430 2024-04-25T15:55:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f4650ff9640 1 -- 172.21.15.67:0/275521738 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f465406dca0 con 0x7f463406f430 2024-04-25T15:55:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T15:55:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f463406f430 msgr2=0x7f4634071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f463406f430 0x7f4634071940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b2000 tx=0x7f463c004740 comp rx=0 tx=0).stop 2024-04-25T15:55:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46540ca6d0 msgr2=0x7f46541f2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46540ca6d0 0x7f46541f2b10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f464400c3c0 tx=0x7f464400c880 comp rx=0 tx=0).stop 2024-04-25T15:55:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 shutdown_connections 2024-04-25T15:55:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f463406f430 0x7f4634071940 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f465415d3f0 0x7f46540c7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46540ca6d0 0x7f46541f2b10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 --2- 172.21.15.67:0/275521738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46540ca2c0 0x7f46541e8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 >> 172.21.15.67:0/275521738 conn(0x7f4654073c50 msgr2=0x7f46541569a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 shutdown_connections 2024-04-25T15:55:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.349+0000 7f465a5b2640 1 -- 172.21.15.67:0/275521738 wait complete. 2024-04-25T15:55:11.372 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T15:55:11.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:11.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.517+0000 7fd80ad50640 1 Processor -- start 2024-04-25T15:55:11.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.517+0000 7fd80ad50640 1 -- start start 2024-04-25T15:55:11.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd80ad50640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd80406dff0 0x7fd8041569b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd80ad50640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd80406e400 con 0x7fd8041608b0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd80ad50640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd80415fe10 con 0x7fd8040c9e90 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd80ad50640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd8040c5a00 con 0x7fd80406dff0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd8041608b0 0x7fd8040723e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57626/0 (socket says 172.21.15.67:57626) 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 -- 172.21.15.67:0/3526275632 learned_addr learned my addr 172.21.15.67:0/3526275632 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418539299 0 0) 0x7fd7f8001120 con 0x7fd8041608b0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd803fff640 1 --2- 172.21.15.67:0/3526275632 >> v2:172.21.15.176:3300/0 conn(0x7fd80406dff0 0x7fd8041569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f4003530 con 0x7fd8041608b0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2002912827 0 0) 0x7fd7f4003530 con 0x7fd8041608b0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 >> v2:172.21.15.176:3300/0 conn(0x7fd80406dff0 msgr2=0x7fd8041569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 --2- 172.21.15.67:0/3526275632 >> v2:172.21.15.176:3300/0 conn(0x7fd80406dff0 0x7fd8041569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 >> v1:172.21.15.176:6789/0 conn(0x7fd8040c9e90 legacy=0x7fd804167070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8040cef10 con 0x7fd8041608b0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fd7f8001850 con 0x7fd8041608b0 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd803fff640 1 --2- 172.21.15.67:0/3526275632 >> v2:172.21.15.176:3300/0 conn(0x7fd80406dff0 0x7fd8041569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 >> v1:172.21.15.67:6789/0 conn(0x7fd8041608b0 legacy=0x7fd8040723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd7f4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd7f4004810 0x7fd7f4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7f4005280 0x7fd7f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f400c1c0 con 0x7fd7f40039c0 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f400c370 con 0x7fd7f4005280 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f400c570 con 0x7fd7f4004810 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd7f4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8092ed640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7f4005280 0x7fd7f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 -- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd7f4004810 msgr2=0x7fd7f4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd803fff640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd7f4004810 0x7fd7f4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd7f4004810 0x7fd7f4004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 -- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7f4005280 msgr2=0x7fd7f400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7f4005280 0x7fd7f400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd808aec640 1 -- 172.21.15.67:0/3526275632 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f4003530 con 0x7fd7f40039c0 2024-04-25T15:55:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8092ed640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7f4005280 0x7fd7f400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:11.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.521+0000 7fd8092ed640 1 -- 172.21.15.67:0/3526275632 reap_dead start 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd808aec640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd7f4003e10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c4c0 tx=0x7fd7f801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d948a4d67b5cd8df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800c0e0 con 0x7fd7f40039c0 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f80a9020 con 0x7fd7f40039c0 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd8037fe640 1 -- 172.21.15.67:0/3526275632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8005b00 con 0x7fd7f40039c0 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd80ad50640 1 -- 172.21.15.67:0/3526275632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 msgr2=0x7fd7f4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd80ad50640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd7f4003e10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c4c0 tx=0x7fd7f801cca0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd80ad50640 1 -- 172.21.15.67:0/3526275632 shutdown_connections 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd80ad50640 1 --2- 172.21.15.67:0/3526275632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd7f4003e10 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd80ad50640 1 -- 172.21.15.67:0/3526275632 >> 172.21.15.67:0/3526275632 conn(0x7fd8040ce0d0 msgr2=0x7fd8040ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd80ad50640 1 -- 172.21.15.67:0/3526275632 shutdown_connections 2024-04-25T15:55:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.525+0000 7fd80ad50640 1 -- 172.21.15.67:0/3526275632 wait complete. 2024-04-25T15:55:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 Processor -- start 2024-04-25T15:55:11.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 -- start start 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd804147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8040cca50 0x7fd804146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd8040cce60 0x7fd804146ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd8040ca6b0 con 0x7fd7f40039c0 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd8040c94e0 con 0x7fd8040cca50 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd80406ec40 con 0x7fd8040cce60 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd804147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd804147cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55254/0 (socket says 172.21.15.67:55254) 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 -- 172.21.15.67:0/3815256316 learned_addr learned my addr 172.21.15.67:0/3815256316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:11.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 -- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd8040cce60 msgr2=0x7fd804146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd8040cce60 0x7fd804146ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd803fff640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8040cca50 0x7fd804146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 -- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8040cca50 msgr2=0x7fd804146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8040cca50 0x7fd804146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 -- 172.21.15.67:0/3815256316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd804148600 con 0x7fd7f40039c0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd808aec640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd804147cb0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd7f801cad0 tx=0x7fd7f4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80a5a70 con 0x7fd7f40039c0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f80b2070 con 0x7fd7f40039c0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8041f9c70 con 0x7fd7f40039c0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80a5e50 con 0x7fd7f40039c0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8041fa230 con 0x7fd7f40039c0 2024-04-25T15:55:11.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.529+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd80406a480 con 0x7fd7f40039c0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd7f80047f0 con 0x7fd7f40039c0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8017fa640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd7e806f5a0 0x7fd7e8071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fd7f8128a50 con 0x7fd7f40039c0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8017fa640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd7e8077cd0 0x7fd7e807a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7f80c0e50 con 0x7fd7e8077cd0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7fd7f8128ee0 con 0x7fd7f40039c0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd803fff640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd7e806f5a0 0x7fd7e8071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8092ed640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd7e8077cd0 0x7fd7e807a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8092ed640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd7e8077cd0 0x7fd7e807a1c0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd803fff640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd7e806f5a0 0x7fd7e8071ab0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd8041546a0 tx=0x7fd7f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.533+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== osd.1 v2:172.21.15.67:6809/2360705508 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd7f80c0e50 con 0x7fd7e8077cd0 2024-04-25T15:55:11.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 --> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd80414aad0 con 0x7fd7e8077cd0 2024-04-25T15:55:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd8017fa640 1 -- 172.21.15.67:0/3815256316 <== osd.1 v2:172.21.15.67:6809/2360705508 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd80414aad0 con 0x7fd7e8077cd0 2024-04-25T15:55:11.547 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:11.547 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd7e8077cd0 msgr2=0x7fd7e807a1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:6809/2360705508,v1:172.21.15.67:6812/2360705508] conn(0x7fd7e8077cd0 0x7fd7e807a1c0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd7e806f5a0 msgr2=0x7fd7e8071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd7e806f5a0 0x7fd7e8071ab0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd8041546a0 tx=0x7fd7f0008040 comp rx=0 tx=0).stop 2024-04-25T15:55:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 msgr2=0x7fd804147cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 --2- 172.21.15.67:0/3815256316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7f40039c0 0x7fd804147cb0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd7f801cad0 tx=0x7fd7f4005840 comp rx=0 tx=0).stop 2024-04-25T15:55:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd8092ed640 1 -- 172.21.15.67:0/3815256316 reap_dead start 2024-04-25T15:55:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 shutdown_connections 2024-04-25T15:55:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 >> 172.21.15.67:0/3815256316 conn(0x7fd8040ce0d0 msgr2=0x7fd8040c87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 shutdown_connections 2024-04-25T15:55:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.545+0000 7fd80ad50640 1 -- 172.21.15.67:0/3815256316 wait complete. 2024-04-25T15:55:11.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T15:55:11.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f8713640 1 Processor -- start 2024-04-25T15:55:11.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f8713640 1 -- start start 2024-04-25T15:55:11.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f8713640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 0x7f27f015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f8713640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f0072dc0 con 0x7f27f015e020 2024-04-25T15:55:11.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f8713640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f015b430 con 0x7f27f00c6fb0 2024-04-25T15:55:11.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f8713640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27f015a670 con 0x7f27f0074790 2024-04-25T15:55:11.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f5cae640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27f015e020 0x7f27f00723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57634/0 (socket says 172.21.15.67:57634) 2024-04-25T15:55:11.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f5cae640 1 -- 172.21.15.67:0/3764551469 learned_addr learned my addr 172.21.15.67:0/3764551469 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.709+0000 7f27f64af640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 0x7f27f015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370274745 0 0) 0x7f27f0072dc0 con 0x7f27f015e020 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27dc003530 con 0x7f27f015e020 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059197602 0 0) 0x7f27f015b430 con 0x7f27f00c6fb0 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27f0072dc0 con 0x7f27f00c6fb0 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 440735040 0 0) 0x7f27dc003530 con 0x7f27f015e020 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 msgr2=0x7f27f015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 0x7f27f015dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 >> v1:172.21.15.176:6789/0 conn(0x7f27f00c6fb0 legacy=0x7f27f0164e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f015a950 con 0x7f27f015e020 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f27e0001680 con 0x7f27f015e020 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 >> v1:172.21.15.67:6789/0 conn(0x7f27f015e020 legacy=0x7f27f00723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27dc003b80 0x7f27dc003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27dc004810 0x7f27dc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 0x7f27dc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27f015a670 con 0x7f27dc003b80 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc00bae0 con 0x7f27dc004810 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27dc00bce0 con 0x7f27f0074790 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f64af640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27dc003b80 0x7f27dc003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f5cae640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27dc004810 0x7f27dc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f6cb0640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 0x7f27dc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f64af640 1 -- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 msgr2=0x7f27dc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f64af640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f0074790 0x7f27dc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f64af640 1 -- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27dc004810 msgr2=0x7f27dc004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f64af640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27dc004810 0x7f27dc004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f64af640 1 -- 172.21.15.67:0/3764551469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc003530 con 0x7f27dc003b80 2024-04-25T15:55:11.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f64af640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27dc003b80 0x7f27dc003fd0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009ae0 tx=0x7f27ec01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4e40feade4692c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.713+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec0b2070 con 0x7f27dc003b80 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27ec0a5680 con 0x7f27dc003b80 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f54ad640 1 -- 172.21.15.67:0/3764551469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec0096b0 con 0x7f27dc003b80 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 -- 172.21.15.67:0/3764551469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27dc003b80 msgr2=0x7f27dc003fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 --2- 172.21.15.67:0/3764551469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27dc003b80 0x7f27dc003fd0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009ae0 tx=0x7f27ec01ca90 comp rx=0 tx=0).stop 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f6cb0640 1 -- 172.21.15.67:0/3764551469 reap_dead start 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 -- 172.21.15.67:0/3764551469 shutdown_connections 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 -- 172.21.15.67:0/3764551469 >> 172.21.15.67:0/3764551469 conn(0x7f27f006fe70 msgr2=0x7f27f00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 -- 172.21.15.67:0/3764551469 shutdown_connections 2024-04-25T15:55:11.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 -- 172.21.15.67:0/3764551469 wait complete. 2024-04-25T15:55:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 Processor -- start 2024-04-25T15:55:11.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 -- start start 2024-04-25T15:55:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27f01574f0 0x7f27f014d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.717+0000 7f27f8713640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f00c6af0 0x7f27f01f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f8713640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f00c51a0 0x7f27f01e7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f8713640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0073080 con 0x7f27f00c51a0 2024-04-25T15:55:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f8713640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f27f015af10 con 0x7f27f01574f0 2024-04-25T15:55:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f8713640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27f0072ab0 con 0x7f27f00c6af0 2024-04-25T15:55:11.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f00c51a0 0x7f27f01e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f00c51a0 0x7f27f01e7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55274/0 (socket says 172.21.15.67:55274) 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 -- 172.21.15.67:0/2881648397 learned_addr learned my addr 172.21.15.67:0/2881648397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f5cae640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f00c6af0 0x7f27f01f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f64af640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27f01574f0 0x7f27f014d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 -- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f00c6af0 msgr2=0x7f27f01f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f00c6af0 0x7f27f01f2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 -- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27f01574f0 msgr2=0x7f27f014d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27f01574f0 0x7f27f014d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 -- 172.21.15.67:0/2881648397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f00c5d10 con 0x7f27f00c51a0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f5cae640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f00c6af0 0x7f27f01f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f6cb0640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f00c51a0 0x7f27f01e7550 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f27f00c8350 tx=0x7f27e8011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27e6ffd640 1 -- 172.21.15.67:0/2881648397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8010040 con 0x7f27f00c51a0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27e6ffd640 1 -- 172.21.15.67:0/2881648397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e801a070 con 0x7f27f00c51a0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f00c5750 con 0x7f27f00c51a0 2024-04-25T15:55:11.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27e6ffd640 1 -- 172.21.15.67:0/2881648397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8015410 con 0x7f27f00c51a0 2024-04-25T15:55:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f0155900 con 0x7f27f00c51a0 2024-04-25T15:55:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f006a480 con 0x7f27f00c51a0 2024-04-25T15:55:11.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27e6ffd640 1 -- 172.21.15.67:0/2881648397 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f27e80155b0 con 0x7f27f00c51a0 2024-04-25T15:55:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.721+0000 7f27e6ffd640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f27d406f430 0x7f27d4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.725+0000 7f27e6ffd640 1 -- 172.21.15.67:0/2881648397 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f27e8093090 con 0x7f27f00c51a0 2024-04-25T15:55:11.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.725+0000 7f27f64af640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f27d406f430 0x7f27d4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:11.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.725+0000 7f27f64af640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f27d406f430 0x7f27d4071940 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0af000 tx=0x7f27ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.725+0000 7f27e6ffd640 1 -- 172.21.15.67:0/2881648397 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f27e8060040 con 0x7f27f00c51a0 2024-04-25T15:55:11.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.889+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f27f01560e0 con 0x7f27d406f430 2024-04-25T15:55:11.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.889+0000 7f27e6ffd640 1 -- 172.21.15.67:0/2881648397 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f27ec0a5680 con 0x7f27d406f430 2024-04-25T15:55:11.890 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T15:55:11.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f27d406f430 msgr2=0x7f27d4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f27d406f430 0x7f27d4071940 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0af000 tx=0x7f27ec008040 comp rx=0 tx=0).stop 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f00c51a0 msgr2=0x7f27f01e7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f00c51a0 0x7f27f01e7550 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f27f00c8350 tx=0x7f27e8011f90 comp rx=0 tx=0).stop 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 shutdown_connections 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f27d406f430 0x7f27d4071940 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f00c51a0 0x7f27f01e7550 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27f00c6af0 0x7f27f01f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 --2- 172.21.15.67:0/2881648397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27f01574f0 0x7f27f014d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 >> 172.21.15.67:0/2881648397 conn(0x7f27f006fe70 msgr2=0x7f27f006cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 shutdown_connections 2024-04-25T15:55:11.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:11.893+0000 7f27f8713640 1 -- 172.21.15.67:0/2881648397 wait complete. 2024-04-25T15:55:11.915 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T15:55:11.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.069+0000 7f17d2b55640 1 Processor -- start 2024-04-25T15:55:12.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.069+0000 7f17d2b55640 1 -- start start 2024-04-25T15:55:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d2b55640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f17cc157940 0x7f17cc0ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d2b55640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17cc0c3980 con 0x7f17cc0caef0 2024-04-25T15:55:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d2b55640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17cc0c2fb0 con 0x7f17cc15ac40 2024-04-25T15:55:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d2b55640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f17cc0c3290 con 0x7f17cc157940 2024-04-25T15:55:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17cc0caef0 0x7f17cc15a530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57642/0 (socket says 172.21.15.67:57642) 2024-04-25T15:55:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 -- 172.21.15.67:0/3382175465 learned_addr learned my addr 172.21.15.67:0/3382175465 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906434035 0 0) 0x7f17cc0c3980 con 0x7f17cc0caef0 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17b4003530 con 0x7f17cc0caef0 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d08f1640 1 --2- 172.21.15.67:0/3382175465 >> v2:172.21.15.176:3300/0 conn(0x7f17cc157940 0x7f17cc0ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1886842453 0 0) 0x7f17b4003530 con 0x7f17cc0caef0 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 >> v2:172.21.15.176:3300/0 conn(0x7f17cc157940 msgr2=0x7f17cc0ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 --2- 172.21.15.67:0/3382175465 >> v2:172.21.15.176:3300/0 conn(0x7f17cc157940 0x7f17cc0ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 >> v1:172.21.15.176:6789/0 conn(0x7f17cc15ac40 legacy=0x7f17cc164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc0c3c40 con 0x7f17cc0caef0 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f17c0001680 con 0x7f17cc0caef0 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d08f1640 1 --2- 172.21.15.67:0/3382175465 >> v2:172.21.15.176:3300/0 conn(0x7f17cc157940 0x7f17cc0ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 >> v1:172.21.15.67:6789/0 conn(0x7f17cc0caef0 legacy=0x7f17cc15a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17b40039c0 0x7f17b4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4004810 0x7f17b4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4005280 0x7f17b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc0c3290 con 0x7f17b4004810 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f17b400c200 con 0x7f17b40039c0 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17b400c400 con 0x7f17b4005280 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4004810 0x7f17b4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d08f1640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17b40039c0 0x7f17b4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d10f2640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4005280 0x7f17b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 -- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4005280 msgr2=0x7f17b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4005280 0x7f17b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 -- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17b40039c0 msgr2=0x7f17b4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17b40039c0 0x7f17b4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 -- 172.21.15.67:0/3382175465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b4003530 con 0x7f17b4004810 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d10f2640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4005280 0x7f17b400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d08f1640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17b40039c0 0x7f17b4003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:12.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17d10f2640 1 -- 172.21.15.67:0/3382175465 reap_dead start 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cbfff640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4004810 0x7f17b4004c20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c190 tx=0x7f17c001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618fb7763e6e166d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0007d00 con 0x7f17b4004810 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17c00a9020 con 0x7f17b4004810 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.073+0000 7f17cb7fe640 1 -- 172.21.15.67:0/3382175465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c00a5b40 con 0x7f17b4004810 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- 172.21.15.67:0/3382175465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4004810 msgr2=0x7f17b4004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4004810 0x7f17b4004c20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c190 tx=0x7f17c001cc40 comp rx=0 tx=0).stop 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- 172.21.15.67:0/3382175465 shutdown_connections 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 --2- 172.21.15.67:0/3382175465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17b4004810 0x7f17b4004c20 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- 172.21.15.67:0/3382175465 >> 172.21.15.67:0/3382175465 conn(0x7f17cc06fd30 msgr2=0x7f17cc070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- 172.21.15.67:0/3382175465 shutdown_connections 2024-04-25T15:55:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- 172.21.15.67:0/3382175465 wait complete. 2024-04-25T15:55:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 Processor -- start 2024-04-25T15:55:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- start start 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4004810 0x7f17cc1f0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0c7910 0x7f17cc0c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17cc153cd0 0x7f17cc148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc0c84b0 con 0x7f17cc0c7910 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc06d9f0 con 0x7f17cc153cd0 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d2b55640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17cc1f1080 con 0x7f17b4004810 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0c7910 0x7f17cc0c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0c7910 0x7f17cc0c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55278/0 (socket says 172.21.15.67:55278) 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 -- 172.21.15.67:0/3725165596 learned_addr learned my addr 172.21.15.67:0/3725165596 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d10f2640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17cc153cd0 0x7f17cc148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d08f1640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4004810 0x7f17cc1f0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 -- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4004810 msgr2=0x7f17cc1f0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17b4004810 0x7f17cc1f0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 -- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17cc153cd0 msgr2=0x7f17cc148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17cc153cd0 0x7f17cc148290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17cbfff640 1 -- 172.21.15.67:0/3725165596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc14a5d0 con 0x7f17cc0c7910 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.077+0000 7f17d10f2640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17cc153cd0 0x7f17cc148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17cbfff640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0c7910 0x7f17cc0c5f50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f17c001ca70 tx=0x7f17b4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0007d00 con 0x7f17cc0c7910 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17c00049f0 con 0x7f17cc0c7910 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17cc1e38e0 con 0x7f17cc0c7910 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0001c40 con 0x7f17cc0c7910 2024-04-25T15:55:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17cc1e3b80 con 0x7f17cc0c7910 2024-04-25T15:55:12.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f17cc06a480 con 0x7f17cc0c7910 2024-04-25T15:55:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f17c00bd060 con 0x7f17cc0c7910 2024-04-25T15:55:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f17b806f400 0x7f17b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f17c0128960 con 0x7f17cc0c7910 2024-04-25T15:55:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f17b8077b30 0x7f17b807a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17c00a9e50 con 0x7f17b8077b30 2024-04-25T15:55:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f17c0128df0 con 0x7f17cc0c7910 2024-04-25T15:55:12.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17d10f2640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f17b8077b30 0x7f17b807a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17d08f1640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f17b806f400 0x7f17b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17d10f2640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f17b8077b30 0x7f17b807a020 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.081+0000 7f17d08f1640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f17b806f400 0x7f17b8071910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009aa0 tx=0x7f17bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.085+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== osd.2 v2:172.21.15.67:6810/2903206440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f17c00a9e50 con 0x7f17b8077b30 2024-04-25T15:55:12.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 --> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f17cc0c59a0 con 0x7f17b8077b30 2024-04-25T15:55:12.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17c97fa640 1 -- 172.21.15.67:0/3725165596 <== osd.2 v2:172.21.15.67:6810/2903206440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f17cc0c59a0 con 0x7f17b8077b30 2024-04-25T15:55:12.097 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:12.097 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:12.097 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:12.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f17b8077b30 msgr2=0x7f17b807a020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:6810/2903206440,v1:172.21.15.67:6813/2903206440] conn(0x7f17b8077b30 0x7f17b807a020 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f17b806f400 msgr2=0x7f17b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f17b806f400 0x7f17b8071910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009aa0 tx=0x7f17bc008040 comp rx=0 tx=0).stop 2024-04-25T15:55:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0c7910 msgr2=0x7f17cc0c5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 --2- 172.21.15.67:0/3725165596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17cc0c7910 0x7f17cc0c5f50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f17c001ca70 tx=0x7f17b4005840 comp rx=0 tx=0).stop 2024-04-25T15:55:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d10f2640 1 -- 172.21.15.67:0/3725165596 reap_dead start 2024-04-25T15:55:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 shutdown_connections 2024-04-25T15:55:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 >> 172.21.15.67:0/3725165596 conn(0x7f17cc06fd30 msgr2=0x7f17cc14d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 shutdown_connections 2024-04-25T15:55:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.097+0000 7f17d2b55640 1 -- 172.21.15.67:0/3725165596 wait complete. 2024-04-25T15:55:12.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T15:55:12.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0383d2e640 1 Processor -- start 2024-04-25T15:55:12.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0383d2e640 1 -- start start 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0383d2e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 0x7f037c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0383d2e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f037c06d780 con 0x7f037c0b5660 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0383d2e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f037c160920 con 0x7f037c15fe00 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0383d2e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f037c15d510 con 0x7f037c06ca60 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f03812c9640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f037c0b5660 0x7f037c0b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57654/0 (socket says 172.21.15.67:57654) 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f03812c9640 1 -- 172.21.15.67:0/1368057358 learned_addr learned my addr 172.21.15.67:0/1368057358 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0381aca640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 0x7f037c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3457883052 0 0) 0x7f037c06d780 con 0x7f037c0b5660 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.253+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0368003530 con 0x7f037c0b5660 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 625366123 0 0) 0x7f0368003530 con 0x7f037c0b5660 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 msgr2=0x7f037c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 0x7f037c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 >> v1:172.21.15.176:6789/0 conn(0x7f037c15fe00 legacy=0x7f037c160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037c160be0 con 0x7f037c0b5660 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f036c0016a0 con 0x7f037c0b5660 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 >> v1:172.21.15.67:6789/0 conn(0x7f037c0b5660 legacy=0x7f037c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03680039f0 0x7f0368003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 0x7f0368004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368004d50 0x7f0368005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f037c15d510 con 0x7f03680039f0 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f036800bb90 con 0x7f0368004d50 2024-04-25T15:55:12.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f036800bd90 con 0x7f037c06ca60 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0381aca640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03680039f0 0x7f0368003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f03812c9640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 0x7f0368004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f03822cb640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368004d50 0x7f0368005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0381aca640 1 -- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 msgr2=0x7f0368004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0381aca640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f037c06ca60 0x7f0368004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0381aca640 1 -- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368004d50 msgr2=0x7f0368005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0381aca640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0368004d50 0x7f0368005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0381aca640 1 -- 172.21.15.67:0/1368057358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0368003530 con 0x7f03680039f0 2024-04-25T15:55:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0381aca640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03680039f0 0x7f0368003e60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0378009ab0 tx=0x7f037801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2220df44b1a4a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03780096b0 con 0x7f03680039f0 2024-04-25T15:55:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03780a64a0 con 0x7f03680039f0 2024-04-25T15:55:12.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.257+0000 7f0380ac8640 1 -- 172.21.15.67:0/1368057358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03780a6850 con 0x7f03680039f0 2024-04-25T15:55:12.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- 172.21.15.67:0/1368057358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03680039f0 msgr2=0x7f0368003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 --2- 172.21.15.67:0/1368057358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03680039f0 0x7f0368003e60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0378009ab0 tx=0x7f037801c9b0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f03822cb640 1 -- 172.21.15.67:0/1368057358 reap_dead start 2024-04-25T15:55:12.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- 172.21.15.67:0/1368057358 shutdown_connections 2024-04-25T15:55:12.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- 172.21.15.67:0/1368057358 >> 172.21.15.67:0/1368057358 conn(0x7f037c06e420 msgr2=0x7f037c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- 172.21.15.67:0/1368057358 shutdown_connections 2024-04-25T15:55:12.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- 172.21.15.67:0/1368057358 wait complete. 2024-04-25T15:55:12.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 Processor -- start 2024-04-25T15:55:12.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- start start 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03680039f0 0x7f037c153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c159f50 0x7f037c14d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037c1e8060 0x7f037c1f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f037c15d1a0 con 0x7f037c159f50 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f037c15cec0 con 0x7f037c1e8060 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.261+0000 7f0383d2e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f037c15c890 con 0x7f03680039f0 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c159f50 0x7f037c14d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c159f50 0x7f037c14d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55294/0 (socket says 172.21.15.67:55294) 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 -- 172.21.15.67:0/2564957496 learned_addr learned my addr 172.21.15.67:0/2564957496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f0381aca640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03680039f0 0x7f037c153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03822cb640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037c1e8060 0x7f037c1f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 -- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03680039f0 msgr2=0x7f037c153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03680039f0 0x7f037c153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 -- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037c1e8060 msgr2=0x7f037c1f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037c1e8060 0x7f037c1f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 -- 172.21.15.67:0/2564957496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037c1f40c0 con 0x7f037c159f50 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03822cb640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037c1e8060 0x7f037c1f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03812c9640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c159f50 0x7f037c14d350 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f036c0117f0 tx=0x7f036c011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03727fc640 1 -- 172.21.15.67:0/2564957496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c010040 con 0x7f037c159f50 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03727fc640 1 -- 172.21.15.67:0/2564957496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f036c01a070 con 0x7f037c159f50 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037c153e80 con 0x7f037c159f50 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03727fc640 1 -- 172.21.15.67:0/2564957496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c00fda0 con 0x7f037c159f50 2024-04-25T15:55:12.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f037c154c50 con 0x7f037c159f50 2024-04-25T15:55:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037c06a480 con 0x7f037c159f50 2024-04-25T15:55:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03727fc640 1 -- 172.21.15.67:0/2564957496 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f036c01f460 con 0x7f037c159f50 2024-04-25T15:55:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03727fc640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f036006f400 0x7f0360071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f03727fc640 1 -- 172.21.15.67:0/2564957496 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f036c092620 con 0x7f037c159f50 2024-04-25T15:55:12.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.265+0000 7f0381aca640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f036006f400 0x7f0360071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.269+0000 7f0381aca640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f036006f400 0x7f0360071910 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f03780ad000 tx=0x7f03780b2040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.269+0000 7f03727fc640 1 -- 172.21.15.67:0/2564957496 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f036c05f3d0 con 0x7f037c159f50 2024-04-25T15:55:12.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.429+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f037c1536a0 con 0x7f036006f400 2024-04-25T15:55:12.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.429+0000 7f03727fc640 1 -- 172.21.15.67:0/2564957496 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f037c1536a0 con 0x7f036006f400 2024-04-25T15:55:12.431 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f036006f400 msgr2=0x7f0360071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f036006f400 0x7f0360071910 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f03780ad000 tx=0x7f03780b2040 comp rx=0 tx=0).stop 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c159f50 msgr2=0x7f037c14d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c159f50 0x7f037c14d350 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f036c0117f0 tx=0x7f036c011cb0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 shutdown_connections 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f036006f400 0x7f0360071910 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f037c1e8060 0x7f037c1f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f037c159f50 0x7f037c14d350 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 --2- 172.21.15.67:0/2564957496 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03680039f0 0x7f037c153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 >> 172.21.15.67:0/2564957496 conn(0x7f037c06e420 msgr2=0x7f037c0c7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 shutdown_connections 2024-04-25T15:55:12.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.433+0000 7f0383d2e640 1 -- 172.21.15.67:0/2564957496 wait complete. 2024-04-25T15:55:12.453 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T15:55:12.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:12.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.597+0000 7f984ac5a640 1 Processor -- start 2024-04-25T15:55:12.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.597+0000 7f984ac5a640 1 -- start start 2024-04-25T15:55:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f984ac5a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f98440c8e50 0x7f98440ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f984ac5a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9844159b80 con 0x7f98441532f0 2024-04-25T15:55:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f984ac5a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9844159e20 con 0x7f98440caef0 2024-04-25T15:55:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f984ac5a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f98440c89c0 con 0x7f98440c8e50 2024-04-25T15:55:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98489f6640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f98441532f0 0x7f9844159360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44936/0 (socket says 172.21.15.67:44936) 2024-04-25T15:55:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98489f6640 1 -- 172.21.15.67:0/238905097 learned_addr learned my addr 172.21.15.67:0/238905097 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f9843fff640 1 --2- 172.21.15.67:0/238905097 >> v2:172.21.15.176:3300/0 conn(0x7f98440c8e50 0x7f98440ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528318833 0 0) 0x7f9844159b80 con 0x7f98441532f0 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f982c003530 con 0x7f98441532f0 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2531188069 0 0) 0x7f982c003530 con 0x7f98441532f0 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 >> v2:172.21.15.176:3300/0 conn(0x7f98440c8e50 msgr2=0x7f98440ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 --2- 172.21.15.67:0/238905097 >> v2:172.21.15.176:3300/0 conn(0x7f98440c8e50 0x7f98440ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 >> v1:172.21.15.176:6789/0 conn(0x7f98440caef0 legacy=0x7f9844163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984415a0e0 con 0x7f98441532f0 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f9834001a20 con 0x7f98441532f0 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 >> v1:172.21.15.67:6789/0 conn(0x7f98441532f0 legacy=0x7f9844159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f982c0039c0 0x7f982c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f982c004810 0x7f982c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f982c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98440c89c0 con 0x7f982c0051d0 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f982c00c170 con 0x7f982c004810 2024-04-25T15:55:12.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f982c00c370 con 0x7f982c0039c0 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98491f7640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f982c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f9843fff640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f982c004810 0x7f982c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98489f6640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f982c0039c0 0x7f982c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98491f7640 1 -- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f982c0039c0 msgr2=0x7f982c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98491f7640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f982c0039c0 0x7f982c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98491f7640 1 -- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f982c004810 msgr2=0x7f982c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98491f7640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f982c004810 0x7f982c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.601+0000 7f98491f7640 1 -- 172.21.15.67:0/238905097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c003530 con 0x7f982c0051d0 2024-04-25T15:55:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f98491f7640 1 -- 172.21.15.67:0/238905097 reap_dead start 2024-04-25T15:55:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f98491f7640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f982c00b9d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9844070950 tx=0x7f983c01c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460dcd2c4ffc212c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c0013a0 con 0x7f982c0051d0 2024-04-25T15:55:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f983c0052d0 con 0x7f982c0051d0 2024-04-25T15:55:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f98437fe640 1 -- 172.21.15.67:0/238905097 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c003840 con 0x7f982c0051d0 2024-04-25T15:55:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 -- 172.21.15.67:0/238905097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 msgr2=0x7f982c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f982c00b9d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9844070950 tx=0x7f983c01c300 comp rx=0 tx=0).stop 2024-04-25T15:55:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 -- 172.21.15.67:0/238905097 shutdown_connections 2024-04-25T15:55:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 --2- 172.21.15.67:0/238905097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f982c00b9d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 -- 172.21.15.67:0/238905097 >> 172.21.15.67:0/238905097 conn(0x7f984406fd30 msgr2=0x7f9844070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 -- 172.21.15.67:0/238905097 shutdown_connections 2024-04-25T15:55:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 -- 172.21.15.67:0/238905097 wait complete. 2024-04-25T15:55:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 Processor -- start 2024-04-25T15:55:12.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.605+0000 7f984ac5a640 1 -- start start 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f98441f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98440c5830 0x7f9844150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98441e1ca0 0x7f98440cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9844153700 con 0x7f982c0051d0 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9844073f80 con 0x7f98440c5830 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9844153880 con 0x7f98441e1ca0 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f98441f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f98441f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36790/0 (socket says 172.21.15.67:36790) 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 -- 172.21.15.67:0/2082740190 learned_addr learned my addr 172.21.15.67:0/2082740190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98491f7640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98441e1ca0 0x7f98440cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f9843fff640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98440c5830 0x7f9844150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 -- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98441e1ca0 msgr2=0x7f98440cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98441e1ca0 0x7f98440cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 -- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98440c5830 msgr2=0x7f9844150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98440c5830 0x7f9844150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 -- 172.21.15.67:0/2082740190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984406ce70 con 0x7f982c0051d0 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98489f6640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f98441f1740 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f98340015b0 tx=0x7f983400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983400edf0 con 0x7f982c0051d0 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9834018070 con 0x7f982c0051d0 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98441ee500 con 0x7f982c0051d0 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9834013640 con 0x7f982c0051d0 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98441eeac0 con 0x7f982c0051d0 2024-04-25T15:55:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f984406a480 con 0x7f982c0051d0 2024-04-25T15:55:12.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f98340137e0 con 0x7f982c0051d0 2024-04-25T15:55:12.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f983006f400 0x7f9830071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f9834090470 con 0x7f982c0051d0 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9830077b30 0x7f983007a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9834028ea0 con 0x7f9830077b30 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f9834090900 con 0x7f982c0051d0 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f9843fff640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f983006f400 0x7f9830071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.609+0000 7f98491f7640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9830077b30 0x7f983007a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.613+0000 7f98491f7640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9830077b30 0x7f983007a020 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.613+0000 7f9843fff640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f983006f400 0x7f9830071910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f98440c4840 tx=0x7f9838008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.613+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== osd.3 v2:172.21.15.67:6808/1688455876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9834028ea0 con 0x7f9830077b30 2024-04-25T15:55:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 --> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f98440cc1c0 con 0x7f9830077b30 2024-04-25T15:55:12.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f98417fa640 1 -- 172.21.15.67:0/2082740190 <== osd.3 v2:172.21.15.67:6808/1688455876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f98440cc1c0 con 0x7f9830077b30 2024-04-25T15:55:12.626 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:12.626 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:12.626 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9830077b30 msgr2=0x7f983007a020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:6808/1688455876,v1:172.21.15.67:6811/1688455876] conn(0x7f9830077b30 0x7f983007a020 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f983006f400 msgr2=0x7f9830071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f983006f400 0x7f9830071910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f98440c4840 tx=0x7f9838008040 comp rx=0 tx=0).stop 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 msgr2=0x7f98441f1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 --2- 172.21.15.67:0/2082740190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c0051d0 0x7f98441f1740 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f98340015b0 tx=0x7f983400e530 comp rx=0 tx=0).stop 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f98491f7640 1 -- 172.21.15.67:0/2082740190 reap_dead start 2024-04-25T15:55:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 shutdown_connections 2024-04-25T15:55:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 >> 172.21.15.67:0/2082740190 conn(0x7f984406fd30 msgr2=0x7f9844071ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 shutdown_connections 2024-04-25T15:55:12.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.625+0000 7f984ac5a640 1 -- 172.21.15.67:0/2082740190 wait complete. 2024-04-25T15:55:12.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T15:55:12.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f805c03c640 1 Processor -- start 2024-04-25T15:55:12.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f805c03c640 1 -- start start 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f805c03c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f805406d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f805c03c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80540c92d0 0x7f8054072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f805c03c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f805406d8e0 con 0x7f80540c96e0 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f805c03c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f805406dbc0 con 0x7f80540c92d0 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f805c03c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8054073c70 con 0x7f8054160800 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f80595d7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80540c92d0 0x7f8054072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f80595d7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80540c92d0 0x7f8054072dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36798/0 (socket says 172.21.15.67:36798) 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f80595d7640 1 -- 172.21.15.67:0/2838607307 learned_addr learned my addr 172.21.15.67:0/2838607307 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f8059dd8640 1 --2- 172.21.15.67:0/2838607307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f805406d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f80595d7640 1 -- 172.21.15.67:0/2838607307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 msgr2=0x7f805406d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f80595d7640 1 --2- 172.21.15.67:0/2838607307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f805406d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f80595d7640 1 -- 172.21.15.67:0/2838607307 >> v1:172.21.15.67:6789/0 conn(0x7f80540c96e0 legacy=0x7f805415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.789+0000 7f80595d7640 1 -- 172.21.15.67:0/2838607307 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8054073f50 con 0x7f80540c92d0 2024-04-25T15:55:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f80595d7640 1 --2- 172.21.15.67:0/2838607307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c92d0 0x7f8054072dc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f805406dd40 tx=0x7f804401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757554c5fe4d93f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f8058dd6640 1 -- 172.21.15.67:0/2838607307 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440b0070 con 0x7f80540c92d0 2024-04-25T15:55:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f8058dd6640 1 -- 172.21.15.67:0/2838607307 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80440ab600 con 0x7f80540c92d0 2024-04-25T15:55:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f8058dd6640 1 -- 172.21.15.67:0/2838607307 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440ab9d0 con 0x7f80540c92d0 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 -- 172.21.15.67:0/2838607307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c92d0 msgr2=0x7f8054072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 --2- 172.21.15.67:0/2838607307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c92d0 0x7f8054072dc0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f805406dd40 tx=0x7f804401c4b0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 -- 172.21.15.67:0/2838607307 shutdown_connections 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 --2- 172.21.15.67:0/2838607307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c92d0 0x7f8054072dc0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 --2- 172.21.15.67:0/2838607307 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f805406d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 -- 172.21.15.67:0/2838607307 >> 172.21.15.67:0/2838607307 conn(0x7f8054070db0 msgr2=0x7f80540ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 -- 172.21.15.67:0/2838607307 shutdown_connections 2024-04-25T15:55:12.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.793+0000 7f805c03c640 1 -- 172.21.15.67:0/2838607307 wait complete. 2024-04-25T15:55:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 Processor -- start 2024-04-25T15:55:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 -- start start 2024-04-25T15:55:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80540c92d0 0x7f80541f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c96e0 0x7f8054157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f80540c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8054160c10 con 0x7f80540c96e0 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f805406d570 con 0x7f80540c92d0 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8054160d90 con 0x7f8054160800 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c96e0 0x7f8054157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c96e0 0x7f8054157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36800/0 (socket says 172.21.15.67:36800) 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 -- 172.21.15.67:0/2269629859 learned_addr learned my addr 172.21.15.67:0/2269629859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805a5d9640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f80540c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 -- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 msgr2=0x7f80540c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f8059dd8640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80540c92d0 0x7f80541f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f80540c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 -- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80540c92d0 msgr2=0x7f80541f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80540c92d0 0x7f80541f6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 -- 172.21.15.67:0/2269629859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8054154af0 con 0x7f80540c96e0 2024-04-25T15:55:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805a5d9640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f80540c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f80595d7640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c96e0 0x7f8054157c40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f804401a0e0 tx=0x7f8044019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f804a7fc640 1 -- 172.21.15.67:0/2269629859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440ba070 con 0x7f80540c96e0 2024-04-25T15:55:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f804a7fc640 1 -- 172.21.15.67:0/2269629859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80440b4620 con 0x7f80540c96e0 2024-04-25T15:55:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8054158180 con 0x7f80540c96e0 2024-04-25T15:55:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f804a7fc640 1 -- 172.21.15.67:0/2269629859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80440b0040 con 0x7f80540c96e0 2024-04-25T15:55:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.797+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80540c88c0 con 0x7f80540c96e0 2024-04-25T15:55:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.801+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80540c8340 con 0x7f80540c96e0 2024-04-25T15:55:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.801+0000 7f804a7fc640 1 -- 172.21.15.67:0/2269629859 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f80440bd070 con 0x7f80540c96e0 2024-04-25T15:55:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.801+0000 7f804a7fc640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f803806f400 0x7f8038071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.801+0000 7f804a7fc640 1 -- 172.21.15.67:0/2269629859 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f8044128490 con 0x7f80540c96e0 2024-04-25T15:55:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.801+0000 7f8059dd8640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f803806f400 0x7f8038071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.801+0000 7f8059dd8640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f803806f400 0x7f8038071910 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f804c0096f0 tx=0x7f804c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:12.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.805+0000 7f804a7fc640 1 -- 172.21.15.67:0/2269629859 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f80540c8340 con 0x7f80540c96e0 2024-04-25T15:55:12.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.973+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f805406a480 con 0x7f803806f400 2024-04-25T15:55:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.973+0000 7f804a7fc640 1 -- 172.21.15.67:0/2269629859 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f805406a480 con 0x7f803806f400 2024-04-25T15:55:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f803806f400 msgr2=0x7f8038071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f803806f400 0x7f8038071910 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f804c0096f0 tx=0x7f804c008040 comp rx=0 tx=0).stop 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c96e0 msgr2=0x7f8054157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c96e0 0x7f8054157c40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f804401a0e0 tx=0x7f8044019250 comp rx=0 tx=0).stop 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 shutdown_connections 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f803806f400 0x7f8038071910 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8054160800 0x7f80540c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80540c96e0 0x7f8054157c40 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 --2- 172.21.15.67:0/2269629859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80540c92d0 0x7f80541f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 >> 172.21.15.67:0/2269629859 conn(0x7f8054070db0 msgr2=0x7f8054153160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 shutdown_connections 2024-04-25T15:55:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:12.977+0000 7f805c03c640 1 -- 172.21.15.67:0/2269629859 wait complete. 2024-04-25T15:55:12.999 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T15:55:12.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.161+0000 7fb8548bb640 1 Processor -- start 2024-04-25T15:55:13.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.161+0000 7fb8548bb640 1 -- start start 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb8548bb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 0x7fb850168590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb8548bb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8500b5d20 con 0x7fb850156ab0 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb8548bb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb850156ec0 con 0x7fb8500c75a0 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb8548bb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb85006ca90 con 0x7fb850166170 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb850156ab0 0x7fb8500b5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44954/0 (socket says 172.21.15.67:44954) 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 -- 172.21.15.67:0/3826450764 learned_addr learned my addr 172.21.15.67:0/3826450764 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84ed9d640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 0x7fb850168590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3232944039 0 0) 0x7fb8500b5d20 con 0x7fb850156ab0 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb838003530 con 0x7fb850156ab0 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 775668310 0 0) 0x7fb838003530 con 0x7fb850156ab0 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 msgr2=0x7fb850168590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:13.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 0x7fb850168590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 >> v1:172.21.15.176:6789/0 conn(0x7fb8500c75a0 legacy=0x7fb8500c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85006cd70 con 0x7fb850156ab0 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7fb840001680 con 0x7fb850156ab0 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 >> v1:172.21.15.67:6789/0 conn(0x7fb850156ab0 legacy=0x7fb8500b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380039f0 0x7fb838003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb838004810 0x7fb838004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 0x7fb838005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb85006ca90 con 0x7fb8380039f0 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb83800bb00 con 0x7fb838004810 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb83800bd00 con 0x7fb850166170 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380039f0 0x7fb838003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84ed9d640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 0x7fb838005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84dd9b640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb838004810 0x7fb838004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 -- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 msgr2=0x7fb838005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb850166170 0x7fb838005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 -- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb838004810 msgr2=0x7fb838004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb838004810 0x7fb838004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.165+0000 7fb84e59c640 1 -- 172.21.15.67:0/3826450764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb838003530 con 0x7fb8380039f0 2024-04-25T15:55:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb84e59c640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380039f0 0x7fb838003e60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c190 tx=0x7fb84001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7debf40025da86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb840007d00 con 0x7fb8380039f0 2024-04-25T15:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8400a9020 con 0x7fb8380039f0 2024-04-25T15:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb84d59a640 1 -- 172.21.15.67:0/3826450764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb840005b00 con 0x7fb8380039f0 2024-04-25T15:55:13.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb8548bb640 1 -- 172.21.15.67:0/3826450764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380039f0 msgr2=0x7fb838003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb8548bb640 1 --2- 172.21.15.67:0/3826450764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8380039f0 0x7fb838003e60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fb84000c190 tx=0x7fb84001cc40 comp rx=0 tx=0).stop 2024-04-25T15:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb84ed9d640 1 -- 172.21.15.67:0/3826450764 reap_dead start 2024-04-25T15:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb8548bb640 1 -- 172.21.15.67:0/3826450764 shutdown_connections 2024-04-25T15:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb8548bb640 1 -- 172.21.15.67:0/3826450764 >> 172.21.15.67:0/3826450764 conn(0x7fb85006e550 msgr2=0x7fb85006e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb8548bb640 1 -- 172.21.15.67:0/3826450764 shutdown_connections 2024-04-25T15:55:13.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.169+0000 7fb8548bb640 1 -- 172.21.15.67:0/3826450764 wait complete. 2024-04-25T15:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 Processor -- start 2024-04-25T15:55:13.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 -- start start 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8380039f0 0x7fb85014ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85015e930 0x7fb85015ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8501467e0 0x7fb850146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8500cc630 con 0x7fb85015e930 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb85006db30 con 0x7fb8380039f0 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb8500cc7b0 con 0x7fb8501467e0 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85015e930 0x7fb85015ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85015e930 0x7fb85015ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36812/0 (socket says 172.21.15.67:36812) 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 -- 172.21.15.67:0/2804540403 learned_addr learned my addr 172.21.15.67:0/2804540403 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84e59c640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8380039f0 0x7fb85014ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84ed9d640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8501467e0 0x7fb850146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 -- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8501467e0 msgr2=0x7fb850146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb8501467e0 0x7fb850146bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 -- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8380039f0 msgr2=0x7fb85014ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8380039f0 0x7fb85014ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 -- 172.21.15.67:0/2804540403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb850153e70 con 0x7fb85015e930 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84e59c640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8380039f0 0x7fb85014ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb84dd9b640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85015e930 0x7fb85015ed40 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fb84400e4e0 tx=0x7fb84400e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8440052e0 con 0x7fb85015e930 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb844018070 con 0x7fb85015e930 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8500d2930 con 0x7fb85015e930 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb844005480 con 0x7fb85015e930 2024-04-25T15:55:13.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8500cbff0 con 0x7fb85015e930 2024-04-25T15:55:13.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.173+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb85015d9b0 con 0x7fb85015e930 2024-04-25T15:55:13.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb844003840 con 0x7fb85015e930 2024-04-25T15:55:13.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb83effd640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb83006f400 0x7fb830071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fb84408faa0 con 0x7fb85015e930 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb83effd640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb830077b30 0x7fb83007a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb83007a6f0 con 0x7fb830077b30 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7fb84408fe50 con 0x7fb85015e930 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb84e59c640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb83006f400 0x7fb830071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb84ed9d640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb830077b30 0x7fb83007a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb84ed9d640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb830077b30 0x7fb83007a020 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.177+0000 7fb84e59c640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb83006f400 0x7fb830071910 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb8400aa000 tx=0x7fb8500b49c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.181+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== osd.4 v2:172.21.15.176:6808/696101260 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb83007a6f0 con 0x7fb830077b30 2024-04-25T15:55:13.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 --> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb85006a480 con 0x7fb830077b30 2024-04-25T15:55:13.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb83effd640 1 -- 172.21.15.67:0/2804540403 <== osd.4 v2:172.21.15.176:6808/696101260 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb85006a480 con 0x7fb830077b30 2024-04-25T15:55:13.195 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:13.195 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:13.195 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:13.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb830077b30 msgr2=0x7fb83007a020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6808/696101260,v1:172.21.15.176:6810/696101260] conn(0x7fb830077b30 0x7fb83007a020 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb83006f400 msgr2=0x7fb830071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb83006f400 0x7fb830071910 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb8400aa000 tx=0x7fb8500b49c0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85015e930 msgr2=0x7fb85015ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 --2- 172.21.15.67:0/2804540403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb85015e930 0x7fb85015ed40 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fb84400e4e0 tx=0x7fb84400e9a0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb84ed9d640 1 -- 172.21.15.67:0/2804540403 reap_dead start 2024-04-25T15:55:13.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 shutdown_connections 2024-04-25T15:55:13.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 >> 172.21.15.67:0/2804540403 conn(0x7fb85006e550 msgr2=0x7fb850151830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:13.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 shutdown_connections 2024-04-25T15:55:13.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.193+0000 7fb8548bb640 1 -- 172.21.15.67:0/2804540403 wait complete. 2024-04-25T15:55:13.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T15:55:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5be216e640 1 Processor -- start 2024-04-25T15:55:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5be216e640 1 -- start start 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5be216e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5bdc073bf0 0x7f5bdc06d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5be216e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5bdc06dc90 0x7f5bdc163ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5be216e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bdc0c77a0 con 0x7f5bdc0caf60 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5be216e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f5bdc0cab60 con 0x7f5bdc073bf0 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5be216e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5bdc159620 con 0x7f5bdc06dc90 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdaffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5bdc073bf0 0x7f5bdc06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdaffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5bdc073bf0 0x7f5bdc06d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36820/0 (socket says 172.21.15.67:36820) 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdaffd640 1 -- 172.21.15.67:0/607235970 learned_addr learned my addr 172.21.15.67:0/607235970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdbfff640 1 --2- 172.21.15.67:0/607235970 >> v2:172.21.15.176:3300/0 conn(0x7f5bdc06dc90 0x7f5bdc163ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdaffd640 1 -- 172.21.15.67:0/607235970 >> v2:172.21.15.176:3300/0 conn(0x7f5bdc06dc90 msgr2=0x7f5bdc163ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdaffd640 1 --2- 172.21.15.67:0/607235970 >> v2:172.21.15.176:3300/0 conn(0x7f5bdc06dc90 0x7f5bdc163ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdaffd640 1 -- 172.21.15.67:0/607235970 >> v1:172.21.15.67:6789/0 conn(0x7f5bdc0caf60 legacy=0x7f5bdc15d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.365+0000 7f5bdaffd640 1 -- 172.21.15.67:0/607235970 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc0c7a60 con 0x7f5bdc073bf0 2024-04-25T15:55:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5bdaffd640 1 --2- 172.21.15.67:0/607235970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc073bf0 0x7f5bdc06d750 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0c61d0 tx=0x7f5bc401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8481a4f6a8ae77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5bda7fc640 1 -- 172.21.15.67:0/607235970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc40a56c0 con 0x7f5bdc073bf0 2024-04-25T15:55:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5bda7fc640 1 -- 172.21.15.67:0/607235970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc40a5860 con 0x7f5bdc073bf0 2024-04-25T15:55:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5bda7fc640 1 -- 172.21.15.67:0/607235970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc40a5c30 con 0x7f5bdc073bf0 2024-04-25T15:55:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 -- 172.21.15.67:0/607235970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc073bf0 msgr2=0x7f5bdc06d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 --2- 172.21.15.67:0/607235970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc073bf0 0x7f5bdc06d750 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0c61d0 tx=0x7f5bc401c9f0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 -- 172.21.15.67:0/607235970 shutdown_connections 2024-04-25T15:55:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 --2- 172.21.15.67:0/607235970 >> v2:172.21.15.176:3300/0 conn(0x7f5bdc06dc90 0x7f5bdc163ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 --2- 172.21.15.67:0/607235970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc073bf0 0x7f5bdc06d750 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 -- 172.21.15.67:0/607235970 >> 172.21.15.67:0/607235970 conn(0x7f5bdc070db0 msgr2=0x7f5bdc0ce950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:13.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 -- 172.21.15.67:0/607235970 shutdown_connections 2024-04-25T15:55:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.369+0000 7f5be216e640 1 -- 172.21.15.67:0/607235970 wait complete. 2024-04-25T15:55:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 Processor -- start 2024-04-25T15:55:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 -- start start 2024-04-25T15:55:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc06dc90 0x7f5bdc150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bdc073bf0 0x7f5bdc14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bdc0caf60 0x7f5bdc1f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc15a570 con 0x7f5bdc06dc90 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc15a3f0 con 0x7f5bdc0caf60 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5bdc166c20 con 0x7f5bdc073bf0 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc06dc90 0x7f5bdc150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc06dc90 0x7f5bdc150da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36836/0 (socket says 172.21.15.67:36836) 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 -- 172.21.15.67:0/534837007 learned_addr learned my addr 172.21.15.67:0/534837007 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdbfff640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bdc0caf60 0x7f5bdc1f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdaffd640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bdc073bf0 0x7f5bdc14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 -- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bdc073bf0 msgr2=0x7f5bdc14e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bdc073bf0 0x7f5bdc14e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 -- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bdc0caf60 msgr2=0x7f5bdc1f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bdc0caf60 0x7f5bdc1f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 -- 172.21.15.67:0/534837007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc155560 con 0x7f5bdc06dc90 2024-04-25T15:55:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bdb7fe640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc06dc90 0x7f5bdc150da0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00e6e0 tx=0x7f5bcc00eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bd3fff640 1 -- 172.21.15.67:0/534837007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc0052d0 con 0x7f5bdc06dc90 2024-04-25T15:55:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bd3fff640 1 -- 172.21.15.67:0/534837007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bcc005470 con 0x7f5bdc06dc90 2024-04-25T15:55:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc154700 con 0x7f5bdc06dc90 2024-04-25T15:55:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5bd3fff640 1 -- 172.21.15.67:0/534837007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc0018b0 con 0x7f5bdc06dc90 2024-04-25T15:55:13.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.373+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc154420 con 0x7f5bdc06dc90 2024-04-25T15:55:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.377+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bdc06a480 con 0x7f5bdc06dc90 2024-04-25T15:55:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.377+0000 7f5bd3fff640 1 -- 172.21.15.67:0/534837007 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5bcc001a50 con 0x7f5bdc06dc90 2024-04-25T15:55:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.377+0000 7f5bd3fff640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5bbc06f430 0x7f5bbc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.377+0000 7f5bd3fff640 1 -- 172.21.15.67:0/534837007 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f5bcc0915f0 con 0x7f5bdc06dc90 2024-04-25T15:55:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.377+0000 7f5bdaffd640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5bbc06f430 0x7f5bbc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.377+0000 7f5bdaffd640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5bbc06f430 0x7f5bbc071940 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40ae000 tx=0x7f5bc40a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.381+0000 7f5bd3fff640 1 -- 172.21.15.67:0/534837007 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5bcc016020 con 0x7f5bdc06dc90 2024-04-25T15:55:13.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.541+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f5bdc0c8a40 con 0x7f5bbc06f430 2024-04-25T15:55:13.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.541+0000 7f5bd3fff640 1 -- 172.21.15.67:0/534837007 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5bdc0c8a40 con 0x7f5bbc06f430 2024-04-25T15:55:13.543 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5bbc06f430 msgr2=0x7f5bbc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5bbc06f430 0x7f5bbc071940 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40ae000 tx=0x7f5bc40a7070 comp rx=0 tx=0).stop 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc06dc90 msgr2=0x7f5bdc150da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc06dc90 0x7f5bdc150da0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00e6e0 tx=0x7f5bcc00eba0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 shutdown_connections 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f5bbc06f430 0x7f5bbc071940 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bdc0caf60 0x7f5bdc1f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bdc073bf0 0x7f5bdc14e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 --2- 172.21.15.67:0/534837007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bdc06dc90 0x7f5bdc150da0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 >> 172.21.15.67:0/534837007 conn(0x7f5bdc070db0 msgr2=0x7f5bdc0c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:13.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 shutdown_connections 2024-04-25T15:55:13.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.545+0000 7f5be216e640 1 -- 172.21.15.67:0/534837007 wait complete. 2024-04-25T15:55:13.565 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T15:55:13.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.709+0000 7fcb10a33640 1 Processor -- start 2024-04-25T15:55:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.709+0000 7fcb10a33640 1 -- start start 2024-04-25T15:55:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb10a33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb0c15a530 0x7fcb0c15a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb10a33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 0x7fcb0c165180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb10a33640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb0c0c3980 con 0x7fcb0c157940 2024-04-25T15:55:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb10a33640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fcb0c0c2fb0 con 0x7fcb0c15a530 2024-04-25T15:55:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb10a33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c0c3290 con 0x7fcb0c162d60 2024-04-25T15:55:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb0c15a530 0x7fcb0c15a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb0c15a530 0x7fcb0c15a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36842/0 (socket says 172.21.15.67:36842) 2024-04-25T15:55:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb0a59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcb0c157940 0x7fcb0c0ca9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44972/0 (socket says 172.21.15.67:44972) 2024-04-25T15:55:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 -- 172.21.15.67:0/1776075363 learned_addr learned my addr 172.21.15.67:0/1776075363 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb0ad9d640 1 --2- 172.21.15.67:0/1776075363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 0x7fcb0c165180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 -- 172.21.15.67:0/1776075363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 msgr2=0x7fcb0c165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 --2- 172.21.15.67:0/1776075363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 0x7fcb0c165180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 -- 172.21.15.67:0/1776075363 >> v1:172.21.15.67:6789/0 conn(0x7fcb0c157940 legacy=0x7fcb0c0ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 -- 172.21.15.67:0/1776075363 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0c3c40 con 0x7fcb0c15a530 2024-04-25T15:55:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb0ad9d640 1 --2- 172.21.15.67:0/1776075363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 0x7fcb0c165180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb03fff640 1 --2- 172.21.15.67:0/1776075363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c15a940 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009870 tx=0x7fcaf401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2303362ac3d05f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb09d9b640 1 -- 172.21.15.67:0/1776075363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40b5070 con 0x7fcb0c15a530 2024-04-25T15:55:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb09d9b640 1 -- 172.21.15.67:0/1776075363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf40a8a50 con 0x7fcb0c15a530 2024-04-25T15:55:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb09d9b640 1 -- 172.21.15.67:0/1776075363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40a8e20 con 0x7fcb0c15a530 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb10a33640 1 -- 172.21.15.67:0/1776075363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 msgr2=0x7fcb0c15a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.713+0000 7fcb10a33640 1 --2- 172.21.15.67:0/1776075363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c15a940 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009870 tx=0x7fcaf401ed50 comp rx=0 tx=0).stop 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- 172.21.15.67:0/1776075363 shutdown_connections 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 --2- 172.21.15.67:0/1776075363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 0x7fcb0c165180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 --2- 172.21.15.67:0/1776075363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c15a940 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- 172.21.15.67:0/1776075363 >> 172.21.15.67:0/1776075363 conn(0x7fcb0c06fd30 msgr2=0x7fcb0c070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- 172.21.15.67:0/1776075363 shutdown_connections 2024-04-25T15:55:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- 172.21.15.67:0/1776075363 wait complete. 2024-04-25T15:55:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 Processor -- start 2024-04-25T15:55:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- start start 2024-04-25T15:55:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c157940 0x7fcb0c153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c1500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 0x7fcb0c151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c06d9f0 con 0x7fcb0c15a530 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c15ae80 con 0x7fcb0c157940 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb10a33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c0c84b0 con 0x7fcb0c162d60 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c1500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c1500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36854/0 (socket says 172.21.15.67:36854) 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 -- 172.21.15.67:0/4117777223 learned_addr learned my addr 172.21.15.67:0/4117777223 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 -- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 msgr2=0x7fcb0c151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb0c162d60 0x7fcb0c151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 -- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c157940 msgr2=0x7fcb0c153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb0c157940 0x7fcb0c153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.717+0000 7fcb03fff640 1 -- 172.21.15.67:0/4117777223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0cb990 con 0x7fcb0c15a530 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb03fff640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c1500d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fcaf401d400 tx=0x7fcaf401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40c0070 con 0x7fcb0c15a530 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf40ae790 con 0x7fcb0c15a530 2024-04-25T15:55:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40b5070 con 0x7fcb0c15a530 2024-04-25T15:55:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c0c71a0 con 0x7fcb0c15a530 2024-04-25T15:55:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c0c7440 con 0x7fcb0c15a530 2024-04-25T15:55:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb0c153550 con 0x7fcb0c15a530 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcaf40c3070 con 0x7fcb0c15a530 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcaec06f430 0x7fcaec071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fcaf412d600 con 0x7fcb0c15a530 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fcaec077b60 0x7fcaec07a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcaf40c4eb0 con 0x7fcaec077b60 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7fcaf412da90 con 0x7fcb0c15a530 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcaec06f430 0x7fcaec071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.721+0000 7fcb0ad9d640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fcaec077b60 0x7fcaec07a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.725+0000 7fcb0a59c640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcaec06f430 0x7fcaec071940 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcafc000f70 tx=0x7fcafc007a10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.725+0000 7fcb0ad9d640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fcaec077b60 0x7fcaec07a050 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.725+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== osd.5 v2:172.21.15.176:6824/2964881609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcaf40c4eb0 con 0x7fcaec077b60 2024-04-25T15:55:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.737+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 --> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcb0c06a480 con 0x7fcaec077b60 2024-04-25T15:55:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.737+0000 7fcb02ffd640 1 -- 172.21.15.67:0/4117777223 <== osd.5 v2:172.21.15.176:6824/2964881609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcb0c06a480 con 0x7fcaec077b60 2024-04-25T15:55:13.740 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:13.740 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:13.740 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fcaec077b60 msgr2=0x7fcaec07a050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6824/2964881609,v1:172.21.15.176:6825/2964881609] conn(0x7fcaec077b60 0x7fcaec07a050 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcaec06f430 msgr2=0x7fcaec071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fcaec06f430 0x7fcaec071940 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fcafc000f70 tx=0x7fcafc007a10 comp rx=0 tx=0).stop 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 msgr2=0x7fcb0c1500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 --2- 172.21.15.67:0/4117777223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb0c15a530 0x7fcb0c1500d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fcaf401d400 tx=0x7fcaf401baf0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb0ad9d640 1 -- 172.21.15.67:0/4117777223 reap_dead start 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 shutdown_connections 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 >> 172.21.15.67:0/4117777223 conn(0x7fcb0c06fd30 msgr2=0x7fcb0c159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:13.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 shutdown_connections 2024-04-25T15:55:13.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.741+0000 7fcb10a33640 1 -- 172.21.15.67:0/4117777223 wait complete. 2024-04-25T15:55:13.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T15:55:13.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.897+0000 7f53f513b640 1 Processor -- start 2024-04-25T15:55:13.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.897+0000 7f53f513b640 1 -- start start 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53f513b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53f015dd00 0x7f53f0068830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53f513b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f0160990 con 0x7f53f0156db0 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53f513b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f0160c70 con 0x7f53f01569a0 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53f513b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f53f015e110 con 0x7f53f015dd00 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53eed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53f015dd00 0x7f53f0068830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53eed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f53f015dd00 0x7f53f0068830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36870/0 (socket says 172.21.15.67:36870) 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53eed9d640 1 -- 172.21.15.67:0/2242476845 learned_addr learned my addr 172.21.15.67:0/2242476845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53eed9d640 1 -- 172.21.15.67:0/2242476845 >> v1:172.21.15.176:6789/0 conn(0x7f53f01569a0 legacy=0x7f53f006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53eed9d640 1 -- 172.21.15.67:0/2242476845 >> v1:172.21.15.67:6789/0 conn(0x7f53f0156db0 legacy=0x7f53f0167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.901+0000 7f53eed9d640 1 -- 172.21.15.67:0/2242476845 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f00c3bf0 con 0x7f53f015dd00 2024-04-25T15:55:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53eed9d640 1 --2- 172.21.15.67:0/2242476845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f015dd00 0x7f53f0068830 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009870 tx=0x7f53e001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15d83d1ff804a02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53edd9b640 1 -- 172.21.15.67:0/2242476845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e00b2070 con 0x7f53f015dd00 2024-04-25T15:55:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53edd9b640 1 -- 172.21.15.67:0/2242476845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e00a9070 con 0x7f53f015dd00 2024-04-25T15:55:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53edd9b640 1 -- 172.21.15.67:0/2242476845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e00ad880 con 0x7f53f015dd00 2024-04-25T15:55:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 -- 172.21.15.67:0/2242476845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f015dd00 msgr2=0x7f53f0068830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 --2- 172.21.15.67:0/2242476845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f015dd00 0x7f53f0068830 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009870 tx=0x7f53e001c560 comp rx=0 tx=0).stop 2024-04-25T15:55:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 -- 172.21.15.67:0/2242476845 shutdown_connections 2024-04-25T15:55:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 --2- 172.21.15.67:0/2242476845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f015dd00 0x7f53f0068830 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 -- 172.21.15.67:0/2242476845 >> 172.21.15.67:0/2242476845 conn(0x7f53f00ce1c0 msgr2=0x7f53f00ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 -- 172.21.15.67:0/2242476845 shutdown_connections 2024-04-25T15:55:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 -- 172.21.15.67:0/2242476845 wait complete. 2024-04-25T15:55:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 Processor -- start 2024-04-25T15:55:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.905+0000 7f53f513b640 1 -- start start 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f01569a0 0x7f53f00b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53f0156db0 0x7f53f00c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53f015dd00 0x7f53f00c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53f006d7e0 con 0x7f53f01569a0 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f53f006d660 con 0x7f53f015dd00 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f53f006cd50 con 0x7f53f0156db0 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f01569a0 0x7f53f00b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f01569a0 0x7f53f00b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36880/0 (socket says 172.21.15.67:36880) 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 -- 172.21.15.67:0/3537005983 learned_addr learned my addr 172.21.15.67:0/3537005983 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53ee59c640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53f0156db0 0x7f53f00c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53ef59e640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53f015dd00 0x7f53f00c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 -- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53f0156db0 msgr2=0x7f53f00c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53f0156db0 0x7f53f00c5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 -- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53f015dd00 msgr2=0x7f53f00c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53f015dd00 0x7f53f00c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 -- 172.21.15.67:0/3537005983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f00c6fa0 con 0x7f53f01569a0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53ef59e640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53f015dd00 0x7f53f00c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53eed9d640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f01569a0 0x7f53f00b3140 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f53e00b0040 tx=0x7f53e0019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53e77fe640 1 -- 172.21.15.67:0/3537005983 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e00bc070 con 0x7f53f01569a0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53e77fe640 1 -- 172.21.15.67:0/3537005983 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e00b6640 con 0x7f53f01569a0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53e77fe640 1 -- 172.21.15.67:0/3537005983 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e00b2040 con 0x7f53f01569a0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f00c79a0 con 0x7f53f01569a0 2024-04-25T15:55:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f00b3ef0 con 0x7f53f01569a0 2024-04-25T15:55:13.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f006a480 con 0x7f53f01569a0 2024-04-25T15:55:13.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.909+0000 7f53e77fe640 1 -- 172.21.15.67:0/3537005983 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f53e00ab070 con 0x7f53f01569a0 2024-04-25T15:55:13.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.913+0000 7f53e77fe640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f53d006f400 0x7f53d0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:13.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.913+0000 7f53e77fe640 1 -- 172.21.15.67:0/3537005983 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f53e01286f0 con 0x7f53f01569a0 2024-04-25T15:55:13.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.913+0000 7f53ee59c640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f53d006f400 0x7f53d0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:13.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.913+0000 7f53ee59c640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f53d006f400 0x7f53d0071910 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f53d8007fd0 tx=0x7f53d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:13.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:13.917+0000 7f53e77fe640 1 -- 172.21.15.67:0/3537005983 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f53e0016150 con 0x7f53f01569a0 2024-04-25T15:55:14.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.077+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f53f00cd770 con 0x7f53d006f400 2024-04-25T15:55:14.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.077+0000 7f53e77fe640 1 -- 172.21.15.67:0/3537005983 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f53f00cd770 con 0x7f53d006f400 2024-04-25T15:55:14.078 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f53d006f400 msgr2=0x7f53d0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f53d006f400 0x7f53d0071910 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f53d8007fd0 tx=0x7f53d8009040 comp rx=0 tx=0).stop 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f01569a0 msgr2=0x7f53f00b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f01569a0 0x7f53f00b3140 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f53e00b0040 tx=0x7f53e0019300 comp rx=0 tx=0).stop 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 shutdown_connections 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f53d006f400 0x7f53d0071910 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53f015dd00 0x7f53f00c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53f0156db0 0x7f53f00c5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 --2- 172.21.15.67:0/3537005983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53f01569a0 0x7f53f00b3140 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 >> 172.21.15.67:0/3537005983 conn(0x7f53f00ce1c0 msgr2=0x7f53f0154ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 shutdown_connections 2024-04-25T15:55:14.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.081+0000 7f53f513b640 1 -- 172.21.15.67:0/3537005983 wait complete. 2024-04-25T15:55:14.100 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T15:55:14.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:14.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.257+0000 7fb7b0a33640 1 Processor -- start 2024-04-25T15:55:14.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.257+0000 7fb7b0a33640 1 -- start start 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7b0a33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac071da0 0x7fb7ac152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7b0a33640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7ac0721b0 0x7fb7ac152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7b0a33640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7ac0b3b60 con 0x7fb7ac0c4630 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7b0a33640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fb7ac0c38b0 con 0x7fb7ac0721b0 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7b0a33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ac06cf20 con 0x7fb7ac071da0 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aa59c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7ac0c4630 0x7fb7ac06c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44998/0 (socket says 172.21.15.67:44998) 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aad9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7ac0721b0 0x7fb7ac152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aa59c640 1 -- 172.21.15.67:0/22664450 learned_addr learned my addr 172.21.15.67:0/22664450 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:14.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7a9d9b640 1 --2- 172.21.15.67:0/22664450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac071da0 0x7fb7ac152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aad9d640 1 -- 172.21.15.67:0/22664450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac071da0 msgr2=0x7fb7ac152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aad9d640 1 --2- 172.21.15.67:0/22664450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac071da0 0x7fb7ac152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aad9d640 1 -- 172.21.15.67:0/22664450 >> v1:172.21.15.67:6789/0 conn(0x7fb7ac0c4630 legacy=0x7fb7ac06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aad9d640 1 -- 172.21.15.67:0/22664450 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ac0b3e40 con 0x7fb7ac0721b0 2024-04-25T15:55:14.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7aad9d640 1 --2- 172.21.15.67:0/22664450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac0721b0 0x7fb7ac152ff0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009d90 tx=0x7fb7a401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7789698fe00b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7a959a640 1 -- 172.21.15.67:0/22664450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a40b4070 con 0x7fb7ac0721b0 2024-04-25T15:55:14.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7a959a640 1 -- 172.21.15.67:0/22664450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a40af600 con 0x7fb7ac0721b0 2024-04-25T15:55:14.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.261+0000 7fb7a959a640 1 -- 172.21.15.67:0/22664450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a40af9d0 con 0x7fb7ac0721b0 2024-04-25T15:55:14.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- 172.21.15.67:0/22664450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac0721b0 msgr2=0x7fb7ac152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 --2- 172.21.15.67:0/22664450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac0721b0 0x7fb7ac152ff0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4009d90 tx=0x7fb7a401f150 comp rx=0 tx=0).stop 2024-04-25T15:55:14.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- 172.21.15.67:0/22664450 shutdown_connections 2024-04-25T15:55:14.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 --2- 172.21.15.67:0/22664450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac0721b0 0x7fb7ac152ff0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 --2- 172.21.15.67:0/22664450 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac071da0 0x7fb7ac152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- 172.21.15.67:0/22664450 >> 172.21.15.67:0/22664450 conn(0x7fb7ac06fd30 msgr2=0x7fb7ac070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:14.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- 172.21.15.67:0/22664450 shutdown_connections 2024-04-25T15:55:14.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- 172.21.15.67:0/22664450 wait complete. 2024-04-25T15:55:14.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 Processor -- start 2024-04-25T15:55:14.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- start start 2024-04-25T15:55:14.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac071da0 0x7fb7ac0c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7ac0721b0 0x7fb7ac154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac0c4630 0x7fb7ac0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac0b3820 con 0x7fb7ac071da0 2024-04-25T15:55:14.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ac0c3de0 con 0x7fb7ac0721b0 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7b0a33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ac15c160 con 0x7fb7ac0c4630 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.265+0000 7fb7aa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac071da0 0x7fb7ac0c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac071da0 0x7fb7ac0c6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36896/0 (socket says 172.21.15.67:36896) 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 -- 172.21.15.67:0/2671846163 learned_addr learned my addr 172.21.15.67:0/2671846163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7a9d9b640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7ac0721b0 0x7fb7ac154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aad9d640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac0c4630 0x7fb7ac0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 -- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac0c4630 msgr2=0x7fb7ac0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac0c4630 0x7fb7ac0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 -- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7ac0721b0 msgr2=0x7fb7ac154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7ac0721b0 0x7fb7ac154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 -- 172.21.15.67:0/2671846163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ac0c7130 con 0x7fb7ac071da0 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aad9d640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7ac0c4630 0x7fb7ac0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:14.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7a9d9b640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7ac0721b0 0x7fb7ac154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:14.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7aa59c640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac071da0 0x7fb7ac0c6bf0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00c3a0 tx=0x7fb79c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c017070 con 0x7fb7ac071da0 2024-04-25T15:55:14.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb79c005cf0 con 0x7fb7ac071da0 2024-04-25T15:55:14.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ac0cd5e0 con 0x7fb7ac071da0 2024-04-25T15:55:14.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0072f0 con 0x7fb7ac071da0 2024-04-25T15:55:14.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ac0c63f0 con 0x7fb7ac071da0 2024-04-25T15:55:14.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb7ac06a480 con 0x7fb7ac071da0 2024-04-25T15:55:14.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb79c01a070 con 0x7fb7ac071da0 2024-04-25T15:55:14.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb78c06f430 0x7fb78c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fb79c08ee90 con 0x7fb7ac071da0 2024-04-25T15:55:14.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb78c077b60 0x7fb78c07a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb79c01dec0 con 0x7fb78c077b60 2024-04-25T15:55:14.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7fb79c08f320 con 0x7fb7ac071da0 2024-04-25T15:55:14.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.269+0000 7fb7a9d9b640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb78c06f430 0x7fb78c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.273+0000 7fb7aad9d640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb78c077b60 0x7fb78c07a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.273+0000 7fb7a9d9b640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb78c06f430 0x7fb78c071940 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac1482a0 tx=0x7fb7a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.273+0000 7fb7aad9d640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb78c077b60 0x7fb78c07a050 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.273+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== osd.6 v2:172.21.15.176:6809/211167612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb79c01dec0 con 0x7fb78c077b60 2024-04-25T15:55:14.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 --> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb7ac0c9bb0 con 0x7fb78c077b60 2024-04-25T15:55:14.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb79affd640 1 -- 172.21.15.67:0/2671846163 <== osd.6 v2:172.21.15.176:6809/211167612 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb7ac0c9bb0 con 0x7fb78c077b60 2024-04-25T15:55:14.288 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:14.288 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:14.289 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb78c077b60 msgr2=0x7fb78c07a050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6809/211167612,v1:172.21.15.176:6811/211167612] conn(0x7fb78c077b60 0x7fb78c07a050 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb78c06f430 msgr2=0x7fb78c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fb78c06f430 0x7fb78c071940 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac1482a0 tx=0x7fb7a0008040 comp rx=0 tx=0).stop 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac071da0 msgr2=0x7fb7ac0c6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 --2- 172.21.15.67:0/2671846163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7ac071da0 0x7fb7ac0c6bf0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00c3a0 tx=0x7fb79c00c860 comp rx=0 tx=0).stop 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7aad9d640 1 -- 172.21.15.67:0/2671846163 reap_dead start 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 shutdown_connections 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 >> 172.21.15.67:0/2671846163 conn(0x7fb7ac06fd30 msgr2=0x7fb7ac150400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 shutdown_connections 2024-04-25T15:55:14.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.289+0000 7fb7b0a33640 1 -- 172.21.15.67:0/2671846163 wait complete. 2024-04-25T15:55:14.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T15:55:14.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd0816f9640 1 Processor -- start 2024-04-25T15:55:14.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd0816f9640 1 -- start start 2024-04-25T15:55:14.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd0816f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c0c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd0816f9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd07c0c5bb0 0x7fd07c0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd0816f9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd07c0c6890 con 0x7fd07c071b20 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd0816f9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd07c15cd40 con 0x7fd07c0c5bb0 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd0816f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd07c0c6b70 con 0x7fd07c0c7610 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd07b7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd07c0c5bb0 0x7fd07c0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd07a7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c0c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd07affd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fd07c071b20 0x7fd07c15d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:46244/0 (socket says 172.21.15.67:46244) 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd07b7fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd07c0c5bb0 0x7fd07c0c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:32984/0 (socket says 172.21.15.67:32984) 2024-04-25T15:55:14.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.453+0000 7fd07affd640 1 -- 172.21.15.67:0/1570507420 learned_addr learned my addr 172.21.15.67:0/1570507420 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd07a7fc640 1 -- 172.21.15.67:0/1570507420 >> v2:172.21.15.176:3300/0 conn(0x7fd07c0c5bb0 msgr2=0x7fd07c0c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd07a7fc640 1 --2- 172.21.15.67:0/1570507420 >> v2:172.21.15.176:3300/0 conn(0x7fd07c0c5bb0 0x7fd07c0c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd07a7fc640 1 -- 172.21.15.67:0/1570507420 >> v1:172.21.15.176:6789/0 conn(0x7fd07c071b20 legacy=0x7fd07c15d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd07a7fc640 1 -- 172.21.15.67:0/1570507420 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c0c6e50 con 0x7fd07c0c7610 2024-04-25T15:55:14.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd07a7fc640 1 --2- 172.21.15.67:0/1570507420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c0c7a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd0700097c0 tx=0x7fd07001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7663c56e46e05f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd079ffb640 1 -- 172.21.15.67:0/1570507420 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0700b1070 con 0x7fd07c0c7610 2024-04-25T15:55:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd079ffb640 1 -- 172.21.15.67:0/1570507420 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0700ac410 con 0x7fd07c0c7610 2024-04-25T15:55:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.457+0000 7fd079ffb640 1 -- 172.21.15.67:0/1570507420 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0700ac7e0 con 0x7fd07c0c7610 2024-04-25T15:55:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 -- 172.21.15.67:0/1570507420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 msgr2=0x7fd07c0c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1570507420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c0c7a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd0700097c0 tx=0x7fd07001c4a0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 -- 172.21.15.67:0/1570507420 shutdown_connections 2024-04-25T15:55:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1570507420 >> v2:172.21.15.176:3300/0 conn(0x7fd07c0c5bb0 0x7fd07c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1570507420 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c0c7a20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 -- 172.21.15.67:0/1570507420 >> 172.21.15.67:0/1570507420 conn(0x7fd07c06fe70 msgr2=0x7fd07c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:14.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 -- 172.21.15.67:0/1570507420 shutdown_connections 2024-04-25T15:55:14.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 -- 172.21.15.67:0/1570507420 wait complete. 2024-04-25T15:55:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 Processor -- start 2024-04-25T15:55:14.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.461+0000 7fd0816f9640 1 -- start start 2024-04-25T15:55:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd07c071b20 0x7fd07c14aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd07c0c5bb0 0x7fd07c154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c06c640 con 0x7fd07c071b20 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c0734f0 con 0x7fd07c0c5bb0 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd07c06c7c0 con 0x7fd07c0c7610 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd07c071b20 0x7fd07c14aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd07c071b20 0x7fd07c14aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36910/0 (socket says 172.21.15.67:36910) 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 -- 172.21.15.67:0/1117369805 learned_addr learned my addr 172.21.15.67:0/1117369805 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07a7fc640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd07c0c5bb0 0x7fd07c154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07b7fe640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 -- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 msgr2=0x7fd07c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 -- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd07c0c5bb0 msgr2=0x7fd07c154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd07c0c5bb0 0x7fd07c154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 -- 172.21.15.67:0/1117369805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c14b3e0 con 0x7fd07c071b20 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07b7fe640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07affd640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd07c071b20 0x7fd07c14aea0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c370 tx=0x7fd06c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0637fe640 1 -- 172.21.15.67:0/1117369805 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c017070 con 0x7fd07c071b20 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0637fe640 1 -- 172.21.15.67:0/1117369805 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c005cf0 con 0x7fd07c071b20 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07c1551c0 con 0x7fd07c071b20 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0637fe640 1 -- 172.21.15.67:0/1117369805 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0072f0 con 0x7fd07c071b20 2024-04-25T15:55:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd07c155780 con 0x7fd07c071b20 2024-04-25T15:55:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd07c06a480 con 0x7fd07c071b20 2024-04-25T15:55:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0637fe640 1 -- 172.21.15.67:0/1117369805 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd06c01a070 con 0x7fd07c071b20 2024-04-25T15:55:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0637fe640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06806f400 0x7fd068071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd0637fe640 1 -- 172.21.15.67:0/1117369805 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fd06c08e630 con 0x7fd07c071b20 2024-04-25T15:55:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.465+0000 7fd07a7fc640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06806f400 0x7fd068071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.469+0000 7fd07a7fc640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06806f400 0x7fd068071910 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd0700ae000 tx=0x7fd0700a6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.473+0000 7fd0637fe640 1 -- 172.21.15.67:0/1117369805 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd06c05b500 con 0x7fd07c071b20 2024-04-25T15:55:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fd07c0c4700 con 0x7fd06806f400 2024-04-25T15:55:14.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0637fe640 1 -- 172.21.15.67:0/1117369805 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd07c0c4700 con 0x7fd06806f400 2024-04-25T15:55:14.633 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06806f400 msgr2=0x7fd068071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06806f400 0x7fd068071910 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd0700ae000 tx=0x7fd0700a6040 comp rx=0 tx=0).stop 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd07c071b20 msgr2=0x7fd07c14aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd07c071b20 0x7fd07c14aea0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c370 tx=0x7fd06c00c830 comp rx=0 tx=0).stop 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 shutdown_connections 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fd06806f400 0x7fd068071910 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd07c0c7610 0x7fd07c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd07c0c5bb0 0x7fd07c154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 --2- 172.21.15.67:0/1117369805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd07c071b20 0x7fd07c14aea0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 >> 172.21.15.67:0/1117369805 conn(0x7fd07c06fe70 msgr2=0x7fd07c074240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:14.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.633+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 shutdown_connections 2024-04-25T15:55:14.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.637+0000 7fd0816f9640 1 -- 172.21.15.67:0/1117369805 wait complete. 2024-04-25T15:55:14.656 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T15:55:14.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T15:55:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.813+0000 7f40dab98640 1 Processor -- start 2024-04-25T15:55:14.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.813+0000 7f40dab98640 1 -- start start 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40dab98640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f40d415a530 0x7f40d415a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40dab98640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40d4162d60 0x7f40d4165180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40dab98640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d40c3980 con 0x7f40d4157940 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40dab98640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f40d40c2fb0 con 0x7f40d4162d60 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40dab98640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f40d40c3290 con 0x7f40d415a530 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40d4162d60 0x7f40d4165180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40d4162d60 0x7f40d4165180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36926/0 (socket says 172.21.15.67:36926) 2024-04-25T15:55:14.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 -- 172.21.15.67:0/3352968689 learned_addr learned my addr 172.21.15.67:0/3352968689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d3fff640 1 --2- 172.21.15.67:0/3352968689 >> v2:172.21.15.176:3300/0 conn(0x7f40d415a530 0x7f40d415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 -- 172.21.15.67:0/3352968689 >> v2:172.21.15.176:3300/0 conn(0x7f40d415a530 msgr2=0x7f40d415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 --2- 172.21.15.67:0/3352968689 >> v2:172.21.15.176:3300/0 conn(0x7f40d415a530 0x7f40d415a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 -- 172.21.15.67:0/3352968689 >> v1:172.21.15.176:6789/0 conn(0x7f40d4157940 legacy=0x7f40d40ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:14.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 -- 172.21.15.67:0/3352968689 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d40c3c40 con 0x7f40d4162d60 2024-04-25T15:55:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d9135640 1 --2- 172.21.15.67:0/3352968689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d4165180 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f40d40cb640 tx=0x7f40cc01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4795e431e51adc7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d37fe640 1 -- 172.21.15.67:0/3352968689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0b4070 con 0x7f40d4162d60 2024-04-25T15:55:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d37fe640 1 -- 172.21.15.67:0/3352968689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40cc0af400 con 0x7f40d4162d60 2024-04-25T15:55:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.817+0000 7f40d37fe640 1 -- 172.21.15.67:0/3352968689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0af7d0 con 0x7f40d4162d60 2024-04-25T15:55:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- 172.21.15.67:0/3352968689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 msgr2=0x7f40d4165180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 --2- 172.21.15.67:0/3352968689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d4165180 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f40d40cb640 tx=0x7f40cc01f1a0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- 172.21.15.67:0/3352968689 shutdown_connections 2024-04-25T15:55:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 --2- 172.21.15.67:0/3352968689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d4165180 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 --2- 172.21.15.67:0/3352968689 >> v2:172.21.15.176:3300/0 conn(0x7f40d415a530 0x7f40d415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- 172.21.15.67:0/3352968689 >> 172.21.15.67:0/3352968689 conn(0x7f40d406fd30 msgr2=0x7f40d4070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- 172.21.15.67:0/3352968689 shutdown_connections 2024-04-25T15:55:14.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- 172.21.15.67:0/3352968689 wait complete. 2024-04-25T15:55:14.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 Processor -- start 2024-04-25T15:55:14.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- start start 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40d4157940 0x7f40d41f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40d415a530 0x7f40d4161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d415d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f40d406d9f0 con 0x7f40d4162d60 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f40d415ae80 con 0x7f40d415a530 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40dab98640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f40d40c84b0 con 0x7f40d4157940 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40d9135640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d415d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40d9135640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d415d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36928/0 (socket says 172.21.15.67:36928) 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40d9135640 1 -- 172.21.15.67:0/2684450391 learned_addr learned my addr 172.21.15.67:0/2684450391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40d3fff640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40d415a530 0x7f40d4161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.821+0000 7f40d8934640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40d4157940 0x7f40d41f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d9135640 1 -- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40d4157940 msgr2=0x7f40d41f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d9135640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40d4157940 0x7f40d41f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d9135640 1 -- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40d415a530 msgr2=0x7f40d4161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d9135640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40d415a530 0x7f40d4161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d9135640 1 -- 172.21.15.67:0/2684450391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d415b5b0 con 0x7f40d4162d60 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d9135640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d415d2d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f40cc01cdd0 tx=0x7f40cc01bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0b4040 con 0x7f40d4162d60 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40cc0c0070 con 0x7f40d4162d60 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d41f1e40 con 0x7f40d4162d60 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc009510 con 0x7f40d4162d60 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d415e5e0 con 0x7f40d4162d60 2024-04-25T15:55:14.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40d415e390 con 0x7f40d4162d60 2024-04-25T15:55:14.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f40cc0ba660 con 0x7f40d4162d60 2024-04-25T15:55:14.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f40c006f400 0x7f40c0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f40cc12dff0 con 0x7f40d4162d60 2024-04-25T15:55:14.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f40c0077b30 0x7f40c007a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:14.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40cc0af400 con 0x7f40c0077b30 2024-04-25T15:55:14.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=129) ==== 24+0+0 (secure 0 0 0) 0x7f40cc12e480 con 0x7f40d4162d60 2024-04-25T15:55:14.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d8934640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f40c006f400 0x7f40c0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.825+0000 7f40d3fff640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f40c0077b30 0x7f40c007a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:14.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.829+0000 7f40d3fff640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f40c0077b30 0x7f40c007a020 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.829+0000 7f40d8934640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f40c006f400 0x7f40c0071910 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f40c400b810 tx=0x7f40c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:14.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.829+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== osd.7 v2:172.21.15.176:6800/604707640 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40cc0af400 con 0x7f40c0077b30 2024-04-25T15:55:14.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.841+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 --> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f40d4152360 con 0x7f40c0077b30 2024-04-25T15:55:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40d17fa640 1 -- 172.21.15.67:0/2684450391 <== osd.7 v2:172.21.15.176:6800/604707640 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f40d4152360 con 0x7f40c0077b30 2024-04-25T15:55:14.844 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-25T15:55:14.844 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T15:55:14.844 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-25T15:55:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f40c0077b30 msgr2=0x7f40c007a020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6800/604707640,v1:172.21.15.176:6801/604707640] conn(0x7f40c0077b30 0x7f40c007a020 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f40c006f400 msgr2=0x7f40c0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f40c006f400 0x7f40c0071910 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f40c400b810 tx=0x7f40c4008040 comp rx=0 tx=0).stop 2024-04-25T15:55:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 msgr2=0x7f40d415d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 --2- 172.21.15.67:0/2684450391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40d4162d60 0x7f40d415d2d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f40cc01cdd0 tx=0x7f40cc01bf40 comp rx=0 tx=0).stop 2024-04-25T15:55:14.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40d9135640 1 -- 172.21.15.67:0/2684450391 reap_dead start 2024-04-25T15:55:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 shutdown_connections 2024-04-25T15:55:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 >> 172.21.15.67:0/2684450391 conn(0x7f40d406fd30 msgr2=0x7f40d4155a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 shutdown_connections 2024-04-25T15:55:14.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:14.845+0000 7f40dab98640 1 -- 172.21.15.67:0/2684450391 wait complete. 2024-04-25T15:55:14.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T15:55:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af99fd640 1 Processor -- start 2024-04-25T15:55:15.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af99fd640 1 -- start start 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af99fd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 0x7f3af40736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af99fd640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af40caa60 con 0x7f3af41547e0 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af99fd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af415a5f0 con 0x7f3af415aaf0 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af99fd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3af4068590 con 0x7f3af406d460 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3af41547e0 0x7f3af406c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45008/0 (socket says 172.21.15.67:45008) 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 -- 172.21.15.67:0/521753958 learned_addr learned my addr 172.21.15.67:0/521753958 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af2ffd640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 0x7f3af40736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365832605 0 0) 0x7f3af40caa60 con 0x7f3af41547e0 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae0003530 con 0x7f3af41547e0 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 473347502 0 0) 0x7f3ae0003530 con 0x7f3af41547e0 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 msgr2=0x7f3af40736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 0x7f3af40736d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 >> v1:172.21.15.176:6789/0 conn(0x7f3af415aaf0 legacy=0x7f3af415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af40ca100 con 0x7f3af41547e0 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3343474840 0 0) 0x7f3adc001800 con 0x7f3af41547e0 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 >> v1:172.21.15.67:6789/0 conn(0x7f3af41547e0 legacy=0x7f3af406c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ae00039f0 0x7f3ae0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae0004810 0x7f3ae0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 0x7f3ae0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3af4068590 con 0x7f3ae0004810 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae000bb00 con 0x7f3ae00039f0 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae000bd00 con 0x7f3af406d460 2024-04-25T15:55:15.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae0004810 0x7f3ae0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af2ffd640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ae00039f0 0x7f3ae0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af37fe640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 0x7f3ae0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 -- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 msgr2=0x7f3ae0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 0x7f3ae0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 -- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ae00039f0 msgr2=0x7f3ae0003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ae00039f0 0x7f3ae0003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3aebfff640 1 -- 172.21.15.67:0/521753958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae0003530 con 0x7f3ae0004810 2024-04-25T15:55:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.017+0000 7f3af37fe640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3af406d460 0x7f3ae0005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3aebfff640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae0004810 0x7f3ae0004c20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3af4158200 tx=0x7f3adc01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3458e2c69dea371 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc007d10 con 0x7f3ae0004810 2024-04-25T15:55:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3adc007290 con 0x7f3ae0004810 2024-04-25T15:55:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af27fc640 1 -- 172.21.15.67:0/521753958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0057b0 con 0x7f3ae0004810 2024-04-25T15:55:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 -- 172.21.15.67:0/521753958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae0004810 msgr2=0x7f3ae0004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 --2- 172.21.15.67:0/521753958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae0004810 0x7f3ae0004c20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f3af4158200 tx=0x7f3adc01f390 comp rx=0 tx=0).stop 2024-04-25T15:55:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af37fe640 1 -- 172.21.15.67:0/521753958 reap_dead start 2024-04-25T15:55:15.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 -- 172.21.15.67:0/521753958 shutdown_connections 2024-04-25T15:55:15.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 -- 172.21.15.67:0/521753958 >> 172.21.15.67:0/521753958 conn(0x7f3af40ce1c0 msgr2=0x7f3af40ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:15.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 -- 172.21.15.67:0/521753958 shutdown_connections 2024-04-25T15:55:15.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 -- 172.21.15.67:0/521753958 wait complete. 2024-04-25T15:55:15.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 Processor -- start 2024-04-25T15:55:15.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.021+0000 7f3af99fd640 1 -- start start 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ae0004810 0x7f3af41f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3af4147540 0x7f3af4147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af4158d90 0x7f3af41591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3af4068ac0 con 0x7f3af4158d90 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3af40687e0 con 0x7f3af4147540 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3af4154bf0 con 0x7f3ae0004810 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af4158d90 0x7f3af41591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af4158d90 0x7f3af41591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36954/0 (socket says 172.21.15.67:36954) 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 -- 172.21.15.67:0/2067273686 learned_addr learned my addr 172.21.15.67:0/2067273686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3aebfff640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3af4147540 0x7f3af4147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af2ffd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ae0004810 0x7f3af41f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 -- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ae0004810 msgr2=0x7f3af41f10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ae0004810 0x7f3af41f10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 -- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3af4147540 msgr2=0x7f3af4147950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3af4147540 0x7f3af4147950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 -- 172.21.15.67:0/2067273686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af414eee0 con 0x7f3af4158d90 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af37fe640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af4158d90 0x7f3af41591a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f3af4072690 tx=0x7f3aec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af0ff9640 1 -- 172.21.15.67:0/2067273686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec006d60 con 0x7f3af4158d90 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af0ff9640 1 -- 172.21.15.67:0/2067273686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aec003980 con 0x7f3af4158d90 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af41f50d0 con 0x7f3af4158d90 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af0ff9640 1 -- 172.21.15.67:0/2067273686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec003d70 con 0x7f3af4158d90 2024-04-25T15:55:15.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af41f4e10 con 0x7f3af4158d90 2024-04-25T15:55:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af406a480 con 0x7f3af4158d90 2024-04-25T15:55:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.025+0000 7f3af0ff9640 1 -- 172.21.15.67:0/2067273686 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3aec005070 con 0x7f3af4158d90 2024-04-25T15:55:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.029+0000 7f3af0ff9640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ad406f430 0x7f3ad4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.029+0000 7f3af0ff9640 1 -- 172.21.15.67:0/2067273686 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f3aec08f060 con 0x7f3af4158d90 2024-04-25T15:55:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.029+0000 7f3af2ffd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ad406f430 0x7f3ad4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.029+0000 7f3af2ffd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ad406f430 0x7f3ad4071940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009970 tx=0x7f3ae4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:15.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.033+0000 7f3af0ff9640 1 -- 172.21.15.67:0/2067273686 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3af414eee0 con 0x7f3af4158d90 2024-04-25T15:55:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f3af415af00 con 0x7f3ad406f430 2024-04-25T15:55:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af0ff9640 1 -- 172.21.15.67:0/2067273686 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3af415af00 con 0x7f3ad406f430 2024-04-25T15:55:15.197 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ad406f430 msgr2=0x7f3ad4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ad406f430 0x7f3ad4071940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009970 tx=0x7f3ae4008040 comp rx=0 tx=0).stop 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af4158d90 msgr2=0x7f3af41591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af4158d90 0x7f3af41591a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f3af4072690 tx=0x7f3aec00dc00 comp rx=0 tx=0).stop 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 shutdown_connections 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3ad406f430 0x7f3ad4071940 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af4158d90 0x7f3af41591a0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3af4147540 0x7f3af4147950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 --2- 172.21.15.67:0/2067273686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3ae0004810 0x7f3af41f10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.197+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 >> 172.21.15.67:0/2067273686 conn(0x7f3af40ce1c0 msgr2=0x7f3af4153dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.201+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 shutdown_connections 2024-04-25T15:55:15.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.201+0000 7f3af99fd640 1 -- 172.21.15.67:0/2067273686 wait complete. 2024-04-25T15:55:15.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:55:15.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f5973dbf640 1 Processor -- start 2024-04-25T15:55:15.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f5973dbf640 1 -- start start 2024-04-25T15:55:15.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f5973dbf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c15ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f5973dbf640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f596c0ca540 0x7f596c0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f5973dbf640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f596c15de10 con 0x7f596c071bc0 2024-04-25T15:55:15.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f5973dbf640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f596c15ebb0 con 0x7f596c0ca540 2024-04-25T15:55:15.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f5973dbf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f596c15e720 con 0x7f596c15a720 2024-04-25T15:55:15.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597235c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f596c0ca540 0x7f596c0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597235c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f596c0ca540 0x7f596c0ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36966/0 (socket says 172.21.15.67:36966) 2024-04-25T15:55:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597235c640 1 -- 172.21.15.67:0/397153568 learned_addr learned my addr 172.21.15.67:0/397153568 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597135a640 1 --2- 172.21.15.67:0/397153568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c15ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597235c640 1 -- 172.21.15.67:0/397153568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 msgr2=0x7f596c15ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597235c640 1 --2- 172.21.15.67:0/397153568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c15ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597235c640 1 -- 172.21.15.67:0/397153568 >> v1:172.21.15.176:6789/0 conn(0x7f596c071bc0 legacy=0x7f596c0cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.361+0000 7f597235c640 1 -- 172.21.15.67:0/397153568 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596c072390 con 0x7f596c0ca540 2024-04-25T15:55:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f597235c640 1 --2- 172.21.15.67:0/397153568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c0ca950 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f596c15e8a0 tx=0x7f595c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5783c315b9236ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5970b59640 1 -- 172.21.15.67:0/397153568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0a57f0 con 0x7f596c0ca540 2024-04-25T15:55:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5970b59640 1 -- 172.21.15.67:0/397153568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595c0a5990 con 0x7f596c0ca540 2024-04-25T15:55:15.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5970b59640 1 -- 172.21.15.67:0/397153568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0a5d60 con 0x7f596c0ca540 2024-04-25T15:55:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 -- 172.21.15.67:0/397153568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 msgr2=0x7f596c0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 --2- 172.21.15.67:0/397153568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c0ca950 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f596c15e8a0 tx=0x7f595c01ca10 comp rx=0 tx=0).stop 2024-04-25T15:55:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 -- 172.21.15.67:0/397153568 shutdown_connections 2024-04-25T15:55:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 --2- 172.21.15.67:0/397153568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c0ca950 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 --2- 172.21.15.67:0/397153568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 -- 172.21.15.67:0/397153568 >> 172.21.15.67:0/397153568 conn(0x7f596c0700b0 msgr2=0x7f596c0704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 -- 172.21.15.67:0/397153568 shutdown_connections 2024-04-25T15:55:15.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.365+0000 7f5973dbf640 1 -- 172.21.15.67:0/397153568 wait complete. 2024-04-25T15:55:15.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 Processor -- start 2024-04-25T15:55:15.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 -- start start 2024-04-25T15:55:15.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c071bc0 0x7f596c1f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f596c071fd0 con 0x7f596c0ca540 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f596c0cc2d0 con 0x7f596c071bc0 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f596c072150 con 0x7f596c15a720 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36970/0 (socket says 172.21.15.67:36970) 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 -- 172.21.15.67:0/425924175 learned_addr learned my addr 172.21.15.67:0/425924175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5971b5b640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c071bc0 0x7f596c1f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597235c640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 -- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 msgr2=0x7f596c14f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c14f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 -- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c071bc0 msgr2=0x7f596c1f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c071bc0 0x7f596c1f7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 -- 172.21.15.67:0/425924175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596c1f7cc0 con 0x7f596c0ca540 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597235c640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:15.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f597135a640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c145e90 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f596000d7d0 tx=0x7f596000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f596a7fc640 1 -- 172.21.15.67:0/425924175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596000b660 con 0x7f596c0ca540 2024-04-25T15:55:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f596a7fc640 1 -- 172.21.15.67:0/425924175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5960019070 con 0x7f596c0ca540 2024-04-25T15:55:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596c1f3f70 con 0x7f596c0ca540 2024-04-25T15:55:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f596a7fc640 1 -- 172.21.15.67:0/425924175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596000ba20 con 0x7f596c0ca540 2024-04-25T15:55:15.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f596c1f4510 con 0x7f596c0ca540 2024-04-25T15:55:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.369+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f596c06a480 con 0x7f596c0ca540 2024-04-25T15:55:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.373+0000 7f596a7fc640 1 -- 172.21.15.67:0/425924175 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f596000bbc0 con 0x7f596c0ca540 2024-04-25T15:55:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.373+0000 7f596a7fc640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f595806f400 0x7f5958071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.373+0000 7f596a7fc640 1 -- 172.21.15.67:0/425924175 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f59600901f0 con 0x7f596c0ca540 2024-04-25T15:55:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.373+0000 7f5971b5b640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f595806f400 0x7f5958071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:15.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.373+0000 7f5971b5b640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f595806f400 0x7f5958071910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f59640015b0 tx=0x7f5964009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.377+0000 7f596a7fc640 1 -- 172.21.15.67:0/425924175 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f596005d110 con 0x7f596c0ca540 2024-04-25T15:55:15.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.533+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f596c15a500 con 0x7f595806f400 2024-04-25T15:55:15.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.553+0000 7f596a7fc640 1 -- 172.21.15.67:0/425924175 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252364 (secure 0 0 0) 0x7f596c15a500 con 0x7f595806f400 2024-04-25T15:55:15.555 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:15.556 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:55:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f595806f400 msgr2=0x7f5958071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f595806f400 0x7f5958071910 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f59640015b0 tx=0x7f5964009040 comp rx=0 tx=0).stop 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 msgr2=0x7f596c145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c145e90 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f596000d7d0 tx=0x7f596000dc90 comp rx=0 tx=0).stop 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 shutdown_connections 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f595806f400 0x7f5958071910 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c15a720 0x7f596c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f596c0ca540 0x7f596c145e90 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 --2- 172.21.15.67:0/425924175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c071bc0 0x7f596c1f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 >> 172.21.15.67:0/425924175 conn(0x7f596c0700b0 msgr2=0x7f596c1598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 shutdown_connections 2024-04-25T15:55:15.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:15.557+0000 7f5973dbf640 1 -- 172.21.15.67:0/425924175 wait complete. 2024-04-25T15:55:15.576 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-04-25T15:55:13.896884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798320,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":9664,"num_write_kb":25167774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":1627779911,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10102,"ondisk_log_size":10102,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":383984,"kb_used_data":13872,"kb_used_omap":12,"kb_used_meta":370035,"kb_avail":754590736,"statfs":{"total":773094113280,"available":772700913664,"internally_reserved":0,"allocated":14204928,"data_stored":7557880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":378916402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004586"},"pg_stats":[{"pgid":"3.3f","version":"129'146","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831956+0000","last_change":"2024-04-25T15:50:31.207929+0000","last_active":"2024-04-25T15:55:01.831956+0000","last_peered":"2024-04-25T15:55:01.831956+0000","last_clean":"2024-04-25T15:55:01.831956+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:55:01.831956+0000","last_undegraded":"2024-04-25T15:55:01.831956+0000","last_fullsized":"2024-04-25T15:55:01.831956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:22:49.546373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"129'130","reported_seq":256,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.814114+0000","last_change":"2024-04-25T15:54:44.217073+0000","last_active":"2024-04-25T15:55:01.814114+0000","last_peered":"2024-04-25T15:55:01.814114+0000","last_clean":"2024-04-25T15:55:01.814114+0000","last_became_active":"2024-04-25T15:54:44.215261+0000","last_became_peered":"2024-04-25T15:54:44.215261+0000","last_unstale":"2024-04-25T15:55:01.814114+0000","last_undegraded":"2024-04-25T15:55:01.814114+0000","last_fullsized":"2024-04-25T15:55:01.814114+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"129'151","reported_seq":344,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.812595+0000","last_change":"2024-04-25T15:54:40.166712+0000","last_active":"2024-04-25T15:55:01.812595+0000","last_peered":"2024-04-25T15:55:01.812595+0000","last_clean":"2024-04-25T15:55:01.812595+0000","last_became_active":"2024-04-25T15:54:40.164866+0000","last_became_peered":"2024-04-25T15:54:40.164866+0000","last_unstale":"2024-04-25T15:55:01.812595+0000","last_undegraded":"2024-04-25T15:55:01.812595+0000","last_fullsized":"2024-04-25T15:55:01.812595+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"129'173","reported_seq":405,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831009+0000","last_change":"2024-04-25T15:50:31.202805+0000","last_active":"2024-04-25T15:55:01.831009+0000","last_peered":"2024-04-25T15:55:01.831009+0000","last_clean":"2024-04-25T15:55:01.831009+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:55:01.831009+0000","last_undegraded":"2024-04-25T15:55:01.831009+0000","last_fullsized":"2024-04-25T15:55:01.831009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:00.043031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"129'148","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:50:31.208344+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:59.420815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"129'135","reported_seq":366,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830601+0000","last_change":"2024-04-25T15:50:31.190629+0000","last_active":"2024-04-25T15:55:01.830601+0000","last_peered":"2024-04-25T15:55:01.830601+0000","last_clean":"2024-04-25T15:55:01.830601+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:55:01.830601+0000","last_undegraded":"2024-04-25T15:55:01.830601+0000","last_fullsized":"2024-04-25T15:55:01.830601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:19:29.247989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"129'164","reported_seq":393,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.804532+0000","last_change":"2024-04-25T15:50:31.193405+0000","last_active":"2024-04-25T15:55:01.804532+0000","last_peered":"2024-04-25T15:55:01.804532+0000","last_clean":"2024-04-25T15:55:01.804532+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:55:01.804532+0000","last_undegraded":"2024-04-25T15:55:01.804532+0000","last_fullsized":"2024-04-25T15:55:01.804532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:12:35.793298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"129'186","reported_seq":417,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833169+0000","last_change":"2024-04-25T15:50:31.202281+0000","last_active":"2024-04-25T15:55:01.833169+0000","last_peered":"2024-04-25T15:55:01.833169+0000","last_clean":"2024-04-25T15:55:01.833169+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:55:01.833169+0000","last_undegraded":"2024-04-25T15:55:01.833169+0000","last_fullsized":"2024-04-25T15:55:01.833169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:02.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"129'152","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836597+0000","last_change":"2024-04-25T15:51:11.145800+0000","last_active":"2024-04-25T15:55:01.836597+0000","last_peered":"2024-04-25T15:55:01.836597+0000","last_clean":"2024-04-25T15:55:01.836597+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:55:01.836597+0000","last_undegraded":"2024-04-25T15:55:01.836597+0000","last_fullsized":"2024-04-25T15:55:01.836597+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"129'156","reported_seq":458,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.831117+0000","last_change":"2024-04-25T15:54:54.355672+0000","last_active":"2024-04-25T15:55:01.831117+0000","last_peered":"2024-04-25T15:55:01.831117+0000","last_clean":"2024-04-25T15:55:01.831117+0000","last_became_active":"2024-04-25T15:54:54.352223+0000","last_became_peered":"2024-04-25T15:54:54.352223+0000","last_unstale":"2024-04-25T15:55:01.831117+0000","last_undegraded":"2024-04-25T15:55:01.831117+0000","last_fullsized":"2024-04-25T15:55:01.831117+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:39.782280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"129'189","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830161+0000","last_change":"2024-04-25T15:50:31.173940+0000","last_active":"2024-04-25T15:55:01.830161+0000","last_peered":"2024-04-25T15:55:01.830161+0000","last_clean":"2024-04-25T15:55:01.830161+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:55:01.830161+0000","last_undegraded":"2024-04-25T15:55:01.830161+0000","last_fullsized":"2024-04-25T15:55:01.830161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:36:39.428030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"129'142","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827413+0000","last_change":"2024-04-25T15:53:16.275917+0000","last_active":"2024-04-25T15:55:01.827413+0000","last_peered":"2024-04-25T15:55:01.827413+0000","last_clean":"2024-04-25T15:55:01.827413+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:55:01.827413+0000","last_undegraded":"2024-04-25T15:55:01.827413+0000","last_fullsized":"2024-04-25T15:55:01.827413+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:28.859703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"129'151","reported_seq":691,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833332+0000","last_change":"2024-04-25T15:54:54.357489+0000","last_active":"2024-04-25T15:55:01.833332+0000","last_peered":"2024-04-25T15:55:01.833332+0000","last_clean":"2024-04-25T15:55:01.833332+0000","last_became_active":"2024-04-25T15:54:54.356694+0000","last_became_peered":"2024-04-25T15:54:54.356694+0000","last_unstale":"2024-04-25T15:55:01.833332+0000","last_undegraded":"2024-04-25T15:55:01.833332+0000","last_fullsized":"2024-04-25T15:55:01.833332+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:38.653628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"129'149","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:11.902005+0000","last_change":"2024-04-25T15:55:11.902061+0000","last_active":"2024-04-25T15:55:11.902005+0000","last_peered":"2024-04-25T15:55:11.902005+0000","last_clean":"2024-04-25T15:55:11.902005+0000","last_became_active":"2024-04-25T15:54:40.159879+0000","last_became_peered":"2024-04-25T15:54:40.159879+0000","last_unstale":"2024-04-25T15:55:11.902005+0000","last_undegraded":"2024-04-25T15:55:11.902005+0000","last_fullsized":"2024-04-25T15:55:11.902005+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'149","last_scrub_stamp":"2024-04-25T15:55:11.901847+0000","last_deep_scrub":"129'149","last_deep_scrub_stamp":"2024-04-25T15:55:11.901847+0000","last_clean_scrub_stamp":"2024-04-25T15:55:11.901847+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:42:28.576752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"129'151","reported_seq":271,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.808892+0000","last_change":"2024-04-25T15:54:44.208852+0000","last_active":"2024-04-25T15:55:01.808892+0000","last_peered":"2024-04-25T15:55:01.808892+0000","last_clean":"2024-04-25T15:55:01.808892+0000","last_became_active":"2024-04-25T15:54:44.207144+0000","last_became_peered":"2024-04-25T15:54:44.207144+0000","last_unstale":"2024-04-25T15:55:01.808892+0000","last_undegraded":"2024-04-25T15:55:01.808892+0000","last_fullsized":"2024-04-25T15:55:01.808892+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"129'147","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.832833+0000","last_change":"2024-04-25T15:50:31.185911+0000","last_active":"2024-04-25T15:55:01.832833+0000","last_peered":"2024-04-25T15:55:01.832833+0000","last_clean":"2024-04-25T15:55:01.832833+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:55:01.832833+0000","last_undegraded":"2024-04-25T15:55:01.832833+0000","last_fullsized":"2024-04-25T15:55:01.832833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:58:46.019345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"129'146","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.833273+0000","last_change":"2024-04-25T15:52:26.279503+0000","last_active":"2024-04-25T15:55:01.833273+0000","last_peered":"2024-04-25T15:55:01.833273+0000","last_clean":"2024-04-25T15:55:01.833273+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:55:01.833273+0000","last_undegraded":"2024-04-25T15:55:01.833273+0000","last_fullsized":"2024-04-25T15:55:01.833273+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:59.952743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"129'157","reported_seq":603,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:12.434004+0000","last_change":"2024-04-25T15:54:54.355989+0000","last_active":"2024-04-25T15:55:12.434004+0000","last_peered":"2024-04-25T15:55:12.434004+0000","last_clean":"2024-04-25T15:55:12.434004+0000","last_became_active":"2024-04-25T15:54:54.354849+0000","last_became_peered":"2024-04-25T15:54:54.354849+0000","last_unstale":"2024-04-25T15:55:12.434004+0000","last_undegraded":"2024-04-25T15:55:12.434004+0000","last_fullsized":"2024-04-25T15:55:12.434004+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"129'172","reported_seq":402,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.818938+0000","last_change":"2024-04-25T15:50:31.210882+0000","last_active":"2024-04-25T15:55:01.818938+0000","last_peered":"2024-04-25T15:55:01.818938+0000","last_clean":"2024-04-25T15:55:01.818938+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:55:01.818938+0000","last_undegraded":"2024-04-25T15:55:01.818938+0000","last_fullsized":"2024-04-25T15:55:01.818938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:49.767071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"129'161","reported_seq":409,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.829727+0000","last_change":"2024-04-25T15:53:17.234347+0000","last_active":"2024-04-25T15:55:01.829727+0000","last_peered":"2024-04-25T15:55:01.829727+0000","last_clean":"2024-04-25T15:55:01.829727+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:55:01.829727+0000","last_undegraded":"2024-04-25T15:55:01.829727+0000","last_fullsized":"2024-04-25T15:55:01.829727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:00:06.437156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"129'137","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827457+0000","last_change":"2024-04-25T15:54:40.169787+0000","last_active":"2024-04-25T15:55:01.827457+0000","last_peered":"2024-04-25T15:55:01.827457+0000","last_clean":"2024-04-25T15:55:01.827457+0000","last_became_active":"2024-04-25T15:54:40.167469+0000","last_became_peered":"2024-04-25T15:54:40.167469+0000","last_unstale":"2024-04-25T15:55:01.827457+0000","last_undegraded":"2024-04-25T15:55:01.827457+0000","last_fullsized":"2024-04-25T15:55:01.827457+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"129'166","reported_seq":391,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827149+0000","last_change":"2024-04-25T15:51:11.145107+0000","last_active":"2024-04-25T15:55:01.827149+0000","last_peered":"2024-04-25T15:55:01.827149+0000","last_clean":"2024-04-25T15:55:01.827149+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:55:01.827149+0000","last_undegraded":"2024-04-25T15:55:01.827149+0000","last_fullsized":"2024-04-25T15:55:01.827149+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"129'134","reported_seq":382,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.815642+0000","last_change":"2024-04-25T15:53:17.287218+0000","last_active":"2024-04-25T15:55:01.815642+0000","last_peered":"2024-04-25T15:55:01.815642+0000","last_clean":"2024-04-25T15:55:01.815642+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:55:01.815642+0000","last_undegraded":"2024-04-25T15:55:01.815642+0000","last_fullsized":"2024-04-25T15:55:01.815642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:14.450349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"129'180","reported_seq":412,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827531+0000","last_change":"2024-04-25T15:50:31.203077+0000","last_active":"2024-04-25T15:55:01.827531+0000","last_peered":"2024-04-25T15:55:01.827531+0000","last_clean":"2024-04-25T15:55:01.827531+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:55:01.827531+0000","last_undegraded":"2024-04-25T15:55:01.827531+0000","last_fullsized":"2024-04-25T15:55:01.827531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:00.467371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"129'134","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819773+0000","last_change":"2024-04-25T15:53:35.493710+0000","last_active":"2024-04-25T15:55:01.819773+0000","last_peered":"2024-04-25T15:55:01.819773+0000","last_clean":"2024-04-25T15:55:01.819773+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:55:01.819773+0000","last_undegraded":"2024-04-25T15:55:01.819773+0000","last_fullsized":"2024-04-25T15:55:01.819773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:44.030962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"129'198","reported_seq":430,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834987+0000","last_change":"2024-04-25T15:50:31.212678+0000","last_active":"2024-04-25T15:55:01.834987+0000","last_peered":"2024-04-25T15:55:01.834987+0000","last_clean":"2024-04-25T15:55:01.834987+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:55:01.834987+0000","last_undegraded":"2024-04-25T15:55:01.834987+0000","last_fullsized":"2024-04-25T15:55:01.834987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:21.270451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"129'144","reported_seq":357,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.809546+0000","last_change":"2024-04-25T15:54:44.212386+0000","last_active":"2024-04-25T15:55:01.809546+0000","last_peered":"2024-04-25T15:55:01.809546+0000","last_clean":"2024-04-25T15:55:01.809546+0000","last_became_active":"2024-04-25T15:54:44.210520+0000","last_became_peered":"2024-04-25T15:54:44.210520+0000","last_unstale":"2024-04-25T15:55:01.809546+0000","last_undegraded":"2024-04-25T15:55:01.809546+0000","last_fullsized":"2024-04-25T15:55:01.809546+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"129'133","reported_seq":400,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819037+0000","last_change":"2024-04-25T15:53:17.549084+0000","last_active":"2024-04-25T15:55:01.819037+0000","last_peered":"2024-04-25T15:55:01.819037+0000","last_clean":"2024-04-25T15:55:01.819037+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:55:01.819037+0000","last_undegraded":"2024-04-25T15:55:01.819037+0000","last_fullsized":"2024-04-25T15:55:01.819037+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:31:20.684670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"129'118","reported_seq":312,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.813293+0000","last_change":"2024-04-25T15:54:40.174462+0000","last_active":"2024-04-25T15:55:01.813293+0000","last_peered":"2024-04-25T15:55:01.813293+0000","last_clean":"2024-04-25T15:55:01.813293+0000","last_became_active":"2024-04-25T15:54:40.173209+0000","last_became_peered":"2024-04-25T15:54:40.173209+0000","last_unstale":"2024-04-25T15:55:01.813293+0000","last_undegraded":"2024-04-25T15:55:01.813293+0000","last_fullsized":"2024-04-25T15:55:01.813293+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"129'131","reported_seq":308,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816416+0000","last_change":"2024-04-25T15:52:11.994661+0000","last_active":"2024-04-25T15:55:01.816416+0000","last_peered":"2024-04-25T15:55:01.816416+0000","last_clean":"2024-04-25T15:55:01.816416+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:55:01.816416+0000","last_undegraded":"2024-04-25T15:55:01.816416+0000","last_fullsized":"2024-04-25T15:55:01.816416+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"129'135","reported_seq":377,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836623+0000","last_change":"2024-04-25T15:51:29.140252+0000","last_active":"2024-04-25T15:55:01.836623+0000","last_peered":"2024-04-25T15:55:01.836623+0000","last_clean":"2024-04-25T15:55:01.836623+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:55:01.836623+0000","last_undegraded":"2024-04-25T15:55:01.836623+0000","last_fullsized":"2024-04-25T15:55:01.836623+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"129'150","reported_seq":479,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826757+0000","last_change":"2024-04-25T15:54:54.359681+0000","last_active":"2024-04-25T15:55:01.826757+0000","last_peered":"2024-04-25T15:55:01.826757+0000","last_clean":"2024-04-25T15:55:01.826757+0000","last_became_active":"2024-04-25T15:54:54.353824+0000","last_became_peered":"2024-04-25T15:54:54.353824+0000","last_unstale":"2024-04-25T15:55:01.826757+0000","last_undegraded":"2024-04-25T15:55:01.826757+0000","last_fullsized":"2024-04-25T15:55:01.826757+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:52.297640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"129'130","reported_seq":246,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816679+0000","last_change":"2024-04-25T15:53:16.209950+0000","last_active":"2024-04-25T15:55:01.816679+0000","last_peered":"2024-04-25T15:55:01.816679+0000","last_clean":"2024-04-25T15:55:01.816679+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:55:01.816679+0000","last_undegraded":"2024-04-25T15:55:01.816679+0000","last_fullsized":"2024-04-25T15:55:01.816679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:11.891019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"129'7","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351493+0000","last_change":"2024-04-25T15:50:30.183298+0000","last_active":"2024-04-25T15:54:54.351493+0000","last_peered":"2024-04-25T15:54:54.351493+0000","last_clean":"2024-04-25T15:54:54.351493+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:54:54.351493+0000","last_undegraded":"2024-04-25T15:54:54.351493+0000","last_fullsized":"2024-04-25T15:54:54.351493+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:17.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"129'137","reported_seq":267,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:12.886596+0000","last_change":"2024-04-25T15:55:12.886663+0000","last_active":"2024-04-25T15:55:12.886596+0000","last_peered":"2024-04-25T15:55:12.886596+0000","last_clean":"2024-04-25T15:55:12.886596+0000","last_became_active":"2024-04-25T15:54:40.162667+0000","last_became_peered":"2024-04-25T15:54:40.162667+0000","last_unstale":"2024-04-25T15:55:12.886596+0000","last_undegraded":"2024-04-25T15:55:12.886596+0000","last_fullsized":"2024-04-25T15:55:12.886596+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'137","last_scrub_stamp":"2024-04-25T15:55:12.886374+0000","last_deep_scrub":"129'137","last_deep_scrub_stamp":"2024-04-25T15:55:12.886374+0000","last_clean_scrub_stamp":"2024-04-25T15:55:12.886374+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:37.419326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":24,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347310+0000","last_change":"2024-04-25T15:54:51.317865+0000","last_active":"2024-04-25T15:54:54.347310+0000","last_peered":"2024-04-25T15:54:54.347310+0000","last_clean":"2024-04-25T15:54:54.347310+0000","last_became_active":"2024-04-25T15:54:51.302422+0000","last_became_peered":"2024-04-25T15:54:51.302422+0000","last_unstale":"2024-04-25T15:54:54.347310+0000","last_undegraded":"2024-04-25T15:54:54.347310+0000","last_fullsized":"2024-04-25T15:54:54.347310+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:36.956598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":258,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.350037+0000","last_change":"2024-04-25T15:54:51.329012+0000","last_active":"2024-04-25T15:54:54.350037+0000","last_peered":"2024-04-25T15:54:54.350037+0000","last_clean":"2024-04-25T15:54:54.350037+0000","last_became_active":"2024-04-25T15:54:51.307662+0000","last_became_peered":"2024-04-25T15:54:51.307662+0000","last_unstale":"2024-04-25T15:54:54.350037+0000","last_undegraded":"2024-04-25T15:54:54.350037+0000","last_fullsized":"2024-04-25T15:54:54.350037+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:41.566474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"129'170","reported_seq":297,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.805606+0000","last_change":"2024-04-25T15:53:17.007081+0000","last_active":"2024-04-25T15:55:01.805606+0000","last_peered":"2024-04-25T15:55:01.805606+0000","last_clean":"2024-04-25T15:55:01.805606+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:55:01.805606+0000","last_undegraded":"2024-04-25T15:55:01.805606+0000","last_fullsized":"2024-04-25T15:55:01.805606+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.345518+0000","last_change":"2024-04-25T15:53:16.093934+0000","last_active":"2024-04-25T15:54:54.345518+0000","last_peered":"2024-04-25T15:54:54.345518+0000","last_clean":"2024-04-25T15:54:54.345518+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:54:54.345518+0000","last_undegraded":"2024-04-25T15:54:54.345518+0000","last_fullsized":"2024-04-25T15:54:54.345518+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:06:00.523635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"129'157","reported_seq":383,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816758+0000","last_change":"2024-04-25T15:54:44.212095+0000","last_active":"2024-04-25T15:55:01.816758+0000","last_peered":"2024-04-25T15:55:01.816758+0000","last_clean":"2024-04-25T15:55:01.816758+0000","last_became_active":"2024-04-25T15:54:44.209985+0000","last_became_peered":"2024-04-25T15:54:44.209985+0000","last_unstale":"2024-04-25T15:55:01.816758+0000","last_undegraded":"2024-04-25T15:55:01.816758+0000","last_fullsized":"2024-04-25T15:55:01.816758+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"129'8","reported_seq":241,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740819+0000","last_change":"2024-04-25T15:50:30.198608+0000","last_active":"2024-04-25T15:55:01.740819+0000","last_peered":"2024-04-25T15:55:01.740819+0000","last_clean":"2024-04-25T15:55:01.740819+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:55:01.740819+0000","last_undegraded":"2024-04-25T15:55:01.740819+0000","last_fullsized":"2024-04-25T15:55:01.740819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:28:05.043212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"129'229","reported_seq":460,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834138+0000","last_change":"2024-04-25T15:50:31.193471+0000","last_active":"2024-04-25T15:55:01.834138+0000","last_peered":"2024-04-25T15:55:01.834138+0000","last_clean":"2024-04-25T15:55:01.834138+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:55:01.834138+0000","last_undegraded":"2024-04-25T15:55:01.834138+0000","last_fullsized":"2024-04-25T15:55:01.834138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:51.227926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346366+0000","last_change":"2024-04-25T15:50:30.203017+0000","last_active":"2024-04-25T15:54:54.346366+0000","last_peered":"2024-04-25T15:54:54.346366+0000","last_clean":"2024-04-25T15:54:54.346366+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:54:54.346366+0000","last_undegraded":"2024-04-25T15:54:54.346366+0000","last_fullsized":"2024-04-25T15:54:54.346366+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:17.257769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"129'141","reported_seq":335,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836492+0000","last_change":"2024-04-25T15:54:40.169299+0000","last_active":"2024-04-25T15:55:01.836492+0000","last_peered":"2024-04-25T15:55:01.836492+0000","last_clean":"2024-04-25T15:55:01.836492+0000","last_became_active":"2024-04-25T15:54:40.168203+0000","last_became_peered":"2024-04-25T15:54:40.168203+0000","last_unstale":"2024-04-25T15:55:01.836492+0000","last_undegraded":"2024-04-25T15:55:01.836492+0000","last_fullsized":"2024-04-25T15:55:01.836492+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"114'11","reported_seq":47,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342212+0000","last_change":"2024-04-25T15:54:41.197909+0000","last_active":"2024-04-25T15:54:54.342212+0000","last_peered":"2024-04-25T15:54:54.342212+0000","last_clean":"2024-04-25T15:54:54.342212+0000","last_became_active":"2024-04-25T15:54:41.171190+0000","last_became_peered":"2024-04-25T15:54:41.171190+0000","last_unstale":"2024-04-25T15:54:54.342212+0000","last_undegraded":"2024-04-25T15:54:54.342212+0000","last_fullsized":"2024-04-25T15:54:54.342212+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2016,"num_keys_recovered":30,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"129'149","reported_seq":265,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825846+0000","last_change":"2024-04-25T15:53:17.058501+0000","last_active":"2024-04-25T15:55:01.825846+0000","last_peered":"2024-04-25T15:55:01.825846+0000","last_clean":"2024-04-25T15:55:01.825846+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:55:01.825846+0000","last_undegraded":"2024-04-25T15:55:01.825846+0000","last_fullsized":"2024-04-25T15:55:01.825846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.822067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"129'93","reported_seq":33,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.740355+0000","last_change":"2024-04-25T15:54:51.358160+0000","last_active":"2024-04-25T15:55:01.740355+0000","last_peered":"2024-04-25T15:55:01.740355+0000","last_clean":"2024-04-25T15:55:01.740355+0000","last_became_active":"2024-04-25T15:54:51.303172+0000","last_became_peered":"2024-04-25T15:54:51.303172+0000","last_unstale":"2024-04-25T15:55:01.740355+0000","last_undegraded":"2024-04-25T15:55:01.740355+0000","last_fullsized":"2024-04-25T15:55:01.740355+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:32:39.823475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199279,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":380717,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"129'150","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.819314+0000","last_change":"2024-04-25T15:50:31.193458+0000","last_active":"2024-04-25T15:55:01.819314+0000","last_peered":"2024-04-25T15:55:01.819314+0000","last_clean":"2024-04-25T15:55:01.819314+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:55:01.819314+0000","last_undegraded":"2024-04-25T15:55:01.819314+0000","last_fullsized":"2024-04-25T15:55:01.819314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:26.501349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.352617+0000","last_change":"2024-04-25T15:54:40.166458+0000","last_active":"2024-04-25T15:54:54.352617+0000","last_peered":"2024-04-25T15:54:54.352617+0000","last_clean":"2024-04-25T15:54:54.352617+0000","last_became_active":"2024-04-25T15:54:40.165701+0000","last_became_peered":"2024-04-25T15:54:40.165701+0000","last_unstale":"2024-04-25T15:54:54.352617+0000","last_undegraded":"2024-04-25T15:54:54.352617+0000","last_fullsized":"2024-04-25T15:54:54.352617+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"129'157","reported_seq":349,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.827608+0000","last_change":"2024-04-25T15:54:40.168672+0000","last_active":"2024-04-25T15:55:01.827608+0000","last_peered":"2024-04-25T15:55:01.827608+0000","last_clean":"2024-04-25T15:55:01.827608+0000","last_became_active":"2024-04-25T15:54:40.167338+0000","last_became_peered":"2024-04-25T15:54:40.167338+0000","last_unstale":"2024-04-25T15:55:01.827608+0000","last_undegraded":"2024-04-25T15:55:01.827608+0000","last_fullsized":"2024-04-25T15:55:01.827608+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344723+0000","last_change":"2024-04-25T15:50:30.198897+0000","last_active":"2024-04-25T15:54:54.344723+0000","last_peered":"2024-04-25T15:54:54.344723+0000","last_clean":"2024-04-25T15:54:54.344723+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:54:54.344723+0000","last_undegraded":"2024-04-25T15:54:54.344723+0000","last_fullsized":"2024-04-25T15:54:54.344723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:52.853449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"129'171","reported_seq":421,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828742+0000","last_change":"2024-04-25T15:53:16.687646+0000","last_active":"2024-04-25T15:55:01.828742+0000","last_peered":"2024-04-25T15:55:01.828742+0000","last_clean":"2024-04-25T15:55:01.828742+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:55:01.828742+0000","last_undegraded":"2024-04-25T15:55:01.828742+0000","last_fullsized":"2024-04-25T15:55:01.828742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:02.098502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"129'7","reported_seq":25,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.347111+0000","last_change":"2024-04-25T15:54:51.333252+0000","last_active":"2024-04-25T15:54:54.347111+0000","last_peered":"2024-04-25T15:54:54.347111+0000","last_clean":"2024-04-25T15:54:54.347111+0000","last_became_active":"2024-04-25T15:54:51.302160+0000","last_became_peered":"2024-04-25T15:54:51.302160+0000","last_unstale":"2024-04-25T15:54:54.347111+0000","last_undegraded":"2024-04-25T15:54:54.347111+0000","last_fullsized":"2024-04-25T15:54:54.347111+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:25.724351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4116,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"129'165","reported_seq":397,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.824740+0000","last_change":"2024-04-25T15:50:31.187468+0000","last_active":"2024-04-25T15:55:01.824740+0000","last_peered":"2024-04-25T15:55:01.824740+0000","last_clean":"2024-04-25T15:55:01.824740+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:55:01.824740+0000","last_undegraded":"2024-04-25T15:55:01.824740+0000","last_fullsized":"2024-04-25T15:55:01.824740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:16:17.867377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"114'18","reported_seq":344,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:12.434117+0000","last_change":"2024-04-25T15:54:41.193381+0000","last_active":"2024-04-25T15:55:12.434117+0000","last_peered":"2024-04-25T15:55:12.434117+0000","last_clean":"2024-04-25T15:55:12.434117+0000","last_became_active":"2024-04-25T15:54:41.166653+0000","last_became_peered":"2024-04-25T15:54:41.166653+0000","last_unstale":"2024-04-25T15:55:12.434117+0000","last_undegraded":"2024-04-25T15:55:12.434117+0000","last_fullsized":"2024-04-25T15:55:12.434117+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":448,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"129'189","reported_seq":447,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826190+0000","last_change":"2024-04-25T15:53:17.460515+0000","last_active":"2024-04-25T15:55:01.826190+0000","last_peered":"2024-04-25T15:55:01.826190+0000","last_clean":"2024-04-25T15:55:01.826190+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:55:01.826190+0000","last_undegraded":"2024-04-25T15:55:01.826190+0000","last_fullsized":"2024-04-25T15:55:01.826190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:00.375965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":120,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341794+0000","last_change":"2024-04-25T15:53:16.069066+0000","last_active":"2024-04-25T15:54:54.341794+0000","last_peered":"2024-04-25T15:54:54.341794+0000","last_clean":"2024-04-25T15:54:54.341794+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:54:54.341794+0000","last_undegraded":"2024-04-25T15:54:54.341794+0000","last_fullsized":"2024-04-25T15:54:54.341794+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:43.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"129'146","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.825150+0000","last_change":"2024-04-25T15:53:17.178176+0000","last_active":"2024-04-25T15:55:01.825150+0000","last_peered":"2024-04-25T15:55:01.825150+0000","last_clean":"2024-04-25T15:55:01.825150+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:55:01.825150+0000","last_undegraded":"2024-04-25T15:55:01.825150+0000","last_fullsized":"2024-04-25T15:55:01.825150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:18.547426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:02.889355+0000","last_change":"2024-04-25T15:50:30.186440+0000","last_active":"2024-04-25T15:55:02.889355+0000","last_peered":"2024-04-25T15:55:02.889355+0000","last_clean":"2024-04-25T15:55:02.889355+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:55:02.889355+0000","last_undegraded":"2024-04-25T15:55:02.889355+0000","last_fullsized":"2024-04-25T15:55:02.889355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"129'138","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828396+0000","last_change":"2024-04-25T15:53:16.333032+0000","last_active":"2024-04-25T15:55:01.828396+0000","last_peered":"2024-04-25T15:55:01.828396+0000","last_clean":"2024-04-25T15:55:01.828396+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:55:01.828396+0000","last_undegraded":"2024-04-25T15:55:01.828396+0000","last_fullsized":"2024-04-25T15:55:01.828396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:47:17.231244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":41,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.342326+0000","last_change":"2024-04-25T15:54:41.186336+0000","last_active":"2024-04-25T15:54:54.342326+0000","last_peered":"2024-04-25T15:54:54.342326+0000","last_clean":"2024-04-25T15:54:54.342326+0000","last_became_active":"2024-04-25T15:54:41.165683+0000","last_became_peered":"2024-04-25T15:54:41.165683+0000","last_unstale":"2024-04-25T15:54:54.342326+0000","last_undegraded":"2024-04-25T15:54:54.342326+0000","last_fullsized":"2024-04-25T15:54:54.342326+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"129'144","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.830924+0000","last_change":"2024-04-25T15:50:31.190600+0000","last_active":"2024-04-25T15:55:01.830924+0000","last_peered":"2024-04-25T15:55:01.830924+0000","last_clean":"2024-04-25T15:55:01.830924+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:55:01.830924+0000","last_undegraded":"2024-04-25T15:55:01.830924+0000","last_fullsized":"2024-04-25T15:55:01.830924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:41.474235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"129'7","reported_seq":241,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:02.889305+0000","last_change":"2024-04-25T15:50:30.198675+0000","last_active":"2024-04-25T15:55:02.889305+0000","last_peered":"2024-04-25T15:55:02.889305+0000","last_clean":"2024-04-25T15:55:02.889305+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:55:02.889305+0000","last_undegraded":"2024-04-25T15:55:02.889305+0000","last_fullsized":"2024-04-25T15:55:02.889305+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"129'151","reported_seq":381,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:02.222181+0000","last_change":"2024-04-25T15:50:31.208294+0000","last_active":"2024-04-25T15:55:02.222181+0000","last_peered":"2024-04-25T15:55:02.222181+0000","last_clean":"2024-04-25T15:55:02.222181+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:55:02.222181+0000","last_undegraded":"2024-04-25T15:55:02.222181+0000","last_fullsized":"2024-04-25T15:55:02.222181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:34:27.121760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630501+0000","last_change":"2024-04-25T15:50:19.519025+0000","last_active":"2024-04-25T15:54:54.630501+0000","last_peered":"2024-04-25T15:54:54.630501+0000","last_clean":"2024-04-25T15:54:54.630501+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:54:54.630501+0000","last_undegraded":"2024-04-25T15:54:54.630501+0000","last_fullsized":"2024-04-25T15:54:54.630501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:18.057441+0000","last_clean_scrub_stamp":"2024-04-25T15:50:18.057441+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:45.694436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":240,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344831+0000","last_change":"2024-04-25T15:50:30.195841+0000","last_active":"2024-04-25T15:54:54.344831+0000","last_peered":"2024-04-25T15:54:54.344831+0000","last_clean":"2024-04-25T15:54:54.344831+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:54:54.344831+0000","last_undegraded":"2024-04-25T15:54:54.344831+0000","last_fullsized":"2024-04-25T15:54:54.344831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:21.529525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"129'158","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.817195+0000","last_change":"2024-04-25T15:50:31.189199+0000","last_active":"2024-04-25T15:55:01.817195+0000","last_peered":"2024-04-25T15:55:01.817195+0000","last_clean":"2024-04-25T15:55:01.817195+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:55:01.817195+0000","last_undegraded":"2024-04-25T15:55:01.817195+0000","last_fullsized":"2024-04-25T15:55:01.817195+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:04:01.336474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":272,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630140+0000","last_change":"2024-04-25T15:54:41.173406+0000","last_active":"2024-04-25T15:54:54.630140+0000","last_peered":"2024-04-25T15:54:54.630140+0000","last_clean":"2024-04-25T15:54:54.630140+0000","last_became_active":"2024-04-25T15:54:41.166720+0000","last_became_peered":"2024-04-25T15:54:41.166720+0000","last_unstale":"2024-04-25T15:54:54.630140+0000","last_undegraded":"2024-04-25T15:54:54.630140+0000","last_fullsized":"2024-04-25T15:54:54.630140+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:41.861800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"129'131","reported_seq":253,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.820923+0000","last_change":"2024-04-25T15:54:40.164755+0000","last_active":"2024-04-25T15:55:01.820923+0000","last_peered":"2024-04-25T15:55:01.820923+0000","last_clean":"2024-04-25T15:55:01.820923+0000","last_became_active":"2024-04-25T15:54:40.163587+0000","last_became_peered":"2024-04-25T15:54:40.163587+0000","last_unstale":"2024-04-25T15:55:01.820923+0000","last_undegraded":"2024-04-25T15:55:01.820923+0000","last_fullsized":"2024-04-25T15:55:01.820923+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"129'13","reported_seq":252,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.738594+0000","last_change":"2024-04-25T15:50:30.185925+0000","last_active":"2024-04-25T15:55:01.738594+0000","last_peered":"2024-04-25T15:55:01.738594+0000","last_clean":"2024-04-25T15:55:01.738594+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:55:01.738594+0000","last_undegraded":"2024-04-25T15:55:01.738594+0000","last_fullsized":"2024-04-25T15:55:01.738594+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"129'131","reported_seq":432,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.809569+0000","last_change":"2024-04-25T15:54:44.216921+0000","last_active":"2024-04-25T15:55:01.809569+0000","last_peered":"2024-04-25T15:55:01.809569+0000","last_clean":"2024-04-25T15:55:01.809569+0000","last_became_active":"2024-04-25T15:54:44.215186+0000","last_became_peered":"2024-04-25T15:54:44.215186+0000","last_unstale":"2024-04-25T15:55:01.809569+0000","last_undegraded":"2024-04-25T15:55:01.809569+0000","last_fullsized":"2024-04-25T15:55:01.809569+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.344849+0000","last_change":"2024-04-25T15:51:47.301818+0000","last_active":"2024-04-25T15:54:54.344849+0000","last_peered":"2024-04-25T15:54:54.344849+0000","last_clean":"2024-04-25T15:54:54.344849+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:54:54.344849+0000","last_undegraded":"2024-04-25T15:54:54.344849+0000","last_fullsized":"2024-04-25T15:54:54.344849+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:32.228404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"129'145","reported_seq":328,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.787876+0000","last_change":"2024-04-25T15:53:54.208189+0000","last_active":"2024-04-25T15:55:01.787876+0000","last_peered":"2024-04-25T15:55:01.787876+0000","last_clean":"2024-04-25T15:55:01.787876+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:55:01.787876+0000","last_undegraded":"2024-04-25T15:55:01.787876+0000","last_fullsized":"2024-04-25T15:55:01.787876+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:02.459575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":37,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341962+0000","last_change":"2024-04-25T15:54:41.168286+0000","last_active":"2024-04-25T15:54:54.341962+0000","last_peered":"2024-04-25T15:54:54.341962+0000","last_clean":"2024-04-25T15:54:54.341962+0000","last_became_active":"2024-04-25T15:54:41.166643+0000","last_became_peered":"2024-04-25T15:54:41.166643+0000","last_unstale":"2024-04-25T15:54:54.341962+0000","last_undegraded":"2024-04-25T15:54:54.341962+0000","last_fullsized":"2024-04-25T15:54:54.341962+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.828419+0000","last_change":"2024-04-25T15:50:31.188140+0000","last_active":"2024-04-25T15:55:01.828419+0000","last_peered":"2024-04-25T15:55:01.828419+0000","last_clean":"2024-04-25T15:55:01.828419+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:55:01.828419+0000","last_undegraded":"2024-04-25T15:55:01.828419+0000","last_fullsized":"2024-04-25T15:55:01.828419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:54.834661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631647+0000","last_change":"2024-04-25T15:50:30.208210+0000","last_active":"2024-04-25T15:54:54.631647+0000","last_peered":"2024-04-25T15:54:54.631647+0000","last_clean":"2024-04-25T15:54:54.631647+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:54:54.631647+0000","last_undegraded":"2024-04-25T15:54:54.631647+0000","last_fullsized":"2024-04-25T15:54:54.631647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:03:43.550553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"129'144","reported_seq":345,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.812420+0000","last_change":"2024-04-25T15:54:44.217114+0000","last_active":"2024-04-25T15:55:01.812420+0000","last_peered":"2024-04-25T15:55:01.812420+0000","last_clean":"2024-04-25T15:55:01.812420+0000","last_became_active":"2024-04-25T15:54:44.215121+0000","last_became_peered":"2024-04-25T15:54:44.215121+0000","last_unstale":"2024-04-25T15:55:01.812420+0000","last_undegraded":"2024-04-25T15:55:01.812420+0000","last_fullsized":"2024-04-25T15:55:01.812420+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"114'26","reported_seq":62,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.341582+0000","last_change":"2024-04-25T15:54:51.320611+0000","last_active":"2024-04-25T15:54:54.341582+0000","last_peered":"2024-04-25T15:54:54.341582+0000","last_clean":"2024-04-25T15:54:54.341582+0000","last_became_active":"2024-04-25T15:54:51.307056+0000","last_became_peered":"2024-04-25T15:54:51.307056+0000","last_unstale":"2024-04-25T15:54:54.341582+0000","last_undegraded":"2024-04-25T15:54:54.341582+0000","last_fullsized":"2024-04-25T15:54:54.341582+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":13,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"129'143","reported_seq":375,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.822154+0000","last_change":"2024-04-25T15:50:31.212034+0000","last_active":"2024-04-25T15:55:01.822154+0000","last_peered":"2024-04-25T15:55:01.822154+0000","last_clean":"2024-04-25T15:55:01.822154+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:55:01.822154+0000","last_undegraded":"2024-04-25T15:55:01.822154+0000","last_fullsized":"2024-04-25T15:55:01.822154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:58.590095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"110'8","reported_seq":242,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.348406+0000","last_change":"2024-04-25T15:50:30.199448+0000","last_active":"2024-04-25T15:54:54.348406+0000","last_peered":"2024-04-25T15:54:54.348406+0000","last_clean":"2024-04-25T15:54:54.348406+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:54:54.348406+0000","last_undegraded":"2024-04-25T15:54:54.348406+0000","last_fullsized":"2024-04-25T15:54:54.348406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:32:17.101784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"129'163","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.816914+0000","last_change":"2024-04-25T15:54:44.208533+0000","last_active":"2024-04-25T15:55:01.816914+0000","last_peered":"2024-04-25T15:55:01.816914+0000","last_clean":"2024-04-25T15:55:01.816914+0000","last_became_active":"2024-04-25T15:54:44.207033+0000","last_became_peered":"2024-04-25T15:54:44.207033+0000","last_unstale":"2024-04-25T15:55:01.816914+0000","last_undegraded":"2024-04-25T15:55:01.816914+0000","last_fullsized":"2024-04-25T15:55:01.816914+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":234,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.630110+0000","last_change":"2024-04-25T15:50:30.197474+0000","last_active":"2024-04-25T15:54:54.630110+0000","last_peered":"2024-04-25T15:54:54.630110+0000","last_clean":"2024-04-25T15:54:54.630110+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:54:54.630110+0000","last_undegraded":"2024-04-25T15:54:54.630110+0000","last_fullsized":"2024-04-25T15:54:54.630110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:08:54.004291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"129'134","reported_seq":250,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.751001+0000","last_change":"2024-04-25T15:53:16.396248+0000","last_active":"2024-04-25T15:55:01.751001+0000","last_peered":"2024-04-25T15:55:01.751001+0000","last_clean":"2024-04-25T15:55:01.751001+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:55:01.751001+0000","last_undegraded":"2024-04-25T15:55:01.751001+0000","last_fullsized":"2024-04-25T15:55:01.751001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:20:33.339112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"129'11","reported_seq":219,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:12.816406+0000","last_change":"2024-04-25T15:55:12.816469+0000","last_active":"2024-04-25T15:55:12.816406+0000","last_peered":"2024-04-25T15:55:12.816406+0000","last_clean":"2024-04-25T15:55:12.816406+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:55:12.816406+0000","last_undegraded":"2024-04-25T15:55:12.816406+0000","last_fullsized":"2024-04-25T15:55:12.816406+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'11","last_scrub_stamp":"2024-04-25T15:55:12.816300+0000","last_deep_scrub":"129'11","last_deep_scrub_stamp":"2024-04-25T15:55:12.816300+0000","last_clean_scrub_stamp":"2024-04-25T15:55:12.816300+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:40.666540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"129'153","reported_seq":521,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.811691+0000","last_change":"2024-04-25T15:54:44.207954+0000","last_active":"2024-04-25T15:55:01.811691+0000","last_peered":"2024-04-25T15:55:01.811691+0000","last_clean":"2024-04-25T15:55:01.811691+0000","last_became_active":"2024-04-25T15:54:44.206595+0000","last_became_peered":"2024-04-25T15:54:44.206595+0000","last_unstale":"2024-04-25T15:55:01.811691+0000","last_undegraded":"2024-04-25T15:55:01.811691+0000","last_fullsized":"2024-04-25T15:55:01.811691+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.107940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":70,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.346165+0000","last_change":"2024-04-25T15:54:22.056875+0000","last_active":"2024-04-25T15:54:54.346165+0000","last_peered":"2024-04-25T15:54:54.346165+0000","last_clean":"2024-04-25T15:54:54.346165+0000","last_became_active":"2024-04-25T15:54:22.037756+0000","last_became_peered":"2024-04-25T15:54:22.037756+0000","last_unstale":"2024-04-25T15:54:54.346165+0000","last_undegraded":"2024-04-25T15:54:54.346165+0000","last_fullsized":"2024-04-25T15:54:54.346165+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"129'143","reported_seq":368,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.822842+0000","last_change":"2024-04-25T15:51:11.141537+0000","last_active":"2024-04-25T15:55:01.822842+0000","last_peered":"2024-04-25T15:55:01.822842+0000","last_clean":"2024-04-25T15:55:01.822842+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:55:01.822842+0000","last_undegraded":"2024-04-25T15:55:01.822842+0000","last_fullsized":"2024-04-25T15:55:01.822842+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"81'5","reported_seq":259,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631676+0000","last_change":"2024-04-25T15:53:16.090212+0000","last_active":"2024-04-25T15:54:54.631676+0000","last_peered":"2024-04-25T15:54:54.631676+0000","last_clean":"2024-04-25T15:54:54.631676+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:54:54.631676+0000","last_undegraded":"2024-04-25T15:54:54.631676+0000","last_fullsized":"2024-04-25T15:54:54.631676+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:48:22.987255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"129'128","reported_seq":386,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.826678+0000","last_change":"2024-04-25T15:53:17.043612+0000","last_active":"2024-04-25T15:55:01.826678+0000","last_peered":"2024-04-25T15:55:01.826678+0000","last_clean":"2024-04-25T15:55:01.826678+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:55:01.826678+0000","last_undegraded":"2024-04-25T15:55:01.826678+0000","last_fullsized":"2024-04-25T15:55:01.826678+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:50:34.609043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"129'183","reported_seq":440,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.835785+0000","last_change":"2024-04-25T15:53:16.944509+0000","last_active":"2024-04-25T15:55:01.835785+0000","last_peered":"2024-04-25T15:55:01.835785+0000","last_clean":"2024-04-25T15:55:01.835785+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:55:01.835785+0000","last_undegraded":"2024-04-25T15:55:01.835785+0000","last_fullsized":"2024-04-25T15:55:01.835785+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:01:58.794592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631338+0000","last_change":"2024-04-25T15:50:30.206251+0000","last_active":"2024-04-25T15:54:54.631338+0000","last_peered":"2024-04-25T15:54:54.631338+0000","last_clean":"2024-04-25T15:54:54.631338+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:54:54.631338+0000","last_undegraded":"2024-04-25T15:54:54.631338+0000","last_fullsized":"2024-04-25T15:54:54.631338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:13:40.119543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"129'141","reported_seq":415,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.834610+0000","last_change":"2024-04-25T15:53:16.551296+0000","last_active":"2024-04-25T15:55:01.834610+0000","last_peered":"2024-04-25T15:55:01.834610+0000","last_clean":"2024-04-25T15:55:01.834610+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:55:01.834610+0000","last_undegraded":"2024-04-25T15:55:01.834610+0000","last_fullsized":"2024-04-25T15:55:01.834610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:50.596707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"81'13","reported_seq":244,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.631249+0000","last_change":"2024-04-25T15:50:30.207637+0000","last_active":"2024-04-25T15:54:54.631249+0000","last_peered":"2024-04-25T15:54:54.631249+0000","last_clean":"2024-04-25T15:54:54.631249+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:54:54.631249+0000","last_undegraded":"2024-04-25T15:54:54.631249+0000","last_fullsized":"2024-04-25T15:54:54.631249+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:33.524272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"129'135","reported_seq":329,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.836677+0000","last_change":"2024-04-25T15:54:40.169300+0000","last_active":"2024-04-25T15:55:01.836677+0000","last_peered":"2024-04-25T15:55:01.836677+0000","last_clean":"2024-04-25T15:55:01.836677+0000","last_became_active":"2024-04-25T15:54:40.168164+0000","last_became_peered":"2024-04-25T15:54:40.168164+0000","last_unstale":"2024-04-25T15:55:01.836677+0000","last_undegraded":"2024-04-25T15:55:01.836677+0000","last_fullsized":"2024-04-25T15:55:01.836677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":236,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.351250+0000","last_change":"2024-04-25T15:50:30.181396+0000","last_active":"2024-04-25T15:54:54.351250+0000","last_peered":"2024-04-25T15:54:54.351250+0000","last_clean":"2024-04-25T15:54:54.351250+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:54:54.351250+0000","last_undegraded":"2024-04-25T15:54:54.351250+0000","last_fullsized":"2024-04-25T15:54:54.351250+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:29.042099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"129'152","reported_seq":598,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:01.832040+0000","last_change":"2024-04-25T15:54:54.360954+0000","last_active":"2024-04-25T15:55:01.832040+0000","last_peered":"2024-04-25T15:55:01.832040+0000","last_clean":"2024-04-25T15:55:01.832040+0000","last_became_active":"2024-04-25T15:54:54.360118+0000","last_became_peered":"2024-04-25T15:54:54.360118+0000","last_unstale":"2024-04-25T15:55:01.832040+0000","last_undegraded":"2024-04-25T15:55:01.832040+0000","last_fullsized":"2024-04-25T15:55:01.832040+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:30.137946+0000","last_clean_scrub_stamp":"2024-04-25T15:50:30.137946+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:37.822818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":104,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:54:54.353131+0000","last_change":"2024-04-25T15:54:40.170535+0000","last_active":"2024-04-25T15:54:54.353131+0000","last_peered":"2024-04-25T15:54:54.353131+0000","last_clean":"2024-04-25T15:54:54.353131+0000","last_became_active":"2024-04-25T15:54:40.169964+0000","last_became_peered":"2024-04-25T15:54:40.169964+0000","last_unstale":"2024-04-25T15:54:54.353131+0000","last_undegraded":"2024-04-25T15:54:54.353131+0000","last_fullsized":"2024-04-25T15:54:54.353131+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:50:29.125604+0000","last_clean_scrub_stamp":"2024-04-25T15:50:29.125604+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":1627389952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9717,"ondisk_log_size":9717,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":207952,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":347,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":389959,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":415904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59668,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312172,"statfs":{"total":96636764160,"available":96575664128,"internally_reserved":0,"allocated":2174976,"data_stored":1338205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41224,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1769472,"data_stored":945806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63260,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308580,"statfs":{"total":96636764160,"available":96571985920,"internally_reserved":0,"allocated":1593344,"data_stored":749669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1568768,"data_stored":747067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1777664,"data_stored":946345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1581056,"data_stored":747704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1572864,"data_stored":746687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2166784,"data_stored":1336397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":201190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:55:15.618 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.618 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.618 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.618 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.619 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.620 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.621 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.622 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.623 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.624 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T15:50:18.057441+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.625 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.626 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.627 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-25T15:50:30.137946+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=30, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-04-25T15:50:29.125604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=50, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=15, tm_min=55, tm_sec=9, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T15:55:15.628 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T15:55:29.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:55:29.800 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:55:35.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:55:35.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe7f57640 1 Processor -- start 2024-04-25T15:55:35.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe7f57640 1 -- start start 2024-04-25T15:55:35.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe7f57640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2fe015a720 0x7f2fe015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe7f57640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe00ca540 0x7f2fe00ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe7f57640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fe015de10 con 0x7f2fe0071bc0 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe7f57640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2fe015ebb0 con 0x7f2fe015a720 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe7f57640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe015e720 con 0x7f2fe00ca540 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe54f2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2fe015a720 0x7f2fe015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe64f4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe00ca540 0x7f2fe00ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe54f2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2fe015a720 0x7f2fe015ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.67:52390/0 (socket says 172.21.15.67:52390) 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe5cf3640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2fe0071bc0 0x7f2fe00cbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:49686/0 (socket says 172.21.15.67:49686) 2024-04-25T15:55:35.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe54f2640 1 -- 172.21.15.67:0/2244083122 learned_addr learned my addr 172.21.15.67:0/2244083122 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe64f4640 1 -- 172.21.15.67:0/2244083122 >> v2:172.21.15.176:3300/0 conn(0x7f2fe015a720 msgr2=0x7f2fe015ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe64f4640 1 --2- 172.21.15.67:0/2244083122 >> v2:172.21.15.176:3300/0 conn(0x7f2fe015a720 0x7f2fe015ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe64f4640 1 -- 172.21.15.67:0/2244083122 >> v1:172.21.15.176:6789/0 conn(0x7f2fe0071bc0 legacy=0x7f2fe00cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe64f4640 1 -- 172.21.15.67:0/2244083122 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe0072390 con 0x7f2fe00ca540 2024-04-25T15:55:35.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.797+0000 7f2fe54f2640 1 --2- 172.21.15.67:0/2244083122 >> v2:172.21.15.176:3300/0 conn(0x7f2fe015a720 0x7f2fe015ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:55:35.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe64f4640 1 --2- 172.21.15.67:0/2244083122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe00ca540 0x7f2fe00ca950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80097c0 tx=0x7f2fd801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40ffb39222cee779 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe4cf1640 1 -- 172.21.15.67:0/2244083122 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd80b1070 con 0x7f2fe00ca540 2024-04-25T15:55:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe4cf1640 1 -- 172.21.15.67:0/2244083122 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd80ac410 con 0x7f2fe00ca540 2024-04-25T15:55:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe4cf1640 1 -- 172.21.15.67:0/2244083122 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd80ac7e0 con 0x7f2fe00ca540 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 -- 172.21.15.67:0/2244083122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe00ca540 msgr2=0x7f2fe00ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/2244083122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe00ca540 0x7f2fe00ca950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80097c0 tx=0x7f2fd801c4a0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 -- 172.21.15.67:0/2244083122 shutdown_connections 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/2244083122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe00ca540 0x7f2fe00ca950 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/2244083122 >> v2:172.21.15.176:3300/0 conn(0x7f2fe015a720 0x7f2fe015ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 -- 172.21.15.67:0/2244083122 >> 172.21.15.67:0/2244083122 conn(0x7f2fe00700b0 msgr2=0x7f2fe00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 -- 172.21.15.67:0/2244083122 shutdown_connections 2024-04-25T15:55:35.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.801+0000 7f2fe7f57640 1 -- 172.21.15.67:0/2244083122 wait complete. 2024-04-25T15:55:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 Processor -- start 2024-04-25T15:55:35.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 -- start start 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe0071bc0 0x7f2fe00cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fe00ca540 0x7f2fe01f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe015a720 0x7f2fe014fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0071fd0 con 0x7f2fe0071bc0 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe00cc2d0 con 0x7f2fe00ca540 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe7f57640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2fe0072150 con 0x7f2fe015a720 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe5cf3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe0071bc0 0x7f2fe00cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe5cf3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe0071bc0 0x7f2fe00cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48296/0 (socket says 172.21.15.67:48296) 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe5cf3640 1 -- 172.21.15.67:0/256812239 learned_addr learned my addr 172.21.15.67:0/256812239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe64f4640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe015a720 0x7f2fe014fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.805+0000 7f2fe54f2640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fe00ca540 0x7f2fe01f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe5cf3640 1 -- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe015a720 msgr2=0x7f2fe014fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe5cf3640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe015a720 0x7f2fe014fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe5cf3640 1 -- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fe00ca540 msgr2=0x7f2fe01f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe5cf3640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fe00ca540 0x7f2fe01f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe5cf3640 1 -- 172.21.15.67:0/256812239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe014dad0 con 0x7f2fe0071bc0 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe5cf3640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe0071bc0 0x7f2fe00cef30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00c730 tx=0x7f2fdc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fd67fc640 1 -- 172.21.15.67:0/256812239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc017070 con 0x7f2fe0071bc0 2024-04-25T15:55:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fd67fc640 1 -- 172.21.15.67:0/256812239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fdc00ce40 con 0x7f2fe0071bc0 2024-04-25T15:55:35.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fd67fc640 1 -- 172.21.15.67:0/256812239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc00a880 con 0x7f2fe0071bc0 2024-04-25T15:55:35.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe014dd40 con 0x7f2fe0071bc0 2024-04-25T15:55:35.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe00cd940 con 0x7f2fe0071bc0 2024-04-25T15:55:35.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe006a480 con 0x7f2fe0071bc0 2024-04-25T15:55:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fd67fc640 1 -- 172.21.15.67:0/256812239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2fdc01a070 con 0x7f2fe0071bc0 2024-04-25T15:55:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.809+0000 7f2fd67fc640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2fc406f400 0x7f2fc4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.813+0000 7f2fd67fc640 1 -- 172.21.15.67:0/256812239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f2fdc08e920 con 0x7f2fe0071bc0 2024-04-25T15:55:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.813+0000 7f2fe54f2640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2fc406f400 0x7f2fc4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.813+0000 7f2fe54f2640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2fc406f400 0x7f2fc4071910 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00096d0 tx=0x7f2fd0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:35.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.817+0000 7f2fd67fc640 1 -- 172.21.15.67:0/256812239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2fdc05b120 con 0x7f2fe0071bc0 2024-04-25T15:55:35.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.977+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 --> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2fe0070530 con 0x7f2fc406f400 2024-04-25T15:55:35.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:35.997+0000 7f2fd67fc640 1 -- 172.21.15.67:0/256812239 <== mgr.4103 v2:172.21.15.176:6832/2784768863 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254289 (secure 0 0 0) 0x7f2fe0070530 con 0x7f2fc406f400 2024-04-25T15:55:36.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:36.002 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-25T15:55:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2fc406f400 msgr2=0x7f2fc4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2fc406f400 0x7f2fc4071910 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00096d0 tx=0x7f2fd0009240 comp rx=0 tx=0).stop 2024-04-25T15:55:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe0071bc0 msgr2=0x7f2fe00cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe0071bc0 0x7f2fe00cef30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00c730 tx=0x7f2fdc00cbf0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 shutdown_connections 2024-04-25T15:55:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f2fc406f400 0x7f2fc4071910 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fe015a720 0x7f2fe014fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fe00ca540 0x7f2fe01f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 --2- 172.21.15.67:0/256812239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe0071bc0 0x7f2fe00cef30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 >> 172.21.15.67:0/256812239 conn(0x7f2fe00700b0 msgr2=0x7f2fe0068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 shutdown_connections 2024-04-25T15:55:36.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.001+0000 7f2fe7f57640 1 -- 172.21.15.67:0/256812239 wait complete. 2024-04-25T15:55:36.028 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-04-25T15:55:35.906065+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798320,"num_objects":101,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":9664,"num_write_kb":25167774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":1627779911,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10191,"ondisk_log_size":10191,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":388144,"kb_used_data":13872,"kb_used_omap":12,"kb_used_meta":374195,"kb_avail":754586576,"statfs":{"total":773094113280,"available":772696653824,"internally_reserved":0,"allocated":14204928,"data_stored":7557880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":383176242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005095"},"pg_stats":[{"pgid":"3.3f","version":"129'146","reported_seq":385,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:24.682271+0000","last_change":"2024-04-25T15:55:24.682342+0000","last_active":"2024-04-25T15:55:24.682271+0000","last_peered":"2024-04-25T15:55:24.682271+0000","last_clean":"2024-04-25T15:55:24.682271+0000","last_became_active":"2024-04-25T15:50:31.206952+0000","last_became_peered":"2024-04-25T15:50:31.206952+0000","last_unstale":"2024-04-25T15:55:24.682271+0000","last_undegraded":"2024-04-25T15:55:24.682271+0000","last_fullsized":"2024-04-25T15:55:24.682271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'146","last_scrub_stamp":"2024-04-25T15:55:24.682052+0000","last_deep_scrub":"129'146","last_deep_scrub_stamp":"2024-04-25T15:55:24.682052+0000","last_clean_scrub_stamp":"2024-04-25T15:55:24.682052+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:48.714198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"129'130","reported_seq":264,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:22.882623+0000","last_change":"2024-04-25T15:55:22.882822+0000","last_active":"2024-04-25T15:55:22.882623+0000","last_peered":"2024-04-25T15:55:22.882623+0000","last_clean":"2024-04-25T15:55:22.882623+0000","last_became_active":"2024-04-25T15:54:44.215261+0000","last_became_peered":"2024-04-25T15:54:44.215261+0000","last_unstale":"2024-04-25T15:55:22.882623+0000","last_undegraded":"2024-04-25T15:55:22.882623+0000","last_fullsized":"2024-04-25T15:55:22.882623+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'130","last_scrub_stamp":"2024-04-25T15:55:22.882276+0000","last_deep_scrub":"129'130","last_deep_scrub_stamp":"2024-04-25T15:55:22.882276+0000","last_clean_scrub_stamp":"2024-04-25T15:55:22.882276+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:10.432046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"129'151","reported_seq":352,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.853785+0000","last_change":"2024-04-25T15:55:15.853875+0000","last_active":"2024-04-25T15:55:15.853785+0000","last_peered":"2024-04-25T15:55:15.853785+0000","last_clean":"2024-04-25T15:55:15.853785+0000","last_became_active":"2024-04-25T15:54:40.164866+0000","last_became_peered":"2024-04-25T15:54:40.164866+0000","last_unstale":"2024-04-25T15:55:15.853785+0000","last_undegraded":"2024-04-25T15:55:15.853785+0000","last_fullsized":"2024-04-25T15:55:15.853785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'151","last_scrub_stamp":"2024-04-25T15:55:15.853567+0000","last_deep_scrub":"129'151","last_deep_scrub_stamp":"2024-04-25T15:55:15.853567+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.853567+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:31.854753+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"129'173","reported_seq":414,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:22.658262+0000","last_change":"2024-04-25T15:55:22.658335+0000","last_active":"2024-04-25T15:55:22.658262+0000","last_peered":"2024-04-25T15:55:22.658262+0000","last_clean":"2024-04-25T15:55:22.658262+0000","last_became_active":"2024-04-25T15:50:31.201924+0000","last_became_peered":"2024-04-25T15:50:31.201924+0000","last_unstale":"2024-04-25T15:55:22.658262+0000","last_undegraded":"2024-04-25T15:55:22.658262+0000","last_fullsized":"2024-04-25T15:55:22.658262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'173","last_scrub_stamp":"2024-04-25T15:55:22.658032+0000","last_deep_scrub":"129'173","last_deep_scrub_stamp":"2024-04-25T15:55:22.658032+0000","last_clean_scrub_stamp":"2024-04-25T15:55:22.658032+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:20:47.823461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"129'148","reported_seq":387,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.538224+0000","last_change":"2024-04-25T15:55:15.538224+0000","last_active":"2024-04-25T15:55:15.538224+0000","last_peered":"2024-04-25T15:55:15.538224+0000","last_clean":"2024-04-25T15:55:15.538224+0000","last_became_active":"2024-04-25T15:50:31.207985+0000","last_became_peered":"2024-04-25T15:50:31.207985+0000","last_unstale":"2024-04-25T15:55:15.538224+0000","last_undegraded":"2024-04-25T15:55:15.538224+0000","last_fullsized":"2024-04-25T15:55:15.538224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'148","last_scrub_stamp":"2024-04-25T15:55:15.537674+0000","last_deep_scrub":"129'148","last_deep_scrub_stamp":"2024-04-25T15:55:15.537674+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.537674+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:55.461163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"129'135","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:20.585347+0000","last_change":"2024-04-25T15:55:20.585347+0000","last_active":"2024-04-25T15:55:20.585347+0000","last_peered":"2024-04-25T15:55:20.585347+0000","last_clean":"2024-04-25T15:55:20.585347+0000","last_became_active":"2024-04-25T15:50:31.186164+0000","last_became_peered":"2024-04-25T15:50:31.186164+0000","last_unstale":"2024-04-25T15:55:20.585347+0000","last_undegraded":"2024-04-25T15:55:20.585347+0000","last_fullsized":"2024-04-25T15:55:20.585347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'135","last_scrub_stamp":"2024-04-25T15:55:20.584934+0000","last_deep_scrub":"129'135","last_deep_scrub_stamp":"2024-04-25T15:55:20.584934+0000","last_clean_scrub_stamp":"2024-04-25T15:55:20.584934+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:12.898511+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"129'164","reported_seq":403,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:22.671099+0000","last_change":"2024-04-25T15:55:22.671099+0000","last_active":"2024-04-25T15:55:22.671099+0000","last_peered":"2024-04-25T15:55:22.671099+0000","last_clean":"2024-04-25T15:55:22.671099+0000","last_became_active":"2024-04-25T15:50:31.192523+0000","last_became_peered":"2024-04-25T15:50:31.192523+0000","last_unstale":"2024-04-25T15:55:22.671099+0000","last_undegraded":"2024-04-25T15:55:22.671099+0000","last_fullsized":"2024-04-25T15:55:22.671099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'164","last_scrub_stamp":"2024-04-25T15:55:22.670803+0000","last_deep_scrub":"129'164","last_deep_scrub_stamp":"2024-04-25T15:55:22.670803+0000","last_clean_scrub_stamp":"2024-04-25T15:55:22.670803+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:22.767270+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"129'186","reported_seq":427,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.210507+0000","last_change":"2024-04-25T15:55:16.210507+0000","last_active":"2024-04-25T15:55:16.210507+0000","last_peered":"2024-04-25T15:55:16.210507+0000","last_clean":"2024-04-25T15:55:16.210507+0000","last_became_active":"2024-04-25T15:50:31.196455+0000","last_became_peered":"2024-04-25T15:50:31.196455+0000","last_unstale":"2024-04-25T15:55:16.210507+0000","last_undegraded":"2024-04-25T15:55:16.210507+0000","last_fullsized":"2024-04-25T15:55:16.210507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'186","last_scrub_stamp":"2024-04-25T15:55:16.210180+0000","last_deep_scrub":"129'186","last_deep_scrub_stamp":"2024-04-25T15:55:16.210180+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.210180+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:17:50.044467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"129'152","reported_seq":385,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.931583+0000","last_change":"2024-04-25T15:55:17.931651+0000","last_active":"2024-04-25T15:55:17.931583+0000","last_peered":"2024-04-25T15:55:17.931583+0000","last_clean":"2024-04-25T15:55:17.931583+0000","last_became_active":"2024-04-25T15:51:11.145226+0000","last_became_peered":"2024-04-25T15:51:11.145226+0000","last_unstale":"2024-04-25T15:55:17.931583+0000","last_undegraded":"2024-04-25T15:55:17.931583+0000","last_fullsized":"2024-04-25T15:55:17.931583+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'152","last_scrub_stamp":"2024-04-25T15:55:17.931354+0000","last_deep_scrub":"129'152","last_deep_scrub_stamp":"2024-04-25T15:55:17.931354+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.931354+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:54.699169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.36","version":"129'156","reported_seq":467,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:25.265733+0000","last_change":"2024-04-25T15:55:25.265807+0000","last_active":"2024-04-25T15:55:25.265733+0000","last_peered":"2024-04-25T15:55:25.265733+0000","last_clean":"2024-04-25T15:55:25.265733+0000","last_became_active":"2024-04-25T15:54:54.352223+0000","last_became_peered":"2024-04-25T15:54:54.352223+0000","last_unstale":"2024-04-25T15:55:25.265733+0000","last_undegraded":"2024-04-25T15:55:25.265733+0000","last_fullsized":"2024-04-25T15:55:25.265733+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'156","last_scrub_stamp":"2024-04-25T15:55:25.265505+0000","last_deep_scrub":"129'156","last_deep_scrub_stamp":"2024-04-25T15:55:25.265505+0000","last_clean_scrub_stamp":"2024-04-25T15:55:25.265505+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:32.310169+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"129'189","reported_seq":430,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.111820+0000","last_change":"2024-04-25T15:55:17.111820+0000","last_active":"2024-04-25T15:55:17.111820+0000","last_peered":"2024-04-25T15:55:17.111820+0000","last_clean":"2024-04-25T15:55:17.111820+0000","last_became_active":"2024-04-25T15:50:31.173010+0000","last_became_peered":"2024-04-25T15:50:31.173010+0000","last_unstale":"2024-04-25T15:55:17.111820+0000","last_undegraded":"2024-04-25T15:55:17.111820+0000","last_fullsized":"2024-04-25T15:55:17.111820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'189","last_scrub_stamp":"2024-04-25T15:55:17.111497+0000","last_deep_scrub":"129'189","last_deep_scrub_stamp":"2024-04-25T15:55:17.111497+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.111497+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:50:57.522246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"129'142","reported_seq":267,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.639173+0000","last_change":"2024-04-25T15:55:15.639248+0000","last_active":"2024-04-25T15:55:15.639173+0000","last_peered":"2024-04-25T15:55:15.639173+0000","last_clean":"2024-04-25T15:55:15.639173+0000","last_became_active":"2024-04-25T15:53:16.037590+0000","last_became_peered":"2024-04-25T15:53:16.037590+0000","last_unstale":"2024-04-25T15:55:15.639173+0000","last_undegraded":"2024-04-25T15:55:15.639173+0000","last_fullsized":"2024-04-25T15:55:15.639173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'142","last_scrub_stamp":"2024-04-25T15:55:15.638894+0000","last_deep_scrub":"129'142","last_deep_scrub_stamp":"2024-04-25T15:55:15.638894+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.638894+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:38:29.970156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"129'151","reported_seq":701,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:22.190367+0000","last_change":"2024-04-25T15:55:22.190367+0000","last_active":"2024-04-25T15:55:22.190367+0000","last_peered":"2024-04-25T15:55:22.190367+0000","last_clean":"2024-04-25T15:55:22.190367+0000","last_became_active":"2024-04-25T15:54:54.356694+0000","last_became_peered":"2024-04-25T15:54:54.356694+0000","last_unstale":"2024-04-25T15:55:22.190367+0000","last_undegraded":"2024-04-25T15:55:22.190367+0000","last_fullsized":"2024-04-25T15:55:22.190367+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'151","last_scrub_stamp":"2024-04-25T15:55:22.190046+0000","last_deep_scrub":"129'151","last_deep_scrub_stamp":"2024-04-25T15:55:22.190046+0000","last_clean_scrub_stamp":"2024-04-25T15:55:22.190046+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:32.891665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"129'149","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:11.902005+0000","last_change":"2024-04-25T15:55:11.902061+0000","last_active":"2024-04-25T15:55:11.902005+0000","last_peered":"2024-04-25T15:55:11.902005+0000","last_clean":"2024-04-25T15:55:11.902005+0000","last_became_active":"2024-04-25T15:54:40.159879+0000","last_became_peered":"2024-04-25T15:54:40.159879+0000","last_unstale":"2024-04-25T15:55:11.902005+0000","last_undegraded":"2024-04-25T15:55:11.902005+0000","last_fullsized":"2024-04-25T15:55:11.902005+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'149","last_scrub_stamp":"2024-04-25T15:55:11.901847+0000","last_deep_scrub":"129'149","last_deep_scrub_stamp":"2024-04-25T15:55:11.901847+0000","last_clean_scrub_stamp":"2024-04-25T15:55:11.901847+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:42:28.576752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"129'151","reported_seq":279,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.875500+0000","last_change":"2024-04-25T15:55:18.875579+0000","last_active":"2024-04-25T15:55:18.875500+0000","last_peered":"2024-04-25T15:55:18.875500+0000","last_clean":"2024-04-25T15:55:18.875500+0000","last_became_active":"2024-04-25T15:54:44.207144+0000","last_became_peered":"2024-04-25T15:54:44.207144+0000","last_unstale":"2024-04-25T15:55:18.875500+0000","last_undegraded":"2024-04-25T15:55:18.875500+0000","last_fullsized":"2024-04-25T15:55:18.875500+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'151","last_scrub_stamp":"2024-04-25T15:55:18.875271+0000","last_deep_scrub":"129'151","last_deep_scrub_stamp":"2024-04-25T15:55:18.875271+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.875271+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:14:33.251622+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"129'147","reported_seq":388,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.622427+0000","last_change":"2024-04-25T15:55:17.622604+0000","last_active":"2024-04-25T15:55:17.622427+0000","last_peered":"2024-04-25T15:55:17.622427+0000","last_clean":"2024-04-25T15:55:17.622427+0000","last_became_active":"2024-04-25T15:50:31.184154+0000","last_became_peered":"2024-04-25T15:50:31.184154+0000","last_unstale":"2024-04-25T15:55:17.622427+0000","last_undegraded":"2024-04-25T15:55:17.622427+0000","last_fullsized":"2024-04-25T15:55:17.622427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'147","last_scrub_stamp":"2024-04-25T15:55:17.622092+0000","last_deep_scrub":"129'147","last_deep_scrub_stamp":"2024-04-25T15:55:17.622092+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.622092+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:31.367766+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"129'146","reported_seq":320,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:25.652687+0000","last_change":"2024-04-25T15:55:25.652757+0000","last_active":"2024-04-25T15:55:25.652687+0000","last_peered":"2024-04-25T15:55:25.652687+0000","last_clean":"2024-04-25T15:55:25.652687+0000","last_became_active":"2024-04-25T15:52:26.279081+0000","last_became_peered":"2024-04-25T15:52:26.279081+0000","last_unstale":"2024-04-25T15:55:25.652687+0000","last_undegraded":"2024-04-25T15:55:25.652687+0000","last_fullsized":"2024-04-25T15:55:25.652687+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'146","last_scrub_stamp":"2024-04-25T15:55:25.652469+0000","last_deep_scrub":"129'146","last_deep_scrub_stamp":"2024-04-25T15:55:25.652469+0000","last_clean_scrub_stamp":"2024-04-25T15:55:25.652469+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:41.040302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"129'157","reported_seq":611,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:21.901378+0000","last_change":"2024-04-25T15:55:21.901462+0000","last_active":"2024-04-25T15:55:21.901378+0000","last_peered":"2024-04-25T15:55:21.901378+0000","last_clean":"2024-04-25T15:55:21.901378+0000","last_became_active":"2024-04-25T15:54:54.354849+0000","last_became_peered":"2024-04-25T15:54:54.354849+0000","last_unstale":"2024-04-25T15:55:21.901378+0000","last_undegraded":"2024-04-25T15:55:21.901378+0000","last_fullsized":"2024-04-25T15:55:21.901378+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'157","last_scrub_stamp":"2024-04-25T15:55:21.901150+0000","last_deep_scrub":"129'157","last_deep_scrub_stamp":"2024-04-25T15:55:21.901150+0000","last_clean_scrub_stamp":"2024-04-25T15:55:21.901150+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:15.451783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":184549376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"129'172","reported_seq":411,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:25.640697+0000","last_change":"2024-04-25T15:55:25.640798+0000","last_active":"2024-04-25T15:55:25.640697+0000","last_peered":"2024-04-25T15:55:25.640697+0000","last_clean":"2024-04-25T15:55:25.640697+0000","last_became_active":"2024-04-25T15:50:31.209933+0000","last_became_peered":"2024-04-25T15:50:31.209933+0000","last_unstale":"2024-04-25T15:55:25.640697+0000","last_undegraded":"2024-04-25T15:55:25.640697+0000","last_fullsized":"2024-04-25T15:55:25.640697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'172","last_scrub_stamp":"2024-04-25T15:55:25.640450+0000","last_deep_scrub":"129'172","last_deep_scrub_stamp":"2024-04-25T15:55:25.640450+0000","last_clean_scrub_stamp":"2024-04-25T15:55:25.640450+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:40.344979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"129'161","reported_seq":418,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:21.194854+0000","last_change":"2024-04-25T15:55:21.194923+0000","last_active":"2024-04-25T15:55:21.194854+0000","last_peered":"2024-04-25T15:55:21.194854+0000","last_clean":"2024-04-25T15:55:21.194854+0000","last_became_active":"2024-04-25T15:53:16.053165+0000","last_became_peered":"2024-04-25T15:53:16.053165+0000","last_unstale":"2024-04-25T15:55:21.194854+0000","last_undegraded":"2024-04-25T15:55:21.194854+0000","last_fullsized":"2024-04-25T15:55:21.194854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'161","last_scrub_stamp":"2024-04-25T15:55:21.194593+0000","last_deep_scrub":"129'161","last_deep_scrub_stamp":"2024-04-25T15:55:21.194593+0000","last_clean_scrub_stamp":"2024-04-25T15:55:21.194593+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:09:14.749968+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":153,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"129'137","reported_seq":336,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:19.916776+0000","last_change":"2024-04-25T15:55:19.916852+0000","last_active":"2024-04-25T15:55:19.916776+0000","last_peered":"2024-04-25T15:55:19.916776+0000","last_clean":"2024-04-25T15:55:19.916776+0000","last_became_active":"2024-04-25T15:54:40.167469+0000","last_became_peered":"2024-04-25T15:54:40.167469+0000","last_unstale":"2024-04-25T15:55:19.916776+0000","last_undegraded":"2024-04-25T15:55:19.916776+0000","last_fullsized":"2024-04-25T15:55:19.916776+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'137","last_scrub_stamp":"2024-04-25T15:55:19.916552+0000","last_deep_scrub":"129'137","last_deep_scrub_stamp":"2024-04-25T15:55:19.916552+0000","last_clean_scrub_stamp":"2024-04-25T15:55:19.916552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:53.586531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"129'166","reported_seq":399,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.957233+0000","last_change":"2024-04-25T15:55:16.957307+0000","last_active":"2024-04-25T15:55:16.957233+0000","last_peered":"2024-04-25T15:55:16.957233+0000","last_clean":"2024-04-25T15:55:16.957233+0000","last_became_active":"2024-04-25T15:51:11.144582+0000","last_became_peered":"2024-04-25T15:51:11.144582+0000","last_unstale":"2024-04-25T15:55:16.957233+0000","last_undegraded":"2024-04-25T15:55:16.957233+0000","last_fullsized":"2024-04-25T15:55:16.957233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'166","last_scrub_stamp":"2024-04-25T15:55:16.957013+0000","last_deep_scrub":"129'166","last_deep_scrub_stamp":"2024-04-25T15:55:16.957013+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.957013+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:11:57.392038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.29","version":"129'134","reported_seq":391,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.184241+0000","last_change":"2024-04-25T15:55:18.184395+0000","last_active":"2024-04-25T15:55:18.184241+0000","last_peered":"2024-04-25T15:55:18.184241+0000","last_clean":"2024-04-25T15:55:18.184241+0000","last_became_active":"2024-04-25T15:53:16.049913+0000","last_became_peered":"2024-04-25T15:53:16.049913+0000","last_unstale":"2024-04-25T15:55:18.184241+0000","last_undegraded":"2024-04-25T15:55:18.184241+0000","last_fullsized":"2024-04-25T15:55:18.184241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'134","last_scrub_stamp":"2024-04-25T15:55:18.183902+0000","last_deep_scrub":"129'134","last_deep_scrub_stamp":"2024-04-25T15:55:18.183902+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.183902+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:18:29.536808+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"129'180","reported_seq":421,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:23.639959+0000","last_change":"2024-04-25T15:55:23.640154+0000","last_active":"2024-04-25T15:55:23.639959+0000","last_peered":"2024-04-25T15:55:23.639959+0000","last_clean":"2024-04-25T15:55:23.639959+0000","last_became_active":"2024-04-25T15:50:31.202191+0000","last_became_peered":"2024-04-25T15:50:31.202191+0000","last_unstale":"2024-04-25T15:55:23.639959+0000","last_undegraded":"2024-04-25T15:55:23.639959+0000","last_fullsized":"2024-04-25T15:55:23.639959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'180","last_scrub_stamp":"2024-04-25T15:55:23.639603+0000","last_deep_scrub":"129'180","last_deep_scrub_stamp":"2024-04-25T15:55:23.639603+0000","last_clean_scrub_stamp":"2024-04-25T15:55:23.639603+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:43.761948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":475136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"129'134","reported_seq":246,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:31.523842+0000","last_change":"2024-04-25T15:55:31.523981+0000","last_active":"2024-04-25T15:55:31.523842+0000","last_peered":"2024-04-25T15:55:31.523842+0000","last_clean":"2024-04-25T15:55:31.523842+0000","last_became_active":"2024-04-25T15:53:35.293980+0000","last_became_peered":"2024-04-25T15:53:35.293980+0000","last_unstale":"2024-04-25T15:55:31.523842+0000","last_undegraded":"2024-04-25T15:55:31.523842+0000","last_fullsized":"2024-04-25T15:55:31.523842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'134","last_scrub_stamp":"2024-04-25T15:55:31.523496+0000","last_deep_scrub":"129'134","last_deep_scrub_stamp":"2024-04-25T15:55:31.523496+0000","last_clean_scrub_stamp":"2024-04-25T15:55:31.523496+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.769400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.26","version":"129'198","reported_seq":439,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:19.171816+0000","last_change":"2024-04-25T15:55:19.171885+0000","last_active":"2024-04-25T15:55:19.171816+0000","last_peered":"2024-04-25T15:55:19.171816+0000","last_clean":"2024-04-25T15:55:19.171816+0000","last_became_active":"2024-04-25T15:50:31.212104+0000","last_became_peered":"2024-04-25T15:50:31.212104+0000","last_unstale":"2024-04-25T15:55:19.171816+0000","last_undegraded":"2024-04-25T15:55:19.171816+0000","last_fullsized":"2024-04-25T15:55:19.171816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'198","last_scrub_stamp":"2024-04-25T15:55:19.171559+0000","last_deep_scrub":"129'198","last_deep_scrub_stamp":"2024-04-25T15:55:19.171559+0000","last_clean_scrub_stamp":"2024-04-25T15:55:19.171559+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:56.988729+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"129'144","reported_seq":365,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:20.883698+0000","last_change":"2024-04-25T15:55:20.883793+0000","last_active":"2024-04-25T15:55:20.883698+0000","last_peered":"2024-04-25T15:55:20.883698+0000","last_clean":"2024-04-25T15:55:20.883698+0000","last_became_active":"2024-04-25T15:54:44.210520+0000","last_became_peered":"2024-04-25T15:54:44.210520+0000","last_unstale":"2024-04-25T15:55:20.883698+0000","last_undegraded":"2024-04-25T15:55:20.883698+0000","last_fullsized":"2024-04-25T15:55:20.883698+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'144","last_scrub_stamp":"2024-04-25T15:55:20.883473+0000","last_deep_scrub":"129'144","last_deep_scrub_stamp":"2024-04-25T15:55:20.883473+0000","last_clean_scrub_stamp":"2024-04-25T15:55:20.883473+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:06:47.183620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"129'133","reported_seq":409,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:19.179799+0000","last_change":"2024-04-25T15:55:19.179870+0000","last_active":"2024-04-25T15:55:19.179799+0000","last_peered":"2024-04-25T15:55:19.179799+0000","last_clean":"2024-04-25T15:55:19.179799+0000","last_became_active":"2024-04-25T15:53:16.050065+0000","last_became_peered":"2024-04-25T15:53:16.050065+0000","last_unstale":"2024-04-25T15:55:19.179799+0000","last_undegraded":"2024-04-25T15:55:19.179799+0000","last_fullsized":"2024-04-25T15:55:19.179799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'133","last_scrub_stamp":"2024-04-25T15:55:19.179571+0000","last_deep_scrub":"129'133","last_deep_scrub_stamp":"2024-04-25T15:55:19.179571+0000","last_clean_scrub_stamp":"2024-04-25T15:55:19.179571+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:32.591850+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"129'118","reported_seq":320,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:21.911711+0000","last_change":"2024-04-25T15:55:21.911860+0000","last_active":"2024-04-25T15:55:21.911711+0000","last_peered":"2024-04-25T15:55:21.911711+0000","last_clean":"2024-04-25T15:55:21.911711+0000","last_became_active":"2024-04-25T15:54:40.173209+0000","last_became_peered":"2024-04-25T15:54:40.173209+0000","last_unstale":"2024-04-25T15:55:21.911711+0000","last_undegraded":"2024-04-25T15:55:21.911711+0000","last_fullsized":"2024-04-25T15:55:21.911711+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'118","last_scrub_stamp":"2024-04-25T15:55:21.911366+0000","last_deep_scrub":"129'118","last_deep_scrub_stamp":"2024-04-25T15:55:21.911366+0000","last_clean_scrub_stamp":"2024-04-25T15:55:21.911366+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:53:15.352789+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"129'131","reported_seq":316,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:13.808894+0000","last_change":"2024-04-25T15:55:13.808919+0000","last_active":"2024-04-25T15:55:13.808894+0000","last_peered":"2024-04-25T15:55:13.808894+0000","last_clean":"2024-04-25T15:55:13.808894+0000","last_became_active":"2024-04-25T15:52:11.994283+0000","last_became_peered":"2024-04-25T15:52:11.994283+0000","last_unstale":"2024-04-25T15:55:13.808894+0000","last_undegraded":"2024-04-25T15:55:13.808894+0000","last_fullsized":"2024-04-25T15:55:13.808894+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'131","last_scrub_stamp":"2024-04-25T15:55:13.808796+0000","last_deep_scrub":"129'131","last_deep_scrub_stamp":"2024-04-25T15:55:13.808796+0000","last_clean_scrub_stamp":"2024-04-25T15:55:13.808796+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:46:40.368155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"129'135","reported_seq":385,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.899072+0000","last_change":"2024-04-25T15:55:18.899146+0000","last_active":"2024-04-25T15:55:18.899072+0000","last_peered":"2024-04-25T15:55:18.899072+0000","last_clean":"2024-04-25T15:55:18.899072+0000","last_became_active":"2024-04-25T15:51:29.139554+0000","last_became_peered":"2024-04-25T15:51:29.139554+0000","last_unstale":"2024-04-25T15:55:18.899072+0000","last_undegraded":"2024-04-25T15:55:18.899072+0000","last_fullsized":"2024-04-25T15:55:18.899072+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'135","last_scrub_stamp":"2024-04-25T15:55:18.898845+0000","last_deep_scrub":"129'135","last_deep_scrub_stamp":"2024-04-25T15:55:18.898845+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.898845+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:39.313539+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"129'150","reported_seq":488,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:26.220901+0000","last_change":"2024-04-25T15:55:26.220975+0000","last_active":"2024-04-25T15:55:26.220901+0000","last_peered":"2024-04-25T15:55:26.220901+0000","last_clean":"2024-04-25T15:55:26.220901+0000","last_became_active":"2024-04-25T15:54:54.353824+0000","last_became_peered":"2024-04-25T15:54:54.353824+0000","last_unstale":"2024-04-25T15:55:26.220901+0000","last_undegraded":"2024-04-25T15:55:26.220901+0000","last_fullsized":"2024-04-25T15:55:26.220901+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'150","last_scrub_stamp":"2024-04-25T15:55:26.220665+0000","last_deep_scrub":"129'150","last_deep_scrub_stamp":"2024-04-25T15:55:26.220665+0000","last_clean_scrub_stamp":"2024-04-25T15:55:26.220665+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:06:06.857152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":146800640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"129'130","reported_seq":255,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:14.609775+0000","last_change":"2024-04-25T15:55:14.609845+0000","last_active":"2024-04-25T15:55:14.609775+0000","last_peered":"2024-04-25T15:55:14.609775+0000","last_clean":"2024-04-25T15:55:14.609775+0000","last_became_active":"2024-04-25T15:53:16.039692+0000","last_became_peered":"2024-04-25T15:53:16.039692+0000","last_unstale":"2024-04-25T15:55:14.609775+0000","last_undegraded":"2024-04-25T15:55:14.609775+0000","last_fullsized":"2024-04-25T15:55:14.609775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'130","last_scrub_stamp":"2024-04-25T15:55:14.609559+0000","last_deep_scrub":"129'130","last_deep_scrub_stamp":"2024-04-25T15:55:14.609559+0000","last_clean_scrub_stamp":"2024-04-25T15:55:14.609559+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:16:12.694771+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"129'8","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:14.154588+0000","last_change":"2024-04-25T15:55:14.154588+0000","last_active":"2024-04-25T15:55:14.154588+0000","last_peered":"2024-04-25T15:55:14.154588+0000","last_clean":"2024-04-25T15:55:14.154588+0000","last_became_active":"2024-04-25T15:50:30.181717+0000","last_became_peered":"2024-04-25T15:50:30.181717+0000","last_unstale":"2024-04-25T15:55:14.154588+0000","last_undegraded":"2024-04-25T15:55:14.154588+0000","last_fullsized":"2024-04-25T15:55:14.154588+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8","last_scrub_stamp":"2024-04-25T15:55:14.154488+0000","last_deep_scrub":"129'8","last_deep_scrub_stamp":"2024-04-25T15:55:14.154488+0000","last_clean_scrub_stamp":"2024-04-25T15:55:14.154488+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:47.140761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1204,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"129'137","reported_seq":267,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:12.886596+0000","last_change":"2024-04-25T15:55:12.886663+0000","last_active":"2024-04-25T15:55:12.886596+0000","last_peered":"2024-04-25T15:55:12.886596+0000","last_clean":"2024-04-25T15:55:12.886596+0000","last_became_active":"2024-04-25T15:54:40.162667+0000","last_became_peered":"2024-04-25T15:54:40.162667+0000","last_unstale":"2024-04-25T15:55:12.886596+0000","last_undegraded":"2024-04-25T15:55:12.886596+0000","last_fullsized":"2024-04-25T15:55:12.886596+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'137","last_scrub_stamp":"2024-04-25T15:55:12.886374+0000","last_deep_scrub":"129'137","last_deep_scrub_stamp":"2024-04-25T15:55:12.886374+0000","last_clean_scrub_stamp":"2024-04-25T15:55:12.886374+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:37.419326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"129'10","reported_seq":36,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:22.196356+0000","last_change":"2024-04-25T15:55:22.196464+0000","last_active":"2024-04-25T15:55:22.196356+0000","last_peered":"2024-04-25T15:55:22.196356+0000","last_clean":"2024-04-25T15:55:22.196356+0000","last_became_active":"2024-04-25T15:54:51.302422+0000","last_became_peered":"2024-04-25T15:54:51.302422+0000","last_unstale":"2024-04-25T15:55:22.196356+0000","last_undegraded":"2024-04-25T15:55:22.196356+0000","last_fullsized":"2024-04-25T15:55:22.196356+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'10","last_scrub_stamp":"2024-04-25T15:55:22.196131+0000","last_deep_scrub":"129'10","last_deep_scrub_stamp":"2024-04-25T15:55:22.196131+0000","last_clean_scrub_stamp":"2024-04-25T15:55:22.196131+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:18:29.569224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"129'5","reported_seq":270,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:21.210505+0000","last_change":"2024-04-25T15:55:21.210505+0000","last_active":"2024-04-25T15:55:21.210505+0000","last_peered":"2024-04-25T15:55:21.210505+0000","last_clean":"2024-04-25T15:55:21.210505+0000","last_became_active":"2024-04-25T15:54:51.307662+0000","last_became_peered":"2024-04-25T15:54:51.307662+0000","last_unstale":"2024-04-25T15:55:21.210505+0000","last_undegraded":"2024-04-25T15:55:21.210505+0000","last_fullsized":"2024-04-25T15:55:21.210505+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5","last_scrub_stamp":"2024-04-25T15:55:21.210196+0000","last_deep_scrub":"129'5","last_deep_scrub_stamp":"2024-04-25T15:55:21.210196+0000","last_clean_scrub_stamp":"2024-04-25T15:55:21.210196+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:50.637337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":20,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"129'170","reported_seq":305,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:14.852354+0000","last_change":"2024-04-25T15:55:14.852383+0000","last_active":"2024-04-25T15:55:14.852354+0000","last_peered":"2024-04-25T15:55:14.852354+0000","last_clean":"2024-04-25T15:55:14.852354+0000","last_became_active":"2024-04-25T15:53:16.029596+0000","last_became_peered":"2024-04-25T15:53:16.029596+0000","last_unstale":"2024-04-25T15:55:14.852354+0000","last_undegraded":"2024-04-25T15:55:14.852354+0000","last_fullsized":"2024-04-25T15:55:14.852354+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'170","last_scrub_stamp":"2024-04-25T15:55:14.852242+0000","last_deep_scrub":"129'170","last_deep_scrub_stamp":"2024-04-25T15:55:14.852242+0000","last_clean_scrub_stamp":"2024-04-25T15:55:14.852242+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:13.709272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":12582912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"129'4","reported_seq":265,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:26.653135+0000","last_change":"2024-04-25T15:55:26.653214+0000","last_active":"2024-04-25T15:55:26.653135+0000","last_peered":"2024-04-25T15:55:26.653135+0000","last_clean":"2024-04-25T15:55:26.653135+0000","last_became_active":"2024-04-25T15:53:16.060811+0000","last_became_peered":"2024-04-25T15:53:16.060811+0000","last_unstale":"2024-04-25T15:55:26.653135+0000","last_undegraded":"2024-04-25T15:55:26.653135+0000","last_fullsized":"2024-04-25T15:55:26.653135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'4","last_scrub_stamp":"2024-04-25T15:55:26.652920+0000","last_deep_scrub":"129'4","last_deep_scrub_stamp":"2024-04-25T15:55:26.652920+0000","last_clean_scrub_stamp":"2024-04-25T15:55:26.652920+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:46.979416+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"129'157","reported_seq":391,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:19.877248+0000","last_change":"2024-04-25T15:55:19.877321+0000","last_active":"2024-04-25T15:55:19.877248+0000","last_peered":"2024-04-25T15:55:19.877248+0000","last_clean":"2024-04-25T15:55:19.877248+0000","last_became_active":"2024-04-25T15:54:44.209985+0000","last_became_peered":"2024-04-25T15:54:44.209985+0000","last_unstale":"2024-04-25T15:55:19.877248+0000","last_undegraded":"2024-04-25T15:55:19.877248+0000","last_fullsized":"2024-04-25T15:55:19.877248+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'157","last_scrub_stamp":"2024-04-25T15:55:19.877017+0000","last_deep_scrub":"129'157","last_deep_scrub_stamp":"2024-04-25T15:55:19.877017+0000","last_clean_scrub_stamp":"2024-04-25T15:55:19.877017+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:18:35.946115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"129'9","reported_seq":252,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.110405+0000","last_change":"2024-04-25T15:55:15.110405+0000","last_active":"2024-04-25T15:55:15.110405+0000","last_peered":"2024-04-25T15:55:15.110405+0000","last_clean":"2024-04-25T15:55:15.110405+0000","last_became_active":"2024-04-25T15:50:30.197742+0000","last_became_peered":"2024-04-25T15:50:30.197742+0000","last_unstale":"2024-04-25T15:55:15.110405+0000","last_undegraded":"2024-04-25T15:55:15.110405+0000","last_fullsized":"2024-04-25T15:55:15.110405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-04-25T15:55:15.110072+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-04-25T15:55:15.110072+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.110072+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:04.152045+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"129'229","reported_seq":470,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:21.633588+0000","last_change":"2024-04-25T15:55:21.633588+0000","last_active":"2024-04-25T15:55:21.633588+0000","last_peered":"2024-04-25T15:55:21.633588+0000","last_clean":"2024-04-25T15:55:21.633588+0000","last_became_active":"2024-04-25T15:50:31.192615+0000","last_became_peered":"2024-04-25T15:50:31.192615+0000","last_unstale":"2024-04-25T15:55:21.633588+0000","last_undegraded":"2024-04-25T15:55:21.633588+0000","last_fullsized":"2024-04-25T15:55:21.633588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'229","last_scrub_stamp":"2024-04-25T15:55:21.633252+0000","last_deep_scrub":"129'229","last_deep_scrub_stamp":"2024-04-25T15:55:21.633252+0000","last_clean_scrub_stamp":"2024-04-25T15:55:21.633252+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:13.887854+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":598016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"129'6","reported_seq":249,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.637512+0000","last_change":"2024-04-25T15:55:16.637512+0000","last_active":"2024-04-25T15:55:16.637512+0000","last_peered":"2024-04-25T15:55:16.637512+0000","last_clean":"2024-04-25T15:55:16.637512+0000","last_became_active":"2024-04-25T15:50:30.202352+0000","last_became_peered":"2024-04-25T15:50:30.202352+0000","last_unstale":"2024-04-25T15:55:16.637512+0000","last_undegraded":"2024-04-25T15:55:16.637512+0000","last_fullsized":"2024-04-25T15:55:16.637512+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'6","last_scrub_stamp":"2024-04-25T15:55:16.637161+0000","last_deep_scrub":"129'6","last_deep_scrub_stamp":"2024-04-25T15:55:16.637161+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.637161+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:10.859805+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"129'141","reported_seq":343,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.879049+0000","last_change":"2024-04-25T15:55:17.879124+0000","last_active":"2024-04-25T15:55:17.879049+0000","last_peered":"2024-04-25T15:55:17.879049+0000","last_clean":"2024-04-25T15:55:17.879049+0000","last_became_active":"2024-04-25T15:54:40.168203+0000","last_became_peered":"2024-04-25T15:54:40.168203+0000","last_unstale":"2024-04-25T15:55:17.879049+0000","last_undegraded":"2024-04-25T15:55:17.879049+0000","last_fullsized":"2024-04-25T15:55:17.879049+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'141","last_scrub_stamp":"2024-04-25T15:55:17.878801+0000","last_deep_scrub":"129'141","last_deep_scrub_stamp":"2024-04-25T15:55:17.878801+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.878801+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:38:26.476996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"129'15","reported_seq":59,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.891378+0000","last_change":"2024-04-25T15:55:17.891460+0000","last_active":"2024-04-25T15:55:17.891378+0000","last_peered":"2024-04-25T15:55:17.891378+0000","last_clean":"2024-04-25T15:55:17.891378+0000","last_became_active":"2024-04-25T15:54:41.171190+0000","last_became_peered":"2024-04-25T15:54:41.171190+0000","last_unstale":"2024-04-25T15:55:17.891378+0000","last_undegraded":"2024-04-25T15:55:17.891378+0000","last_fullsized":"2024-04-25T15:55:17.891378+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-04-25T15:55:17.891160+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-04-25T15:55:17.891160+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.891160+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:49.673246+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2016,"num_keys_recovered":30,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"129'149","reported_seq":274,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.596121+0000","last_change":"2024-04-25T15:55:17.596260+0000","last_active":"2024-04-25T15:55:17.596121+0000","last_peered":"2024-04-25T15:55:17.596121+0000","last_clean":"2024-04-25T15:55:17.596121+0000","last_became_active":"2024-04-25T15:53:16.056737+0000","last_became_peered":"2024-04-25T15:53:16.056737+0000","last_unstale":"2024-04-25T15:55:17.596121+0000","last_undegraded":"2024-04-25T15:55:17.596121+0000","last_fullsized":"2024-04-25T15:55:17.596121+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'149","last_scrub_stamp":"2024-04-25T15:55:17.595755+0000","last_deep_scrub":"129'149","last_deep_scrub_stamp":"2024-04-25T15:55:17.595755+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.595755+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:18.260381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"129'97","reported_seq":46,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:24.226169+0000","last_change":"2024-04-25T15:55:24.226271+0000","last_active":"2024-04-25T15:55:24.226169+0000","last_peered":"2024-04-25T15:55:24.226169+0000","last_clean":"2024-04-25T15:55:24.226169+0000","last_became_active":"2024-04-25T15:54:51.303172+0000","last_became_peered":"2024-04-25T15:54:51.303172+0000","last_unstale":"2024-04-25T15:55:24.226169+0000","last_undegraded":"2024-04-25T15:55:24.226169+0000","last_fullsized":"2024-04-25T15:55:24.226169+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'97","last_scrub_stamp":"2024-04-25T15:55:24.225954+0000","last_deep_scrub":"129'97","last_deep_scrub_stamp":"2024-04-25T15:55:24.225954+0000","last_clean_scrub_stamp":"2024-04-25T15:55:24.225954+0000","objects_scrubbed":6,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:10:52.366582+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199279,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":380717,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"129'150","reported_seq":389,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:23.682307+0000","last_change":"2024-04-25T15:55:23.682375+0000","last_active":"2024-04-25T15:55:23.682307+0000","last_peered":"2024-04-25T15:55:23.682307+0000","last_clean":"2024-04-25T15:55:23.682307+0000","last_became_active":"2024-04-25T15:50:31.192599+0000","last_became_peered":"2024-04-25T15:50:31.192599+0000","last_unstale":"2024-04-25T15:55:23.682307+0000","last_undegraded":"2024-04-25T15:55:23.682307+0000","last_fullsized":"2024-04-25T15:55:23.682307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'150","last_scrub_stamp":"2024-04-25T15:55:23.682077+0000","last_deep_scrub":"129'150","last_deep_scrub_stamp":"2024-04-25T15:55:23.682077+0000","last_clean_scrub_stamp":"2024-04-25T15:55:23.682077+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:50:16.421739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"129'15","reported_seq":114,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:14.888718+0000","last_change":"2024-04-25T15:55:14.888755+0000","last_active":"2024-04-25T15:55:14.888718+0000","last_peered":"2024-04-25T15:55:14.888718+0000","last_clean":"2024-04-25T15:55:14.888718+0000","last_became_active":"2024-04-25T15:54:40.165701+0000","last_became_peered":"2024-04-25T15:54:40.165701+0000","last_unstale":"2024-04-25T15:55:14.888718+0000","last_undegraded":"2024-04-25T15:55:14.888718+0000","last_fullsized":"2024-04-25T15:55:14.888718+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-04-25T15:55:14.888626+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-04-25T15:55:14.888626+0000","last_clean_scrub_stamp":"2024-04-25T15:55:14.888626+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:45:46.812506+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"129'157","reported_seq":357,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.844790+0000","last_change":"2024-04-25T15:55:16.844862+0000","last_active":"2024-04-25T15:55:16.844790+0000","last_peered":"2024-04-25T15:55:16.844790+0000","last_clean":"2024-04-25T15:55:16.844790+0000","last_became_active":"2024-04-25T15:54:40.167338+0000","last_became_peered":"2024-04-25T15:54:40.167338+0000","last_unstale":"2024-04-25T15:55:16.844790+0000","last_undegraded":"2024-04-25T15:55:16.844790+0000","last_fullsized":"2024-04-25T15:55:16.844790+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'157","last_scrub_stamp":"2024-04-25T15:55:16.844519+0000","last_deep_scrub":"129'157","last_deep_scrub_stamp":"2024-04-25T15:55:16.844519+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.844519+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:45.131878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"129'8","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.613343+0000","last_change":"2024-04-25T15:55:15.613423+0000","last_active":"2024-04-25T15:55:15.613343+0000","last_peered":"2024-04-25T15:55:15.613343+0000","last_clean":"2024-04-25T15:55:15.613343+0000","last_became_active":"2024-04-25T15:50:30.198187+0000","last_became_peered":"2024-04-25T15:50:30.198187+0000","last_unstale":"2024-04-25T15:55:15.613343+0000","last_undegraded":"2024-04-25T15:55:15.613343+0000","last_fullsized":"2024-04-25T15:55:15.613343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8","last_scrub_stamp":"2024-04-25T15:55:15.613091+0000","last_deep_scrub":"129'8","last_deep_scrub_stamp":"2024-04-25T15:55:15.613091+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.613091+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:30:15.277250+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"129'171","reported_seq":430,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:28.550030+0000","last_change":"2024-04-25T15:55:28.550195+0000","last_active":"2024-04-25T15:55:28.550030+0000","last_peered":"2024-04-25T15:55:28.550030+0000","last_clean":"2024-04-25T15:55:28.550030+0000","last_became_active":"2024-04-25T15:53:16.054828+0000","last_became_peered":"2024-04-25T15:53:16.054828+0000","last_unstale":"2024-04-25T15:55:28.550030+0000","last_undegraded":"2024-04-25T15:55:28.550030+0000","last_fullsized":"2024-04-25T15:55:28.550030+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'171","last_scrub_stamp":"2024-04-25T15:55:28.549655+0000","last_deep_scrub":"129'171","last_deep_scrub_stamp":"2024-04-25T15:55:28.549655+0000","last_clean_scrub_stamp":"2024-04-25T15:55:28.549655+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:37.204863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"129'9","reported_seq":36,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:23.213047+0000","last_change":"2024-04-25T15:55:23.213202+0000","last_active":"2024-04-25T15:55:23.213047+0000","last_peered":"2024-04-25T15:55:23.213047+0000","last_clean":"2024-04-25T15:55:23.213047+0000","last_became_active":"2024-04-25T15:54:51.302160+0000","last_became_peered":"2024-04-25T15:54:51.302160+0000","last_unstale":"2024-04-25T15:55:23.213047+0000","last_undegraded":"2024-04-25T15:55:23.213047+0000","last_fullsized":"2024-04-25T15:55:23.213047+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-04-25T15:55:23.212756+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-04-25T15:55:23.212756+0000","last_clean_scrub_stamp":"2024-04-25T15:55:23.212756+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:01:04.004388+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4116,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"129'165","reported_seq":406,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.147676+0000","last_change":"2024-04-25T15:55:18.147850+0000","last_active":"2024-04-25T15:55:18.147676+0000","last_peered":"2024-04-25T15:55:18.147676+0000","last_clean":"2024-04-25T15:55:18.147676+0000","last_became_active":"2024-04-25T15:50:31.186599+0000","last_became_peered":"2024-04-25T15:50:31.186599+0000","last_unstale":"2024-04-25T15:55:18.147676+0000","last_undegraded":"2024-04-25T15:55:18.147676+0000","last_fullsized":"2024-04-25T15:55:18.147676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'165","last_scrub_stamp":"2024-04-25T15:55:18.147402+0000","last_deep_scrub":"129'165","last_deep_scrub_stamp":"2024-04-25T15:55:18.147402+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.147402+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:57:57.256980+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"129'26","reported_seq":360,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:20.957225+0000","last_change":"2024-04-25T15:55:20.957316+0000","last_active":"2024-04-25T15:55:20.957225+0000","last_peered":"2024-04-25T15:55:20.957225+0000","last_clean":"2024-04-25T15:55:20.957225+0000","last_became_active":"2024-04-25T15:54:41.166653+0000","last_became_peered":"2024-04-25T15:54:41.166653+0000","last_unstale":"2024-04-25T15:55:20.957225+0000","last_undegraded":"2024-04-25T15:55:20.957225+0000","last_fullsized":"2024-04-25T15:55:20.957225+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'26","last_scrub_stamp":"2024-04-25T15:55:20.956996+0000","last_deep_scrub":"129'26","last_deep_scrub_stamp":"2024-04-25T15:55:20.956996+0000","last_clean_scrub_stamp":"2024-04-25T15:55:20.956996+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:58.783677+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":448,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"129'189","reported_seq":456,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:30.498771+0000","last_change":"2024-04-25T15:55:30.498864+0000","last_active":"2024-04-25T15:55:30.498771+0000","last_peered":"2024-04-25T15:55:30.498771+0000","last_clean":"2024-04-25T15:55:30.498771+0000","last_became_active":"2024-04-25T15:53:16.054872+0000","last_became_peered":"2024-04-25T15:53:16.054872+0000","last_unstale":"2024-04-25T15:55:30.498771+0000","last_undegraded":"2024-04-25T15:55:30.498771+0000","last_fullsized":"2024-04-25T15:55:30.498771+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'189","last_scrub_stamp":"2024-04-25T15:55:30.498526+0000","last_deep_scrub":"129'189","last_deep_scrub_stamp":"2024-04-25T15:55:30.498526+0000","last_clean_scrub_stamp":"2024-04-25T15:55:30.498526+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:03:24.072794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":33554432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"129'4","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:13.639502+0000","last_change":"2024-04-25T15:55:13.639502+0000","last_active":"2024-04-25T15:55:13.639502+0000","last_peered":"2024-04-25T15:55:13.639502+0000","last_clean":"2024-04-25T15:55:13.639502+0000","last_became_active":"2024-04-25T15:53:16.041516+0000","last_became_peered":"2024-04-25T15:53:16.041516+0000","last_unstale":"2024-04-25T15:55:13.639502+0000","last_undegraded":"2024-04-25T15:55:13.639502+0000","last_fullsized":"2024-04-25T15:55:13.639502+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'4","last_scrub_stamp":"2024-04-25T15:55:13.639215+0000","last_deep_scrub":"129'4","last_deep_scrub_stamp":"2024-04-25T15:55:13.639215+0000","last_clean_scrub_stamp":"2024-04-25T15:55:13.639215+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:09.560112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"129'146","reported_seq":401,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:29.537048+0000","last_change":"2024-04-25T15:55:29.537115+0000","last_active":"2024-04-25T15:55:29.537048+0000","last_peered":"2024-04-25T15:55:29.537048+0000","last_clean":"2024-04-25T15:55:29.537048+0000","last_became_active":"2024-04-25T15:53:16.054896+0000","last_became_peered":"2024-04-25T15:53:16.054896+0000","last_unstale":"2024-04-25T15:55:29.537048+0000","last_undegraded":"2024-04-25T15:55:29.537048+0000","last_fullsized":"2024-04-25T15:55:29.537048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'146","last_scrub_stamp":"2024-04-25T15:55:29.536817+0000","last_deep_scrub":"129'146","last_deep_scrub_stamp":"2024-04-25T15:55:29.536817+0000","last_clean_scrub_stamp":"2024-04-25T15:55:29.536817+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:05:53.575064+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":385024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"129'7","reported_seq":249,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:14.040067+0000","last_change":"2024-04-25T15:55:14.040198+0000","last_active":"2024-04-25T15:55:14.040067+0000","last_peered":"2024-04-25T15:55:14.040067+0000","last_clean":"2024-04-25T15:55:14.040067+0000","last_became_active":"2024-04-25T15:50:30.184580+0000","last_became_peered":"2024-04-25T15:50:30.184580+0000","last_unstale":"2024-04-25T15:55:14.040067+0000","last_undegraded":"2024-04-25T15:55:14.040067+0000","last_fullsized":"2024-04-25T15:55:14.040067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7","last_scrub_stamp":"2024-04-25T15:55:14.039853+0000","last_deep_scrub":"129'7","last_deep_scrub_stamp":"2024-04-25T15:55:14.039853+0000","last_clean_scrub_stamp":"2024-04-25T15:55:14.039853+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:17.350158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"129'138","reported_seq":397,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:27.565551+0000","last_change":"2024-04-25T15:55:27.565629+0000","last_active":"2024-04-25T15:55:27.565551+0000","last_peered":"2024-04-25T15:55:27.565551+0000","last_clean":"2024-04-25T15:55:27.565551+0000","last_became_active":"2024-04-25T15:53:16.050958+0000","last_became_peered":"2024-04-25T15:53:16.050958+0000","last_unstale":"2024-04-25T15:55:27.565551+0000","last_undegraded":"2024-04-25T15:55:27.565551+0000","last_fullsized":"2024-04-25T15:55:27.565551+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'138","last_scrub_stamp":"2024-04-25T15:55:27.565317+0000","last_deep_scrub":"129'138","last_deep_scrub_stamp":"2024-04-25T15:55:27.565317+0000","last_clean_scrub_stamp":"2024-04-25T15:55:27.565317+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:15.205390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"129'4","reported_seq":50,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.859440+0000","last_change":"2024-04-25T15:55:16.859525+0000","last_active":"2024-04-25T15:55:16.859440+0000","last_peered":"2024-04-25T15:55:16.859440+0000","last_clean":"2024-04-25T15:55:16.859440+0000","last_became_active":"2024-04-25T15:54:41.165683+0000","last_became_peered":"2024-04-25T15:54:41.165683+0000","last_unstale":"2024-04-25T15:55:16.859440+0000","last_undegraded":"2024-04-25T15:55:16.859440+0000","last_fullsized":"2024-04-25T15:55:16.859440+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'4","last_scrub_stamp":"2024-04-25T15:55:16.859210+0000","last_deep_scrub":"129'4","last_deep_scrub_stamp":"2024-04-25T15:55:16.859210+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.859210+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:58:07.025606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"129'144","reported_seq":385,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:19.602696+0000","last_change":"2024-04-25T15:55:19.602878+0000","last_active":"2024-04-25T15:55:19.602696+0000","last_peered":"2024-04-25T15:55:19.602696+0000","last_clean":"2024-04-25T15:55:19.602696+0000","last_became_active":"2024-04-25T15:50:31.189758+0000","last_became_peered":"2024-04-25T15:50:31.189758+0000","last_unstale":"2024-04-25T15:55:19.602696+0000","last_undegraded":"2024-04-25T15:55:19.602696+0000","last_fullsized":"2024-04-25T15:55:19.602696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'144","last_scrub_stamp":"2024-04-25T15:55:19.602443+0000","last_deep_scrub":"129'144","last_deep_scrub_stamp":"2024-04-25T15:55:19.602443+0000","last_clean_scrub_stamp":"2024-04-25T15:55:19.602443+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:07:26.245741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"129'9","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.006762+0000","last_change":"2024-04-25T15:55:15.006821+0000","last_active":"2024-04-25T15:55:15.006762+0000","last_peered":"2024-04-25T15:55:15.006762+0000","last_clean":"2024-04-25T15:55:15.006762+0000","last_became_active":"2024-04-25T15:50:30.198222+0000","last_became_peered":"2024-04-25T15:50:30.198222+0000","last_unstale":"2024-04-25T15:55:15.006762+0000","last_undegraded":"2024-04-25T15:55:15.006762+0000","last_fullsized":"2024-04-25T15:55:15.006762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-04-25T15:55:15.006599+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-04-25T15:55:15.006599+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.006599+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:05.045292+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"129'151","reported_seq":390,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.574910+0000","last_change":"2024-04-25T15:55:16.575065+0000","last_active":"2024-04-25T15:55:16.574910+0000","last_peered":"2024-04-25T15:55:16.574910+0000","last_clean":"2024-04-25T15:55:16.574910+0000","last_became_active":"2024-04-25T15:50:31.207861+0000","last_became_peered":"2024-04-25T15:50:31.207861+0000","last_unstale":"2024-04-25T15:55:16.574910+0000","last_undegraded":"2024-04-25T15:55:16.574910+0000","last_fullsized":"2024-04-25T15:55:16.574910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'151","last_scrub_stamp":"2024-04-25T15:55:16.574564+0000","last_deep_scrub":"129'151","last_deep_scrub_stamp":"2024-04-25T15:55:16.574564+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.574564+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:34:31.446929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"129'74","reported_seq":339,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.536273+0000","last_change":"2024-04-25T15:55:17.536421+0000","last_active":"2024-04-25T15:55:17.536273+0000","last_peered":"2024-04-25T15:55:17.536273+0000","last_clean":"2024-04-25T15:55:17.536273+0000","last_became_active":"2024-04-25T15:50:19.516912+0000","last_became_peered":"2024-04-25T15:50:19.516912+0000","last_unstale":"2024-04-25T15:55:17.536273+0000","last_undegraded":"2024-04-25T15:55:17.536273+0000","last_fullsized":"2024-04-25T15:55:17.536273+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'74","last_scrub_stamp":"2024-04-25T15:55:17.535995+0000","last_deep_scrub":"129'74","last_deep_scrub_stamp":"2024-04-25T15:55:17.535995+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.535995+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:00.842766+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"129'11","reported_seq":254,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:14.583235+0000","last_change":"2024-04-25T15:55:14.583322+0000","last_active":"2024-04-25T15:55:14.583235+0000","last_peered":"2024-04-25T15:55:14.583235+0000","last_clean":"2024-04-25T15:55:14.583235+0000","last_became_active":"2024-04-25T15:50:30.195014+0000","last_became_peered":"2024-04-25T15:50:30.195014+0000","last_unstale":"2024-04-25T15:55:14.583235+0000","last_undegraded":"2024-04-25T15:55:14.583235+0000","last_fullsized":"2024-04-25T15:55:14.583235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'11","last_scrub_stamp":"2024-04-25T15:55:14.583011+0000","last_deep_scrub":"129'11","last_deep_scrub_stamp":"2024-04-25T15:55:14.583011+0000","last_clean_scrub_stamp":"2024-04-25T15:55:14.583011+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:50:28.548327+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"129'158","reported_seq":399,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.170216+0000","last_change":"2024-04-25T15:55:15.170268+0000","last_active":"2024-04-25T15:55:15.170216+0000","last_peered":"2024-04-25T15:55:15.170216+0000","last_clean":"2024-04-25T15:55:15.170216+0000","last_became_active":"2024-04-25T15:50:31.187914+0000","last_became_peered":"2024-04-25T15:50:31.187914+0000","last_unstale":"2024-04-25T15:55:15.170216+0000","last_undegraded":"2024-04-25T15:55:15.170216+0000","last_fullsized":"2024-04-25T15:55:15.170216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'158","last_scrub_stamp":"2024-04-25T15:55:15.170075+0000","last_deep_scrub":"129'158","last_deep_scrub_stamp":"2024-04-25T15:55:15.170075+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.170075+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:58.657475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":409600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"129'5","reported_seq":282,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:32.508243+0000","last_change":"2024-04-25T15:55:32.508324+0000","last_active":"2024-04-25T15:55:32.508243+0000","last_peered":"2024-04-25T15:55:32.508243+0000","last_clean":"2024-04-25T15:55:32.508243+0000","last_became_active":"2024-04-25T15:54:41.166720+0000","last_became_peered":"2024-04-25T15:54:41.166720+0000","last_unstale":"2024-04-25T15:55:32.508243+0000","last_undegraded":"2024-04-25T15:55:32.508243+0000","last_fullsized":"2024-04-25T15:55:32.508243+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5","last_scrub_stamp":"2024-04-25T15:55:32.507995+0000","last_deep_scrub":"129'5","last_deep_scrub_stamp":"2024-04-25T15:55:32.507995+0000","last_clean_scrub_stamp":"2024-04-25T15:55:32.507995+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:31.249571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"129'131","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:13.901478+0000","last_change":"2024-04-25T15:55:13.901526+0000","last_active":"2024-04-25T15:55:13.901478+0000","last_peered":"2024-04-25T15:55:13.901478+0000","last_clean":"2024-04-25T15:55:13.901478+0000","last_became_active":"2024-04-25T15:54:40.163587+0000","last_became_peered":"2024-04-25T15:54:40.163587+0000","last_unstale":"2024-04-25T15:55:13.901478+0000","last_undegraded":"2024-04-25T15:55:13.901478+0000","last_fullsized":"2024-04-25T15:55:13.901478+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'131","last_scrub_stamp":"2024-04-25T15:55:13.901339+0000","last_deep_scrub":"129'131","last_deep_scrub_stamp":"2024-04-25T15:55:13.901339+0000","last_clean_scrub_stamp":"2024-04-25T15:55:13.901339+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:14:24.375683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"129'16","reported_seq":263,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:13.052958+0000","last_change":"2024-04-25T15:55:13.053033+0000","last_active":"2024-04-25T15:55:13.052958+0000","last_peered":"2024-04-25T15:55:13.052958+0000","last_clean":"2024-04-25T15:55:13.052958+0000","last_became_active":"2024-04-25T15:50:30.184763+0000","last_became_peered":"2024-04-25T15:50:30.184763+0000","last_unstale":"2024-04-25T15:55:13.052958+0000","last_undegraded":"2024-04-25T15:55:13.052958+0000","last_fullsized":"2024-04-25T15:55:13.052958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'16","last_scrub_stamp":"2024-04-25T15:55:13.052842+0000","last_deep_scrub":"129'16","last_deep_scrub_stamp":"2024-04-25T15:55:13.052842+0000","last_clean_scrub_stamp":"2024-04-25T15:55:13.052842+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:34:37.953499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":617,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"129'131","reported_seq":440,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:21.870304+0000","last_change":"2024-04-25T15:55:21.870451+0000","last_active":"2024-04-25T15:55:21.870304+0000","last_peered":"2024-04-25T15:55:21.870304+0000","last_clean":"2024-04-25T15:55:21.870304+0000","last_became_active":"2024-04-25T15:54:44.215186+0000","last_became_peered":"2024-04-25T15:54:44.215186+0000","last_unstale":"2024-04-25T15:55:21.870304+0000","last_undegraded":"2024-04-25T15:55:21.870304+0000","last_fullsized":"2024-04-25T15:55:21.870304+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'131","last_scrub_stamp":"2024-04-25T15:55:21.869922+0000","last_deep_scrub":"129'131","last_deep_scrub_stamp":"2024-04-25T15:55:21.869922+0000","last_clean_scrub_stamp":"2024-04-25T15:55:21.869922+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:59:35.306009+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":159383552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"129'7","reported_seq":272,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:24.647962+0000","last_change":"2024-04-25T15:55:24.648151+0000","last_active":"2024-04-25T15:55:24.647962+0000","last_peered":"2024-04-25T15:55:24.647962+0000","last_clean":"2024-04-25T15:55:24.647962+0000","last_became_active":"2024-04-25T15:51:47.269095+0000","last_became_peered":"2024-04-25T15:51:47.269095+0000","last_unstale":"2024-04-25T15:55:24.647962+0000","last_undegraded":"2024-04-25T15:55:24.647962+0000","last_fullsized":"2024-04-25T15:55:24.647962+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7","last_scrub_stamp":"2024-04-25T15:55:24.647686+0000","last_deep_scrub":"129'7","last_deep_scrub_stamp":"2024-04-25T15:55:24.647686+0000","last_clean_scrub_stamp":"2024-04-25T15:55:24.647686+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:24.631506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1176,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"129'145","reported_seq":338,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.573136+0000","last_change":"2024-04-25T15:55:18.573136+0000","last_active":"2024-04-25T15:55:18.573136+0000","last_peered":"2024-04-25T15:55:18.573136+0000","last_clean":"2024-04-25T15:55:18.573136+0000","last_became_active":"2024-04-25T15:53:53.055595+0000","last_became_peered":"2024-04-25T15:53:53.055595+0000","last_unstale":"2024-04-25T15:55:18.573136+0000","last_undegraded":"2024-04-25T15:55:18.573136+0000","last_fullsized":"2024-04-25T15:55:18.573136+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'145","last_scrub_stamp":"2024-04-25T15:55:18.572620+0000","last_deep_scrub":"129'145","last_deep_scrub_stamp":"2024-04-25T15:55:18.572620+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.572620+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:47.685877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":45,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.825293+0000","last_change":"2024-04-25T15:55:15.825441+0000","last_active":"2024-04-25T15:55:15.825293+0000","last_peered":"2024-04-25T15:55:15.825293+0000","last_clean":"2024-04-25T15:55:15.825293+0000","last_became_active":"2024-04-25T15:54:41.166643+0000","last_became_peered":"2024-04-25T15:54:41.166643+0000","last_unstale":"2024-04-25T15:55:15.825293+0000","last_undegraded":"2024-04-25T15:55:15.825293+0000","last_fullsized":"2024-04-25T15:55:15.825293+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'1","last_scrub_stamp":"2024-04-25T15:55:15.825003+0000","last_deep_scrub":"20'1","last_deep_scrub_stamp":"2024-04-25T15:55:15.825003+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.825003+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:10.273404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"129'143","reported_seq":384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.620238+0000","last_change":"2024-04-25T15:55:18.620306+0000","last_active":"2024-04-25T15:55:18.620238+0000","last_peered":"2024-04-25T15:55:18.620238+0000","last_clean":"2024-04-25T15:55:18.620238+0000","last_became_active":"2024-04-25T15:50:31.187308+0000","last_became_peered":"2024-04-25T15:50:31.187308+0000","last_unstale":"2024-04-25T15:55:18.620238+0000","last_undegraded":"2024-04-25T15:55:18.620238+0000","last_fullsized":"2024-04-25T15:55:18.620238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'143","last_scrub_stamp":"2024-04-25T15:55:18.619976+0000","last_deep_scrub":"129'143","last_deep_scrub_stamp":"2024-04-25T15:55:18.619976+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.619976+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:08.988831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"129'7","reported_seq":248,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:21.658635+0000","last_change":"2024-04-25T15:55:21.658721+0000","last_active":"2024-04-25T15:55:21.658635+0000","last_peered":"2024-04-25T15:55:21.658635+0000","last_clean":"2024-04-25T15:55:21.658635+0000","last_became_active":"2024-04-25T15:50:30.207471+0000","last_became_peered":"2024-04-25T15:50:30.207471+0000","last_unstale":"2024-04-25T15:55:21.658635+0000","last_undegraded":"2024-04-25T15:55:21.658635+0000","last_fullsized":"2024-04-25T15:55:21.658635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7","last_scrub_stamp":"2024-04-25T15:55:21.658428+0000","last_deep_scrub":"129'7","last_deep_scrub_stamp":"2024-04-25T15:55:21.658428+0000","last_clean_scrub_stamp":"2024-04-25T15:55:21.658428+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:24:19.550984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"129'144","reported_seq":353,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:23.858797+0000","last_change":"2024-04-25T15:55:23.859018+0000","last_active":"2024-04-25T15:55:23.858797+0000","last_peered":"2024-04-25T15:55:23.858797+0000","last_clean":"2024-04-25T15:55:23.858797+0000","last_became_active":"2024-04-25T15:54:44.215121+0000","last_became_peered":"2024-04-25T15:54:44.215121+0000","last_unstale":"2024-04-25T15:55:23.858797+0000","last_undegraded":"2024-04-25T15:55:23.858797+0000","last_fullsized":"2024-04-25T15:55:23.858797+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'144","last_scrub_stamp":"2024-04-25T15:55:23.858466+0000","last_deep_scrub":"129'144","last_deep_scrub_stamp":"2024-04-25T15:55:23.858466+0000","last_clean_scrub_stamp":"2024-04-25T15:55:23.858466+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:16.600046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"129'30","reported_seq":74,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:24.864887+0000","last_change":"2024-04-25T15:55:24.864978+0000","last_active":"2024-04-25T15:55:24.864887+0000","last_peered":"2024-04-25T15:55:24.864887+0000","last_clean":"2024-04-25T15:55:24.864887+0000","last_became_active":"2024-04-25T15:54:51.307056+0000","last_became_peered":"2024-04-25T15:54:51.307056+0000","last_unstale":"2024-04-25T15:55:24.864887+0000","last_undegraded":"2024-04-25T15:55:24.864887+0000","last_fullsized":"2024-04-25T15:55:24.864887+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'30","last_scrub_stamp":"2024-04-25T15:55:24.864666+0000","last_deep_scrub":"129'30","last_deep_scrub_stamp":"2024-04-25T15:55:24.864666+0000","last_clean_scrub_stamp":"2024-04-25T15:55:24.864666+0000","objects_scrubbed":4,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:17.435016+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":13,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"129'143","reported_seq":384,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:17.223557+0000","last_change":"2024-04-25T15:55:17.223710+0000","last_active":"2024-04-25T15:55:17.223557+0000","last_peered":"2024-04-25T15:55:17.223557+0000","last_clean":"2024-04-25T15:55:17.223557+0000","last_became_active":"2024-04-25T15:50:31.211677+0000","last_became_peered":"2024-04-25T15:50:31.211677+0000","last_unstale":"2024-04-25T15:55:17.223557+0000","last_undegraded":"2024-04-25T15:55:17.223557+0000","last_fullsized":"2024-04-25T15:55:17.223557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'143","last_scrub_stamp":"2024-04-25T15:55:17.223181+0000","last_deep_scrub":"129'143","last_deep_scrub_stamp":"2024-04-25T15:55:17.223181+0000","last_clean_scrub_stamp":"2024-04-25T15:55:17.223181+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:54:25.063285+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":138,"num_write_kb":376832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"129'12","reported_seq":255,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.102130+0000","last_change":"2024-04-25T15:55:16.102214+0000","last_active":"2024-04-25T15:55:16.102130+0000","last_peered":"2024-04-25T15:55:16.102130+0000","last_clean":"2024-04-25T15:55:16.102130+0000","last_became_active":"2024-04-25T15:50:30.196528+0000","last_became_peered":"2024-04-25T15:50:30.196528+0000","last_unstale":"2024-04-25T15:55:16.102130+0000","last_undegraded":"2024-04-25T15:55:16.102130+0000","last_fullsized":"2024-04-25T15:55:16.102130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'12","last_scrub_stamp":"2024-04-25T15:55:16.101900+0000","last_deep_scrub":"129'12","last_deep_scrub_stamp":"2024-04-25T15:55:16.101900+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.101900+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:00.176202+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"129'163","reported_seq":400,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:22.914351+0000","last_change":"2024-04-25T15:55:22.914419+0000","last_active":"2024-04-25T15:55:22.914351+0000","last_peered":"2024-04-25T15:55:22.914351+0000","last_clean":"2024-04-25T15:55:22.914351+0000","last_became_active":"2024-04-25T15:54:44.207033+0000","last_became_peered":"2024-04-25T15:54:44.207033+0000","last_unstale":"2024-04-25T15:55:22.914351+0000","last_undegraded":"2024-04-25T15:55:22.914351+0000","last_fullsized":"2024-04-25T15:55:22.914351+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'163","last_scrub_stamp":"2024-04-25T15:55:22.914123+0000","last_deep_scrub":"129'163","last_deep_scrub_stamp":"2024-04-25T15:55:22.914123+0000","last_clean_scrub_stamp":"2024-04-25T15:55:22.914123+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:47:13.051937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"129'3","reported_seq":244,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.532775+0000","last_change":"2024-04-25T15:55:18.532858+0000","last_active":"2024-04-25T15:55:18.532775+0000","last_peered":"2024-04-25T15:55:18.532775+0000","last_clean":"2024-04-25T15:55:18.532775+0000","last_became_active":"2024-04-25T15:50:30.196681+0000","last_became_peered":"2024-04-25T15:50:30.196681+0000","last_unstale":"2024-04-25T15:55:18.532775+0000","last_undegraded":"2024-04-25T15:55:18.532775+0000","last_fullsized":"2024-04-25T15:55:18.532775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'3","last_scrub_stamp":"2024-04-25T15:55:18.532535+0000","last_deep_scrub":"129'3","last_deep_scrub_stamp":"2024-04-25T15:55:18.532535+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.532535+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:10.750241+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"129'134","reported_seq":260,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:16.640066+0000","last_change":"2024-04-25T15:55:16.640066+0000","last_active":"2024-04-25T15:55:16.640066+0000","last_peered":"2024-04-25T15:55:16.640066+0000","last_clean":"2024-04-25T15:55:16.640066+0000","last_became_active":"2024-04-25T15:53:16.043160+0000","last_became_peered":"2024-04-25T15:53:16.043160+0000","last_unstale":"2024-04-25T15:55:16.640066+0000","last_undegraded":"2024-04-25T15:55:16.640066+0000","last_fullsized":"2024-04-25T15:55:16.640066+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'134","last_scrub_stamp":"2024-04-25T15:55:16.639557+0000","last_deep_scrub":"129'134","last_deep_scrub_stamp":"2024-04-25T15:55:16.639557+0000","last_clean_scrub_stamp":"2024-04-25T15:55:16.639557+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:15:11.657204+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"129'11","reported_seq":219,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:12.816406+0000","last_change":"2024-04-25T15:55:12.816469+0000","last_active":"2024-04-25T15:55:12.816406+0000","last_peered":"2024-04-25T15:55:12.816406+0000","last_clean":"2024-04-25T15:55:12.816406+0000","last_became_active":"2024-04-25T15:51:47.263122+0000","last_became_peered":"2024-04-25T15:51:47.263122+0000","last_unstale":"2024-04-25T15:55:12.816406+0000","last_undegraded":"2024-04-25T15:55:12.816406+0000","last_fullsized":"2024-04-25T15:55:12.816406+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'11","last_scrub_stamp":"2024-04-25T15:55:12.816300+0000","last_deep_scrub":"129'11","last_deep_scrub_stamp":"2024-04-25T15:55:12.816300+0000","last_clean_scrub_stamp":"2024-04-25T15:55:12.816300+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:40.666540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"129'153","reported_seq":530,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:20.209500+0000","last_change":"2024-04-25T15:55:20.209570+0000","last_active":"2024-04-25T15:55:20.209500+0000","last_peered":"2024-04-25T15:55:20.209500+0000","last_clean":"2024-04-25T15:55:20.209500+0000","last_became_active":"2024-04-25T15:54:44.206595+0000","last_became_peered":"2024-04-25T15:54:44.206595+0000","last_unstale":"2024-04-25T15:55:20.209500+0000","last_undegraded":"2024-04-25T15:55:20.209500+0000","last_fullsized":"2024-04-25T15:55:20.209500+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'153","last_scrub_stamp":"2024-04-25T15:55:20.209268+0000","last_deep_scrub":"129'153","last_deep_scrub_stamp":"2024-04-25T15:55:20.209268+0000","last_clean_scrub_stamp":"2024-04-25T15:55:20.209268+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:38:40.678510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":393216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"129'5","reported_seq":80,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:19.941034+0000","last_change":"2024-04-25T15:55:19.941110+0000","last_active":"2024-04-25T15:55:19.941034+0000","last_peered":"2024-04-25T15:55:19.941034+0000","last_clean":"2024-04-25T15:55:19.941034+0000","last_became_active":"2024-04-25T15:54:22.037756+0000","last_became_peered":"2024-04-25T15:54:22.037756+0000","last_unstale":"2024-04-25T15:55:19.941034+0000","last_undegraded":"2024-04-25T15:55:19.941034+0000","last_fullsized":"2024-04-25T15:55:19.941034+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5","last_scrub_stamp":"2024-04-25T15:55:19.940817+0000","last_deep_scrub":"129'5","last_deep_scrub_stamp":"2024-04-25T15:55:19.940817+0000","last_clean_scrub_stamp":"2024-04-25T15:55:19.940817+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:00:21.897191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"129'143","reported_seq":376,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:15.962872+0000","last_change":"2024-04-25T15:55:15.962951+0000","last_active":"2024-04-25T15:55:15.962872+0000","last_peered":"2024-04-25T15:55:15.962872+0000","last_clean":"2024-04-25T15:55:15.962872+0000","last_became_active":"2024-04-25T15:51:11.140916+0000","last_became_peered":"2024-04-25T15:51:11.140916+0000","last_unstale":"2024-04-25T15:55:15.962872+0000","last_undegraded":"2024-04-25T15:55:15.962872+0000","last_fullsized":"2024-04-25T15:55:15.962872+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'143","last_scrub_stamp":"2024-04-25T15:55:15.962585+0000","last_deep_scrub":"129'143","last_deep_scrub_stamp":"2024-04-25T15:55:15.962585+0000","last_clean_scrub_stamp":"2024-04-25T15:55:15.962585+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:55.227035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":135,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"129'7","reported_seq":270,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:26.600618+0000","last_change":"2024-04-25T15:55:26.600801+0000","last_active":"2024-04-25T15:55:26.600618+0000","last_peered":"2024-04-25T15:55:26.600618+0000","last_clean":"2024-04-25T15:55:26.600618+0000","last_became_active":"2024-04-25T15:53:16.060500+0000","last_became_peered":"2024-04-25T15:53:16.060500+0000","last_unstale":"2024-04-25T15:55:26.600618+0000","last_undegraded":"2024-04-25T15:55:26.600618+0000","last_fullsized":"2024-04-25T15:55:26.600618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7","last_scrub_stamp":"2024-04-25T15:55:26.600365+0000","last_deep_scrub":"129'7","last_deep_scrub_stamp":"2024-04-25T15:55:26.600365+0000","last_clean_scrub_stamp":"2024-04-25T15:55:26.600365+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:42.108223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"129'128","reported_seq":395,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:20.203382+0000","last_change":"2024-04-25T15:55:20.203457+0000","last_active":"2024-04-25T15:55:20.203382+0000","last_peered":"2024-04-25T15:55:20.203382+0000","last_clean":"2024-04-25T15:55:20.203382+0000","last_became_active":"2024-04-25T15:53:16.045355+0000","last_became_peered":"2024-04-25T15:53:16.045355+0000","last_unstale":"2024-04-25T15:55:20.203382+0000","last_undegraded":"2024-04-25T15:55:20.203382+0000","last_fullsized":"2024-04-25T15:55:20.203382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'128","last_scrub_stamp":"2024-04-25T15:55:20.203141+0000","last_deep_scrub":"129'128","last_deep_scrub_stamp":"2024-04-25T15:55:20.203141+0000","last_clean_scrub_stamp":"2024-04-25T15:55:20.203141+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:19:55.893649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"129'183","reported_seq":450,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:28.638029+0000","last_change":"2024-04-25T15:55:28.638029+0000","last_active":"2024-04-25T15:55:28.638029+0000","last_peered":"2024-04-25T15:55:28.638029+0000","last_clean":"2024-04-25T15:55:28.638029+0000","last_became_active":"2024-04-25T15:53:16.062448+0000","last_became_peered":"2024-04-25T15:53:16.062448+0000","last_unstale":"2024-04-25T15:55:28.638029+0000","last_undegraded":"2024-04-25T15:55:28.638029+0000","last_fullsized":"2024-04-25T15:55:28.638029+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'183","last_scrub_stamp":"2024-04-25T15:55:28.637572+0000","last_deep_scrub":"129'183","last_deep_scrub_stamp":"2024-04-25T15:55:28.637572+0000","last_clean_scrub_stamp":"2024-04-25T15:55:28.637572+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:17:24.465945+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":168,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"129'8","reported_seq":249,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:19.573367+0000","last_change":"2024-04-25T15:55:19.573367+0000","last_active":"2024-04-25T15:55:19.573367+0000","last_peered":"2024-04-25T15:55:19.573367+0000","last_clean":"2024-04-25T15:55:19.573367+0000","last_became_active":"2024-04-25T15:50:30.205552+0000","last_became_peered":"2024-04-25T15:50:30.205552+0000","last_unstale":"2024-04-25T15:55:19.573367+0000","last_undegraded":"2024-04-25T15:55:19.573367+0000","last_fullsized":"2024-04-25T15:55:19.573367+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8","last_scrub_stamp":"2024-04-25T15:55:19.573036+0000","last_deep_scrub":"129'8","last_deep_scrub_stamp":"2024-04-25T15:55:19.573036+0000","last_clean_scrub_stamp":"2024-04-25T15:55:19.573036+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:20.779884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"129'141","reported_seq":425,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:27.616004+0000","last_change":"2024-04-25T15:55:27.616004+0000","last_active":"2024-04-25T15:55:27.616004+0000","last_peered":"2024-04-25T15:55:27.616004+0000","last_clean":"2024-04-25T15:55:27.616004+0000","last_became_active":"2024-04-25T15:53:16.060579+0000","last_became_peered":"2024-04-25T15:53:16.060579+0000","last_unstale":"2024-04-25T15:55:27.616004+0000","last_undegraded":"2024-04-25T15:55:27.616004+0000","last_fullsized":"2024-04-25T15:55:27.616004+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'141","last_scrub_stamp":"2024-04-25T15:55:27.615702+0000","last_deep_scrub":"129'141","last_deep_scrub_stamp":"2024-04-25T15:55:27.615702+0000","last_clean_scrub_stamp":"2024-04-25T15:55:27.615702+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:11:49.598635+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":50331648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"129'20","reported_seq":261,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:20.623281+0000","last_change":"2024-04-25T15:55:20.623281+0000","last_active":"2024-04-25T15:55:20.623281+0000","last_peered":"2024-04-25T15:55:20.623281+0000","last_clean":"2024-04-25T15:55:20.623281+0000","last_became_active":"2024-04-25T15:50:30.207115+0000","last_became_peered":"2024-04-25T15:50:30.207115+0000","last_unstale":"2024-04-25T15:55:20.623281+0000","last_undegraded":"2024-04-25T15:55:20.623281+0000","last_fullsized":"2024-04-25T15:55:20.623281+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'20","last_scrub_stamp":"2024-04-25T15:55:20.622945+0000","last_deep_scrub":"129'20","last_deep_scrub_stamp":"2024-04-25T15:55:20.622945+0000","last_clean_scrub_stamp":"2024-04-25T15:55:20.622945+0000","objects_scrubbed":7,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:43:26.715966+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"129'135","reported_seq":337,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:18.926262+0000","last_change":"2024-04-25T15:55:18.926403+0000","last_active":"2024-04-25T15:55:18.926262+0000","last_peered":"2024-04-25T15:55:18.926262+0000","last_clean":"2024-04-25T15:55:18.926262+0000","last_became_active":"2024-04-25T15:54:40.168164+0000","last_became_peered":"2024-04-25T15:54:40.168164+0000","last_unstale":"2024-04-25T15:55:18.926262+0000","last_undegraded":"2024-04-25T15:55:18.926262+0000","last_fullsized":"2024-04-25T15:55:18.926262+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'135","last_scrub_stamp":"2024-04-25T15:55:18.925913+0000","last_deep_scrub":"129'135","last_deep_scrub_stamp":"2024-04-25T15:55:18.925913+0000","last_clean_scrub_stamp":"2024-04-25T15:55:18.925913+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:52.152297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":352256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":245,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:13.161188+0000","last_change":"2024-04-25T15:55:13.161227+0000","last_active":"2024-04-25T15:55:13.161188+0000","last_peered":"2024-04-25T15:55:13.161188+0000","last_clean":"2024-04-25T15:55:13.161188+0000","last_became_active":"2024-04-25T15:50:30.180850+0000","last_became_peered":"2024-04-25T15:50:30.180850+0000","last_unstale":"2024-04-25T15:55:13.161188+0000","last_undegraded":"2024-04-25T15:55:13.161188+0000","last_fullsized":"2024-04-25T15:55:13.161188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'2","last_scrub_stamp":"2024-04-25T15:55:13.161045+0000","last_deep_scrub":"20'2","last_deep_scrub_stamp":"2024-04-25T15:55:13.161045+0000","last_clean_scrub_stamp":"2024-04-25T15:55:13.161045+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:10.729160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"129'152","reported_seq":607,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:23.227279+0000","last_change":"2024-04-25T15:55:23.227358+0000","last_active":"2024-04-25T15:55:23.227279+0000","last_peered":"2024-04-25T15:55:23.227279+0000","last_clean":"2024-04-25T15:55:23.227279+0000","last_became_active":"2024-04-25T15:54:54.360118+0000","last_became_peered":"2024-04-25T15:54:54.360118+0000","last_unstale":"2024-04-25T15:55:23.227279+0000","last_undegraded":"2024-04-25T15:55:23.227279+0000","last_fullsized":"2024-04-25T15:55:23.227279+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'152","last_scrub_stamp":"2024-04-25T15:55:23.227044+0000","last_deep_scrub":"129'152","last_deep_scrub_stamp":"2024-04-25T15:55:23.227044+0000","last_clean_scrub_stamp":"2024-04-25T15:55:23.227044+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:56:04.867396+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"129'19","reported_seq":118,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-04-25T15:55:20.964284+0000","last_change":"2024-04-25T15:55:20.964393+0000","last_active":"2024-04-25T15:55:20.964284+0000","last_peered":"2024-04-25T15:55:20.964284+0000","last_clean":"2024-04-25T15:55:20.964284+0000","last_became_active":"2024-04-25T15:54:40.169964+0000","last_became_peered":"2024-04-25T15:54:40.169964+0000","last_unstale":"2024-04-25T15:55:20.964284+0000","last_undegraded":"2024-04-25T15:55:20.964284+0000","last_fullsized":"2024-04-25T15:55:20.964284+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'19","last_scrub_stamp":"2024-04-25T15:55:20.964051+0000","last_deep_scrub":"129'19","last_deep_scrub_stamp":"2024-04-25T15:55:20.964051+0000","last_clean_scrub_stamp":"2024-04-25T15:55:20.964051+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:07.796305+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9216,"num_write_kb":25165824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":1627389952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9717,"ondisk_log_size":9717,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":207952,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":347,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":389959,"num_keys_recovered":63,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":415904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574917,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63828,"kb_used_data":2124,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308012,"statfs":{"total":96636764160,"available":96571404288,"internally_reserved":0,"allocated":2174976,"data_stored":1338205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41224,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330616,"statfs":{"total":96636764160,"available":96594550784,"internally_reserved":0,"allocated":1769472,"data_stored":945806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63260,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308580,"statfs":{"total":96636764160,"available":96571985920,"internally_reserved":0,"allocated":1593344,"data_stored":749669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1568768,"data_stored":747067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41232,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330608,"statfs":{"total":96636764160,"available":96594542592,"internally_reserved":0,"allocated":1777664,"data_stored":946345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41040,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330800,"statfs":{"total":96636764160,"available":96594739200,"internally_reserved":0,"allocated":1581056,"data_stored":747704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1572864,"data_stored":746687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":2166784,"data_stored":1336397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":201190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":200651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:55:36.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T15:55:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d27a4640 1 Processor -- start 2024-04-25T15:55:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d27a4640 1 -- start start 2024-04-25T15:55:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d27a4640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6cc154ac0 0x7fe6cc0cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d27a4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe6cc15e210 0x7fe6cc165190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d27a4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6cc0ca8a0 con 0x7fe6cc15de00 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d27a4640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fe6cc0cab80 con 0x7fe6cc154ac0 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d27a4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe6cc15a920 con 0x7fe6cc15e210 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6cbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6cc154ac0 0x7fe6cc0cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6cbfff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6cc154ac0 0x7fe6cc0cbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48304/0 (socket says 172.21.15.67:48304) 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6cbfff640 1 -- 172.21.15.67:0/2030014158 learned_addr learned my addr 172.21.15.67:0/2030014158 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d0d41640 1 --2- 172.21.15.67:0/2030014158 >> v2:172.21.15.176:3300/0 conn(0x7fe6cc15e210 0x7fe6cc165190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6cbfff640 1 -- 172.21.15.67:0/2030014158 >> v2:172.21.15.176:3300/0 conn(0x7fe6cc15e210 msgr2=0x7fe6cc165190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6cbfff640 1 --2- 172.21.15.67:0/2030014158 >> v2:172.21.15.176:3300/0 conn(0x7fe6cc15e210 0x7fe6cc165190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6cbfff640 1 -- 172.21.15.67:0/2030014158 >> v1:172.21.15.176:6789/0 conn(0x7fe6cc15de00 legacy=0x7fe6cc0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6cbfff640 1 -- 172.21.15.67:0/2030014158 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6cc15ac00 con 0x7fe6cc154ac0 2024-04-25T15:55:36.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.233+0000 7fe6d0d41640 1 --2- 172.21.15.67:0/2030014158 >> v2:172.21.15.176:3300/0 conn(0x7fe6cc15e210 0x7fe6cc165190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:36.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6cbfff640 1 --2- 172.21.15.67:0/2030014158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc154ac0 0x7fe6cc0cbe90 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0cda20 tx=0x7fe6bc01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbff291709cc0763 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6caffd640 1 -- 172.21.15.67:0/2030014158 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0a56d0 con 0x7fe6cc154ac0 2024-04-25T15:55:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6caffd640 1 -- 172.21.15.67:0/2030014158 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6bc0a5870 con 0x7fe6cc154ac0 2024-04-25T15:55:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6caffd640 1 -- 172.21.15.67:0/2030014158 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0a5c40 con 0x7fe6cc154ac0 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 -- 172.21.15.67:0/2030014158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc154ac0 msgr2=0x7fe6cc0cbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/2030014158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc154ac0 0x7fe6cc0cbe90 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0cda20 tx=0x7fe6bc01c920 comp rx=0 tx=0).stop 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 -- 172.21.15.67:0/2030014158 shutdown_connections 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/2030014158 >> v2:172.21.15.176:3300/0 conn(0x7fe6cc15e210 0x7fe6cc165190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/2030014158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc154ac0 0x7fe6cc0cbe90 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 -- 172.21.15.67:0/2030014158 >> 172.21.15.67:0/2030014158 conn(0x7fe6cc068c70 msgr2=0x7fe6cc071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 -- 172.21.15.67:0/2030014158 shutdown_connections 2024-04-25T15:55:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.237+0000 7fe6d27a4640 1 -- 172.21.15.67:0/2030014158 wait complete. 2024-04-25T15:55:36.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 Processor -- start 2024-04-25T15:55:36.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 -- start start 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6cc154ac0 0x7fe6cc15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc15de00 0x7fe6cc146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6cc15e210 0x7fe6cc0cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc0ccd10 con 0x7fe6cc15de00 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc15b6e0 con 0x7fe6cc15e210 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d27a4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe6cc0cce90 con 0x7fe6cc154ac0 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc15de00 0x7fe6cc146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc15de00 0x7fe6cc146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48316/0 (socket says 172.21.15.67:48316) 2024-04-25T15:55:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 -- 172.21.15.67:0/3226888469 learned_addr learned my addr 172.21.15.67:0/3226888469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cbfff640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6cc154ac0 0x7fe6cc15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d0d41640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6cc15e210 0x7fe6cc0cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 -- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6cc154ac0 msgr2=0x7fe6cc15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6cc154ac0 0x7fe6cc15b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 -- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6cc15e210 msgr2=0x7fe6cc0cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6cc15e210 0x7fe6cc0cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cb7fe640 1 -- 172.21.15.67:0/3226888469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6cc1f72a0 con 0x7fe6cc15de00 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6cbfff640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6cc154ac0 0x7fe6cc15b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.241+0000 7fe6d0d41640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6cc15e210 0x7fe6cc0cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6cb7fe640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc15de00 0x7fe6cc146520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000b3f0 tx=0x7fe6c000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0015020 con 0x7fe6cc15de00 2024-04-25T15:55:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6c0011830 con 0x7fe6cc15de00 2024-04-25T15:55:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0011c00 con 0x7fe6cc15de00 2024-04-25T15:55:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6cc14ab50 con 0x7fe6cc15de00 2024-04-25T15:55:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6cc14a530 con 0x7fe6cc15de00 2024-04-25T15:55:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6cc06a480 con 0x7fe6cc15de00 2024-04-25T15:55:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe6c0018020 con 0x7fe6cc15de00 2024-04-25T15:55:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6c8ff9640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6b806f400 0x7fe6b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fe6c008f310 con 0x7fe6cc15de00 2024-04-25T15:55:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6cbfff640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6b806f400 0x7fe6b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.245+0000 7fe6cbfff640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6b806f400 0x7fe6b8071910 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0ae000 tx=0x7fe6bc0a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:36.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.249+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe6c001f020 con 0x7fe6cc15de00 2024-04-25T15:55:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.413+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fe6cc151df0 con 0x7fe6cc15de00 2024-04-25T15:55:36.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.421+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7fe6c005c150 con 0x7fe6cc15de00 2024-04-25T15:55:36.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6c8ff9640 1 -- 172.21.15.67:0/3226888469 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe6c005b560 con 0x7fe6cc15de00 2024-04-25T15:55:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6b806f400 msgr2=0x7fe6b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6b806f400 0x7fe6b8071910 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0ae000 tx=0x7fe6bc0a7070 comp rx=0 tx=0).stop 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc15de00 msgr2=0x7fe6cc146520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc15de00 0x7fe6cc146520 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000b3f0 tx=0x7fe6c000b8b0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 shutdown_connections 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fe6b806f400 0x7fe6b8071910 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6cc15e210 0x7fe6cc0cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6cc15de00 0x7fe6cc146520 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 --2- 172.21.15.67:0/3226888469 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6cc154ac0 0x7fe6cc15b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 >> 172.21.15.67:0/3226888469 conn(0x7fe6cc068c70 msgr2=0x7fe6cc0c7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 shutdown_connections 2024-04-25T15:55:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.425+0000 7fe6d27a4640 1 -- 172.21.15.67:0/3226888469 wait complete. 2024-04-25T15:55:36.449 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T15:55:36.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:55:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75fd74f640 1 Processor -- start 2024-04-25T15:55:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75fd74f640 1 -- start start 2024-04-25T15:55:36.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75fd74f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f75f80b5d70 0x7f75f806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75fd74f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f815fe00 0x7f75f8160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75fd74f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f806d780 con 0x7f75f806ca60 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75fd74f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f75f815d510 con 0x7f75f815fe00 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75fd74f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f75f8160be0 con 0x7f75f80b5d70 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f815fe00 0x7f75f8160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75f815fe00 0x7f75f8160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48330/0 (socket says 172.21.15.67:48330) 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75f77fe640 1 -- 172.21.15.67:0/1319971665 learned_addr learned my addr 172.21.15.67:0/1319971665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75f67fc640 1 --2- 172.21.15.67:0/1319971665 >> v2:172.21.15.176:3300/0 conn(0x7f75f80b5d70 0x7f75f806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75f77fe640 1 -- 172.21.15.67:0/1319971665 >> v2:172.21.15.176:3300/0 conn(0x7f75f80b5d70 msgr2=0x7f75f806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.597+0000 7f75f77fe640 1 --2- 172.21.15.67:0/1319971665 >> v2:172.21.15.176:3300/0 conn(0x7f75f80b5d70 0x7f75f806ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75f77fe640 1 -- 172.21.15.67:0/1319971665 >> v1:172.21.15.67:6789/0 conn(0x7f75f806ca60 legacy=0x7f75f80b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75f77fe640 1 -- 172.21.15.67:0/1319971665 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f815cbe0 con 0x7f75f815fe00 2024-04-25T15:55:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75f77fe640 1 --2- 172.21.15.67:0/1319971665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f8160210 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009e50 tx=0x7f75f001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ed6898cb26ebae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75f5ffb640 1 -- 172.21.15.67:0/1319971665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00a57a0 con 0x7f75f815fe00 2024-04-25T15:55:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75f5ffb640 1 -- 172.21.15.67:0/1319971665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f75f00a5940 con 0x7f75f815fe00 2024-04-25T15:55:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75f5ffb640 1 -- 172.21.15.67:0/1319971665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00a5d30 con 0x7f75f815fe00 2024-04-25T15:55:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 -- 172.21.15.67:0/1319971665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 msgr2=0x7f75f8160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 --2- 172.21.15.67:0/1319971665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f8160210 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009e50 tx=0x7f75f001c9f0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 -- 172.21.15.67:0/1319971665 shutdown_connections 2024-04-25T15:55:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 --2- 172.21.15.67:0/1319971665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f8160210 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 --2- 172.21.15.67:0/1319971665 >> v2:172.21.15.176:3300/0 conn(0x7f75f80b5d70 0x7f75f806ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 -- 172.21.15.67:0/1319971665 >> 172.21.15.67:0/1319971665 conn(0x7f75f806e420 msgr2=0x7f75f806e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 -- 172.21.15.67:0/1319971665 shutdown_connections 2024-04-25T15:55:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.601+0000 7f75fd74f640 1 -- 172.21.15.67:0/1319971665 wait complete. 2024-04-25T15:55:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 Processor -- start 2024-04-25T15:55:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 -- start start 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f806ca60 0x7f75f814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f80b5d70 0x7f75f8153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f80c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80b7010 con 0x7f75f815fe00 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80b6e90 con 0x7f75f80b5d70 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f75f8160750 con 0x7f75f806ca60 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f80c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f80c9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48342/0 (socket says 172.21.15.67:48342) 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 -- 172.21.15.67:0/4195804012 learned_addr learned my addr 172.21.15.67:0/4195804012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f67fc640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f80b5d70 0x7f75f8153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f6ffd640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f806ca60 0x7f75f814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 -- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f806ca60 msgr2=0x7f75f814ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f806ca60 0x7f75f814ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 -- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f80b5d70 msgr2=0x7f75f8153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f80b5d70 0x7f75f8153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 -- 172.21.15.67:0/4195804012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f814f270 con 0x7f75f815fe00 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f6ffd640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f806ca60 0x7f75f814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75f77fe640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f80c9ca0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f75f00af040 tx=0x7f75f0019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75ef7fe640 1 -- 172.21.15.67:0/4195804012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00b6480 con 0x7f75f815fe00 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75ef7fe640 1 -- 172.21.15.67:0/4195804012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f75f00b6640 con 0x7f75f815fe00 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f80c8e30 con 0x7f75f815fe00 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75ef7fe640 1 -- 172.21.15.67:0/4195804012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00b6a50 con 0x7f75f815fe00 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.605+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f814e530 con 0x7f75f815fe00 2024-04-25T15:55:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.609+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f806a480 con 0x7f75f815fe00 2024-04-25T15:55:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.609+0000 7f75ef7fe640 1 -- 172.21.15.67:0/4195804012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f75f00be020 con 0x7f75f815fe00 2024-04-25T15:55:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.609+0000 7f75ef7fe640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75d806f400 0x7f75d8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.609+0000 7f75ef7fe640 1 -- 172.21.15.67:0/4195804012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f75f012a5e0 con 0x7f75f815fe00 2024-04-25T15:55:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.609+0000 7f75f6ffd640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75d806f400 0x7f75d8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.609+0000 7f75f6ffd640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75d806f400 0x7f75d8071910 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f75e8000f70 tx=0x7f75e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:36.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.613+0000 7f75ef7fe640 1 -- 172.21.15.67:0/4195804012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f75f00165e0 con 0x7f75f815fe00 2024-04-25T15:55:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.813+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75f806dd20 con 0x7f75f815fe00 2024-04-25T15:55:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75ef7fe640 1 -- 172.21.15.67:0/4195804012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6911 (secure 0 0 0) 0x7f75f00f7500 con 0x7f75f815fe00 2024-04-25T15:55:36.816 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/920422475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":920422475},{"type":"v1","addr":"172.21.15.67:6835","nonce":920422475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:50:30.570677+0000","modified":"2024-04-25T15:51:04.945545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4294,"mds_1":4288,"mds_2":4276,"mds_3":4272,"mds_4":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"a","rank":4,"incarnation":16,"state":"up:active","state_seq":12,"addr":"172.21.15.67:6834/144842397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":144842397},{"type":"v1","addr":"172.21.15.67:6834","nonce":144842397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4272":{"gid":4272,"name":"d","rank":3,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.176:6838/2023672746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2023672746},{"type":"v1","addr":"172.21.15.176:6838","nonce":2023672746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4276":{"gid":4276,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":9,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6835/2385091301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2385091301},{"type":"v1","addr":"172.21.15.176:6835","nonce":2385091301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.176:6839/1886671623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6837","nonce":1886671623},{"type":"v1","addr":"172.21.15.176:6839","nonce":1886671623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4270,4272,4276,4288,4294]},"id":1}]} 2024-04-25T15:55:36.817 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75d806f400 msgr2=0x7f75d8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75d806f400 0x7f75d8071910 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f75e8000f70 tx=0x7f75e8009040 comp rx=0 tx=0).stop 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 msgr2=0x7f75f80c9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f80c9ca0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f75f00af040 tx=0x7f75f0019790 comp rx=0 tx=0).stop 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 shutdown_connections 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f75d806f400 0x7f75d8071910 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75f815fe00 0x7f75f80c9ca0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f80b5d70 0x7f75f8153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 --2- 172.21.15.67:0/4195804012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f806ca60 0x7f75f814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 >> 172.21.15.67:0/4195804012 conn(0x7f75f806e420 msgr2=0x7f75f815ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 shutdown_connections 2024-04-25T15:55:36.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.817+0000 7f75fd74f640 1 -- 172.21.15.67:0/4195804012 wait complete. 2024-04-25T15:55:36.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:55:36.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48c4cc5640 1 Processor -- start 2024-04-25T15:55:36.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48c4cc5640 1 -- start start 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48c4cc5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48c0159520 0x7f48c00c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48c4cc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c00c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48c4cc5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48c00c7340 con 0x7f48c00c6140 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48c4cc5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f48c00c67c0 con 0x7f48c0159520 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48c4cc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f48c00c6aa0 con 0x7f48c0073bd0 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48be59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48c0159520 0x7f48c00c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48be59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48c0159520 0x7f48c00c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48356/0 (socket says 172.21.15.67:48356) 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48be59c640 1 -- 172.21.15.67:0/4159648270 learned_addr learned my addr 172.21.15.67:0/4159648270 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48bdd9b640 1 --2- 172.21.15.67:0/4159648270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c00c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48be59c640 1 -- 172.21.15.67:0/4159648270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 msgr2=0x7f48c00c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48be59c640 1 --2- 172.21.15.67:0/4159648270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c00c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48be59c640 1 -- 172.21.15.67:0/4159648270 >> v1:172.21.15.176:6789/0 conn(0x7f48c00c6140 legacy=0x7f48c00c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48be59c640 1 -- 172.21.15.67:0/4159648270 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c00c6d80 con 0x7f48c0159520 2024-04-25T15:55:36.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.993+0000 7f48bdd9b640 1 --2- 172.21.15.67:0/4159648270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c00c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:36.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48be59c640 1 --2- 172.21.15.67:0/4159648270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c00c4760 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009870 tx=0x7f48b001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819498d8ff2d8a30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:36.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48bd59a640 1 -- 172.21.15.67:0/4159648270 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b00b5070 con 0x7f48c0159520 2024-04-25T15:55:36.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48bd59a640 1 -- 172.21.15.67:0/4159648270 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f48b00a89d0 con 0x7f48c0159520 2024-04-25T15:55:36.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48bd59a640 1 -- 172.21.15.67:0/4159648270 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b00a8dc0 con 0x7f48c0159520 2024-04-25T15:55:36.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 -- 172.21.15.67:0/4159648270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 msgr2=0x7f48c00c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:36.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/4159648270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c00c4760 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009870 tx=0x7f48b001ed10 comp rx=0 tx=0).stop 2024-04-25T15:55:36.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 -- 172.21.15.67:0/4159648270 shutdown_connections 2024-04-25T15:55:36.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/4159648270 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c00c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/4159648270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c00c4760 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:36.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 -- 172.21.15.67:0/4159648270 >> 172.21.15.67:0/4159648270 conn(0x7f48c006fe70 msgr2=0x7f48c00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:36.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 -- 172.21.15.67:0/4159648270 shutdown_connections 2024-04-25T15:55:36.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:36.997+0000 7f48c4cc5640 1 -- 172.21.15.67:0/4159648270 wait complete. 2024-04-25T15:55:37.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 Processor -- start 2024-04-25T15:55:37.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 -- start start 2024-04-25T15:55:37.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c0148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48c00c6140 0x7f48c01f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c014af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f48c0073fe0 con 0x7f48c0159520 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f48c00736b0 con 0x7f48c00c6140 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f48c0074160 con 0x7f48c0073bd0 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c014af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c014af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48358/0 (socket says 172.21.15.67:48358) 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 -- 172.21.15.67:0/2682080980 learned_addr learned my addr 172.21.15.67:0/2682080980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48be59c640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c0148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bdd9b640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48c00c6140 0x7f48c01f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 -- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 msgr2=0x7f48c0148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c0148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 -- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48c00c6140 msgr2=0x7f48c01f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48c00c6140 0x7f48c01f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 -- 172.21.15.67:0/2682080980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c01f4630 con 0x7f48c0159520 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48be59c640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c0148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48bed9d640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c014af20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f48b800b9e0 tx=0x7f48b800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48aeffd640 1 -- 172.21.15.67:0/2682080980 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b8003b80 con 0x7f48c0159520 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48aeffd640 1 -- 172.21.15.67:0/2682080980 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f48b8003d20 con 0x7f48c0159520 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c0157ae0 con 0x7f48c0159520 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48aeffd640 1 -- 172.21.15.67:0/2682080980 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b800cb80 con 0x7f48c0159520 2024-04-25T15:55:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c0152280 con 0x7f48c0159520 2024-04-25T15:55:37.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.001+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c006a480 con 0x7f48c0159520 2024-04-25T15:55:37.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.005+0000 7f48aeffd640 1 -- 172.21.15.67:0/2682080980 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f48b8006d10 con 0x7f48c0159520 2024-04-25T15:55:37.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.005+0000 7f48aeffd640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f48a006f400 0x7f48a0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.005+0000 7f48aeffd640 1 -- 172.21.15.67:0/2682080980 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f48b808eff0 con 0x7f48c0159520 2024-04-25T15:55:37.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.005+0000 7f48be59c640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f48a006f400 0x7f48a0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.005+0000 7f48be59c640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f48a006f400 0x7f48a0071910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f48b00b2000 tx=0x7f48b00aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.009+0000 7f48aeffd640 1 -- 172.21.15.67:0/2682080980 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f48b805bd40 con 0x7f48c0159520 2024-04-25T15:55:37.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.205+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48c014baa0 con 0x7f48c0159520 2024-04-25T15:55:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48aeffd640 1 -- 172.21.15.67:0/2682080980 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6911 (secure 0 0 0) 0x7f48b805b640 con 0x7f48c0159520 2024-04-25T15:55:37.208 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/920422475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":920422475},{"type":"v1","addr":"172.21.15.67:6835","nonce":920422475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T15:50:30.570677+0000","modified":"2024-04-25T15:51:04.945545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4294,"mds_1":4288,"mds_2":4276,"mds_3":4272,"mds_4":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"a","rank":4,"incarnation":16,"state":"up:active","state_seq":12,"addr":"172.21.15.67:6834/144842397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":144842397},{"type":"v1","addr":"172.21.15.67:6834","nonce":144842397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4272":{"gid":4272,"name":"d","rank":3,"incarnation":14,"state":"up:active","state_seq":10,"addr":"172.21.15.176:6838/2023672746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2023672746},{"type":"v1","addr":"172.21.15.176:6838","nonce":2023672746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4276":{"gid":4276,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":9,"addr":"172.21.15.67:6837/2429063224","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":2429063224},{"type":"v1","addr":"172.21.15.67:6837","nonce":2429063224}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4288":{"gid":4288,"name":"f","rank":1,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6835/2385091301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":2385091301},{"type":"v1","addr":"172.21.15.176:6835","nonce":2385091301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.176:6839/1886671623","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6837","nonce":1886671623},{"type":"v1","addr":"172.21.15.176:6839","nonce":1886671623}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4270,4272,4276,4288,4294]},"id":1}]} 2024-04-25T15:55:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 17 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f48a006f400 msgr2=0x7f48a0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f48a006f400 0x7f48a0071910 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f48b00b2000 tx=0x7f48b00aa070 comp rx=0 tx=0).stop 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 msgr2=0x7f48c014af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c014af20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f48b800b9e0 tx=0x7f48b800bea0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 shutdown_connections 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f48a006f400 0x7f48a0071910 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48c0159520 0x7f48c014af20 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48c00c6140 0x7f48c01f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 --2- 172.21.15.67:0/2682080980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48c0073bd0 0x7f48c0148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 >> 172.21.15.67:0/2682080980 conn(0x7f48c006fe70 msgr2=0x7f48c00c4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 shutdown_connections 2024-04-25T15:55:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.209+0000 7f48c4cc5640 1 -- 172.21.15.67:0/2682080980 wait complete. 2024-04-25T15:55:37.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:55:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.365+0000 7f5140bfb640 1 Processor -- start 2024-04-25T15:55:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.365+0000 7f5140bfb640 1 -- start start 2024-04-25T15:55:37.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f5140bfb640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f513c159520 0x7f513c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f5140bfb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f5140bfb640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f513c0c7340 con 0x7f513c0c6140 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f5140bfb640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f513c0c67c0 con 0x7f513c159520 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f5140bfb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f513c0c6aa0 con 0x7f513c073bd0 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f513c159520 0x7f513c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513a59c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f513c159520 0x7f513c0c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48370/0 (socket says 172.21.15.67:48370) 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513a59c640 1 -- 172.21.15.67:0/1659253987 learned_addr learned my addr 172.21.15.67:0/1659253987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513ad9d640 1 --1- 172.21.15.67:0/1659253987 >> v1:172.21.15.67:6789/0 conn(0x7f513c0c6140 0x7f513c0c4ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41858/0 (socket says 172.21.15.67:41858) 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f5139d9b640 1 --2- 172.21.15.67:0/1659253987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513a59c640 1 -- 172.21.15.67:0/1659253987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 msgr2=0x7f513c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513a59c640 1 --2- 172.21.15.67:0/1659253987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c0c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513a59c640 1 -- 172.21.15.67:0/1659253987 >> v1:172.21.15.67:6789/0 conn(0x7f513c0c6140 legacy=0x7f513c0c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.369+0000 7f513a59c640 1 -- 172.21.15.67:0/1659253987 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c0c6d80 con 0x7f513c159520 2024-04-25T15:55:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f513a59c640 1 --2- 172.21.15.67:0/1659253987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c159520 0x7f513c0c4760 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f512c009870 tx=0x7f512c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f0a5b61a5fda18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f513959a640 1 -- 172.21.15.67:0/1659253987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c0b5070 con 0x7f513c159520 2024-04-25T15:55:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f513959a640 1 -- 172.21.15.67:0/1659253987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f512c0a8a50 con 0x7f513c159520 2024-04-25T15:55:37.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f513959a640 1 -- 172.21.15.67:0/1659253987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c0a8e40 con 0x7f513c159520 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 -- 172.21.15.67:0/1659253987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c159520 msgr2=0x7f513c0c4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 --2- 172.21.15.67:0/1659253987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c159520 0x7f513c0c4760 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f512c009870 tx=0x7f512c01ed50 comp rx=0 tx=0).stop 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 -- 172.21.15.67:0/1659253987 shutdown_connections 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 --2- 172.21.15.67:0/1659253987 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c0c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 --2- 172.21.15.67:0/1659253987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c159520 0x7f513c0c4760 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 -- 172.21.15.67:0/1659253987 >> 172.21.15.67:0/1659253987 conn(0x7f513c06fe70 msgr2=0x7f513c0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 -- 172.21.15.67:0/1659253987 shutdown_connections 2024-04-25T15:55:37.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 -- 172.21.15.67:0/1659253987 wait complete. 2024-04-25T15:55:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 Processor -- start 2024-04-25T15:55:37.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.373+0000 7f5140bfb640 1 -- start start 2024-04-25T15:55:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c14bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0c6140 0x7f513c14ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f513c159520 0x7f513c145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f513c073fe0 con 0x7f513c0c6140 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f513c0736b0 con 0x7f513c159520 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f513c074160 con 0x7f513c073bd0 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0c6140 0x7f513c14ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0c6140 0x7f513c14ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48372/0 (socket says 172.21.15.67:48372) 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 -- 172.21.15.67:0/346370256 learned_addr learned my addr 172.21.15.67:0/346370256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 -- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 msgr2=0x7f513c14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f513ad9d640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f513c159520 0x7f513c145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f513a59c640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c14bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c14bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 -- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f513c159520 msgr2=0x7f513c145250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f513c159520 0x7f513c145250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 -- 172.21.15.67:0/346370256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c147a60 con 0x7f513c0c6140 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5139d9b640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0c6140 0x7f513c14ce00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f513000d620 tx=0x7f513000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f512affd640 1 -- 172.21.15.67:0/346370256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513000b620 con 0x7f513c0c6140 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f512affd640 1 -- 172.21.15.67:0/346370256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5130019070 con 0x7f513c0c6140 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513c1599f0 con 0x7f513c0c6140 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f512affd640 1 -- 172.21.15.67:0/346370256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513000ba00 con 0x7f513c0c6140 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513c1f2290 con 0x7f513c0c6140 2024-04-25T15:55:37.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f513c06a480 con 0x7f513c0c6140 2024-04-25T15:55:37.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f512affd640 1 -- 172.21.15.67:0/346370256 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f513000bba0 con 0x7f513c0c6140 2024-04-25T15:55:37.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f512affd640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f511c06f430 0x7f511c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f512affd640 1 -- 172.21.15.67:0/346370256 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7f5130090260 con 0x7f513c0c6140 2024-04-25T15:55:37.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.377+0000 7f513a59c640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f511c06f430 0x7f511c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.381+0000 7f513a59c640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f511c06f430 0x7f511c071940 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f512c0b2000 tx=0x7f512c0aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.381+0000 7f512affd640 1 -- 172.21.15.67:0/346370256 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f513005d0f0 con 0x7f513c0c6140 2024-04-25T15:55:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.541+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f513c159c60 con 0x7f513c0c6140 2024-04-25T15:55:37.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f512affd640 1 -- 172.21.15.67:0/346370256 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+17051 (secure 0 0 0) 0x7f513005c9f0 con 0x7f513c0c6140 2024-04-25T15:55:37.546 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-25T15:55:37.547 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":129,"fsid":"fd5b4894-828e-4cb5-af55-b54a791f04c8","created":"2024-04-25T15:50:05.181255+0000","modified":"2024-04-25T15:54:54.321839+0000","last_up_change":"2024-04-25T15:50:17.054376+0000","last_in_change":"2024-04-25T15:53:14.990625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:50:17.846623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:50:28.408547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:50:29.569527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ae04396-30a5-4c1d-86e5-63bc4f26f4c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6801","nonce":851806635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6803","nonce":851806635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6807","nonce":851806635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":851806635},{"type":"v1","addr":"172.21.15.67:6805","nonce":851806635}]},"public_addr":"172.21.15.67:6801/851806635","cluster_addr":"172.21.15.67:6803/851806635","heartbeat_back_addr":"172.21.15.67:6807/851806635","heartbeat_front_addr":"172.21.15.67:6805/851806635","state":["exists","up"]},{"osd":1,"uuid":"efacae28-481c-42e9-9d26-5a019cb078b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6812","nonce":2360705508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6817","nonce":2360705508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6825","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6828","nonce":2360705508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2360705508},{"type":"v1","addr":"172.21.15.67:6822","nonce":2360705508}]},"public_addr":"172.21.15.67:6812/2360705508","cluster_addr":"172.21.15.67:6817/2360705508","heartbeat_back_addr":"172.21.15.67:6828/2360705508","heartbeat_front_addr":"172.21.15.67:6822/2360705508","state":["exists","up"]},{"osd":2,"uuid":"60e87127-223e-4fd7-aefb-f00326f9deca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6813","nonce":2903206440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6819","nonce":2903206440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6831","nonce":2903206440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2903206440},{"type":"v1","addr":"172.21.15.67:6827","nonce":2903206440}]},"public_addr":"172.21.15.67:6813/2903206440","cluster_addr":"172.21.15.67:6819/2903206440","heartbeat_back_addr":"172.21.15.67:6831/2903206440","heartbeat_front_addr":"172.21.15.67:6827/2903206440","state":["exists","up"]},{"osd":3,"uuid":"85ccf60b-0609-40a0-aae5-1e6a37eebdf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6811","nonce":1688455876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6818","nonce":1688455876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6829","nonce":1688455876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1688455876},{"type":"v1","addr":"172.21.15.67:6824","nonce":1688455876}]},"public_addr":"172.21.15.67:6811/1688455876","cluster_addr":"172.21.15.67:6818/1688455876","heartbeat_back_addr":"172.21.15.67:6829/1688455876","heartbeat_front_addr":"172.21.15.67:6824/1688455876","state":["exists","up"]},{"osd":4,"uuid":"ce4919c0-d2fb-4210-b281-e843e84c94c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6810","nonce":696101260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6814","nonce":696101260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6822","nonce":696101260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":696101260},{"type":"v1","addr":"172.21.15.176:6818","nonce":696101260}]},"public_addr":"172.21.15.176:6810/696101260","cluster_addr":"172.21.15.176:6814/696101260","heartbeat_back_addr":"172.21.15.176:6822/696101260","heartbeat_front_addr":"172.21.15.176:6818/696101260","state":["exists","up"]},{"osd":5,"uuid":"47dcf3c9-5c23-4584-922c-b774345f2175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6825","nonce":2964881609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6827","nonce":2964881609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6831","nonce":2964881609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":2964881609},{"type":"v1","addr":"172.21.15.176:6829","nonce":2964881609}]},"public_addr":"172.21.15.176:6825/2964881609","cluster_addr":"172.21.15.176:6827/2964881609","heartbeat_back_addr":"172.21.15.176:6831/2964881609","heartbeat_front_addr":"172.21.15.176:6829/2964881609","state":["exists","up"]},{"osd":6,"uuid":"312d2166-f5ad-4cf3-8339-16a49bf48b19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6811","nonce":211167612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6813","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6815","nonce":211167612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6821","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6823","nonce":211167612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6817","nonce":211167612},{"type":"v1","addr":"172.21.15.176:6819","nonce":211167612}]},"public_addr":"172.21.15.176:6811/211167612","cluster_addr":"172.21.15.176:6815/211167612","heartbeat_back_addr":"172.21.15.176:6823/211167612","heartbeat_front_addr":"172.21.15.176:6819/211167612","state":["exists","up"]},{"osd":7,"uuid":"922d3949-fd7b-4d1e-8b59-7311998bbc0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6801","nonce":604707640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6803","nonce":604707640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6807","nonce":604707640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":604707640},{"type":"v1","addr":"172.21.15.176:6805","nonce":604707640}]},"public_addr":"172.21.15.176:6801/604707640","cluster_addr":"172.21.15.176:6803/604707640","heartbeat_back_addr":"172.21.15.176:6807/604707640","heartbeat_front_addr":"172.21.15.176:6805/604707640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.418696+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.395641+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.361155+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.573487+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.449326+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.423627+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.434102+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:50:13.384476+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.a","osds":[2,6]},{"pgid":"3.18","osds":[1,4]},{"pgid":"3.27","osds":[7,6]}],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":5}]},{"pgid":"3.22","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"3.3e","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f511c06f430 msgr2=0x7f511c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f511c06f430 0x7f511c071940 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f512c0b2000 tx=0x7f512c0aa070 comp rx=0 tx=0).stop 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0c6140 msgr2=0x7f513c14ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0c6140 0x7f513c14ce00 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f513000d620 tx=0x7f513000dae0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 shutdown_connections 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f511c06f430 0x7f511c071940 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f513c159520 0x7f513c145250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f513c0c6140 0x7f513c14ce00 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 --2- 172.21.15.67:0/346370256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f513c073bd0 0x7f513c14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 >> 172.21.15.67:0/346370256 conn(0x7f513c06fe70 msgr2=0x7f513c14a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 shutdown_connections 2024-04-25T15:55:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.545+0000 7f5140bfb640 1 -- 172.21.15.67:0/346370256 wait complete. 2024-04-25T15:55:37.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T15:55:37.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcda728640 1 Processor -- start 2024-04-25T15:55:37.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcda728640 1 -- start start 2024-04-25T15:55:37.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcda728640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdcd4071b20 0x7fdcd415d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcda728640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd40c5bb0 0x7fdcd40c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcda728640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcd40c6890 con 0x7fdcd415d710 2024-04-25T15:55:37.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcda728640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7fdcd415cd40 con 0x7fdcd4071b20 2024-04-25T15:55:37.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcda728640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdcd40c6b70 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd3fff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdcd4071b20 0x7fdcd415d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd37fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdcd415d710 0x7fdcd40c7610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41864/0 (socket says 172.21.15.67:41864) 2024-04-25T15:55:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd37fe640 1 -- 172.21.15.67:0/982840359 learned_addr learned my addr 172.21.15.67:0/982840359 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd8cc5640 1 --2- 172.21.15.67:0/982840359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd40c5bb0 0x7fdcd40c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd3fff640 1 -- 172.21.15.67:0/982840359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd40c5bb0 msgr2=0x7fdcd40c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd3fff640 1 --2- 172.21.15.67:0/982840359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd40c5bb0 0x7fdcd40c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd3fff640 1 -- 172.21.15.67:0/982840359 >> v1:172.21.15.67:6789/0 conn(0x7fdcd415d710 legacy=0x7fdcd40c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.709+0000 7fdcd3fff640 1 -- 172.21.15.67:0/982840359 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd40c6e50 con 0x7fdcd4071b20 2024-04-25T15:55:37.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcd3fff640 1 --2- 172.21.15.67:0/982840359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd4071b20 0x7fdcd415d1d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40097c0 tx=0x7fdcc401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac11aae5c7919487 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcd2ffd640 1 -- 172.21.15.67:0/982840359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc40b1070 con 0x7fdcd4071b20 2024-04-25T15:55:37.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcd2ffd640 1 -- 172.21.15.67:0/982840359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fdcc40ac410 con 0x7fdcd4071b20 2024-04-25T15:55:37.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcd2ffd640 1 -- 172.21.15.67:0/982840359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc40ac800 con 0x7fdcd4071b20 2024-04-25T15:55:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 -- 172.21.15.67:0/982840359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd4071b20 msgr2=0x7fdcd415d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 --2- 172.21.15.67:0/982840359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd4071b20 0x7fdcd415d1d0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40097c0 tx=0x7fdcc401c4a0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 -- 172.21.15.67:0/982840359 shutdown_connections 2024-04-25T15:55:37.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 --2- 172.21.15.67:0/982840359 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd40c5bb0 0x7fdcd40c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 --2- 172.21.15.67:0/982840359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd4071b20 0x7fdcd415d1d0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 -- 172.21.15.67:0/982840359 >> 172.21.15.67:0/982840359 conn(0x7fdcd406fe70 msgr2=0x7fdcd40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 -- 172.21.15.67:0/982840359 shutdown_connections 2024-04-25T15:55:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.713+0000 7fdcda728640 1 -- 172.21.15.67:0/982840359 wait complete. 2024-04-25T15:55:37.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 Processor -- start 2024-04-25T15:55:37.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 -- start start 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd4071b20 0x7fdcd41625f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd40c5bb0 0x7fdcd4149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdcd415d710 0x7fdcd4140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4166580 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd40c7d20 con 0x7fdcd415d710 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdcd4166700 con 0x7fdcd4071b20 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd40c5bb0 0x7fdcd4149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd40c5bb0 0x7fdcd4149ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48390/0 (socket says 172.21.15.67:48390) 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 -- 172.21.15.67:0/1754104552 learned_addr learned my addr 172.21.15.67:0/1754104552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd8cc5640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdcd415d710 0x7fdcd4140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd3fff640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd4071b20 0x7fdcd41625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 -- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd4071b20 msgr2=0x7fdcd41625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd4071b20 0x7fdcd41625f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 -- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdcd415d710 msgr2=0x7fdcd4140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdcd415d710 0x7fdcd4140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 -- 172.21.15.67:0/1754104552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd4162b30 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd8cc5640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdcd415d710 0x7fdcd4140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd37fe640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd40c5bb0 0x7fdcd4149ce0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8001ec0 tx=0x7fdcc800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd0ff9640 1 -- 172.21.15.67:0/1754104552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8017070 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd0ff9640 1 -- 172.21.15.67:0/1754104552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fdcc8005cf0 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd41f6690 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcd0ff9640 1 -- 172.21.15.67:0/1754104552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80072f0 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd41f6c50 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.717+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd41f6e90 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.721+0000 7fdcd0ff9640 1 -- 172.21.15.67:0/1754104552 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdcc801a070 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.721+0000 7fdcd0ff9640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcc006f430 0x7fdcc0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.721+0000 7fdcd0ff9640 1 -- 172.21.15.67:0/1754104552 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 6031+0+0 (secure 0 0 0) 0x7fdcc808ecf0 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.721+0000 7fdcd3fff640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcc006f430 0x7fdcc0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.721+0000 7fdcd3fff640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcc006f430 0x7fdcc0071940 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40ae000 tx=0x7fdcc40a6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:37.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.725+0000 7fdcd0ff9640 1 -- 172.21.15.67:0/1754104552 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdcc8021020 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.913+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fdcd4148540 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.917+0000 7fdcd0ff9640 1 -- 172.21.15.67:0/1754104552 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7fdcc805ba80 con 0x7fdcd40c5bb0 2024-04-25T15:55:37.918 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T15:55:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcc006f430 msgr2=0x7fdcc0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcc006f430 0x7fdcc0071940 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40ae000 tx=0x7fdcc40a6040 comp rx=0 tx=0).stop 2024-04-25T15:55:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd40c5bb0 msgr2=0x7fdcd4149ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:37.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd40c5bb0 0x7fdcd4149ce0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8001ec0 tx=0x7fdcc800c890 comp rx=0 tx=0).stop 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 shutdown_connections 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7fdcc006f430 0x7fdcc0071940 secure :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40ae000 tx=0x7fdcc40a6040 comp rx=0 tx=0).stop 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdcd415d710 0x7fdcd4140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdcd40c5bb0 0x7fdcd4149ce0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 --2- 172.21.15.67:0/1754104552 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdcd4071b20 0x7fdcd41625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 >> 172.21.15.67:0/1754104552 conn(0x7fdcd406fe70 msgr2=0x7fdcd40c7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 shutdown_connections 2024-04-25T15:55:37.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:37.921+0000 7fdcda728640 1 -- 172.21.15.67:0/1754104552 wait complete. 2024-04-25T15:55:37.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T15:55:37.987 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-25T15:55:38.004 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-25T15:55:38.024 INFO:tasks.ceph.mds.f.smithi176.stdout:starting mds.f at 2024-04-25T15:55:38.028 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-04-25T15:55:38.041 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-04-25T15:55:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.085+0000 7f4d07b17640 1 Processor -- start 2024-04-25T15:55:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.085+0000 7f4d07b17640 1 -- start start 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d07b17640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d0006ca60 0x7f4d0006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d07b17640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4d0015fe00 0x7f4d00160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d07b17640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d0006d780 con 0x7f4d000b5660 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d07b17640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4d0015d510 con 0x7f4d0015fe00 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d07b17640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d00160be0 con 0x7f4d0006ca60 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d058b3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d0006ca60 0x7f4d0006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d050b2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f4d000b5660 0x7f4d000b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.67:49722/0 (socket says 172.21.15.67:49722) 2024-04-25T15:55:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d060b4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4d0015fe00 0x7f4d00160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d058b3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d0006ca60 0x7f4d0006ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.67:32808/0 (socket says 172.21.15.67:32808) 2024-04-25T15:55:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d050b2640 1 -- 172.21.15.67:0/253077545 learned_addr learned my addr 172.21.15.67:0/253077545 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d058b3640 1 -- 172.21.15.67:0/253077545 >> v2:172.21.15.176:3300/0 conn(0x7f4d0015fe00 msgr2=0x7f4d00160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d058b3640 1 --2- 172.21.15.67:0/253077545 >> v2:172.21.15.176:3300/0 conn(0x7f4d0015fe00 0x7f4d00160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d048b1640 1 -- 172.21.15.67:0/253077545 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1830698171 0 0) 0x7f4d0006d780 con 0x7f4d000b5660 2024-04-25T15:55:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d058b3640 1 -- 172.21.15.67:0/253077545 >> v1:172.21.15.176:6789/0 conn(0x7f4d000b5660 legacy=0x7f4d000b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d058b3640 1 -- 172.21.15.67:0/253077545 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0015cbe0 con 0x7f4d0006ca60 2024-04-25T15:55:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d058b3640 1 --2- 172.21.15.67:0/253077545 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d0006ca60 0x7f4d0006ce70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0097c0 tx=0x7f4cfc01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba73c1712175cbc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.089+0000 7f4d048b1640 1 -- 172.21.15.67:0/253077545 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc0b5070 con 0x7f4d0006ca60 2024-04-25T15:55:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d048b1640 1 -- 172.21.15.67:0/253077545 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4cfc0a8990 con 0x7f4d0006ca60 2024-04-25T15:55:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d048b1640 1 -- 172.21.15.67:0/253077545 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc0a8d80 con 0x7f4d0006ca60 2024-04-25T15:55:38.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- 172.21.15.67:0/253077545 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d0006ca60 msgr2=0x7f4d0006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 --2- 172.21.15.67:0/253077545 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d0006ca60 0x7f4d0006ce70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0097c0 tx=0x7f4cfc01ec90 comp rx=0 tx=0).stop 2024-04-25T15:55:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- 172.21.15.67:0/253077545 shutdown_connections 2024-04-25T15:55:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 --2- 172.21.15.67:0/253077545 >> v2:172.21.15.176:3300/0 conn(0x7f4d0015fe00 0x7f4d00160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 --2- 172.21.15.67:0/253077545 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d0006ca60 0x7f4d0006ce70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- 172.21.15.67:0/253077545 >> 172.21.15.67:0/253077545 conn(0x7f4d0006e420 msgr2=0x7f4d0006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- 172.21.15.67:0/253077545 shutdown_connections 2024-04-25T15:55:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- 172.21.15.67:0/253077545 wait complete. 2024-04-25T15:55:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 Processor -- start 2024-04-25T15:55:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- start start 2024-04-25T15:55:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0006ca60 0x7f4d001f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d000b5660 0x7f4d000ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d0015fe00 0x7f4d00155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0015c890 con 0x7f4d0006ca60 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0015d1a0 con 0x7f4d0015fe00 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d07b17640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d000b6e90 con 0x7f4d000b5660 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d058b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0006ca60 0x7f4d001f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d058b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0006ca60 0x7f4d001f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48440/0 (socket says 172.21.15.67:48440) 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d058b3640 1 -- 172.21.15.67:0/1424290742 learned_addr learned my addr 172.21.15.67:0/1424290742 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d050b2640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d000b5660 0x7f4d000ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.093+0000 7f4d060b4640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d0015fe00 0x7f4d00155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d058b3640 1 -- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d000b5660 msgr2=0x7f4d000ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d058b3640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d000b5660 0x7f4d000ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d058b3640 1 -- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d0015fe00 msgr2=0x7f4d00155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d058b3640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d0015fe00 0x7f4d00155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d058b3640 1 -- 172.21.15.67:0/1424290742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d001f3780 con 0x7f4d0006ca60 2024-04-25T15:55:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d058b3640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0006ca60 0x7f4d001f3240 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0b3040 tx=0x7f4cfc01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4cf67fc640 1 -- 172.21.15.67:0/1424290742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc0bf070 con 0x7f4d0006ca60 2024-04-25T15:55:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4cf67fc640 1 -- 172.21.15.67:0/1424290742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4cfc0b0820 con 0x7f4d0006ca60 2024-04-25T15:55:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d001f6730 con 0x7f4d0006ca60 2024-04-25T15:55:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4cf67fc640 1 -- 172.21.15.67:0/1424290742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc0b5040 con 0x7f4d0006ca60 2024-04-25T15:55:38.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d001f6cf0 con 0x7f4d0006ca60 2024-04-25T15:55:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d0006a480 con 0x7f4d0006ca60 2024-04-25T15:55:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4cf67fc640 1 -- 172.21.15.67:0/1424290742 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4cfc0c2070 con 0x7f4d0006ca60 2024-04-25T15:55:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4cf67fc640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4ce406f400 0x7f4ce4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4cf67fc640 1 -- 172.21.15.67:0/1424290742 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6461+0+0 (secure 0 0 0) 0x7f4cfc12bc80 con 0x7f4d0006ca60 2024-04-25T15:55:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d050b2640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4ce406f400 0x7f4ce4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:38.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.097+0000 7f4d050b2640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4ce406f400 0x7f4ce4071910 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000f70 tx=0x7f4cf0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:38.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.101+0000 7f4cf67fc640 1 -- 172.21.15.67:0/1424290742 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4cfc0187f0 con 0x7f4d0006ca60 2024-04-25T15:55:38.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.301+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4d000cd380 con 0x7f4d0006ca60 2024-04-25T15:55:38.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.921+0000 7f4cf67fc640 1 -- 172.21.15.67:0/1424290742 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f4cfc0f82d0 con 0x7f4d0006ca60 2024-04-25T15:55:38.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4ce406f400 msgr2=0x7f4ce4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:38.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4ce406f400 0x7f4ce4071910 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0000f70 tx=0x7f4cf0009040 comp rx=0 tx=0).stop 2024-04-25T15:55:38.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0006ca60 msgr2=0x7f4d001f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0006ca60 0x7f4d001f3240 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0b3040 tx=0x7f4cfc01ba30 comp rx=0 tx=0).stop 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 shutdown_connections 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f4ce406f400 0x7f4ce4071910 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d0015fe00 0x7f4d00155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d000b5660 0x7f4d000ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 --2- 172.21.15.67:0/1424290742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0006ca60 0x7f4d001f3240 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 >> 172.21.15.67:0/1424290742 conn(0x7f4d0006e420 msgr2=0x7f4d00152a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 shutdown_connections 2024-04-25T15:55:38.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:38.925+0000 7f4d07b17640 1 -- 172.21.15.67:0/1424290742 wait complete. 2024-04-25T15:55:38.955 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T15:55:39.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.156+0000 7f0e570e9640 1 Processor -- start 2024-04-25T15:55:39.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.156+0000 7f0e570e9640 1 -- start start 2024-04-25T15:55:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e50071da0 0x7f0e50152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0e500721b0 0x7f0e50152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e500b3b60 con 0x7f0e500c4630 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f0e500c38b0 con 0x7f0e50071da0 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0e5006cf20 con 0x7f0e500721b0 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e54e85640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e500c4630 0x7f0e5006c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41890/0 (socket says 172.21.15.67:41890) 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e54e85640 1 -- 172.21.15.67:0/2145131966 learned_addr learned my addr 172.21.15.67:0/2145131966 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4ffff640 1 --2- 172.21.15.67:0/2145131966 >> v2:172.21.15.67:3300/0 conn(0x7f0e50071da0 0x7f0e50152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e55686640 1 --2- 172.21.15.67:0/2145131966 >> v2:172.21.15.176:3300/0 conn(0x7f0e500721b0 0x7f0e50152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4f7fe640 1 -- 172.21.15.67:0/2145131966 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3816137077 0 0) 0x7f0e500b3b60 con 0x7f0e500c4630 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4f7fe640 1 -- 172.21.15.67:0/2145131966 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e3c003530 con 0x7f0e500c4630 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4ffff640 1 -- 172.21.15.67:0/2145131966 >> v2:172.21.15.176:3300/0 conn(0x7f0e500721b0 msgr2=0x7f0e50152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4ffff640 1 --2- 172.21.15.67:0/2145131966 >> v2:172.21.15.176:3300/0 conn(0x7f0e500721b0 0x7f0e50152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4ffff640 1 -- 172.21.15.67:0/2145131966 >> v1:172.21.15.67:6789/0 conn(0x7f0e500c4630 legacy=0x7f0e5006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:39.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4ffff640 1 -- 172.21.15.67:0/2145131966 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e500b3e40 con 0x7f0e50071da0 2024-04-25T15:55:39.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4ffff640 1 --2- 172.21.15.67:0/2145131966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e50152ab0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f0e50074df0 tx=0x7f0e4001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39735672a0949479 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:39.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4f7fe640 1 -- 172.21.15.67:0/2145131966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e400b5070 con 0x7f0e50071da0 2024-04-25T15:55:39.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4f7fe640 1 -- 172.21.15.67:0/2145131966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0e400a8930 con 0x7f0e50071da0 2024-04-25T15:55:39.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e4f7fe640 1 -- 172.21.15.67:0/2145131966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e400a8ce0 con 0x7f0e50071da0 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- 172.21.15.67:0/2145131966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 msgr2=0x7f0e50152ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 --2- 172.21.15.67:0/2145131966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e50152ab0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f0e50074df0 tx=0x7f0e4001ec30 comp rx=0 tx=0).stop 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- 172.21.15.67:0/2145131966 shutdown_connections 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 --2- 172.21.15.67:0/2145131966 >> v2:172.21.15.176:3300/0 conn(0x7f0e500721b0 0x7f0e50152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 --2- 172.21.15.67:0/2145131966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e50152ab0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- 172.21.15.67:0/2145131966 >> 172.21.15.67:0/2145131966 conn(0x7f0e5006fd30 msgr2=0x7f0e50070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- 172.21.15.67:0/2145131966 shutdown_connections 2024-04-25T15:55:39.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.160+0000 7f0e570e9640 1 -- 172.21.15.67:0/2145131966 wait complete. 2024-04-25T15:55:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 Processor -- start 2024-04-25T15:55:39.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 -- start start 2024-04-25T15:55:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e5015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e500721b0 0x7f0e501385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e500c4630 0x7f0e501f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e500b3820 con 0x7f0e50071da0 2024-04-25T15:55:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0e500c3de0 con 0x7f0e500721b0 2024-04-25T15:55:39.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e5015c160 con 0x7f0e500c4630 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e5015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e5015b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48458/0 (socket says 172.21.15.67:48458) 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 -- 172.21.15.67:0/1737495565 learned_addr learned my addr 172.21.15.67:0/1737495565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e4ffff640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e500721b0 0x7f0e501385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 -- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e500c4630 msgr2=0x7f0e501f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e55686640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e500c4630 0x7f0e501f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e500c4630 0x7f0e501f46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 -- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e500721b0 msgr2=0x7f0e501385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e500721b0 0x7f0e501385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 -- 172.21.15.67:0/1737495565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5015bb00 con 0x7f0e50071da0 2024-04-25T15:55:39.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e4ffff640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e500721b0 0x7f0e501385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:55:39.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e54e85640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e5015b5c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800c480 tx=0x7f0e3800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:39.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e4d7fa640 1 -- 172.21.15.67:0/1737495565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38018070 con 0x7f0e50071da0 2024-04-25T15:55:39.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e4d7fa640 1 -- 172.21.15.67:0/1737495565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0e380074e0 con 0x7f0e50071da0 2024-04-25T15:55:39.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e4d7fa640 1 -- 172.21.15.67:0/1737495565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38004a50 con 0x7f0e50071da0 2024-04-25T15:55:39.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e501391d0 con 0x7f0e50071da0 2024-04-25T15:55:39.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.164+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e500cd160 con 0x7f0e50071da0 2024-04-25T15:55:39.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.168+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e5006a480 con 0x7f0e50071da0 2024-04-25T15:55:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.168+0000 7f0e4d7fa640 1 -- 172.21.15.67:0/1737495565 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0e3801b070 con 0x7f0e50071da0 2024-04-25T15:55:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.168+0000 7f0e4d7fa640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e3006f400 0x7f0e30071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.168+0000 7f0e4d7fa640 1 -- 172.21.15.67:0/1737495565 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 6461+0+0 (secure 0 0 0) 0x7f0e3808eb80 con 0x7f0e50071da0 2024-04-25T15:55:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.168+0000 7f0e4ffff640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e3006f400 0x7f0e30071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:39.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.168+0000 7f0e4ffff640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e3006f400 0x7f0e30071910 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0e400b2000 tx=0x7f0e400aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.172+0000 7f0e4d7fa640 1 -- 172.21.15.67:0/1737495565 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0e38022080 con 0x7f0e50071da0 2024-04-25T15:55:39.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:39.340+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0e5006d970 con 0x7f0e50071da0 2024-04-25T15:55:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.136+0000 7f0e4d7fa640 1 -- 172.21.15.67:0/1737495565 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v131) ==== 181+0+0 (secure 0 0 0) 0x7f0e3805b980 con 0x7f0e50071da0 2024-04-25T15:55:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-25T15:55:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e3006f400 msgr2=0x7f0e30071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e3006f400 0x7f0e30071910 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0e400b2000 tx=0x7f0e400aa070 comp rx=0 tx=0).stop 2024-04-25T15:55:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 msgr2=0x7f0e5015b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:40.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e5015b5c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800c480 tx=0x7f0e3800c940 comp rx=0 tx=0).stop 2024-04-25T15:55:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 shutdown_connections 2024-04-25T15:55:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f0e3006f400 0x7f0e30071910 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e500c4630 0x7f0e501f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e500721b0 0x7f0e501385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 --2- 172.21.15.67:0/1737495565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e50071da0 0x7f0e5015b5c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.140+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 >> 172.21.15.67:0/1737495565 conn(0x7f0e5006fd30 msgr2=0x7f0e50152880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.144+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 shutdown_connections 2024-04-25T15:55:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.144+0000 7f0e570e9640 1 -- 172.21.15.67:0/1737495565 wait complete. 2024-04-25T15:55:40.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T15:55:40.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.324+0000 7f3b463a2640 1 Processor -- start 2024-04-25T15:55:40.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.324+0000 7f3b463a2640 1 -- start start 2024-04-25T15:55:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b463a2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b4015a530 0x7f3b4015a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:40.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b463a2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b400c3980 con 0x7f3b40157940 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b463a2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b400c2fb0 con 0x7f3b40162d60 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b463a2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 -- 0x7f3b400c3290 con 0x7f3b4015a530 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b4015a530 0x7f3b4015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3ffff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3b40157940 0x7f3b400ca9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41906/0 (socket says 172.21.15.67:41906) 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3ffff640 1 -- 172.21.15.67:0/3399369553 learned_addr learned my addr 172.21.15.67:0/3399369553 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3f7fe640 1 -- 172.21.15.67:0/3399369553 >> v1:172.21.15.176:6789/0 conn(0x7f3b40162d60 legacy=0x7f3b40165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3effd640 1 -- 172.21.15.67:0/3399369553 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413050351 0 0) 0x7f3b400c3980 con 0x7f3b40157940 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3f7fe640 1 -- 172.21.15.67:0/3399369553 >> v1:172.21.15.67:6789/0 conn(0x7f3b40157940 legacy=0x7f3b400ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:40.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3f7fe640 1 -- 172.21.15.67:0/3399369553 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b400c3c40 con 0x7f3b4015a530 2024-04-25T15:55:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3f7fe640 1 --2- 172.21.15.67:0/3399369553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b4015a530 0x7f3b4015a940 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3b340097c0 tx=0x7f3b3401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbaf0458cfd4a107 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3effd640 1 -- 172.21.15.67:0/3399369553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340b5070 con 0x7f3b4015a530 2024-04-25T15:55:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3effd640 1 -- 172.21.15.67:0/3399369553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3b340a8990 con 0x7f3b4015a530 2024-04-25T15:55:40.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.328+0000 7f3b3effd640 1 -- 172.21.15.67:0/3399369553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340a8d80 con 0x7f3b4015a530 2024-04-25T15:55:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- 172.21.15.67:0/3399369553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b4015a530 msgr2=0x7f3b4015a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3399369553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b4015a530 0x7f3b4015a940 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3b340097c0 tx=0x7f3b3401ec90 comp rx=0 tx=0).stop 2024-04-25T15:55:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- 172.21.15.67:0/3399369553 shutdown_connections 2024-04-25T15:55:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3399369553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b4015a530 0x7f3b4015a940 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- 172.21.15.67:0/3399369553 >> 172.21.15.67:0/3399369553 conn(0x7f3b4006fd30 msgr2=0x7f3b40070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- 172.21.15.67:0/3399369553 shutdown_connections 2024-04-25T15:55:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- 172.21.15.67:0/3399369553 wait complete. 2024-04-25T15:55:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 Processor -- start 2024-04-25T15:55:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- start start 2024-04-25T15:55:40.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b40157940 0x7f3b400c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b4015a530 0x7f3b40073830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b40162d60 0x7f3b401f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4006d9f0 con 0x7f3b40162d60 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4015ae80 con 0x7f3b4015a530 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b463a2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3b400c84b0 con 0x7f3b40157940 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b4493f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b40162d60 0x7f3b401f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b4493f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b40162d60 0x7f3b401f3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48476/0 (socket says 172.21.15.67:48476) 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b4493f640 1 -- 172.21.15.67:0/3814095392 learned_addr learned my addr 172.21.15.67:0/3814095392 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b3f7fe640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b4015a530 0x7f3b40073830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.332+0000 7f3b3ffff640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b40157940 0x7f3b400c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:40.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b4493f640 1 -- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b40157940 msgr2=0x7f3b400c5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b4493f640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b40157940 0x7f3b400c5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b4493f640 1 -- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b4015a530 msgr2=0x7f3b40073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b4493f640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b4015a530 0x7f3b40073830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b4493f640 1 -- 172.21.15.67:0/3814095392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b400c6100 con 0x7f3b40162d60 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3f7fe640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b4015a530 0x7f3b40073830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b4493f640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b40162d60 0x7f3b401f3600 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c730 tx=0x7f3b3800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3cff9640 1 -- 172.21.15.67:0/3814095392 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b38003b80 con 0x7f3b40162d60 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3cff9640 1 -- 172.21.15.67:0/3814095392 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3b38003d20 con 0x7f3b40162d60 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b400c6a90 con 0x7f3b40162d60 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3cff9640 1 -- 172.21.15.67:0/3814095392 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3800aa50 con 0x7f3b40162d60 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b400c7050 con 0x7f3b40162d60 2024-04-25T15:55:40.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b4006a480 con 0x7f3b40162d60 2024-04-25T15:55:40.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3cff9640 1 -- 172.21.15.67:0/3814095392 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3b3800abf0 con 0x7f3b40162d60 2024-04-25T15:55:40.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3cff9640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3b2c06f400 0x7f3b2c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3cff9640 1 -- 172.21.15.67:0/3814095392 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 5994+0+0 (secure 0 0 0) 0x7f3b3808fd60 con 0x7f3b40162d60 2024-04-25T15:55:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.336+0000 7f3b3ffff640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3b2c06f400 0x7f3b2c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:40.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.340+0000 7f3b3ffff640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3b2c06f400 0x7f3b2c071910 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3b30000f70 tx=0x7f3b30007d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.340+0000 7f3b3cff9640 1 -- 172.21.15.67:0/3814095392 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3b3805cc10 con 0x7f3b40162d60 2024-04-25T15:55:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:40.508+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3b400ce8e0 con 0x7f3b40162d60 2024-04-25T15:55:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.148+0000 7f3b3cff9640 1 -- 172.21.15.67:0/3814095392 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v132) ==== 169+0+0 (secure 0 0 0) 0x7f3b3805c510 con 0x7f3b40162d60 2024-04-25T15:55:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-25T15:55:41.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.152+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3b2c06f400 msgr2=0x7f3b2c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:41.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.152+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3b2c06f400 0x7f3b2c071910 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3b30000f70 tx=0x7f3b30007d10 comp rx=0 tx=0).stop 2024-04-25T15:55:41.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.152+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b40162d60 msgr2=0x7f3b401f3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:41.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.152+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b40162d60 0x7f3b401f3600 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800c730 tx=0x7f3b3800cbf0 comp rx=0 tx=0).stop 2024-04-25T15:55:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 shutdown_connections 2024-04-25T15:55:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:6832/2784768863,v1:172.21.15.176:6833/2784768863] conn(0x7f3b2c06f400 0x7f3b2c071910 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b40162d60 0x7f3b401f3600 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b4015a530 0x7f3b40073830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 --2- 172.21.15.67:0/3814095392 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b40157940 0x7f3b400c5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 >> 172.21.15.67:0/3814095392 conn(0x7f3b4006fd30 msgr2=0x7f3b400ca980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 shutdown_connections 2024-04-25T15:55:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-25T15:55:41.156+0000 7f3b463a2640 1 -- 172.21.15.67:0/3814095392 wait complete. 2024-04-25T15:55:41.188 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T15:55:41.188 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T15:55:41.188 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:41.206 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T15:55:41.206 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T15:55:41.207 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:41.262 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T15:55:41.262 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-25T15:55:41.263 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:41.344 INFO:tasks.ceph.mds.e:Stopped 2024-04-25T15:55:41.344 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T15:55:41.345 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:41.363 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T15:55:41.363 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T15:55:41.363 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:41.420 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T15:55:41.420 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-25T15:55:41.420 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:41.477 INFO:tasks.ceph.mds.f:Stopped 2024-04-25T15:55:41.558 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T15:55:41.559 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T15:55:41.559 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:41.631 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T15:55:41.632 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T15:55:41.632 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:45.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-25T15:55:47.734 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T15:55:47.734 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T15:55:47.734 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:47.816 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T15:55:47.816 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T15:55:47.816 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:51.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-25T15:55:51.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-25T15:55:53.918 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T15:55:53.918 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T15:55:53.918 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:53.984 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T15:55:53.984 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T15:55:53.984 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:54.065 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T15:55:54.066 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T15:55:54.066 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:54.156 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T15:55:54.156 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T15:55:54.156 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:55:56.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-25T15:55:56.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-25T15:55:59.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:55:59.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:56:00.257 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T15:56:00.257 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T15:56:00.257 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T15:56:00.257 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:56:00.285 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T15:56:00.285 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T15:56:00.285 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:56:00.316 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T15:56:00.316 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T15:56:00.316 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T15:56:00.317 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:56:00.348 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T15:56:00.348 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T15:56:00.348 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:56:00.375 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T15:56:00.376 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T15:56:00.376 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T15:56:00.434 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T15:56:00.435 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T15:56:00.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v MDS_SLOW_METADATA_IO | egrep -v MDS_TRIM | head -n 1 2024-04-25T15:56:00.460 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:56:00.460 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T15:56:00.669 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:56:00.669 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T15:56:01.043 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:56:01.044 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T15:56:01.202 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:56:01.203 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T15:56:01.396 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:56:01.396 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T15:56:01.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-25T15:56:01.782 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:56:01.782 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T15:56:02.003 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:56:02.003 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T15:56:02.177 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:56:02.177 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T15:56:02.354 INFO:tasks.ceph:Archiving mon data... 2024-04-25T15:56:02.355 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/data/mon.a.tgz 2024-04-25T15:56:02.355 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-25T15:56:02.362 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.ucGfjUAixs 2024-04-25T15:56:02.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ucGfjUAixs 2024-04-25T15:56:02.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.ucGfjUAixs 2024-04-25T15:56:02.577 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.ucGfjUAixs is 446KB 2024-04-25T15:56:02.637 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.ucGfjUAixs 2024-04-25T15:56:02.643 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/data/mon.b.tgz 2024-04-25T15:56:02.643 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-04-25T15:56:02.649 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.jSUTTXhC6R 2024-04-25T15:56:02.649 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jSUTTXhC6R 2024-04-25T15:56:02.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.jSUTTXhC6R 2024-04-25T15:56:02.903 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.jSUTTXhC6R is 632KB 2024-04-25T15:56:02.966 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.jSUTTXhC6R 2024-04-25T15:56:02.971 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/data/mon.c.tgz 2024-04-25T15:56:02.971 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-04-25T15:56:03.019 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.KYFDkXdqUP 2024-04-25T15:56:03.020 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.KYFDkXdqUP 2024-04-25T15:56:03.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.KYFDkXdqUP 2024-04-25T15:56:03.279 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.KYFDkXdqUP is 632KB 2024-04-25T15:56:03.341 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.KYFDkXdqUP 2024-04-25T15:56:03.346 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T15:56:03.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T15:56:03.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T15:56:03.412 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T15:56:03.452 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T15:56:03.452 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T15:56:03.453 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/remote/smithi067/crash 2024-04-25T15:56:03.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T15:56:03.469 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/remote/smithi176/crash 2024-04-25T15:56:03.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T15:56:03.507 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T15:56:03.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:56:03.517 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T15:56:03.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T15:56:03.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T15:56:03.623 INFO:tasks.ceph:Compressing logs... 2024-04-25T15:56:03.623 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T15:56:03.657 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T15:56:03.674 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2024-04-25T15:56:03.674 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45330.log 2024-04-25T15:56:03.674 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2024-04-25T15:56:03.676 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2024-04-25T15:56:03.676 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44979.log 2024-04-25T15:56:03.676 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45330.log: /var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-04-25T15:56:03.677 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34760.log 2024-04-25T15:56:03.677 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2024-04-25T15:56:03.677 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45330.log.gz 2024-04-25T15:56:03.677 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46239.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44979.log.gz 2024-04-25T15:56:03.677 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44991.log 2024-04-25T15:56:03.677 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-04-25T15:56:03.677 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36922.log 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34760.log: /var/log/ceph/ceph-client.admin.44991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44023.log 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34760.log.gz 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44991.log.gz 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42944.log 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36922.log.gz 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38970.log 2024-04-25T15:56:03.681 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44023.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42944.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1%/var/log/ceph/ceph-client.admin.38970.log: -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38354.log 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34254.log 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.38970.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41542.log 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34254.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38354.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34254.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41542.log.gz 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39149.log 2024-04-25T15:56:03.682 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36061.log 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39149.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38072.log 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36061.log.gz 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39149.log.gz 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43876.log 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43748.log: /var/log/ceph/ceph-client.admin.38072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45536.log 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38072.log.gz 2024-04-25T15:56:03.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38431.log 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43876.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45536.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.43246.log 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43876.log.gz 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38431.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45536.log.gz 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44989.log 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38431.log.gz 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43246.log.gz 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38313.log 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44989.log.gz 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42221.log 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43383.log 2024-04-25T15:56:03.691 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38313.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45978.log 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42221.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39609.log 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42766.log 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.43383.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39609.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45978.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44575.log 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39609.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27807.log 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45614.log 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42766.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44575.log.gz 2024-04-25T15:56:03.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.27807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36948.log 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27807.log.gz 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37183.log 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36948.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45614.log.gz 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36948.log.gz 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.37183.log.gz 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45796.log 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-04-25T15:56:03.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44392.log 2024-04-25T15:56:03.710 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27764.log 2024-04-25T15:56:03.710 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27867.log 2024-04-25T15:56:03.710 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27764.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27764.log.gz 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27867.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27867.log.gz 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27836.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27692.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.5.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27836.log.gz 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27716.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27812.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27692.log.gz 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27716.log: /var/log/ceph/ceph-client.admin.27812.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-04-25T15:56:03.711 INFO:teuthology.orchestra.run.smithi176.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27716.log.gz 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27812.log.gz 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: 92.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T15:56:03.712 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27740.log 2024-04-25T15:56:03.722 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T15:56:03.722 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27740.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27740.log.gz 2024-04-25T15:56:03.742 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T15:56:03.742 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.b.log: 92.4% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-04-25T15:56:03.744 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45796.log.gz 2024-04-25T15:56:03.744 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz 2024-04-25T15:56:03.744 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2024-04-25T15:56:03.744 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-04-25T15:56:03.744 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33750.log 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44392.log.gz 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44992.log 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35198.log 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44992.log: /var/log/ceph/ceph-client.admin.34786.log 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33750.log.gz 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35198.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44992.log.gz 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41861.log 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34786.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.35198.log.gz 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40560.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.34786.log.gz --verbose 2024-04-25T15:56:03.745 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.41995.log 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44574.log 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41861.log.gz 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44968.log 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41995.log: /var/log/ceph/ceph-client.admin.44574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33827.log 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44574.log.gz 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38770.log 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44968.log.gz 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.33827.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36183.log 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33827.log.gz 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36183.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38770.log.gz 2024-04-25T15:56:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36183.log.gz 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34192.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45463.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34192.log.gz 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38464.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40433.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45463.log.gz 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41887.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38116.log 2024-04-25T15:56:03.747 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38464.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40433.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36740.log 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41699.log 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38116.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41887.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42936.log 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36740.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43167.log 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41699.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45900.log 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42936.log: /var/log/ceph/ceph-client.admin.43167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42453.log 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42936.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45900.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43167.log.gz 2024-04-25T15:56:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35253.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45900.log.gz 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.42453.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37818.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36693.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37818.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42453.log.gz 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35253.log.gz 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44418.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37818.log.gz 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42536.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36693.log: /var/log/ceph/ceph-client.admin.44418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42196.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36693.log.gz 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44418.log.gz 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43064.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42536.log: /var/log/ceph/ceph-client.admin.42196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39922.log 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42196.log.gz 2024-04-25T15:56:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43064.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42536.log.gz 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44340.log 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39922.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43064.log.gz 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44184.log 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39922.log.gz 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37739.log 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44340.log: /var/log/ceph/ceph-client.admin.44184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44522.log 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37739.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44184.log.gz 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44340.log.gz 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43788.log 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37739.log.gz 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44522.log: /var/log/ceph/ceph-client.admin.43788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43981.log 2024-04-25T15:56:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40510.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44522.log.gz 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44366.log 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43788.log.gz 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37434.log 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36381.log 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43981.log.gz 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44366.log: /var/log/ceph/ceph-client.admin.37434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36562.log 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37434.log.gz 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36381.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44366.log.gz 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35434.log 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36381.log.gz 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44049.log 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36562.log: /var/log/ceph/ceph-client.admin.35434.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T15:56:03.751 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35434.log.gz 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44049.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36562.log.gz 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35148.log 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43204.log 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.44049.log.gz 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44496.log 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.43204.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35148.log.gz 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33823.log 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44496.log.gz 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43204.log.gz 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44770.log 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38044.log: /var/log/ceph/ceph-client.admin.33823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45640.log 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-04-25T15:56:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44770.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.33823.log.gz 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34969.log 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44770.log.gz 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45640.log: /var/log/ceph/ceph-client.admin.34969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35587.log 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34969.log.gz 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45640.log.gz 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34227.log 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34504.log 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35587.log.gz 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34227.log: /var/log/ceph/ceph-client.admin.34504.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T15:56:03.753 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34227.log.gz 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43665.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34504.log.gz 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42994.log 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46135.log 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44758.log 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42994.log: /var/log/ceph/ceph-client.admin.46135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39492.log 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42994.log.gz 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46135.log.gz 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42498.log 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39492.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44758.log.gz 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34222.log 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39492.log.gz 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34218.log 2024-04-25T15:56:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42498.log: /var/log/ceph/ceph-client.admin.34222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38849.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34222.log.gz 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42498.log.gz 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45770.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38849.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34218.log.gz 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44763.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38849.log.gz 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36298.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38875.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45770.log.gz 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44763.log.gz 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45744.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36298.log.gz 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39193.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39893.log 2024-04-25T15:56:03.755 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38875.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45744.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39193.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34734.log 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36245.log 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41647.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39893.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37947.log 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36245.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34734.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37947.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.36245.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44980.log 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37632.log 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38708.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37947.log.gz 2024-04-25T15:56:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36766.log 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44980.log.gz 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42577.log 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37632.log.gz 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38040.log 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36766.log: /var/log/ceph/ceph-client.admin.42577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37393.log 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36766.log.gz 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5%/var/log/ceph/ceph-client.admin.38040.log: -- replaced with /var/log/ceph/ceph-client.admin.42577.log.gz 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38602.log 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38040.log.gz 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36532.log 2024-04-25T15:56:03.757 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36324.log 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38602.log: /var/log/ceph/ceph-client.admin.36532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37393.log.gz 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5%/var/log/ceph/ceph-client.admin.36324.log: -- replaced with /var/log/ceph/ceph-client.admin.36532.log.gz 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38602.log.gz -5 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33672.log 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.36324.log.gz 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42865.log 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35542.log 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33672.log: 91.3%/var/log/ceph/ceph-client.admin.42865.log: -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33354.log 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42865.log.gz 2024-04-25T15:56:03.758 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33672.log.gz 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.35542.log: --verbose -- /var/log/ceph/ceph-client.admin.41210.log 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44288.log 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33354.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.35542.log.gz 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41210.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.33302.log 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33354.log.gz 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39435.log 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33302.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44288.log.gz 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38931.log 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45437.log 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33302.log.gz 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45588.log 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39435.log.gz 2024-04-25T15:56:03.759 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38931.log.gz 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46213.log 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35745.log 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45588.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45437.log.gz 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45822.log 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45588.log.gz 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46109.log 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45822.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35745.log.gz 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35957.log 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45822.log.gz 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33276.log 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45926.log 2024-04-25T15:56:03.760 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35957.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46109.log.gz 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44106.log 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33276.log.gz 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45926.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.35957.log.gz 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39548.log 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44106.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45926.log.gz 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46161.log 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44106.log.gz 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33776.log 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46161.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39548.log.gz 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38212.log 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46161.log.gz 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35381.log 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36653.log 2024-04-25T15:56:03.761 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38212.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33776.log.gz 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.35381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37018.log 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38212.log.gz 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36653.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41410.log 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37018.log.gz 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43691.log 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36653.log.gz 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40389.log 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41410.log: /var/log/ceph/ceph-client.admin.43691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40389.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41410.log.gz 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.43691.log.gz 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35771.log 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40389.log.gz 2024-04-25T15:56:03.762 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37492.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33594.log 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35771.log: /var/log/ceph/ceph-client.admin.33594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37129.log 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33594.log.gz/var/log/ceph/ceph-client.admin.40625.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.35771.log.gz 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40059.log 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34020.log 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34031.log 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.37129.log.gz 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3%/var/log/ceph/ceph-client.admin.34020.log: -- replaced with /var/log/ceph/ceph-client.admin.40059.log.gz 2024-04-25T15:56:03.763 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34020.log.gz 2024-04-25T15:56:03.765 INFO:teuthology.orchestra.run.smithi176.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T15:56:03.765 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27788.log 2024-04-25T15:56:03.766 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.27788.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27788.log.gz 2024-04-25T15:56:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35932.log 2024-04-25T15:56:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45385.log 2024-04-25T15:56:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34031.log: /var/log/ceph/ceph-client.admin.35932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42280.log 2024-04-25T15:56:03.785 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34031.log.gz 2024-04-25T15:56:03.785 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8%/var/log/ceph/ceph-client.admin.45385.log: -- replaced with /var/log/ceph/ceph-client.admin.35932.log.gz 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45385.log.gz 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45278.log 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33828.log 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45278.log: 91.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.42280.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.42765.log 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45278.log.gz 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33828.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33828.log.gz 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45562.log 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41132.log 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42765.log: /var/log/ceph/ceph-client.admin.45562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41611.log 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4%/var/log/ceph/ceph-client.admin.41132.log: -- replaced with /var/log/ceph/ceph-client.admin.45562.log.gz 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.42765.log.gz 2024-04-25T15:56:03.786 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41132.log.gz 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37278.log 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41611.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45666.log 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41611.log.gz 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37278.log.gz 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44560.log 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33138.log 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.45666.log.gz 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41369.log 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44560.log.gz 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45252.log 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33138.log.gz 2024-04-25T15:56:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37579.log 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41369.log: /var/log/ceph/ceph-client.admin.45252.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45252.log.gz 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41369.log.gz 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40385.log 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45224.log 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37579.log: /var/log/ceph/ceph-client.admin.40385.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40385.log.gz 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.37579.log.gz 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34251.log 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37819.log 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45224.log: /var/log/ceph/ceph-client.admin.34251.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45224.log.gz 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34251.log.gz 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41773.log 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37819.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.37819.log.gz 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39756.log 2024-04-25T15:56:03.788 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40737.log 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41773.log: /var/log/ceph/ceph-client.admin.39756.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41773.log.gz 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39756.log.gz 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38216.log 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40737.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40737.log.gz 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33490.log 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37912.log 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38216.log: /var/log/ceph/ceph-client.admin.33490.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38216.log.gz 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33490.log.gz 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35704.log 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36471.log 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37912.log: /var/log/ceph/ceph-client.admin.35704.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.37912.log.gz 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35704.log.gz 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41438.log 2024-04-25T15:56:03.789 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36471.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36471.log.gz 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34262.log 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41438.log: /var/log/ceph/ceph-client.admin.39781.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41438.log.gz 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.35512.log 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34262.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34262.log.gz 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33328.log 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46303.log 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35512.log: /var/log/ceph/ceph-client.admin.33328.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35512.log.gz 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33328.log.gz 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35460.log 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46303.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46303.log.gz 2024-04-25T15:56:03.790 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46004.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43850.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35460.log: /var/log/ceph/ceph-client.admin.46004.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35460.log.gz 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46004.log.gz 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42350.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43850.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43850.log.gz 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39825.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44755.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42350.log: /var/log/ceph/ceph-client.admin.39825.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42350.log.gz 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39825.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39082.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44755.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44755.log.gz 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40596.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35833.log 2024-04-25T15:56:03.791 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39082.log: /var/log/ceph/ceph-client.admin.40596.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40596.log.gz 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38594.log 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35833.log: /var/log/ceph/ceph.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35833.log.gz 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43880.log 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38594.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38594.log.gz 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45692.log 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43880.log: 85.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43880.log.gz 2024-04-25T15:56:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42090.log 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43340.log 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45692.log: /var/log/ceph/ceph-client.admin.42090.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42090.log.gz 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45692.log.gz 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33824.log 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37340.log 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43340.log: /var/log/ceph/ceph-client.admin.33824.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43340.log.gz 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33824.log.gz 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44262.log 2024-04-25T15:56:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34943.log 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37340.log: /var/log/ceph/ceph-client.admin.44262.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37340.log.gz 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44262.log.gz 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34943.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34943.log.gz 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33248.log 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42256.log 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39367.log: /var/log/ceph/ceph-client.admin.33248.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33248.log.gz 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41255.log 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34891.log 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42256.log: /var/log/ceph/ceph-client.admin.41255.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42256.log.gz 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41255.log.gz 2024-04-25T15:56:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45952.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43379.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34891.log: /var/log/ceph/ceph-client.admin.45952.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34891.log.gz 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45952.log.gz 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42135.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43379.log: /var/log/ceph/ceph-client.admin.41969.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43379.log.gz 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44573.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42135.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42135.log.gz 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37673.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41061.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44573.log: /var/log/ceph/ceph-client.admin.37673.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44573.log.gz 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37673.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45848.log 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.803 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41061.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41061.log.gz 2024-04-25T15:56:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2024-04-25T15:56:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38182.log 2024-04-25T15:56:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45848.log: /var/log/ceph/ceph-client.admin.40169.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45848.log.gz 2024-04-25T15:56:03.804 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2024-04-25T15:56:03.804 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33825.log 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34538.log 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38182.log: /var/log/ceph/ceph-client.admin.33825.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38182.log.gz 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44576.log 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33825.log.gz 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34538.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34538.log.gz 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44132.log 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44576.log: 87.3%/var/log/ceph/ceph-client.admin.44132.log: -- replaced with /var/log/ceph/ceph-client.admin.44576.log.gz 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33222.log 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44132.log.gz 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43270.log 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33222.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-04-25T15:56:03.809 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33222.log.gz 2024-04-25T15:56:03.810 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41816.log 2024-04-25T15:56:03.810 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44158.log 2024-04-25T15:56:03.810 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41816.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.43270.log.gz 2024-04-25T15:56:03.810 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41816.log.gz 2024-04-25T15:56:03.810 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33826.log 2024-04-25T15:56:03.810 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33438.log 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44158.log: /var/log/ceph/ceph-client.admin.33826.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44158.log.gz 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33826.log.gz 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34030.log 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33438.log: /var/log/ceph/ceph-client.admin.34030.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33438.log.gz 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34030.log.gz 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44561.log 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44210.log: /var/log/ceph/ceph-client.admin.44561.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44561.log.gz 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41492.log 2024-04-25T15:56:03.813 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35048.log 2024-04-25T15:56:03.814 INFO:teuthology.orchestra.run.smithi176.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45718.log: /var/log/ceph/ceph-client.admin.41492.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41492.log.gz 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34452.log 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40270.log 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35048.log: /var/log/ceph/ceph-client.admin.34452.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34452.log.gz 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.35048.log.gz 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37765.log 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45304.log 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40270.log: /var/log/ceph/ceph-client.admin.37765.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40270.log.gz 2024-04-25T15:56:03.817 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42354.log 2024-04-25T15:56:03.818 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37765.log.gz 2024-04-25T15:56:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45304.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45304.log.gz 2024-04-25T15:56:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38552.log 2024-04-25T15:56:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38389.log 2024-04-25T15:56:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42354.log: /var/log/ceph/ceph-client.admin.38552.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42354.log.gz 2024-04-25T15:56:03.819 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38552.log.gz 2024-04-25T15:56:03.819 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43608.log 2024-04-25T15:56:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39522.log 2024-04-25T15:56:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38389.log: /var/log/ceph/ceph-client.admin.43608.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38389.log.gz 2024-04-25T15:56:03.821 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27588.log 2024-04-25T15:56:03.821 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43608.log.gz 2024-04-25T15:56:03.822 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39522.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39522.log.gz 2024-04-25T15:56:03.822 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37081.log 2024-04-25T15:56:03.822 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45356.log 2024-04-25T15:56:03.823 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.27588.log: /var/log/ceph/ceph-client.admin.37081.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.27588.log.gz 2024-04-25T15:56:03.823 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37081.log.gz 2024-04-25T15:56:03.824 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T15:56:03.824 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40511.log 2024-04-25T15:56:03.824 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45356.log: /var/log/ceph/ceph.audit.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45356.log.gz 2024-04-25T15:56:03.825 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44778.log 2024-04-25T15:56:03.825 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40511.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40511.log.gz 2024-04-25T15:56:03.825 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34258.log 2024-04-25T15:56:03.826 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44778.log: 88.9% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44778.log.gz 2024-04-25T15:56:03.827 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T15:56:03.827 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T15:56:03.827 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37207.log 2024-04-25T15:56:03.827 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34258.log: /var/log/ceph/ceph-mds.c.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34258.log.gz 2024-04-25T15:56:03.828 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46085.log 2024-04-25T15:56:03.829 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37207.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.37207.log.gz 2024-04-25T15:56:03.829 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45172.log 2024-04-25T15:56:03.830 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46085.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46085.log.gz 2024-04-25T15:56:03.831 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T15:56:03.831 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45172.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45172.log.gz 2024-04-25T15:56:03.832 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41022.log 2024-04-25T15:56:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33380.log 2024-04-25T15:56:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41022.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41022.log.gz 2024-04-25T15:56:03.856 INFO:teuthology.orchestra.run.smithi176.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T15:56:03.864 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45411.log 2024-04-25T15:56:03.864 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33380.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33380.log.gz 2024-04-25T15:56:03.865 INFO:teuthology.orchestra.run.smithi067.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T15:56:03.869 INFO:teuthology.orchestra.run.smithi067.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T15:56:03.869 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34032.log 2024-04-25T15:56:03.870 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39027.log 2024-04-25T15:56:03.871 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34032.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45411.log.gz 2024-04-25T15:56:03.871 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34032.log.gz 2024-04-25T15:56:03.872 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34019.log 2024-04-25T15:56:03.872 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36979.log 2024-04-25T15:56:03.872 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39027.log.gz 2024-04-25T15:56:03.873 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34019.log.gz 2024-04-25T15:56:03.873 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40108.log 2024-04-25T15:56:03.873 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34682.log 2024-04-25T15:56:03.874 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36979.log: /var/log/ceph/ceph-client.admin.40108.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36979.log.gz 2024-04-25T15:56:03.874 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40108.log.gz 2024-04-25T15:56:03.874 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40307.log 2024-04-25T15:56:03.875 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34682.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34682.log.gz 2024-04-25T15:56:03.875 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42720.log 2024-04-25T15:56:03.875 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36007.log 2024-04-25T15:56:03.876 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40307.log: /var/log/ceph/ceph-client.admin.42720.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.40307.log.gz 2024-04-25T15:56:03.876 INFO:teuthology.orchestra.run.smithi067.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42720.log.gz 2024-04-25T15:56:03.876 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44572.log 2024-04-25T15:56:03.877 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42695.log 2024-04-25T15:56:03.877 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36007.log: /var/log/ceph/ceph-client.admin.44572.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36007.log.gz 2024-04-25T15:56:03.878 INFO:teuthology.orchestra.run.smithi067.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40018.log 2024-04-25T15:56:03.878 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44572.log.gz 2024-04-25T15:56:03.878 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42695.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42695.log.gz 2024-04-25T15:56:03.879 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43549.log 2024-04-25T15:56:03.879 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36142.log 2024-04-25T15:56:03.880 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40018.log: /var/log/ceph/ceph-client.admin.43549.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43549.log.gz 2024-04-25T15:56:03.880 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40018.log.gz 2024-04-25T15:56:03.881 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33817.log 2024-04-25T15:56:03.881 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37999.log 2024-04-25T15:56:03.882 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36142.log: /var/log/ceph/ceph-client.admin.33817.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33817.log.gz 2024-04-25T15:56:03.882 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36142.log.gz 2024-04-25T15:56:03.882 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39897.log 2024-04-25T15:56:03.882 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34564.log 2024-04-25T15:56:03.883 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37999.log: /var/log/ceph/ceph-client.admin.39897.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.37999.log.gz 2024-04-25T15:56:03.884 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.39897.log.gz 2024-04-25T15:56:03.884 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33698.log 2024-04-25T15:56:03.884 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34564.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34564.log.gz 2024-04-25T15:56:03.884 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34708.log 2024-04-25T15:56:03.885 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33830.log 2024-04-25T15:56:03.885 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33698.log: /var/log/ceph/ceph-client.admin.34708.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33698.log.gz 2024-04-25T15:56:03.886 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34708.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34400.log 2024-04-25T15:56:03.886 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.886 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33830.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33830.log.gz 2024-04-25T15:56:03.887 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22780.log 2024-04-25T15:56:03.887 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44080.log 2024-04-25T15:56:03.888 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34400.log: /var/log/ceph/ceph.tmp-client.admin.22780.log: 0.0% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34400.log.gz 2024-04-25T15:56:03.888 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.22780.log.gz 2024-04-25T15:56:03.888 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44975.log 2024-04-25T15:56:03.888 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33568.log 2024-04-25T15:56:03.889 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44080.log: /var/log/ceph/ceph-client.admin.44975.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44080.log.gz 2024-04-25T15:56:03.890 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44975.log.gz 2024-04-25T15:56:03.890 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44236.log 2024-04-25T15:56:03.890 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33568.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33568.log.gz 2024-04-25T15:56:03.890 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36418.log 2024-04-25T15:56:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36877.log 2024-04-25T15:56:03.891 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44236.log: /var/log/ceph/ceph-client.admin.36418.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44236.log.gz 2024-04-25T15:56:03.892 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.36418.log.gz 2024-04-25T15:56:03.892 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33516.log 2024-04-25T15:56:03.892 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36877.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36877.log.gz 2024-04-25T15:56:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34995.log 2024-04-25T15:56:03.893 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38779.log 2024-04-25T15:56:03.894 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33516.log: /var/log/ceph/ceph-client.admin.34995.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33516.log.gz 2024-04-25T15:56:03.895 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44470.log 2024-04-25T15:56:03.895 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.34995.log.gz 2024-04-25T15:56:03.895 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38779.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38779.log.gz 2024-04-25T15:56:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43374.log 2024-04-25T15:56:03.896 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37306.log 2024-04-25T15:56:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44470.log: /var/log/ceph/ceph-client.admin.43374.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44470.log.gz 2024-04-25T15:56:03.897 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43374.log.gz 2024-04-25T15:56:03.897 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41570.log 2024-04-25T15:56:03.898 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44964.log 2024-04-25T15:56:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37306.log: /var/log/ceph/ceph-client.admin.41570.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37306.log.gz 2024-04-25T15:56:03.899 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41570.log.gz 2024-04-25T15:56:03.899 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33045.log 2024-04-25T15:56:03.899 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T15:56:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44964.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44964.log.gz 2024-04-25T15:56:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27704.log 2024-04-25T15:56:03.900 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33542.log 2024-04-25T15:56:03.901 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33045.log: /var/log/ceph/ceph-client.admin.27704.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33045.log.gz 2024-04-25T15:56:03.902 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27704.log.gz 2024-04-25T15:56:03.902 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33724.log 2024-04-25T15:56:03.902 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46056.log 2024-04-25T15:56:03.903 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33542.log: /var/log/ceph/ceph-client.admin.33724.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33542.log.gz 2024-04-25T15:56:03.903 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33724.log.gz 2024-04-25T15:56:03.903 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35887.log 2024-04-25T15:56:03.903 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33646.log 2024-04-25T15:56:03.904 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46056.log: /var/log/ceph/ceph-client.admin.35887.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46056.log.gz 2024-04-25T15:56:03.904 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35887.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45874.log 2024-04-25T15:56:03.904 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33646.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33646.log.gz 2024-04-25T15:56:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40852.log 2024-04-25T15:56:03.906 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33102.log 2024-04-25T15:56:03.906 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45874.log: /var/log/ceph/ceph-client.admin.40852.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45874.log.gz 2024-04-25T15:56:03.907 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40805.log 2024-04-25T15:56:03.907 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40852.log.gz 2024-04-25T15:56:03.907 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33102.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33102.log.gz 2024-04-25T15:56:03.907 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34865.log 2024-04-25T15:56:03.907 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42871.log 2024-04-25T15:56:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40805.log: 87.6%/var/log/ceph/ceph-client.admin.34865.log: -- replaced with /var/log/ceph/ceph-client.admin.40805.log.gz 2024-04-25T15:56:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44777.log 2024-04-25T15:56:03.909 INFO:teuthology.orchestra.run.smithi067.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34865.log.gz 2024-04-25T15:56:03.909 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42049.log 2024-04-25T15:56:03.910 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44777.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42871.log.gz 2024-04-25T15:56:03.910 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44777.log.gz 2024-04-25T15:56:03.910 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-04-25T15:56:03.910 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33620.log 2024-04-25T15:56:03.911 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42049.log: /var/log/ceph/ceph-client.admin.41261.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42049.log.gz 2024-04-25T15:56:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46329.log 2024-04-25T15:56:03.912 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-04-25T15:56:03.912 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33620.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33620.log.gz 2024-04-25T15:56:03.913 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44876.log 2024-04-25T15:56:03.913 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2024-04-25T15:56:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46329.log: /var/log/ceph/ceph-client.admin.44876.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46329.log.gz 2024-04-25T15:56:03.914 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44876.log.gz 2024-04-25T15:56:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-04-25T15:56:03.914 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34011.log 2024-04-25T15:56:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40897.log: /var/log/ceph/ceph-client.admin.41183.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2024-04-25T15:56:03.916 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-04-25T15:56:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46030.log 2024-04-25T15:56:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34011.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34011.log.gz 2024-04-25T15:56:03.916 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34617.log 2024-04-25T15:56:03.917 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44444.log 2024-04-25T15:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46030.log: /var/log/ceph/ceph-client.admin.34617.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46030.log.gz 2024-04-25T15:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34617.log.gz 2024-04-25T15:56:03.918 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43102.log 2024-04-25T15:56:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42625.log 2024-04-25T15:56:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44444.log: /var/log/ceph/ceph-client.admin.43102.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44444.log.gz 2024-04-25T15:56:03.920 INFO:teuthology.orchestra.run.smithi067.stderr: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43023.log 2024-04-25T15:56:03.920 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43102.log.gz 2024-04-25T15:56:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42625.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42625.log.gz 2024-04-25T15:56:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38518.log 2024-04-25T15:56:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34838.log 2024-04-25T15:56:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43023.log: /var/log/ceph/ceph-client.admin.38518.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.43023.log.gz 2024-04-25T15:56:03.922 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38518.log.gz 2024-04-25T15:56:03.922 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34426.log 2024-04-25T15:56:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36065.log 2024-04-25T15:56:03.923 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34838.log: /var/log/ceph/ceph-client.admin.34426.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34838.log.gz 2024-04-25T15:56:03.924 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34426.log.gz 2024-04-25T15:56:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44314.log 2024-04-25T15:56:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36065.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.36065.log.gz 2024-04-25T15:56:03.924 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34478.log 2024-04-25T15:56:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37789.log 2024-04-25T15:56:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44314.log: /var/log/ceph/ceph-client.admin.34478.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44314.log.gz 2024-04-25T15:56:03.926 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34018.log 2024-04-25T15:56:03.926 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34478.log.gz 2024-04-25T15:56:03.927 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37789.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37789.log.gz 2024-04-25T15:56:03.927 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36506.log 2024-04-25T15:56:03.927 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36770.log 2024-04-25T15:56:03.928 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34018.log: /var/log/ceph/ceph-client.admin.36506.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34018.log.gz 2024-04-25T15:56:03.928 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36506.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-04-25T15:56:03.928 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.929 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36770.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36770.log.gz 2024-04-25T15:56:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-04-25T15:56:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34656.log 2024-04-25T15:56:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39340.log: /var/log/ceph/ceph-mds.e.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-04-25T15:56:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44570.log 2024-04-25T15:56:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34656.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34656.log.gz 2024-04-25T15:56:03.932 INFO:teuthology.orchestra.run.smithi067.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-04-25T15:56:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34917.log 2024-04-25T15:56:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44570.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44570.log.gz 2024-04-25T15:56:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39057.log 2024-04-25T15:56:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45198.log 2024-04-25T15:56:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34917.log: /var/log/ceph/ceph-client.admin.39057.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34917.log.gz 2024-04-25T15:56:03.935 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39057.log.gz 2024-04-25T15:56:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35486.log 2024-04-25T15:56:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44766.log 2024-04-25T15:56:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45198.log: /var/log/ceph/ceph-client.admin.35486.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45198.log.gz 2024-04-25T15:56:03.936 INFO:teuthology.orchestra.run.smithi067.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35486.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34812.log 2024-04-25T15:56:03.936 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44766.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2024-04-25T15:56:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2024-04-25T15:56:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40947.log 2024-04-25T15:56:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34812.log: /var/log/ceph/ceph-client.admin.41060.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34812.log.gz 2024-04-25T15:56:03.938 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2024-04-25T15:56:03.938 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-04-25T15:56:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40947.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40947.log.gz 2024-04-25T15:56:03.939 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36561.log 2024-04-25T15:56:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38267.log 2024-04-25T15:56:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40458.log: /var/log/ceph/ceph-client.admin.36561.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-04-25T15:56:03.941 INFO:teuthology.orchestra.run.smithi067.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36561.log.gz 2024-04-25T15:56:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33464.log 2024-04-25T15:56:03.941 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38267.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38267.log.gz 2024-04-25T15:56:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37051.log 2024-04-25T15:56:03.942 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38150.log 2024-04-25T15:56:03.943 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33464.log: /var/log/ceph/ceph-client.admin.37051.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33464.log.gz 2024-04-25T15:56:03.943 INFO:teuthology.orchestra.run.smithi067.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37051.log.gz 2024-04-25T15:56:03.944 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37549.log 2024-04-25T15:56:03.944 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33410.log 2024-04-25T15:56:03.945 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38150.log: /var/log/ceph/ceph-client.admin.37549.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38150.log.gz 2024-04-25T15:56:03.945 INFO:teuthology.orchestra.run.smithi067.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2024-04-25T15:56:03.945 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37549.log.gz 2024-04-25T15:56:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33410.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33410.log.gz 2024-04-25T15:56:03.946 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41674.log 2024-04-25T15:56:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34591.log 2024-04-25T15:56:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46187.log: /var/log/ceph/ceph-client.admin.41674.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-04-25T15:56:03.947 INFO:teuthology.orchestra.run.smithi067.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41674.log.gz 2024-04-25T15:56:03.947 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43470.log 2024-04-25T15:56:03.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34591.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34591.log.gz 2024-04-25T15:56:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35098.log 2024-04-25T15:56:03.949 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43470.log: /var/log/ceph/ceph-client.admin.35098.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43470.log.gz 2024-04-25T15:56:03.950 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.35098.log.gz 2024-04-25T15:56:03.996 INFO:teuthology.orchestra.run.smithi176.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T15:56:04.254 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T15:56:04.389 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T15:56:04.462 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T15:56:04.516 INFO:teuthology.orchestra.run.smithi067.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T15:56:04.569 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T15:56:04.690 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T15:56:04.694 INFO:teuthology.orchestra.run.smithi176.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T15:56:04.695 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T15:56:04.737 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T15:56:04.739 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-04-25T15:56:04.740 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m1.069s 2024-04-25T15:56:04.740 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m4.248s 2024-04-25T15:56:04.740 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.205s 2024-04-25T15:56:11.526 INFO:teuthology.orchestra.run.smithi067.stderr: 96.9% -- replaced with /var/log/ceph/ceph-client.0.35381.log.gz 2024-04-25T15:56:11.526 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-04-25T15:56:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m7.868s 2024-04-25T15:56:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m11.725s 2024-04-25T15:56:11.527 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.583s 2024-04-25T15:56:11.528 INFO:tasks.ceph:Archiving logs... 2024-04-25T15:56:11.528 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/remote/smithi067/log 2024-04-25T15:56:11.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T15:56:12.886 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/remote/smithi176/log 2024-04-25T15:56:12.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T15:56:13.401 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T15:56:13.411 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T15:56:13.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T15:56:13.414 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T15:56:13.482 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T15:56:13.491 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T15:56:13.491 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:56:13.494 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:56:13.504 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T15:56:13.504 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-04-25T15:56:13.504 INFO:teuthology.orchestra.run.smithi067.stdout:+hv01.front.sepi 67.205.162.81 3 u 40 64 377 0.094 -2.993 0.960 2024-04-25T15:56:13.504 INFO:teuthology.orchestra.run.smithi067.stdout:*hv02.front.sepi 63.231.80.2 3 u 39 64 377 0.082 -1.787 1.435 2024-04-25T15:56:13.505 INFO:teuthology.orchestra.run.smithi067.stdout:+hv03.front.sepi 74.6.168.72 3 u 39 64 377 0.085 -0.078 1.608 2024-04-25T15:56:13.505 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:56:13.507 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T15:56:13.508 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2024-04-25T15:56:13.508 INFO:teuthology.orchestra.run.smithi176.stdout:+hv01.front.sepi 67.205.162.81 3 u 43 64 377 0.076 -1.435 0.037 2024-04-25T15:56:13.508 INFO:teuthology.orchestra.run.smithi176.stdout:+hv02.front.sepi 63.231.80.2 3 u 46 64 377 0.080 +0.561 0.035 2024-04-25T15:56:13.508 INFO:teuthology.orchestra.run.smithi176.stdout:*hv03.front.sepi 74.6.168.72 3 u 45 64 377 0.104 -0.033 0.023 2024-04-25T15:56:13.508 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T15:56:13.508 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T15:56:13.529 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T15:56:13.530 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T15:56:13.568 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T15:56:13.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T15:56:13.670 INFO:teuthology.task.internal:Duration was 996.642497 seconds 2024-04-25T15:56:13.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T15:56:13.718 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T15:56:13.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T15:56:13.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T15:56:13.759 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T15:56:13.759 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-04-25T15:56:13.759 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T15:56:13.813 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-04-25T15:56:13.813 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T15:56:13.822 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T15:56:13.823 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T15:56:13.856 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T15:56:13.891 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T15:56:13.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T15:56:13.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T15:56:14.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T15:56:14.022 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T15:56:14.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T15:56:14.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T15:56:14.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T15:56:14.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:56:14.080 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:56:14.089 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-04-25T15:56:14.105 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-04-25T15:56:14.141 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:56:14.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:56:14.151 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T15:56:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:56:14.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T15:56:14.199 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T15:56:14.200 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/remote/smithi067 2024-04-25T15:56:14.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T15:56:14.223 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673228/remote/smithi176 2024-04-25T15:56:14.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T15:56:14.252 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T15:56:14.252 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T15:56:14.261 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T15:56:14.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T15:56:14.307 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T15:56:14.307 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T15:56:14.335 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T15:56:14.335 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T15:56:14.337 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T15:56:14.341 INFO:teuthology.orchestra.run.smithi067.stdout: 658643 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 15:56 /home/ubuntu/cephtest 2024-04-25T15:56:14.344 INFO:teuthology.orchestra.run.smithi176.stdout: 658624 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 15:56 /home/ubuntu/cephtest 2024-04-25T15:56:14.345 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T15:56:14.385 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T15:56:14.488 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} duration: 996.6424973011017 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T15:56:14.489 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T15:56:14.581 INFO:teuthology.run:pass